Now the Windows 10 Anniversary update is out, you won't need to go through the Insider Preview rigamarole to get the integrated Bash shell.
The Insider Preview of Windows 10 (Build 14316 and later) has the beta of the new port of Bash shell from Ubuntu included, however it takes a bit of fiddling to get it going.
start > cmd to open a command prompt and type
bash. After a few moments it should have installed the bash shell.
start > bash and you should be presented with a full Ubuntu 14.04 bash prompt, with working apt and all the other command line goodness (no Desktop/Graphical UI).
The process of opting into the Insider Builds did not go smoothly for me. It appears if you opt in after a build has been released you won't update to the Insider Preview until the next preview is pushed out to customers. Patience is key here.
Check which build of Windows you are running, the release build is 10586.
start > run > winver
Check the registry entries for the Insider Preview:
start > run > regedit
Check that BranchName and UserPreferredBranchName strings are set to rs1_release.
Also ensure that EnablePreviewBuilds and ThresholdOptedin DWORDs are set to 1.
If the key
exists it features builds that have been installed and then reverted from. These will be unavailable to install again unless the key is removed.
If you had to change anything in the registry editor, restart your computer and check for updates again.