Downloading the entire NuGet package database

When I was having the serialization problems described in Saving the entire list of NuGet Packages I realized that adding a NuGet IPackage (retrieved from the GetPackages() method) to a SharePackageRepository will also download actual packages :)


This means that this script:



will will get the data for the first 10 NuGet packages:



i.e, this will also download of the actual nupkg (the zip with all the code+data files)



(also stored in the local computer’s NuGet cache)



This means that with this code:



we can download the entire NuGet library :)

On execution, all metadata will be downloaded first:



which took 1m:32s (to get the information about 12197 packages):



and then the download of all packages will start:



and take a while:)

Here it is after after 10m (180 Mb downloaded)



…. after 24m (505 Mb downloaded)



…after 45m (697 Mbs downloaded)



…after 1h (1Gb downloaded)



After 1h 2m and 4100 fetched (1.50 GB downloaded) we hit an exception:





So we need to improve our code to handle errors (and cache the packages list)

The code below will open a new C# script editor (with the packages passed as the _package variable)
which can then be used to trigger another fetch:



and here we go again (with the git-got.0.0.1.nupkg error being catched):



…after 15m (1.78 Gb downloaded)



…after 45m (2.13Gb downloaded)



… after 50m (and 2.47Gb downloaded) I had to stop the current download thread (while changing locations)



Starting again on a new network (and physical location)



Finally (after a couple interruptions caused by the VM going to sleep) 3.99 Gb



the 27 errors were caused by network errors, re-running the scan we got 4 errors:



on these packages:



So now we have a 4Gb of cache folder



and 4 GB NuGet archive



each of these 12,193 folders containing an *.nupkg and *.nuspec file




See also Offline copy of the entire NuGet.org gallery. What should I do with these 4.05 Gbs of amazing .Net Apps/APIs?