full year 2014 corpus - better concurrent continue

coderman coderman at gmail.com
Sun Jan 11 16:32:33 PST 2015


"why this format?"
 - it is lowest common denominator, in a sense. the rest explained later...

but first, a fix. there is an issue with commands that won't continue
past the PDF and small file bundles.  use fix below to run or more
continuation downloaders:


### The gist of a quick continue download: (many can be run in parallel)
#
# Update torsunget.sh and run dl-fy2014.sh to complete.
#
# assumes a copy of https://peertech.org/files/fy2014lst.txt
# or 84fe6b33e5b6f2478523432514fcb24b844105c4a38635ba97543e9c7152f90e
# in current working directory where ./dl-fy2014.sh is run.
#
# E.g.: curl -x http://127.0.0.1:8778/ -o fy2014lst.txt \
#  http://bigsun36arflx75h.onion/shid/84f/e6b/84fe6b33e5b6f247..7543e9c7152f90e
#
rm -f torsunget.sh >/dev/null 2>&1
curl -x http://127.0.0.1:8778/ -o torsunget.sh \
 http://bigsundaawafn36e.onion/torsunget.sh.txt \
  && chmod +x torsunget.sh
curl -x http://127.0.0.1:8778/ -o dl-fy2014.sh \
 http://bigsundaawafn36e.onion/dl-fy2014.sh.txt \
  && chmod +x dl-fy2014.sh
time ./dl-fy2014.sh



More information about the cypherpunks mailing list