Re: full year 2014 corpus - better concurrent continue
"why this format?" - it is lowest common denominator, in a sense. the rest explained later... but first, a fix. there is an issue with commands that won't continue past the PDF and small file bundles. use fix below to run or more continuation downloaders: ### The gist of a quick continue download: (many can be run in parallel) # # Update torsunget.sh and run dl-fy2014.sh to complete. # # assumes a copy of https://peertech.org/files/fy2014lst.txt # or 84fe6b33e5b6f2478523432514fcb24b844105c4a38635ba97543e9c7152f90e # in current working directory where ./dl-fy2014.sh is run. # # E.g.: curl -x http://127.0.0.1:8778/ -o fy2014lst.txt \ # http://bigsun36arflx75h.onion/shid/84f/e6b/84fe6b33e5b6f247..7543e9c7152f90e # rm -f torsunget.sh >/dev/null 2>&1 curl -x http://127.0.0.1:8778/ -o torsunget.sh \ http://bigsundaawafn36e.onion/torsunget.sh.txt \ && chmod +x torsunget.sh curl -x http://127.0.0.1:8778/ -o dl-fy2014.sh \ http://bigsundaawafn36e.onion/dl-fy2014.sh.txt \ && chmod +x dl-fy2014.sh time ./dl-fy2014.sh
participants (1)
-
coderman