This one was easy enough write to fetch, parse, and then fetch the real page data. GUI browsers do all that behind the scenes. Wget '-r' style spidering would be needed for users to copy sites that use js/json frameworks sitewide. I'll look into this thread to handle more complex sites, and to keep from writing and maintaining site specific parsers. Endless scroll, pages with 1MB of var encoded obfuscated js spread across multiple script sources just to display 100 lines of text... such wow. Whatever happened to plain old html, some href links, form fields to submit. Ecom marketing cluttering search... :( Search engines... There might be use for a spider to stream everything it's scraping in realtime into one formatted output pipe, instead of first to disk then into the indexer. Something like continuous WARC to stdout/pipe. Parallel threads working different trees of the namespace. Auto nudge in case one gets stuck in the weeds. Seeding inventory requires the destination market to provide api for loading, or the seeder to write the shim loader, which you could then sell. Connect the CLI to the darknets, one giant API :)