Thinking on different tools. Content scraping also missing, in addition to spidering. What kind of tool would be helpful here? Assuming that we need to build it. Maybe a local http server that provides access to a selenium session as if it is non-js html? Elements with mouse events could be converted to links. My interest in this missing area relates to ecommerce scraping. I'd like for people to be able to work around the product search engines that use marketing algorithms for all their result orders. I'd like for it to be easy for reseller to seed the inventory of decentralized marketplaces from existing platforms. The larger corps have been winning the scraping tech race recently, I believe. PhantomJS, which itself was another cultural polyfill for an ongoing situation, had a number of libraries and tools, but Selenium Webdriver has now replaced by it. A web search for "phantomjs OR casperjs spider" turns up some hits. "selenium spider" appears to as well. For example: https://scalpel.readthedocs.io/en/latest/selenium-spider/

[sorry, I misremembered the question and thought grarpamp was asking regarding spidering. if you just want to archive a page it could be helpful to be aware that interfaces like selenium usually have a single function to take a screenshot, and comparably to dump the rendered page body. so a downloader would be like 3-7 lines of code. that's probably why coderman's reply appeared short.]

This one was easy enough write to fetch, parse, and then fetch the real page data. GUI browsers do all that behind the scenes. Wget '-r' style spidering would be needed for users to copy sites that use js/json frameworks sitewide. I'll look into this thread to handle more complex sites, and to keep from writing and maintaining site specific parsers. Endless scroll, pages with 1MB of var encoded obfuscated js spread across multiple script sources just to display 100 lines of text... such wow. Whatever happened to plain old html, some href links, form fields to submit. Ecom marketing cluttering search... :( Search engines... There might be use for a spider to stream everything it's scraping in realtime into one formatted output pipe, instead of first to disk then into the indexer. Something like continuous WARC to stdout/pipe. Parallel threads working different trees of the namespace. Auto nudge in case one gets stuck in the weeds. Seeding inventory requires the destination market to provide api for loading, or the seeder to write the shim loader, which you could then sell. Connect the CLI to the darknets, one giant API :)