JavaScript Capable Webpage Download Tool?

grarpamp grarpamp at gmail.com
Sat Sep 4 22:39:31 PDT 2021


On 9/3/21, coderman <coderman at protonmail.com> wrote:
> try selenium webdriver:

Wasn't really wanting to write some new tool,
from a library, that someone out there has probably
already done better.

Wget / wget2 / curl / fetch / elinks / * old CLI tools
don't interpret javascript / json, thus the pages they
save to disk are 'source', not pulled out of the webserver
via the pages embedded js yielding the text and links one
would see in a GUI browser. Then there are those pages that
do crazy js based variables mathfuscation, bloatfeaturing, etc.
One of the old tools used to compile against ecma/js, but
it was disclaimed as broken and capable of only basic js.

Maybe can run ./firefox -<options> <url> > <file>
but is inefficient, no spidering options, etc.


In return, people should know that 'gron' and similar tools exist.


More information about the cypherpunks mailing list