On 9/3/21, coderman <coderman@protonmail.com> wrote:
try selenium webdriver:
Wasn't really wanting to write some new tool, from a library, that someone out there has probably already done better. Wget / wget2 / curl / fetch / elinks / * old CLI tools don't interpret javascript / json, thus the pages they save to disk are 'source', not pulled out of the webserver via the pages embedded js yielding the text and links one would see in a GUI browser. Then there are those pages that do crazy js based variables mathfuscation, bloatfeaturing, etc. One of the old tools used to compile against ecma/js, but it was disclaimed as broken and capable of only basic js. Maybe can run ./firefox -<options> <url> > <file> but is inefficient, no spidering options, etc. In return, people should know that 'gron' and similar tools exist.