I want to rip the contents of a pay website, but I have to log in to their web site on a web page to get access
Does anyone have any good tools for Windows for that?
I’m guessing that any such tools must have a built in browser, or be a browser plugin for it to work.
I have an account, so that’s not a problem. The problem is how to automate going into every little content page and downloading the content, including the hi-res files.
I’m on a Mac and use SiteSucker so I know that’s not super helpful but for windows you could try wGet or WebCopy? https://www.cyotek.com/cyotek-webcopy / https://gnuwin32.sourceforge.net/packages/wget.htm
Webcopy looks promising if I can get the crawler part of it to work with this site’s authentication…
edit: I couldn’t get Webcopy’s spider to authenticate correctly.
Webcopy uses the deprecated version of Internet Explorer in Windows 10 as a module, and I can log into the website using the Capture Forms browser dialog, but the cookies or whatever else don’t translate over to the spider.