So, I'm having a small problem with wget. I'm trying to download a website but, due to the nature of the site, not all the images are available all the time and I can't download just the folder with the images I want because of the robots.txt file (I found a way around that).

I need wget to act like a crawler, but it should download the pictures. But from my understanding the --spider command will not download anything. It just crawls the site looking for "stuff".

I'm using the command 'wget -r -e robots=off sitename.com' and that gets some of it, but only the links available. If I try 'wget -r sitename.com/images/' I get a 404.

If I use 'wget --spider -r -e robots=off sitename.com' it doesn't download anything.

Thank you for your time!