Wednesday 15 July 2015

web crawler - wget --spider doesn't get file size for some links -


I have to get some file size. Some people recommend wget --spider, however, when I run it on some links, then he said length: unspecified [text / html] is there any way to solve this or I actually file without downloading Can I use other ways to get the shape? Thanks!

This is because the server content-length header, or it's incorrect Sending as you can ignore it by using the - ignore-length option:

  $ wget --ignore-length http: // autos .cn.yahoo.com /    

No comments:

Post a Comment