Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Isn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?


HEAD requests


Scrapers could also use the "Cache-Control", "Expires" and "If-Modified-Since" headers properly, to reduce their traffic a little, but do they?




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: