Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

No. It prevents pages from being crawled, but references to uncrawled pages can still be shown. See http://www.mattcutts.com/blog/robots-txt-remove-url/ for how it works.


Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: