Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This looks like microsoft is assuming robots.txt is case insensitive?


It is. Only domain names are case-insensitive.

This document explains how Google handles robots.txt : http://code.google.com/web/controlcrawlindex/docs/robots_txt...


Are we reading the same file? Under where it describes matching paths (so replace /fish and /Fish with /search and /Search if you like):

==Example path matches==

[path] /fish

Matches: /fish /fish.html /fish/salmon.html /fishheads /fishheads/yummy.html /fish.php?id=anything

Does not match: /Fish.asp /catfish /?id=fish

Comments: Note the case-sensitive matching.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: