Robots.txt
robots.txt tells search engines which parts of a site they may crawl. It doesn’t remove already indexed URLs by itself.
robots.txt provides crawl guidance to search engines.
- Use to block non-public or low-value paths
- Doesn’t guarantee de-indexing
- Keep it simple and test changes