robots.txt
约 1 分钟阅读
- 字段
- User-agent
- Allow
- Disallow
- Sitemap
- robots.txt
- developer.mozilla.org/robots.txt链接developer.mozilla.org/robots.txt字段 · User-agent · Allow · Disallow · Sitemap · https://developer.mozilla.org/robots.txt · https://www.workday.com/robots.txt笔记:robots.txt
- www.workday.com/robots.txt链接www.workday.com/robots.txt字段 · User-agent · Allow · Disallow · Sitemap · https://developer.mozilla.org/robots.txt · https://www.workday.com/robots.txt笔记:robots.txt
- 参考
User-agent: *
Disallow: /User-agent: *
Disallow:User-agent: *
Sitemap: https://developer.mozilla.org/sitemap.xml
Disallow: /api/
Disallow: /*/files/
Disallow: /media
Disallow: /en-US/search
Disallow: /es/search
Disallow: /fr/search
Disallow: /ja/search
Disallow: /ko/search
Disallow: /pt-BR/search
Disallow: /ru/search
Disallow: /zh-CN/search
Disallow: /zh-TW/search关联信息
反向链接、本文链接的其他页面和外部资料。
参考资料
其他外链4 条
- developer.mozilla.org/robots.txtdeveloper.mozilla.org/robots.txt
- developers.google.com/search/docs/advanced/robots/introdevelopers.google.com/search/docs/advanced/robots/intro
- support.google.com/webmasters/answer/6062598support.google.com/webmasters/answer/6062598
- www.workday.com/robots.txtwww.workday.com/robots.txt