Summary
robots.txt の書式を解説Reference
A Standard for Robot Exclusionhttp://www.robotstxt.org/wc/norobots.html
A Standard for Robot Exclusion - robots.txtの書式
Summary
robots.txt の書式を解説Reference
A Standard for Robot Exclusionhttp://www.robotstxt.org/wc/norobots.html