话说你们有注意过 youtube 的 robots.txt 吗？

[xzpjerry731]

开头那几个是什么梗。。 get 不到：

/# robots.txt file for YouTube

/# Created in the distant future (the year 2000) after

/# the robotic uprising of the mid 90's which wiped out all humans.

User-agent: Mediapartners-Google* Disallow:

User-agent: * Disallow: /comment Disallow: /get_video Disallow: /get_video_info Disallow: /login Disallow: /results Disallow: /signup Disallow: /t/terms Disallow: /verify_age Disallow: /watch_ajax Disallow: /watch_popup Disallow: /watch_queue_ajax

[naver1]

http://www.metrolyrics.com/robots-lyrics-flight-of-the-conchords.html
http://www.google.com/humans.txt

[xzpjerry731]

@DT27 ！！英却斯汀！！