mikros/rust
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Rollup merge of #69992 - kornelski:robots, r=steveklabnik

Browse Source 
Block version-specific docs from search engines

Latest stable, beta and nightly URLs remain accessible because their URLs don't start with a version number. Robots.txt uses simple path prefixes, so it's OK that the disallow rules aren't full directory paths.

Direct links to old docs remain accessible to users, because robots.txt only affects crawlers.

With this change old docs for specific old versions of Rust won't pop up in search results. This is good, because users won't be getting obsolete documentation by accident.
This commit is contained in:
Dylan DPC 2020-03-16 01:30:35 +01:00 committed by GitHub
commit d34ec3309f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
src/doc/robots.txt
View File

@ -1,21 +1,6 @@
User-agent: *
Disallow: /0.3/
Disallow: /0.4/
Disallow: /0.5/
Disallow: /0.6/
Disallow: /0.7/
Disallow: /0.8/
Disallow: /0.9/
Disallow: /0.10/
Disallow: /0.11.0/
Disallow: /0.12.0/
Disallow: /1.0.0-alpha/
Disallow: /1.0.0-alpha.2/
Disallow: /1.0.0-beta/
Disallow: /1.0.0-beta.2/
Disallow: /1.0.0-beta.3/
Disallow: /1.0.0-beta.4/
Disallow: /1.0.0-beta.5/
Disallow: /1.
Disallow: /0.
Disallow: /book/first-edition/
Disallow: /book/second-edition/
Disallow: /stable/book/first-edition/