diff options
| author | Alex Auvolat <alex@adnab.me> | 2018-09-01 16:07:22 +0200 |
|---|---|---|
| committer | Alex Auvolat <alex@adnab.me> | 2018-09-01 16:07:22 +0200 |
| commit | 81b5a844a2a155e28c497a8ce671eb5f02803e5d (patch) | |
| tree | 15e878ebe47d5993e7af46a48a1ccdc1dd1e4b2f /shardweb/assets/static/robots.txt | |
| parent | c6ec33d6e612168e14d77007915a4ea423c55a2e (diff) | |
| download | shard-81b5a844a2a155e28c497a8ce671eb5f02803e5d.tar.gz shard-81b5a844a2a155e28c497a8ce671eb5f02803e5d.zip | |
Import shardweb0.0.2
Diffstat (limited to 'shardweb/assets/static/robots.txt')
| -rw-r--r-- | shardweb/assets/static/robots.txt | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/shardweb/assets/static/robots.txt b/shardweb/assets/static/robots.txt new file mode 100644 index 0000000..3c9c7c0 --- /dev/null +++ b/shardweb/assets/static/robots.txt @@ -0,0 +1,5 @@ +# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file +# +# To ban all spiders from the entire site uncomment the next two lines: +# User-agent: * +# Disallow: / |