diff options
| author | mms <michal@sapka.me> | 2024-06-25 15:43:38 +0200 |
|---|---|---|
| committer | mms <michal@sapka.me> | 2024-06-25 15:43:38 +0200 |
| commit | ab8b812dddce2b05833bb5e2dfed8be876a0a1dc (patch) | |
| tree | 9b720680a1fac5fdf144c9cb9de73627e5ad6261 /static | |
| parent | bda6eb69a7e63d89a8fa04668c3bbfc80a17e6c5 (diff) | |
chore: block bots
Diffstat (limited to 'static')
| -rw-r--r-- | static/robots.txt | 76 |
1 files changed, 73 insertions, 3 deletions
diff --git a/static/robots.txt b/static/robots.txt index 867a36c..5cb00a4 100644 --- a/static/robots.txt +++ b/static/robots.txt @@ -6,9 +6,6 @@ Disallow: /git/ User-agent: ChatGPT-User Disallow: / -User-agent: GPTBot -Disallow: / - User-agent: Mediapartners-Google Disallow: / @@ -21,3 +18,76 @@ Disallow: / User-agent: CCBot Disallow: / +# Dark Visitors Robots.txt + +# AI Data Scraper +# https://darkvisitors.com/agents/applebot-extended + +User-agent: Applebot-Extended +Disallow: / + +# AI Data Scraper +# https://darkvisitors.com/agents/bytespider + +User-agent: Bytespider +Disallow: / + +# AI Data Scraper +# https://darkvisitors.com/agents/ccbot + +User-agent: CCBot +Disallow: / + +# AI Data Scraper +# https://darkvisitors.com/agents/claudebot + +User-agent: ClaudeBot +Disallow: / + +# AI Data Scraper +# https://darkvisitors.com/agents/diffbot + +User-agent: Diffbot +Disallow: / + +# AI Data Scraper +# https://darkvisitors.com/agents/facebookbot + +User-agent: FacebookBot +Disallow: / + +# AI Data Scraper +# https://darkvisitors.com/agents/google-extended + +User-agent: Google-Extended +Disallow: / + +# AI Data Scraper +# https://darkvisitors.com/agents/gptbot + +User-agent: GPTBot +Disallow: / + +# AI Data Scraper +# https://darkvisitors.com/agents/omgili + +User-agent: omgili +Disallow: / + +# Undocumented AI Agent +# https://darkvisitors.com/agents/anthropic-ai + +User-agent: anthropic-ai +Disallow: / + +# Undocumented AI Agent +# https://darkvisitors.com/agents/claude-web + +User-agent: Claude-Web +Disallow: / + +# Undocumented AI Agent +# https://darkvisitors.com/agents/cohere-ai + +User-agent: cohere-ai +Disallow: / |