foster/forgejo-docs
0
0
Fork 0
mirror of https://codeberg.org/forgejo/docs.git synced 2024-12-22 22:17:10 -05:00
Code Issues Activity

developer: infrastructure: blacklist bots that do not obey robots.txt

Browse source 
They get /forgejo/forgejo/archive/100448a007707c049a769cb71cd170fcb5b64a25.bundle

despite https://code.forgejo.org/robots.txt having

Disallow: /*/*/archive/
Disallow: /*/*/archive/*.bundle
This commit is contained in:
Earl Warren 2024-04-13 15:15:30 +02:00 committed by Gusted
parent 798fee5b11
commit d1662f520e
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/developer/infrastructure.md
View file

@ -271,6 +271,8 @@ server {
server_name code.forgejo.org;
location / {
deny 47.76.209.138; # crawler that does not obey robots.txt
deny 47.76.99.127; # crawler that does not obey robots.txt
proxy_pass http://10.6.83.195:8080;
client_max_body_size 2G;
#