2026-02-01 09:46 UTC
Replies (13)
-
@robb@social.lol 2026-02-01 09:58
@matthiasott@mastodon.social This is what I currently do https://rknight.me/blog/blocking-bots-with-nginx/ But also yes, I’m seeing more of it. Specifically Chinese bots that go over every single page on my site one by one.
-
@webrocker@mastodon.social 2026-02-01 10:11
@matthiasott@mastodon.social hard to tell for my personal sites b/c I don't monitor the traffic/stats at all. but in client projects having trackers there's been a huge increase in botish visits in the recent months, and even those who claim to filter out the known bots from their traffic stats have seen a jump in suspicious looking visits
-
@adactio@mastodon.social 2026-02-01 11:11
@matthiasott@mastodon.social Yes, it’s bad: https://adactio.com/journal/21831
-
@HaraldKi@nrw.social 2026-02-01 11:18
@matthiasott@mastodon.social Check more or less recent posts from @ Codeberg@social.anoxinon.de @ lwn@lwn.net @ osm_tech@en.osm.town
-
@hvdklauw@mastodon.social 2026-02-01 13:01
@matthiasott@mastodon.social yes, a while ago, 99% of our load came from AI crawlers, after updating the robots.txt it became less, but there are still a lot of bad actors out there and that's the ones who at least set the user agent to identify themselves, as user agent blocking becomes more common I really wonder how bad they'll get
-
@boldewyn@mastodon.social 2026-02-01 14:47
@matthiasott@mastodon.social hell yes! This is a terrible problem on my site codepoints.net, even with CloudFlare in front of it. On single code point pages I deep-link to my site search for similar code points. In the last year it became worse and worse that “users” from China with Chrome follow those links and bring the site down due to excessive DB load from the search. I hated having to add rate limiting etc, and I know of at least one legitimate user who was bitten by it. Such a pest on the open web!
-
@Doctacosa@social.interordi.com 2026-02-01 15:24
@matthiasott@mastodon.social I only heard stories until this week, when I was hit. I have a pet project that's fetching a lot of data and is a bit slow, but perfectly fine for human use. However, I suddenly had both Baidu and Meta's crawlers loading several pages at once, slowing everything down dramatically. Blocking both bots from that section of my site with robots.txt solved the problem... at least for now!
-
@tylersticka@social.lol 2026-02-01 15:26
@matthiasott@mastodon.social Yep. Not as bad as others, but most recently I’ve seen a surge in traffic from Singapore. Will be updating my `robots.txt` list and looking into blocking soon.
-
@mayank@front-end.social 2026-02-01 16:23
@matthiasott@mastodon.social yeah… and it's one of the big reasons i stopped blogging.
-
@hdv@front-end.social 2026-02-01 16:43
@matthiasott@mastodon.social 👋 yes, after a few times of an unexpectedly larger Netlify bill due to astronomical traffic (I think due to LLM bots) I switched hosts so that I can have a larger plan at lower cost.
-
@cory@follow.coryd.dev 2026-02-01 16:47
@matthiasott@mastodon.social I'm listing AI (and other poorly behaved crawlers) in my robots.txt while also sending them a 403 to anything else they request should they not honor that. I also had to block all traffic from China due to bots fetching endless pages sequentially. https://www.coryd.dev/posts/2026/blocking-entire-countries-because-of-scrapers
-
@dj3ei@mastodon.radio 2026-02-01 23:21
OSM mentioned that as a poblem in https://en.osm.town/@osm_tech/115968544599864782 and other messages. @matthiasott@mastodon.social
-
@mauricerenck@mastodon.online 2026-02-01 16:02
@matthiasott@mastodon.social I have this once or twice a week and now started geo blocking, currently Singapore. There were even some bots saying "sorry we are beta and break things, block us if you want" in their user agent string... So I blocked some user agents too. But most of the requests look like normal desktop browsers, so i needed geo blocking. The requests came so rapidly I couldn't use the Kirby panel anymore even though I already had caching enabled.