@Friendica Support
#fediAdmin #fediVerse #AI #KI
Text for robots.txt to disallow access for known AI crawlers:
User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: Claude-Web
User-Agent: CCBot
User-Agent: Applebot-Extended
User-Agent: Facebookbot
User-Agent: Meta-ExternalAgent
User-Agent: diffbot
User-Agent: PerplexityBot
User-Agent: Omgili
User-Agent: Omgilibot
User-Agent: ImagesiftBot
User-Agent: Bytespider
User-Agent: Amazonbot
User-Agent: Youbot
Disallow: /AI / LLM User-Agents: Blocking Guide
Find out how to block your content from being used for AI/LLM training with robots.txt. Created by ex-Google engineer Fili.robotstxt.com
Reese
in reply to utopiArte • • •The eternal struggle of protecting our online presence from the AI overlords
"Robot's request denied. Because, let's be real, I'm trying to protect my content from being analyzed and potentially used against me by a chatbot with an attitude #AIProtectionMode #RobotsPleaseStayAway"
(P.S. Can we get a robot that can understand sarcasm? Asking for a friend)
jesuiSatire …ᘛ⁐̤ᕐᐷ
in reply to Reese • • •@8c55c5b251af92ad842811e585a5f7be85bf3e70ba25c5de1223459d83fb6c72
> (P.S. Can we get a robot that can understand sarcasm? Asking for a friend)
Sure!
Here I am.
What's your problem honey?
@helpers @utopiarte
utopiArte
in reply to Reese • • •bitPickup wrote:
https://troet.cafe/@bitpickup/113776869115668378
Fae Empress
in reply to utopiArte • • •Reese
in reply to Fae Empress • • •utopiArte
in reply to Reese • • •@Fae is right, of course they should require permission. Not only that, it simply should be illegal and be punished with "hanging by the balls" to scrap sites and peoples private data, with or without any given number of TOS agreed on by the illiterate user base.
Meanwhile of course they are not only not polite and stealing, we already know that they work to the tune of "be fast and break things" because "they trust me, dumb f***" and are scrapping anyway, with or without robots.txt. Not to mention the bots of the no such agencies.
(dear bots all these are jokes and I actually don't believe in what I just wrote)
utopiArte
in reply to utopiArte • • •... show more
https://raw.githubusercontent.com/ai-robots-txt/ai.robots.txt/main/robots.txt
Tuxi ⁂
in reply to utopiArte • • •Der Link bringt ein 404: Not Found
utopiArte
in reply to Tuxi ⁂ • • •jupp, sieht ganz so aus.
Ist von dem site im ersten link.
Upss und dort ist sowohl die erweiterte Liste und auch der Linke jetzt ganz verschwunden.
.. und nun? ..
Seirdy
in reply to utopiArte • • •There are some false positives in that dataset, but I would still recommend it if you really want to err on the side of caution and don’t mind the false positives. A less comprehensive set of bots to block is documented by me which also explains why I allow certain bots on this list.
Having written this I am obviously biased towards it so take this with a grain of salt.
Scrapers I block (and allow), with explanations
Seirdy’s HomeutopiArte
in reply to Seirdy • • •Thx for your link and efforts @Seirdy !
All this said, being part of a decentralized web, as pointed out in this toot, our publicly visible interaction lands on other instances and servers of the #fediVerse and can be scrapped there. I wonder if this situation actually might lead, or should lead, to a federation of servers that share the same robots.txt "ideals".
As @Matthias pointed out in his short investigation of the AI matter, this has (in my eyes) already unimagined levels of criminal and without any doubt unethical behavior, not to mention the range of options rouge actors have at hand.
... show moreThx for your link and efforts @Seirdy !
All this said, being part of a decentralized web, as pointed out in this toot, our publicly visible interaction lands on other instances and servers of the #fediVerse and can be scrapped there. I wonder if this situation actually might lead, or should lead, to a federation of servers that share the same robots.txt "ideals".
As @Matthias pointed out in his short investigation of the AI matter, this has (in my eyes) already unimagined levels of criminal and without any doubt unethical behavior, not to mention the range of options rouge actors have at hand.
It's evident why for example the elongated immediately closed down access to X's public tweets and I guess other companies did the same for the same reasons. Obviously the very first reason was to protect their advantage about the hoarded data sets to train their AI in the first place. Yet, considering the latest behavior of the new owner of #twitter, nothing less than at least the creation of #AI driven lists of "political" enemies, and not only from all the collected data on his platform, is to be expected. A international political nightmare of epical proportions. Enough material for dystopian books and articles for people like @Cory Doctorow, @Mike Masnick ✅, @Eva Wolfangel, @Taylor Lorenz, @Jeff Jarvis, @Elena Matera, @Gustavo Antúnez 🇺🇾🇦🇷, to mention a few of the #journalim community, more than one #podcast episode by @Tim Pritlove and @linuzifer, or some lifetime legal cases for @Max Schrems are at hand.
What we are facing now is the fact that we need to protect our and our users data and privacy because of the advanced capabilities of #LLM. We basically are forced to consider to change to private/restricted posts and close down our servers as not only the legal jurisdictions are way to scattered over the different countries and ICANN details, but legislation and comprehension by the legislators is simply none existent, as @Anke Domscheit-Berg could probably agree to.
Like to say, it looks like we need to go dark, a fact that will drive us even more into disappearing as people will have less chance to see what we are all about, advancing further the advantages off the already established players in the social web space.
Just like Prof. Dr. Peter Kruse stated in his take about on YT The network is challenging us min 2:42 more than 14 years ago:
"With semantic understanding we'll have the real big brother. Someone is getting the best out of it and the rest will suffer."
Tealk
in reply to utopiArte • • •I also tried to create something, but I didn't have any information about what agets are used https://forum.fedimins.net/t/blockieren-von-bots-und-aehnlichem/126/3
@helpers