Pro@programming.dev to Technology@programming.devEnglish · edit-27 months agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comimagemessage-square99fedilinkarrow-up1667arrow-down12file-textcross-posted to: technology@lemmy.world
arrow-up1665arrow-down1imageCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.dev to Technology@programming.devEnglish · edit-27 months agomessage-square99fedilinkfile-textcross-posted to: technology@lemmy.world
minus-squaredeur@feddit.nllinkfedilinkEnglisharrow-up3arrow-down9·edit-29 months agoIf it is scraping it is scraping. If the endpoint canonically is intended to be a machine readable endpoint (do not respond with “but html is machine-readable” as you know that is not the point) it is not. Your scraper would be harmful.
minus-squareFizzyOrange@programming.devlinkfedilinkEnglisharrow-up6·9 months agoSo search engines shouldn’t exist? This is absurdly simplistic.
minus-squaregrue@lemmy.worldlinkfedilinkEnglisharrow-up8·9 months agoBut html is machine-readable and that absolutely is the point! Never forget what they stole from us.
If it is scraping it is scraping. If the endpoint canonically is intended to be a machine readable endpoint (do not respond with “but html is machine-readable” as you know that is not the point) it is not. Your scraper would be harmful.
So search engines shouldn’t exist? This is absurdly simplistic.
But html is machine-readable and that absolutely is the point!
Never forget what they stole from us.