minus-squareProntomomo@lemmy.worldtoSelfhosted@lemmy.world•Cloudflare blocking AI crawlerslinkfedilinkEnglisharrow-up5·5 days agoI don’t think this blocks crawlers. About 1/5 websites uses cloudflare, the significant thing here’s is that AI scraping is now blocked by default on most of those sites, NOT crawling linkfedilink
minus-squareProntomomo@lemmy.worldtomemes@lemmy.world•You changing them or leaving them for months until it's the right time again?linkfedilinkarrow-up1·8 months agoFour months linkfedilink
I don’t think this blocks crawlers. About 1/5 websites uses cloudflare, the significant thing here’s is that AI scraping is now blocked by default on most of those sites, NOT crawling