
When AI engines or LLMs (large-language models) such as ChatGPT started to become popular, one of the main concerns some people had was the widespread crawling and indexing of content that companies like OpenAI and Anthropic engaged in while training these models, and that this is a form of theft. As some readers are probably aware, I’ve written about this before for previous editions of The Torment Nexus, and I’ve argued that instead of theft or copyright infringement, this kind of indexing should be considered a fair use under US copyright law, just as Google’s scanning of millions of physical books was, because of the transformative nature of that use (one of the four factors that judges have to consider in order to make a fair-use ruling). There have been a couple of court decisions that suggest this might become legal precedent — with certain restrictions — but the Supreme Court has yet to weigh in on the question.
In addition to the numerous copyright-infringement lawsuits that have been launched by authors, publishers, and newspaper owners like the New York Times (which tried to negotiate a payment-for-indexing deal with OpenAI but failed to reach an agreement on price), there have been a number of attempts to foil the scraping and indexing bots that AI companies use to hoover up content. Some are of the “spanner in the spokes” variety, like the Iocaine Project, named after the tasteless poison in the movie The Princess Bride (Never go against a Sicilian when death is on the line!). It is an open-source program that allows website owners to trap AI bots and scrapers in a kind of garbage-infested dead-end, where they are forced to go around and around indexing nonsense in a maze with no exit. Other attempts to foil the bots are a little more sophisticated, and Cloudflare has come up with several. But is the solution worse than the problem?
In case you’re not familiar with Cloudflare, it is what’s known as a CDN or content-delivery network, essentially a giant middleman that stands between your blog or website (or the sites and services of hundreds of thousands of corporations, governments and other entities) and the festering swamp of hackers and ne’er-do-wells that is the open internet. If you use Cloudflare, your site will load a lot faster — because the company caches it and then serves it from its own superfast network of servers — and is also protected from hacking attempts and DDoS (distributed denial of service) attacks, and a host of other nasty behavior. In the interests of full disclosure, I should point out that use Cloudflare for my website, but just for DNS hosting and rerouting, so that when you type in mathewingram dot com you get taken to the right website.
Note: This is a version of my Torment Nexus newsletter, which I send out via Ghost, the open-source publishing platform. You can see other issues and sign up here.
Continue reading “Does Cloudflare want to protect the web or control it?”

















