We had a data quality problem. More than half the documents in one of our pipelines, 1,453 out of 2,677, were headline stubs: a title, a publisher name, a link, and a median of about 86 characters of text. The article bodies had never been fetched. One of those stubs was 91 characters long, and the extraction we ran against it produced a project name that appears nowhere in the stub, nowhere in our corpus, and nowhere in the world.
The engineering fix was obvious. Go and fetch the article bodies. Before we built it, we went to read what the publishers actually permit.
At every publisher we checked, robots.txt said yes and the terms of use said no.
The handshake that worked, and why it isn’t the model any more
robots.txt has been holding the web together since 1994, and it has almost nothing to do with the file itself. It worked because search engines and publishers wanted the same outcome. A crawler took the content and sent readers back. Publishers who didn’t want that had a one-line way to say so, and search engines honoured it, because honouring it cost them nearly nothing and defecting would have cost them the trust that made the whole arrangement work.
The bargain underneath was traffic. That is what made the handshake self-enforcing for thirty years, without a court ever having to weigh in on whether a text file constitutes a contract.
AI ingestion doesn’t return traffic. Whatever else has changed, that is the load-bearing thing: the exchange that made the old agreement enforce itself quietly stopped being true, and what’s left is a file designed to express a preference inside a relationship that no longer exists.
Publishers noticed before most builders did.
Four publishers, four different answers
We read seven publishers’ live terms. Here is the spread, and the point of listing them is not that any one is wrong. Each is a rational response to a situation nobody has settled.
The Guardian writes the contradiction out explicitly. Its terms state that “our robots.txt notice does not, and shall not, constitute the Guardian’s permission or approval for the purposes of these terms and conditions”, and that the terms “shall prevail in all respects over our robots.txt notice”. Two documents, deliberately saying different things, with a stated order of precedence between them.
The Associated Press goes the other way and makes robots.txt contractual, requiring that you “comply with any instructions set out in any robots.txt files associated with the AP Works”. Its operative language also reaches past crawlers to “digital engines of any kind”, drafting written after a 2024 court ruling that terms may not bind a logged-out scraper at all.
The BBC puts prose inside the machine file. Its robots.txt carries a human-readable header prohibiting scraping, crawling, systematic extraction, retrieval-augmented generation, AI-powered search, and dataset creation from BBC content.
The New York Times is the only one of the seven that describes an automated-access carve-out in its terms, and it’s worth quoting because it defines the shape of the only safe harbour on offer: “a public search engine’s use of spiders for creating search indices solely for the inclusion of links and short, non-AI synthesized snippets of the Content in search results”.
Then there’s the machine-readable attempt. Cloudflare’s Content Signals, launched September 2025 across more than 3.8 million managed domains, splits the question into separate signals for search, for AI training, and for AI input, and attaches an express reservation of rights under European copyright law. It’s the most serious proposal anyone has put on the table. Cloudflare is also candid that the signals express preferences and “are not technical countermeasures against scraping”, which is the honest framing and also the reason it doesn’t yet settle anything.
Six of the seven publishers prohibit robots, scrapers and crawlers by name in their terms. None of the seven permits automated full-text retrieval for a commercial product. And at all seven, article URLs are not disallowed to a generic crawler in robots.txt. The blocking is done entirely by named user agent: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot. A fetcher that doesn’t announce itself as one of those isn’t disallowed by the file and is prohibited by the terms.
The two layers say different things, and the publishers know it.
Two articles, opposite answers, and no way to tell them apart
Here’s what that looks like at the level of an actual working system.
Two articles came through the same feed within days of each other, about the same kind of project. Same shape, same metadata, indistinguishable to anything automated looking at them.
The first was from World Construction Network, published by GlobalData. Its robots.txt is about as permissive as the open web gets: the generic group blocks only WordPress plumbing and search queries, and exactly two named agents get a full disallow. The article fetched cleanly on the first attempt with a plain, honestly identified client. No gate, no paywall, no complaint. On a robots-only reading it’s a green light.
The terms are where it turns. GlobalData’s site terms permit downloading extracts for personal use and state that you “must not use any part of the content on our site for commercial purposes without obtaining a licence”, and they reserve database rights on top of copyright. Worth being precise here, because we’re naming a company: the governing terms page for that specific site was not retrievable when we looked, and we did not route around the block to read it. What we read were two sibling GlobalData terms pages that both say the same thing and neither of which claims to cover other GlobalData websites. So we have the house position and a reasonable inference, and we don’t have the operative document.
The second was from Solar Power World. Its robots.txt opens with a line that tries to form a contract: “As a condition of accessing this website, you agree to abide by the following content signals.” It carries search=yes, ai-train=no, an express Article 4 reservation of rights, and a full disallow on ClaudeBot by name. The article wouldn’t load at all: three attempts, the same redirect loop each time.
Its actual restriction took some finding. The terms-of-use URL returns a 404. The parent company’s terms link redirects across domains to a rebranded parent whose terms describe themselves as covering a different website entirely and contain no scraping provision at all. The operative clause is in the privacy policy, dated 2009, and it prohibits material being “reproduced, distributed, transmitted, cached or otherwise used” without prior written permission. It names caching specifically, which is exactly what storing an article body is.
Two publishers. Opposite answers. Both arrived in our system as the same kind of object, and nothing in either one told us which was which.
The generalisable finding is worse than the anecdote. On a sample of two articles, terms discovery produced a robots block on the governing terms page, a cross-domain redirect to a rebranded parent, a dead terms URL, terms that disclaimed covering the site in question, and the binding clause hiding inside a document about privacy. No pipeline can perform that, and for one of the two a human researcher couldn’t finish it either.
The line we actually drew
There’s a distinction in all of this that I think matters more than any of the documents, and it took us a while to see it.
There is a real difference between fetching a page that is served to you, and fetching a page after the server declined to serve you and you disguised your client to try again.
The first supports an argument about implied permission. The second attacks it. Commercial unblocking services exist precisely to do the second, and they’re very good at it, which is the problem. A tool that quietly turns a refusal into a success removes the one signal that tells you where the line was.
In July 2026, News Corp filed a counterclaim against Brave that leads with the allegation that Brave masked web crawlers to prevent publisher detection or blocking. Brave is not a distant analogy for what we’re building. It’s an indexer of publisher content that sells derived output to enterprise customers, which is close enough that we read the filing carefully.
So we wrote a rule into the design and I’d recommend it to anyone building in this space. Log what the honestly identified client got back, every time. Never let the pipeline escalate to an unblocker by itself. If it escalates, a person decided that, for that domain, and there is a record of the decision. That log is what distinguishes fetching from circumventing, and it has to be written at the time, because it can’t be reconstructed afterwards.
Nobody has finished writing the rules
Three efforts are running at once. The IETF has a working group on AI preferences with drafts targeting the end of August 2026, which would give machine-readable signals a standards footing for the first time. Cloudflare’s Content Signals is deployed at real scale but expresses preferences rather than enforcing them. RSL launched in September 2025 with more than fifty partners and no enforcement mechanism at all.
The law underneath them is genuinely unsettled, and I’d rather say so than pretend I know how it lands. I’m not a lawyer. But you don’t need to be one to read the clearest illustration available: in late 2025, two judges in the same district reached opposite conclusions three weeks apart on how much of a news article is protected when it’s summarised. One treated news summaries that restate uncopyrightable facts as not substantially similar. The other held that non-verbatim summaries can infringe where they track the structure and expressive choices of the original. The first American appellate ruling on fair use in AI training was argued in June 2026 and hasn’t landed yet.
The part I didn’t expect
We set out to fix the stubs. What the reading showed is that the stub layer, the headline and the link and the short excerpt, sits close to the inside of that New York Times carve-out, and at Solar Power World is expressly granted by search=yes. Full article bodies put us decisively outside it.
The limitation we were about to engineer away was a large part of the reason we’d never had a problem.
So the fix changed shape rather than getting abandoned. We’re fetching bodies for a named list of publishers whose terms we’ve actually read, each registered as its own source, each carrying what the terms said and the date we checked. Unknown publishers keep flowing as stubs and their bodies stay unfetched until somebody has read them. And we split the question of what we fetch from the question of what we store, because retention drives more exposure than access does, and because storage is the one part of this we control completely.
That’s the reusable lesson, and it’s why this was worth writing up: before you fix a long-standing limitation, ask what it has been protecting. Some defects are load-bearing.
Where this leaves us
We’re building a platform that reads public signals and tells firms what work is coming before it arrives. The data underneath it has to be legitimately obtained, and not as a posture. If we can’t account for where a signal came from, it shouldn’t be in the product, and that gets considerably harder to fix once customers are relying on it.
What we have is a standing policy document, a short watchlist of the things that would change our answer, and a monthly check against it. The watchlist runs to five items and two dates. We manage the problem rather than solve it, and given where the ground currently sits, I don’t think anyone honest can offer you solved.
If you want a system that brings you work and can still stand up to a question about where its data came from, that’s what we build and run. We’ve published how the engine actually works if you want the mechanics before the conversation.
If you’d rather move fast and find out later, there are plenty of cowboys who’ll take the job. They’re cheaper right up until the letter arrives.
Every publisher document quoted here was read on 31 July 2026 and terms change often. Where we couldn’t retrieve a governing document, this piece says so rather than filling the gap.