Insight 04

Eight ways to say no to bots · and no winner yet

Say nothing, and EU law treats your pages as fair game for text and data mining: letting a machine read them at scale, including to train one. You can say no. It is the one corner of law where a sign on your own server changes what others may do. But a machine has to be able to read the sign, nobody has agreed on what the sign looks like, and the regulator deciding that has not decided yet.

This is the article where we are the publisher rather than the crawler. UnGovr puts out a corpus, so every question below is one we have already had to answer about our own servers: what our robots.txt says, what our llms.txt says, and what licence rides on each exported record. The last one is a promise to strangers, which concentrates the mind more than a policy page does.

The rule: silence counts as yes

Start with the default, because it is the part people get backwards.

Article 4 of the EU's DSM copyright directive lets anyone mine works they can lawfully reach. The permission is automatic. It stops only if the rightsholder has reserved the work, which is the directive's word for having said no.

The exception or limitation provided for in paragraph 1 shall apply on condition that the use of works and other subject matter referred to in that paragraph has not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online.

Directive (EU) 2019/790, Article 4(3)

Recital 18 says what a reservation has to look like online. It narrows "appropriate" to machine-readable means, "including metadata and terms and conditions of a website or a service".

So the default runs the opposite way to most people's intuition. Mining is permitted unless you say otherwise. Silence is a licence.

Article 3 is worth knowing about because it works differently. That is the research exception, and nobody can reserve against it or write it away by contract. Article 4 is the one where publishing a signal is the only thing that changes the outcome.

The AI Act then puts a duty on the other side of the same fact. Since 2 August 2025, providers of general-purpose AI (GPAI) models must have a policy to identify and comply with reservations expressed under Article 4(3), "including through state-of-the-art technologies".

Somebody is now obliged to go looking for your sign. The open question is what they are obliged to be able to read.

What happened when this was tested in court

The leading case so far is Kneschke v. LAION (Higher Regional Court of Hamburg, 5 U 104/24, 10 December 2025). A photographer sued over a dataset built from his work. He lost: the court dismissed the appeal and held that the dataset's creation did not infringe.

The part that matters here is why his opt-out failed. It was written in ordinary prose, in the site's terms. The court measured that against what technology could reasonably parse at the time of the use, which was 2021, and held it did not qualify.

Two things about that holding are easy to misread.

It is tied to a date rather than settled for good. The court observed that the answer could be different today, given what language models can now read. A prose reservation may be climbing back into scope rather than out of it.

And it is not final. The court granted leave to appeal to Germany's Federal Court of Justice, on the ground that the case has fundamental significance. The most-cited decision in this area is one appeal away from being replaced.

The practical lesson survives either way. A reservation whose readability depends on an argument about the state of the art is one you may have to litigate. A reservation a parser resolves in a single request is not.

Eight ways to say no, and what each is worth

The question is not which format is best designed. It is which one a court, or the AI Office, will accept as appropriate machine-readable means. Standing matters more than features.

• robots.txt

  The only mechanism named in the general-purpose AI Code of Practice's Measure 1.3, "Identify and comply with rights reservations when crawling the World Wide Web". It is also the only one with thirty years of deployment and a published standard. Weakest expressively, strongest institutionally, and if you do exactly one thing, do this.

• The Text and Data Mining Reservation Protocol

  A W3C Community Group final report from May 2024, expressly not a W3C standard. It defines two properties, tdm-reservation and tdm-policy, which you can serve as an HTTP header, an HTML meta element, or a well-known JSON file. It was written specifically to satisfy Article 4(3), which is why it is worth publishing even though nothing yet requires anyone to read it.

• Cloudflare's Content Signals Policy

  A Content-Signal line inside robots.txt carrying search, ai-input and ai-train. It rides in a file every crawler already fetches, which is a real distribution advantage, and Cloudflare is candid that these express preferences rather than technical countermeasures.

• Really Simple Licensing

  Version 1.0 published on 10 December 2025. It supplements robots.txt with usage categories and, unusually, terms on which use is permitted, including compensation. It is the only candidate that answers "yes, for a price" rather than only yes or no.

• The remaining four

  The Commission's own study also names C2PA text-and-data-mining assertions, ai.txt, the Do Not Train registry and JPEG Trust. Those four and the four above are the eight. The study lists two commercial protocols besides. Being on a study's list is not adoption, and none of these currently has enough reach to be your only signal.

The regulator is deciding this right now

Between 1 December 2025 and 23 January 2026 the European Commission ran a stakeholder consultation on machine-readable rights reservations, supported by the AI Office.

What it is meant to produce is a list of generally agreed opt-out solutions, published by the Commission and the EU Intellectual Property Office, and used by the AI Office when it assesses whether general-purpose AI providers have met their Article 53(1)(c) obligation.

As of 19 August 2026 that list has not been published, and the process the Commission describes runs through workshops after the consultation closes. It is the single most load-bearing thing on this page: a format on that list becomes the safe answer, and a format left off it becomes an argument. If you are reading this at any distance from its date, check that first.

What a redundant posture looks like

Nothing has won, which makes redundancy rational rather than lazy. All four of these are cheap, none conflicts with another, and each covers a different failure of the others.

• The reservation in robots.txt

  Per-purpose tokens for the crawlers you mean to exclude. This is the one an obligated reader is most clearly expected to fetch.

• A Text and Data Mining Reservation Protocol assertion

  The well-known JSON file is the least intrusive form and covers your whole origin. It is the one purpose-built for the legal test.

• The same statement in the terms, in plain language

  Recital 18 names terms and conditions, and Kneschke is tied to a date rather than settled for good. Prose is not sufficient on its own today. It is not worthless either, and it costs a paragraph.

• The date it was published, on record

  The test is applied as at the time of the use, so the date you reserved is a fact you may need. A commit in a public repository, or a dated changelog entry, is enough.

What this does not buy you is anything outside the EU's reach, and the differences are not small.

Japan's Article 30-4 permits mining with no opt-out mechanism at all, so there is nothing there for your signal to attach to. Singapore's exception cannot be reserved against or contracted away. The United States has no equivalent provision, so a reservation there is an assertion of intent that may matter to a fair-use analysis rather than a switch that flips one.

Which leaves the question every one of these four articles has quietly avoided. All of this treats a page as a work: who may copy it, who may mine it, what a sign on it means. Some of what you publish is not a work at all. It is a person. That is the last article of this season.

Meanwhile the Law pages carry the per-jurisdiction posture we hold today, and LexLint itself will read your app's declared profile and report which of these attach to it.

Sources and further reading

• Directive (EU) 2019/790, Article 4 and Recital 18 The provision the whole question turns on, and the recital that narrows "appropriate manner" for online content.
• European Commission, stakeholder consultation on AI and copyright compliance What the Commission and the AI Office are trying to produce, and the process by which the agreed list of opt-out solutions is meant to arrive. Check this before relying on anything else here.
• General-Purpose AI Code of Practice, copyright chapter Measure 1.3 and the commitment to identify and comply with rights reservations, including through protocols beyond robots.txt.
• Text and Data Mining Reservation Protocol, Community Group final report The specification, its three delivery mechanisms, and its own statement that it is not a W3C standard.
• EDRLab, TDM Reservation Protocol The implementation guide, which is the shorter route to actually deploying it.
• Cloudflare, the Content Signals Policy The syntax, and a clear statement of what expressing a preference does and does not do.
• Really Simple Licensing Version 1.0, published December 2025. The only candidate here designed to express terms rather than refusal.
• Higher Regional Court of Hamburg confirms AI training was permitted, Kneschke v. LAION The December 2025 appellate decision, why the prose reservation failed, and the grant of leave to appeal further.
• Machine-readable but still not actionable A critical reading of the same decision, arguing the gap between a reservation being readable and being usable is where the real problem sits.

This article is a research summary of published law and published specifications, with every source linked above. It is not legal advice and it creates no attorney-client relationship. Law cited as of 19 August 2026; the Commission process described is in progress and its output will change what this page says.