The LexLint Scraping Brief

Is web scraping legal? · there is no internet law, only the law of every jurisdiction your scraper touches

No body of law governs the internet. A scraper is under the law of every jurisdiction it touches, and the jurisdictions are found by asking where each party is: you, the machine, the site, the people on its pages, and the model if one is in the loop.

Read this if you are about to collect data from other people's websites and want to know which law applies before you start.

You come away with the parties around a scraper, the four questions that find the jurisdictions, and what scraping shares with the law of AI agents and what is its own.

Which country's law applies to my scraper? · three jurisdictions, and how to find the third

Where you are, where the scraper runs, and where the target site has legal standing. The first two you know. The third takes a method, and the method is a ladder of evidence from a legal notice down to a domain ending.

Read this if you know which sites you want and need to say which jurisdictions' law you are under, including for a site whose owner you have never looked up.

You come away with the three jurisdictions, what the target site's location turns on for each body of law, the evidence ladder in the order to trust it, and what to do when you cannot tell.

Can I scrape public data? · public is not the same as permitted

A page anyone can see is a fact about how you got the data, not a permission to have it. Privacy law, copyright and computer-misuse law each answer the question differently, and only one of them is on your side.

Read this if you were told scraping is fine if the data is public and want to know how far that is true.

You come away with three bodies of law, three meanings of public, and the one where public helps you.

Can I get past robots.txt, a CAPTCHA or a login? · what is in the way, and what it means to get past it

A robots.txt file asks. A rate limit pushes back. A bot challenge and a CAPTCHA stand in the way. A login and a terms page make a contract. A letter revokes. Each rung moves you into a different body of law, and a court lit one rung in July 2026.

Read this if you hit something in the way and want to know what stepping around it would mean.

You come away with the eight rungs from weakest to strongest, what each is in law, which body of law each moves you into, and where a careful crawler stops.

Can I scrape names, emails and photos? · the people in the data

Consent is not available to a scraper. Legitimate interest is the basis with a test attached. The duty to tell people does not go away because you cannot tell them one by one. And “they put it online” is an argument about one limb of the test, not a way around it.

Read this if the pages you want have people on them: names, contact details, faces, posts.

You come away with why consent is out, what the legitimate-interest test asks, the notice you owe, where the regimes diverge, and the enforcement record.

So you want to build a scraper · the options, with the legal angle at each fork

Before you write a scraper, ask whether you need one: a bulk download, a feed or a documented API is a permission the scraping question never gives. If you do scrape, the forks after that are the API before the HTML, a service or your own, and, for your own, the tools and the rules.

Read this if you are about to build something that collects data from websites and want to choose the route with the fewest legal problems before you write a line.

You come away with four forks in order, what each route gives you in law and what it leaves with you, and the plain limit of what LexLint can find.

Which tools should I use to scrape? · the standard kit, and the legal angle of each row

The tools are open source, mature and free, and none of them decides how it is used. The standard kit in Python and JavaScript, by job, with the setting at each row that keeps you on the right side of a rule the rest of this brief explains, and the two things no tool should be asked to do.

Read this if you have decided to write your own scraper and want the standard tools, with the legal reason for how each one is set.

You come away with seven jobs, the standard tools for each, the switch that matters in law, the coding-assistant row, and the two refusals.

How to scrape responsibly · what each rule buys you in law

Ask first. Identify yourself. Slow down. Keep a record. Do not take what is clearly not public, and do not pass on what you have no right to. Each is good manners, and each one closes a legal door.

Read this if you have decided to go ahead and want to do it in the way that keeps the obvious problems off the table.

You come away with seven rules, the legal reason for each, the record to keep, and what to take to a lawyer.

What do I need to worry about? · the Scraping Checkup, up to twelve questions

Answer up to twelve questions about what you want to scrape, from where, and what you will do with it. The result is your scraping profile: the short list of things to be concerned about, in the order to deal with them, and a profile you can hand to an engineer or a lawyer.

Read this if you want the short list, not the reading.

You come away with the jurisdictions, the concerns in order, and the profile.

Disallow: / - what a website can actually forbid

Thirty years of asking crawlers to behave, and the three routes by which a request that binds nobody can start to bind someone.

Read this if you want the thirty-year history of robots.txt and the signals that followed it, and the three routes by which a request can come to bind.

You come away with why a crawler control is a request, the six stages the signals went through, and the three routes by which one can bind anyway.

Published 18 August 2026

403 Forbidden · which barriers the law will actually defend

Computer-misuse statutes want a wall, and a request is not one. Where the line sits in four jurisdictions, and the July 2026 ruling that put a rung on the ladder.

Read this if you want the case law behind the rungs: which barriers a computer-misuse statute will treat as a wall, in four jurisdictions.

You come away with one hinge phrased four ways, the July 2026 ruling that lit a rung, the ladder a builder faces, and what it means if you operate a crawler.

Published 23 August 2026

By continuing, you agree · when terms of use bind a crawler

A crawler never reads, clicks or assents to anything, and terms still decide more scraping cases than any statute. How the contract forms, and the two jurisdictions it cannot reach.

Read this if you want to know when a terms page binds a crawler that never read, clicked or assented to anything.

You come away with how the contract forms, where it carries on after intellectual property stops, the two jurisdictions it cannot reach, and what a breach is worth.

Published 28 August 2026

Eight ways to say no to bots · and no winner yet

Say nothing and EU law treats your pages as fair game for AI training. Saying no only counts if a machine can read it, the regulator has not yet said which formats count, and the answer meanwhile is to publish more than one.

Read this if you want to know which of the eight ways to say no to bots the law makes operative, and where the regulator stands.

You come away with why silence counts as yes under EU copyright, what happened when a reservation was tested in court, eight ways to say no and what each is worth.

Published 2 September 2026

There is no robots.txt for people · personal data in a scraped corpus

The lawn is public and the dog walkers are not. A page can be lawfully crawled, lawfully mined, and still unlawful to process, and copyright law has nothing to say about it.

Read this if you want the full treatment of personal data in a scraped corpus, regime by regime, including government pages.

You come away with why a page can be lawfully crawled and lawfully mined and still be unlawful to process, what the regulator said, the enforcement record, and where the regimes diverge.

Published 7 September 2026