Activities / What code does

Crawls the web

crawls_web

Your app runs automated software that visits and downloads web pages at scale, rather than a person browsing them one at a time.

scope code
in the registry since 1.0
1,751 laws in 256 jurisdictions name it

What triggers it

The lint's definition. It fetches pages or files from sites you do not operate, by any automated means, whether or not the code calls itself a crawler.

This covers a search-indexing bot, a price-comparison tool that pulls listings from other sites, and a research crawler that harvests articles for later analysis.

What does not

A single scheduled call to a partner's documented API is not crawling on its own.

Often confused with Reuses other publishers' content and Trains models.

The law it brings into scope

The kinds of law that name this activity in the LexLint law library, with the number of laws and jurisdictions in each, as of 19 Oct 2026. The examples are chosen by rule, laws in force first and the widest jurisdiction first; each section's page lists every law.

Area of law	Laws	Jurisdictions
Privacy law Have a legal basis for personal data, honor people's rights over it, and report a breach. General Data Protection Regulation, Principles Relating to Processing of Personal Data and Accountability (Article 5) European Union UK GDPR Article 82, Data Protection Act 2018 Section 169, and Information Commission Enforcement United Kingdom DOJ Data Security Program (Bulk Sensitive Personal Data Rule) United States	1,073	233
Scraping law Stay within computer-access law, a site's terms, and the copyright in what is copied. Cybercrime Directive, Articles 2(d) and 3 to 12 (offenses against information systems, penalties and jurisdiction) European Union Copyright, Designs and Patents Act 1988, secondary infringement and the meaning of infringing copy United Kingdom Computer Fraud and Abuse Act (unauthorized access and the gates-based authorization test) United States	531	247
Cybersecurity law Build software securely, handle reported flaws, and report incidents on a deadline. National Cybersecurity Incident Reporting Measures China Digital Code, general cybersecurity duty on digital-object owners and holders Kazakhstan Information and Communications Network Act, Report on Computer Security Incidents South Korea	90	66
Reuse law Reproduce other publishers' content only as far as an exception or a license allows. Copyright, Designs and Patents Act 1988, literary works including computer programs, acts restricted by copyright, and adaptation United Kingdom Copyright Act, exclusive rights and infringement, 17 U.S.C. §§ 106 and 501 United States CPI Article L122-5-3, Text and Data Mining Exception France	26	26
Product and AI liability law Answer in damages when defective software or an AI system causes harm. Consumer Rights Act 2015, section 46, remedy for damage that digital content causes to a consumer's device or other digital content United Kingdom Võlaõigusseadus, Sections 1061 to 1067, Liability for a Defective Product (Directive 85/374/EEC) Estonia Cybersecurity Program safe harbor from exemplary damages (S.B. 2610) Texas	23	22
AI law Tell people when they are dealing with an AI system or its output, and control the uses the law calls high-risk. AI Act, Article 5 (prohibited AI practices) European Union Act LXXV of 2025 on the Domestic Implementation of the EU AI Regulation Hungary Lov nr. 467 af 14. maj 2025, National Competent Authorities and Article 5 Enforcement Denmark	14	11
Communications law Get consent before marketing, calling, recording or storing data on a device, and honor an opt-out. Spam Act 2003, Address-Harvesting Software and Harvested-Address Lists Australia	1	1

Counterparties that usually come with it

a proxy or scraping service that fetches pages for it
a headless-browser host

A component that cannot run as documented without one of these names it in its ACTIVITIES.md under talks_to, with required: true. That party's jurisdiction then joins the run of every product that uses the component, and the finding says why it is there. A default the deployer can change is asked about, never added. Talking to a party never makes a product that party.

In a declaration

A library or framework that makes this possible enables it; an application whose shipped code does it performs it.

For a library

activities: - id: crawls_web mentioned: enables enabled: requires_configuration where: [<the module that carries it>] note: <one sentence>

For an application

activities: - id: crawls_web mentioned: performs enabled: by_default where: [<the module that carries it>] note: <one sentence: what it does, and the kind of law it attracts>

History

In the registry since version 1.0, 16 Oct 2026.

In the glossary: Crawls the web.

The activity registry is LexLint's open list; this page's text is licensed under Creative Commons Attribution 4.0.