No. robots.txt is a voluntary convention, not a statute, and no United States court has held that ignoring it is by itself unlawful. It was standardised in 2022 as RFC 9309, and that document describes a protocol for well-behaved crawlers, not a legal instrument. A file on a web server cannot create a contract with a visitor who never agreed to anything.
That is the narrow answer, and stopping there would be misleading. robots.txt still changes your legal position, because it establishes notice. A site that later brings a breach-of-contract or trespass-to-chattels claim will point at its robots.txt to argue you knew you were unwelcome. Notice does not create liability on its own, but it removes the "we had no idea" defence and it colours how a court reads everything else you did.
This page describes how the protocol works and how courts have treated it. It is not legal advice.
robots.txt is fetched by the crawler, parsed by the crawler, and obeyed by the crawler. Nothing enforces it. A Disallow line is a request, and the server will still return the page to a client that ignores it. That is why sites that genuinely need to stop automated access use rate limits, authentication, or an anti-bot vendor instead, all of which are technical controls rather than requests.
The distinction that matters legally is between a request and a gate. The Supreme Court's reading of the Computer Fraud and Abuse Act in Van Buren v. United States (2021) turned on whether a gate was up or down, not on whether a rule was broken. A page served freely to anyone who asks has its gate up, whatever robots.txt says about it. Defeating a login, sharing credentials, or circumventing an access control is a different act and is where CFAA exposure begins.
Three places, none of them a direct robots.txt claim.
As evidence of notice in a contract or trespass claim, as above.
In Content-Signal declarations, a newer convention that lets a site state how fetched content may be used, separately from whether it may be fetched. That is a licensing statement rather than an access rule, and it is the layer most relevant to AI training rather than to search.
In your own vendor's policy. Some providers honour robots.txt on your behalf and some do not, and the difference will not appear in a pricing table. If you are buying collection as a service, ask directly, because the answer becomes part of your compliance story rather than theirs.
String publishes its own crawler policy at usestring.ai/robots.txt as per-agent groups rather than a single wildcard block. Answer-engine crawlers that produce citations are allowed. Training-data crawlers are disallowed. Google-Extended is deliberately left allowed, because blocking it could cost grounding in Gemini's app surface while Google Search is governed by the ordinary Googlebot rules.
That is a statement about our own site, not a claim about what you may collect from others. Our terms of service set out what we require of customers, and the trust center covers our compliance status.
Not by itself in the United States. It is a voluntary protocol standardised as RFC 9309, and no US court has found that disregarding it is independently unlawful. It can still support a site's breach-of-contract or trespass claim by showing you were on notice.
Generally no. A contract needs assent, and a file served to an anonymous client is not something the client agreed to. Terms of service accepted through an account are a much stronger basis for a claim, which is the distinction the court drew in Meta v. Bright Data when it found a logged-out scraper was not bound by Facebook's terms.
Technical controls: authentication, rate limiting, and anti-bot systems such as Cloudflare, DataDome, PerimeterX, Akamai, and Kasada. Those are the measures the Web Data Frontier Benchmark tests against, because they are what a request actually meets.
Usually yes, and for reasons that are not legal. It reduces the chance of being blocked, it keeps you off pages a site has flagged as expensive or useless to crawl, and it is a much better position to argue from if a dispute ever arises.
The major search-and-answer crawlers publish user-agent names and state that they honour it, and sites can allow or disallow each one separately. Compliance varies across the wider field, and robots.txt cannot verify who is actually calling, since a user agent string is self-reported.
