SimilarSiteSearch

robotstxt.org

robots.txt

Who runs robots.txt, what it tracks and what it costs, and the 20 closest web standards reference alternatives.

Who’s behind it
Not stated
Online since
Unknown
Trackers
Prices
No pricing page
Last price change
–
Closest alternative
Sitemaps

Measured from robots.txt’s own pages, the Internet Archive and public records. Hover a dotted value for its source and date.

We’ll send a link to confirm. Your address is only used for these alerts.

Visit robots.txt

01 · Who’s behind it

Who runs robots.txt

robots.txt robotstxt.org

Wikidata: standard used to advise web crawlers and scrapers not to index a web page or site

Company
No imprint we could read
Online since
Not in the Internet Archive

02 · Privacy

What robots.txt tracks

What the homepage loads before you accept or decline cookies. A cookie banner should hold most of it back until you agree, and a tag manager can load more that we don’t see.

trackers that follow visitors

None of the 19 alternatives below has fewer; 9 have more.

Privacy-friendly alternatives →
Cookie banner
None found

We found no trackers, ad pixels or third-party embeds on robots.txt’s homepage.

03 · Alternatives

Alternatives to robots.txt

No ad trackersRegistered in the EUFree plan

None of the 60 sites closest to robots.txt match these filters. We read imprints and pricing pages once a week, so more sites qualify as we get to them.

04 · History

What changed

Price and policy history →

No changes yet. Each new crawl is compared with the last one, and what robots.txt adds, drops or rewrites shows up here.

05 · Details

How we measure

Alternatives are matched by what sites actually offer, from their homepages, Wikidata and a summary by Claude, in any language. Trackers and the rest of the stack come from the homepage’s code and DNS records.

The full technology stack and search numbers · 2 technologies
Neighbours
k = 60
Sites searched
9,216
Vector
384-d
Query
44 ms

Built with

Web servers
SaaS tools (verified in DNS)