robotstxt.org
robots.txt
Who runs robots.txt, what it tracks and what it costs, and the 20 closest web standards reference alternatives.
- Who’s behind it
- Not stated
- Online since
- Unknown
- Trackers
- Prices
- No pricing page
- Last price change
- –
- Closest alternative
- Sitemaps
Measured from robots.txt’s own pages, the Internet Archive and public records. Hover a dotted value for its source and date.
Visit robots.txt01 · Who’s behind it
Who runs robots.txt
robots.txt robotstxt.org
Wikidata: standard used to advise web crawlers and scrapers not to index a web page or site
- Company
- No imprint we could read
- Online since
- Not in the Internet Archive
02 · Privacy
What robots.txt tracks
What the homepage loads before you accept or decline cookies. A cookie banner should hold most of it back until you agree, and a tag manager can load more that we don’t see.
trackers that follow visitors
None of the 19 alternatives below has fewer; 9 have more.
Privacy-friendly alternatives →- Cookie banner
- None found
We found no trackers, ad pixels or third-party embeds on robots.txt’s homepage.
03 · Alternatives
Alternatives to robots.txt
Thumbs up if it’s a real alternative to robots.txt, thumbs down if it isn’t.
20 of the 60 sites closest to robots.txt match.
- 01
sitemaps.org - Home
The Sitemaps protocol enables webmasters to information earch engine about pages on their site that are available for crawling.
Very close - 02
Packet Storm Security
Packet Storm Security provides security news, exploits, advisories, and tools for information security professionals.
Close - 03
The Open Graph protocol
The Open Graph protocol enables any web page to become a rich object in a social graph.
Close - 04
site news
Close - 05
Home • Angular
The web development framework for building modern apps.
Close - 06
Client Challenge
free and open-source web content management system
Close - 07
How to enable JavaScript in your browser and why
Instructions on how to enable (activate) JavaScript in web browser and why.
Close - 08
web.dev
Guidance to build modern web experiences that work on any browser.
Close - 09
Pokémon Database -- the fastest way to get your Pokémon information
The Pokémon Database is your source for information and multimedia for the Pokémon games.
Close - 10
BarracudaCentral.org - Technical Insight for Security Pros
Close - 11
Internet Meme Database | Know Your Meme
Know Your Meme is a website dedicated to documenting Internet phenomena: viral videos, image macros, catchphrases, web celebs and more.
Close - 12
Symfony, High Performance PHP Framework for Web Development
Symfony is a web framework and a set of reusable PHP packages to build web applications, APIs, microservices and CLI apps.
Close - 13
int08h
Close - 14
Build optimized websites quickly, focus on your content | Docusaurus
An optimized site generator in React. Docusaurus helps you to move fast and write content. Build documentation websites, blogs, marketing pages, and more.
Close - 15
WebDAV Resources
Central resource site for the WebDAV community.
Close - 16
Making sure you're not a bot!
Czech and Slovak web project providing a movie database
Close - 17
Welcome | IZA - Institute of Labor Economics
Welcome to the IZA@LISER Network website
Close - 18
All The Tropes
online wiki hosted on Miraheze/WikiTide listing conventions found in fictional works, fork of TV Tropes
Close - 19
PHP
PHP is a popular general-purpose scripting language that powers everything from your blog to the most popular websites in the world.
Close - 20
Google
search engine by Google LLC
Close
Closest first, by how much each site’s offer resembles robots.txt’s. Trackers are counted the same way as above. Highlighted technologies are ones robots.txt uses too. Votes and comments come from visitors and don’t change the order.
04 · History
What changed
Price and policy history →No changes yet. Each new crawl is compared with the last one, and what robots.txt adds, drops or rewrites shows up here.
05 · Details
How we measure
Alternatives are matched by what sites actually offer, from their homepages, Wikidata and a summary by Claude, in any language. Trackers and the rest of the stack come from the homepage’s code and DNS records.
The full technology stack, the competitor map and search numbers · 2 technologies
- Neighbours
- k = 60
- Sites searched
- 9,199
- Vector
- 384-d
- Query
- 55 ms
Competitor map
Each alternative by how close its vector is to robots.txt’s (x) and how much traffic it gets, how much of the stack it shares, or where it sits in the embedding space itself (y). Hover for the numbers, click to jump to it in the list.
similarity × popularity
Built with
- Web servers
- Apache HTTP Server header server: Apache/2.4.68 (Debian)
- SaaS tools (verified in DNS)
- Google Search Console TXT google-site-verification=FuTkdKut4zpT5f61O29qfjnWKg3u5iA292tOglYwZuo
What competitors use
Across the 20 closest sites.
- Google Search Console 14/20
- Google Workspace · not on robotstxt.org 9/20
- nginx · not on robotstxt.org 7/20
- Cloudflare · not on robotstxt.org 7/20
- Cloudflare DNS · not on robotstxt.org 7/20
- jQuery · not on robotstxt.org 4/20
- Google AdSense · not on robotstxt.org 3/20
- Google Analytics · not on robotstxt.org 3/20
- Mailchimp · not on robotstxt.org 3/20
- SendGrid · not on robotstxt.org 3/20
- Microsoft 365 · not on robotstxt.org 3/20
- Fastmail · not on robotstxt.org 3/20
- Cloudflare Web Analytics · not on robotstxt.org 2/20
- Fastly · not on robotstxt.org 2/20
- Google Fonts · not on robotstxt.org 2/20
- OneTrust · not on robotstxt.org 2/20