robotstxt.org
robots.txt
Who runs robots.txt, what it tracks and what it costs, and the 20 closest web standards reference alternatives.
- Who’s behind it
- Not stated
- Online since
- Unknown
- Trackers
- Prices
- No pricing page
- Last price change
- –
- Closest alternative
- Sitemaps
Measured from robots.txt’s own pages, the Internet Archive and public records. Hover a dotted value for its source and date.
Visit robots.txt01 · Who’s behind it
Who runs robots.txt
robots.txt robotstxt.org
Wikidata: standard used to advise web crawlers and scrapers not to index a web page or site
- Company
- No imprint we could read
- Online since
- Not in the Internet Archive
02 · Privacy
What robots.txt tracks
What the homepage loads before you accept or decline cookies. A cookie banner should hold most of it back until you agree, and a tag manager can load more that we don’t see.
trackers that follow visitors
None of the 19 alternatives below has fewer; 9 have more.
Privacy-friendly alternatives →- Cookie banner
- None found
We found no trackers, ad pixels or third-party embeds on robots.txt’s homepage.
03 · Alternatives
Alternatives to robots.txt
Thumbs up if it’s a real alternative to robots.txt, thumbs down if it isn’t.
- 01
sitemaps.org - Home
The Sitemaps protocol enables webmasters to information earch engine about pages on their site that are available for crawling.
Very close - 02
Welcome to Boutell.Com, Inc.!
Close - 03
Packet Storm Security
Packet Storm Security provides security news, exploits, advisories, and tools for information security professionals.
Close - 04
site news
Close - 05
Home • Angular
The web development framework for building modern apps.
Close - 06
Client Challenge
free and open-source web content management system
Close - 07
World of Warcraft News and Guides
The largest World of Warcraft (WoW) information site, featuring guides, news, and information on classes, gearing, professions, raids, transmog, and more.
Close - 08 FanGraphs fangraphs.com
baseball statistics website
Close - 09
Pokémon Database -- the fastest way to get your Pokémon information
The Pokémon Database is your source for information and multimedia for the Pokémon games.
Close - 10
Build optimized websites quickly, focus on your content | Docusaurus
An optimized site generator in React. Docusaurus helps you to move fast and write content. Build documentation websites, blogs, marketing pages, and more.
Close - 11
Google
search engine by Google LLC
Close - 12
Web Hypertext Application Technology Working Group (WHATWG)
Web standards organization overseeing alternate HTML5 specification
Close - 13
Ahrefs—AI Marketing Platform Powered by Big Data
We help marketers drive visibility across AI search, SEO, content, and social – with the largest AI and search databases online.
Close - 14
Sucuri - Complete Website Security, Protection & Monitoring
Website security made simple—Sucuri offers fast malware cleanup, reliable monitoring, and powerful protection for any CMS.
Close - 15
A List Apart – For people who make websites
For people who make websites
Close - 16
LiveInternet @ Статистика и дневники, почта и поиск
LiveInternet - крупный сервис дневников и сообществ, авторитетный сервис статистики для сайтов
Close - 17
What browser? My browser? Is my browser out of date?
What browser do I have? Find out my browser, check that my browser is up to date and what my settings are.
Close - 18
Mojeek
Mojeek is a web search engine that provides unbiased, fast, and relevant search results combined with a no tracking privacy policy.
Close - 19
IETF | Internet Engineering Task Force
The Internet Engineering Task Force (IETF) is the premier Internet standards body, developing open standards through open processes.
Close - 20
Ghostarchive, a website archive
web archiving site
Close
Closest first, by how much each site’s offer resembles robots.txt’s. Trackers are counted the same way as above. Highlighted technologies are ones robots.txt uses too. Votes and comments come from visitors and don’t change the order.
04 · History
What changed
Price and policy history →No changes yet. Each new crawl is compared with the last one, and what robots.txt adds, drops or rewrites shows up here.
05 · Details
How we measure
Alternatives are matched by what sites actually offer, from their homepages, Wikidata and a summary by Claude, in any language. Trackers and the rest of the stack come from the homepage’s code and DNS records.
The full technology stack, the competitor map and search numbers · 2 technologies
- Neighbours
- k = 20
- Sites searched
- 9,192
- Vector
- 384-d
- Query
- 107 ms
Competitor map
Each alternative by how close its vector is to robots.txt’s (x) and how much traffic it gets, how much of the stack it shares, or where it sits in the embedding space itself (y). Hover for the numbers, click to jump to it in the list.
similarity × popularity
Built with
- Web servers
- Apache HTTP Server header server: Apache/2.4.68 (Debian)
- SaaS tools (verified in DNS)
- Google Search Console TXT google-site-verification=FuTkdKut4zpT5f61O29qfjnWKg3u5iA292tOglYwZuo
What competitors use
Across the 20 closest sites.
- Google Search Console 14/20
- Google Workspace · not on robotstxt.org 9/20
- nginx · not on robotstxt.org 7/20
- Cloudflare · not on robotstxt.org 7/20
- Cloudflare DNS · not on robotstxt.org 7/20
- jQuery · not on robotstxt.org 4/20
- Google AdSense · not on robotstxt.org 3/20
- Google Analytics · not on robotstxt.org 3/20
- Mailchimp · not on robotstxt.org 3/20
- SendGrid · not on robotstxt.org 3/20
- Microsoft 365 · not on robotstxt.org 3/20
- Fastmail · not on robotstxt.org 3/20
- Cloudflare Web Analytics · not on robotstxt.org 2/20
- Fastly · not on robotstxt.org 2/20
- Google Fonts · not on robotstxt.org 2/20
- OneTrust · not on robotstxt.org 2/20