Crawler and operator information

What crawlers are served

Every crawler receives the same HTML a person receives, rendered from the same data on the same request. There is no alternate version for search engines, no prerendered copy served only to bots, and no client-side rendering to execute: these pages ship no JavaScript at all. If a crawler cannot read a fact, a person cannot read it either.

Which pages ask not to be indexed

Candidate listings and the state and city pages that link to them are served with noindex, follow, both as an X-Robots-Tag response header and as an HTML robots meta tag. They are deliberately crawlable: you have to be able to fetch a page to see the directive, so these URLs are not blocked in robots.txt. They are also deliberately outside every sitemap we publish, and they carry no structured data.

The reason is honesty rather than secrecy. A candidate listing is an unverified entry, and we do not want an unverified entry to appear in search results as though GovTrack had checked it. When a listing is verified, and when a city page meets the criteria for indexing, that page becomes eligible to be crawled and indexed.

A missing or private listing answers 404 with a noindex directive, and a listing that should not be found is indistinguishable from one that never existed.

What robots.txt disallows

Only the transactional and administrative areas are disallowed:

• /super
• /api
• /claim
• /manage
• /correct
• /search
• /media

Nothing else is disallowed. Every other client, named on our allowlist or not, receives the same HTML under Cloudflare's normal rate limits. Expected clients include Googlebot, Bingbot, Applebot, DuckDuckBot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot.

Sitemaps

sitemap.xml lists the editorial pages only: the home page, this page, the methodology page, the about page, the legal, privacy, and takedown pages. Candidate organizations, state and city pages, claim, management, and administrative URLs never appear in it.

Freshness and caching

Pages are rendered from the database when you ask for them, so there is no artifact timestamp to read and no cache to purge. We send Cache-Control: no-store while the directory is being built, so what you fetch is what the database says at that moment. A listing can change because a firm corrected it, because a record was removed, or because a fact was re-observed.

Contact the operator

If you operate a crawler and something here is breaking your run, tell us what you requested and when, and we will look at it. We would rather serve you correctly than have you work around us.

Write to directory@govtrack.io for anything this page does not cover.