To block us, add a User-agent: healthcare.dev-archiver group to your robots.txt; we pick it up on our next visit. For anything else, including a lower rate, email hello@healthcare.dev with your site's address and we'll reply within two business days.
Crawler
healthcare.dev-archiver
We keep a dated record of public healthcare documents so changes can be traced to the day. Here's how our crawler behaves.
How it works
- User agent
- Every request carries "healthcare.dev-archiver/1.0 (+https://healthcare.dev/crawler)". The product token for robots.txt is healthcare.dev-archiver.
- Public pages only
- We fetch documents anyone can open without an account: published coverage policies, prior authorization lists and public data files. We never log in, and we stop at anything that asks for a sign-in or answers 401 or 403.
- robots.txt
- We read robots.txt before fetching from a site and follow it, including rules that name healthcare.dev-archiver and any Crawl-delay. If robots.txt can't be read because the site is having trouble, we fetch nothing from that site until it can.
- Rate limits
- One request at a time, with a pause of several seconds between requests to the same site (longer if your Crawl-delay asks for it). Most documents are checked once a day or less, and we only re-fetch the documents we track, not whole sites.
- What we keep
- A dated copy of each document version and its response headers, so we can show when a published policy changed. We don't republish the documents themselves.
Contact