Pandalytics
Domainsbot web crawler — bot identification, behaviour and opt-out
Pandalytics is the web crawler operated by Domainsbot Inc.
It fetches a site's robots.txt to check crawl permissions, then performs a single,
low-rate HTTPS GET of the domain's homepage to collect public technical metadata
(HTTP status, title/description, language, detected technologies, TLS and DNS facts) for
Domainsbot's domain-intelligence and analytics products. It is a benign, identifiable crawler
and does not collect personal data, submit forms, log in, or execute JavaScript.
How to identify Pandalytics
| User-Agent | Pandalytics/2.0 (+https://domainsbot.com/network-crawler-pandalytics-kdc68dhsbd/) |
|---|---|
| robots.txt token | Pandalytics |
| Verification | Cloudflare Verified Bot via Web Bot Auth
(RFC 9421 HTTP Message Signatures, Ed25519). Requests carry
Signature, Signature-Input and
Signature-Agent: "https://domainsbot.com" headers. |
| Key directory | https://domainsbot.com/.well-known/http-message-signatures-directory |
| Cloudflare category | Monitoring & Analytics |
| Network | Identity is proven by request signature, not a fixed IP list; egress IP addresses may vary. |
Crawling behaviour
- Requests
/robots.txtfirst and only fetches the homepage if permitted for thePandalyticstoken. - Then fetches only the domain homepage
/— no deep crawling. GETrequests only. No logins, no form submissions, no JavaScript execution.- Re-crawls homepages on a recurring basis to keep metadata current; frequency varies by domain and is kept at a low per-domain rate.
- Honours
robots.txt(Disallow) andCrawl-delay.
How to block or limit Pandalytics
Add the following to your robots.txt to disallow the crawler, or set a
Crawl-delay to slow it down:
User-agent: Pandalytics Disallow: / # or, to allow but rate-limit: User-agent: Pandalytics Crawl-delay: 10
To request a block, exclusion, or for any questions, email crawler@domainsbot.com — we respond to crawler opt-out requests.