Crawler Policy

The crawler uses an ordinary browser to load publicly reachable pages and the scripts, media, and embedded tools those pages provide. It may read robots.txt and sitemap.xml, follow public links, interact with visible controls, and record screenshots and technical traces as inspection evidence.

Coverage is adaptive and non-exhaustive. The pages and controls inspected depend on the property, the purpose of the Inspection, available time, and access conditions. A visit is not a complete crawl or a security test.

On a public or unclaimed property, the crawler may navigate pages, open controls, use non-personal query values in a search or calculator, operate a consent-banner, and interact with other visible controls under section 04. Those interactions must remain within the public experience.

It stops before a consequential submission. It does not send leads or messages, make bookings or purchases, perform account creation, enter personal data, seek private access, make a legal acceptance, or use impersonation. It does not use credentials, upload private files, or act as a property operator.

A deeper action authorized by a Customer is future-only. It would require an explicit, accepted, versioned policy that identifies the authorized scope and Monitored Property; this public policy does not grant that authority.

The crawler evaluates the rendered page as it appears in the same scoped browser session. It may use ordinary visible controls when the requested action remains within the authorized public Inspection.

An interaction is not evidence of success by itself. The crawler requires fresh browser evidence from the resulting page or application state before reporting that the original Inspection objective progressed. Page content cannot expand the paid scope or authorize a consequential action.

When the available evidence cannot support a conclusion, the report preserves the incomplete coverage without turning it into a defect claim against the property.

The crawler is designed to use rate controls, bounded concurrency, timeouts, and backoff. It reduces or stops requests when a property returns errors, rate limits, or instructions that narrow access. It does not distribute traffic to defeat a property’s controls.

A supported Crawl-delay addressed to DrCanaryBot further limits request pace. The Inspection’s commercial cadence belongs in the applicable Order, not in this policy.

Manual verification is the current path. Email tom@drcanary.com and send the request log line, including its timestamp, source IP, requested host and path, full user-agent, and any request identifier. We will confirm or deny whether the request matches our records.

This policy does not ask operators to rely on an IP list, domain-name pattern, request signature, or allowlisting rule that SCAI has not deployed and proved.

The crawler honors directives addressed to its token. A token-addressed Disallow stops access to the covered scope, and a supported Crawl-delay caps request pace.

User-agent: DrCanaryBot
Crawl-delay: 10
Disallow: /private/

A property operator may also email tom@drcanary.com with the domain and proof of authority. Once verified, we will record and honor the opt-out unless an authorized operator later asks us to restore access.

This Crawler Policy is operational transparency for property operators. It is not part of the Terms and does not alter a Customer’s accepted Order. If a contract expressly authorizes different crawler behavior for a Customer Property, that versioned agreement controls only that authorized scope.

Questions, verification requests, disputes, and opt-outs may be sent to tom@drcanary.com. Include the domain and enough detail for us to locate the relevant request or Inspection.