Raw evidence

Website Crawler

Sequential, robots-aware crawl of public HTML on the seed host and its www equivalent. Output is a raw crawl envelope — intermediate evidence, never the final RAG file.

Downloads are named web_<entity>_rag_<date>.json, e.g. web_faix_rag_251026.json

Depth

Max attempts

Redirect hops

Delay (s)

Timeout/request (s)

Elapsed budget (s)

Language

All languages

Include staff directories

Keep 06 270 numbers (override)

Path prefix

Extra allowed hosts (comma)

Exclude regex (comma)