Lab report · methodology
What a legacy query-string catalogue costs you in crawl budget
Method, sample, and the part where we say which of our numbers you should not rely on — which, in this edition, is all of them, because the study has not been run.
The question
A legacy catalogue addressed through a query string — /main/scripts.html?script=… —
can expose one underlying resource under dozens of distinct URLs: different case, different
encoding of the same space, tracking parameters appended by whatever linked to it, and
parameters in different orders. Each variant is, to a crawler, a separate address.
The question is what that multiplication costs on a real site: how much of the crawl allocation goes to fetching variants of pages already fetched, and how much of it is displaced from content that has never been seen.
Method
- Inventory. Collect every legacy URL from server logs, the existing sitemap, external backlink sources and archived indexes. Deduplicate at the string level only — do not normalise yet.
- Normalise and group with the Legacy URL Mapper, which is the same code path published as a tool so this step is reproducible without our data.
- Classify each group as unambiguous, duplicate or manual-review. Record the counts, not just the ratios.
- Segment the crawl log by group. For each group, count requests, unique variants requested, and the response codes served.
- Compute displacement. Compare the request volume absorbed by duplicate and review groups against the volume reaching canonical content, over the same window.
- Report variance. Repeat over at least three non-overlapping windows. A single window on a single site is an anecdote.
Sample design
- Sites
- Not yet recruited. Target: 3–5 sites with a documented legacy query-string catalogue and access to raw server logs.
- Window
- Not yet run. Target: three non-overlapping 30-day windows per site.
- Log source
- Raw origin server logs only. CDN-aggregated summaries are excluded — they collapse exactly the variants we are trying to count.
- Bot identification
- Reverse DNS verification per crawler, not user-agent matching alone.
No results are recorded. When they are, this section will publish the site count, the window dates, the per-site figures, the aggregate, and the variance — or it will stay empty.
There is no chart here. An empty chart outline, or one drawn from illustrative numbers, would imply a measurement that does not exist.
What you can act on today, without the numbers
The method above is worth running on your own logs. But three moves do not depend on the study at all, and none of them requires knowing the magnitude of the effect:
- Collapse the variants you control. Case, encoding and parameter order are yours to normalise at the edge. This is uncontroversial and cheap.
- Do not soft-redirect unknown identifiers to the homepage. An unrecognised
?script=value should return 404. A 200 at the front page tells both the crawler and the visitor that the link worked. - Serve 410 for endpoints that will never return. Executable CGI, order and account paths are gone permanently, and saying so is more useful than 404 to everyone involved.
Those three are exactly what the Archive routing rules on this site implement, and the resolver behind them is tested.
Limitations
- No data. Every figure that would make this report actionable at scale is absent by design in this edition.
- Crawler behaviour is not a published contract and changes without notice. Any measurement is a measurement of one period.
- Site-specific. A catalogue of 40 legacy URLs and one of 40,000 do not behave the same way, and the method does not assume they do.
Method and ownership disclosure
Contextter, a related SEO product under the same ownership as this publication, was used as a working tool while this method was developed. No Contextter crawl or commercial dataset appears in this edition. The proposed inventory and grouping steps can be performed with the local-first Legacy URL Mapper and server logs you control. See the full relationship disclosure.