Historical software reference — discontinued
URLSpider
A command-line crawler that turned linked pages into a URL list. The historical program is not available, supported or recommended.
01
Documented purpose
The 2003 product page describes starting from a page, following its links recursively and building a list of discovered URLs from a Telnet or SSH session. That task survives; the implementation does not.
02
Modern boundary
- Scope
- Restrict hosts and schemes before fetching.
- Politeness
- Identify the crawler, rate-limit requests and honour access rules.
- Safety
- Block private-network destinations and cap depth, pages, bytes and time.
03
Historical URL
- /main/scripts.html?script=URLSpider
04