Collect URL is a tool that functions like a search engine bot by extracting all links on a user-provided URL and recursively scanning them. Users can limit it to collect links from a single domain.
One of the best features of Collect URL is that it allows the user to edit the file extension ignore list. This means that the user can specify file types that are not of interest and have them excluded from the scan. Additionally, the software has adjustable connection and response timeout values that can be changed to suit the user's preference.
Moreover, Collect URL allows the user to modify the user agent value for HTTP and HTTPS connections. This is a great feature if one desires to mimic a search engine bot to avoid getting blocked by some servers. It is worth mentioning that the software is developed by VOVSOFT, a trusted software development company with a track record of producing top-notch software products.
In summary, Collect URL is a reliable and efficient software for URL scanning, listing, and recursive scans. Its ability to ignore certain file extensions, adjust connection and response timeout values, and modify user agent values makes it a preferred choice for many professionals.
Version 2.5: Supported SSLv23, TLSv1, TLSv1_1, TLSv1_2.