This software is a Django-based application that prevents web crawlers from accessing a website's pages, complementing the functionality of Sitemaps. With the Robots Exclusion application, website owners can effectively manage their website's access and visibility to search engines.
Django-robots is an excellent Django application designed for managing robots.txt files, which features the robots exclusion protocol. Additionally, this app complements the Django Sitemap contrib application. The robots exclusion app contains two essential database models—a relationship between them binds the URLS and Rules.
Installing Django-robots involves getting the application source from the app site and following a few simple instructions in the INSTALL.txt file. Another requirement is adding 'robots' to your INSTALLED_APPS setting. Make sure the 'django.template.loaders.app_directories.load_template_source' is included in your TEMPLATE_LOADERS settings. Additionally, verify that you have installed the sites framework.
To activate robots.txt creation on your Django site, input " (r'^robots.txt$', include('robots.urls'))" to your URLconf. This informs Django to build robots.txt when a crawl robot accesses /robots.txt. Syncing your database ensures creating the Rule objects via the admin interface or shell.
The Rule in Django-robots defines an abstract rule employed in responding to web robots using the robots exclusion protocol. Multiple URL patterns link to allow or disallow a robot, defined by its user agent, to access the given URLs. For the crawl delay field supported by search engines, you can set the delay between successive crawlers accesses in seconds. Large values increase the maximum crawl rate to your web server.
The sites framework is essential when enabling multiple robots.txt for a Django instance. If no rule exists, it automatically allows access to every URL to every web robot. In Django-robots, the URL case is sensitive and exact, allowing or denying access to web robots. The absence of the trailing slash marks files starting with the name of the specified pattern.
With caching, you can initiate optional cache generation for robots.txt. Add or make changes to the ROBOTS_CACHE_TIMEOUT value in seconds in your Django settings file to activate caching. Setting ROBOTS_CACHE_TIMEOUT to 60*60*24 caches robots.txt for 24hours. The default value is None, which implies no caching.
Installing Django-robots involves getting the application source from the app site and following a few simple instructions in the INSTALL.txt file. Another requirement is adding 'robots' to your INSTALLED_APPS setting. Make sure the 'django.template.loaders.app_directories.load_template_source' is included in your TEMPLATE_LOADERS settings. Additionally, verify that you have installed the sites framework.
To activate robots.txt creation on your Django site, input " (r'^robots.txt$', include('robots.urls'))" to your URLconf. This informs Django to build robots.txt when a crawl robot accesses /robots.txt. Syncing your database ensures creating the Rule objects via the admin interface or shell.
The Rule in Django-robots defines an abstract rule employed in responding to web robots using the robots exclusion protocol. Multiple URL patterns link to allow or disallow a robot, defined by its user agent, to access the given URLs. For the crawl delay field supported by search engines, you can set the delay between successive crawlers accesses in seconds. Large values increase the maximum crawl rate to your web server.
The sites framework is essential when enabling multiple robots.txt for a Django instance. If no rule exists, it automatically allows access to every URL to every web robot. In Django-robots, the URL case is sensitive and exact, allowing or denying access to web robots. The absence of the trailing slash marks files starting with the name of the specified pattern.
With caching, you can initiate optional cache generation for robots.txt. Add or make changes to the ROBOTS_CACHE_TIMEOUT value in seconds in your Django settings file to activate caching. Setting ROBOTS_CACHE_TIMEOUT to 60*60*24 caches robots.txt for 24hours. The default value is None, which implies no caching.
What's New
Version 0.6.1: N/A