MarkosWeb

covid-19 widget

Markosweb Bot

Markosweb Bot obeys the Robot Exclusion Standard. Specifically, MarkosWeb Bot adheres to the 1996 Robots Exclusion Standard (RES).

Markosweb Bot obeys the first entry in the robots.txt file with a User-agent containing "MarkosWeb"

  • If there is no such record, it will obey the first entry with a User-agent of "*".
  • If it is not able to retrieve a robots.txt file, it will assume there are no restrictions for MarkosWeb Bot. It will keep trying to retrieve the file, and will obey it if becomes available.


Disallowed documents, including slash "/" (the home page of the site), are not crawled, nor are links in those documents followed. MarkosWeb Bot does read the home page at each site and uses it internally, but if it is disallowed, it is neither indexed nor followed. If a page has robots.txt standards disallowing it to be crawled, MarkosWeb Bot will not read or use the contents of that page.

Example robots.txt:
User-agent: SmartViper
Disallow: /