If you are viewing this page, your website was probably visited by the MetaQuerier crawler. The crawler is part of the MetaQuerier project at the University of Illinois in Urbana-Champaign. Its purpose is to collect information for research and development of next-generation search engines.
The crawler fetches HTML pages and associated objects, and store them in our local repository. Our objectives are: 1) to collect information for R&D use, and 2) to experiment with crawling mechanisms.
The crawler follows the robots exclusion protocol, which allows site administrators to prevent robots from visiting all or certain parts of their website. You can learn more about this protocol here. The robots.txt file must be located at www.yoursite.com/robots.txt. There are many ways to block only certain robots or only certain areas of your site, but here is a brief example of a robots.txt entry which would block our crawler from visiting every page on your site:
User-agent: MQBot Disallow: /
In addition, the crawler delays at least 5 seconds between requests to the same website so that it does not overload the server.
We would like the crawler to be as friendly as possible, so if you feel the crawler is not behaving properly or have any other concerns, we would like to hear about it. Please email us at mqbot@cs.uiuc.edu with a description of the problem and any other relevant information such as access times or server logs.