Robot is our web crawler.
A web crawler is an Internet bot that systematically browses the World Wide Web. Crawlers visit publicly accessible domains and, depending on their purpose, perform a range of actions, such as analyzing keywords and hashtags, indexing page content and URLs, extracting data, and following existing redirects to analyze new websites.
User agent is any software, acting on behalf of a user, which retrieves, renders and and provides end-user interaction with websites.
The user-agent of our Crawler is:
This can occur for the following reasons