Robot
Robot is our web crawler.
What is a web crawler
A web crawler is an Internet bot that systematically browses the World Wide Web. Crawlers visit publicly accessible domains and, depending on their purpose, perform a range of actions, such as analyzing keywords and hashtags, indexing page content and URLs, extracting data, and following existing redirects to analyze new websites.
What is user-agent?
User agent is any software, acting on behalf of a user, which retrieves, renders and and provides end-user interaction with websites.
The user-agent of our Crawler is:
Robot/1.0.0 (+http://search.mobilesl.com/robot)
Why the web crawler come to your site?
This can occur for the following reasons
- Your site is indexed by our search engine and the link was shown in the search result
- The search engine indexes your site or check for updates
Why should I index my website?
Search traffic is an important source of visitors for many websites. The more search traffic, the more potential customers or readers a website can attract.
Why might reindexing be necessary?
- Your site is not present in our system
- There have been changes to your site
- The site is not fully indexed
How do I index my site?
- Go to the Aura website at the link below
- Paste a link to your website in the input box
- Pay and after that your site will be indexed
Feedback