‘PropertyBot’ is the name of property.com.au’s web crawler. PropertyBot crawls real estate agency websites to identify ‘on market’ advertised property listings so that we can show a hyperlink to that listing on property.com.au. Consumers visiting property.com.au can click on the hyperlink to be redirected to the agency website. The sole purpose of PropertyBot is to provide this hyperlink.
Controlling how PropertyBot accesses your site
PropertyBot is designed to minimise the number of requests it makes to your website; each website will only be crawled once per day and PropertyBot will never make more than one request per second.
Any parts of your website that bots are prevented from accessing through instructions in your website's robots.txt file will not be accessed by PropertyBot. If there is a particular part of your website that you would prefer PropertyBot not to access, you can include the following directive in your robots.txt file (PropertyBot will be unable to access paths under /wp/wp-admin/ in this example):
User-agent: propertybot
Disallow: /wp/wp-admin/
Stop PropertyBot from accessing your site
If you do not wish for hyperlinks to your website to be displayed on property.com.au, you may instruct PropertyBot to not crawl your website by including the following entry in your website’s robots.txt file:
User-agent: propertybot
Disallow: /
If PropertyBot is prevented from crawling your website over several days, PropertyBot will permanently stop attempting to crawl your website.