Google have today released a message to a number of e-commrce stores where Google has identified crawl errors (inability to read product data) from product listing submissions
We at Feed Manager have previously covered reasons for these types of problems in more depth here. This announcement relates specifically to robot blocking commands in site source data preventing Google from seeing page and image links relating to products.
Google have now announced that starting on 17 September 2013, if they cannot crawl the landing page of a product it will no longer show this item in Google Shopping results. The current position is that the product may be displayed as inactive or of limited availability
Items that Google is unable to crawl will be displayed in the Data Quality tab of the Merchant Center account.
To prevent this situation occurring so that Google can access your site and images, ensure that your robots.txt file allows both user agents “Googlebot” and “Googlebot-image” to crawl your site. You can do this by adding the following two lines to your file:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
To learn more about robots.txt files, in respect of Google’s bot, read
To learn more about Google’s crawling process on common issues, read
To learn more about “Fetch as Google” (test your website as Google sees it), read
We will be contacting all our client merchants where this is a specific issue on their account, but all customers of Shopping are advised to pass this post onto their web support to ensure that the website code base has made the necessary changes to allow Google’s bots free access to all product links.
Additional Tracking Scripts – Check that these do not get blocked also!
For Feed Manager customers, we always include utm_source and utm_medium tracking scripts with every product and image link sent to all channels (unless instructed otherwise). This is to enable visibility of clicks and performance in Google Analytics. Please ensure that these scripts are also allowed by your system.
Safe Encoding within Links
Feed Manager also encodes links for special or unsafe characters within image and product URLs as advised by Google and other channels. We recommend that you ensure that your website is able to interpret these characters and/or that your export file encodes these characters before we even process your product file. The Degreave website is a useful source of these characters and their safe translation.
Very important to avoid are spaces,”+”, “.”, special symbols such as ©, ®, ™, accented vowels and the like. Altough Feed Manager will encode these values, we have found that for images Google does not interpret them correctly, and we see disapproved products as a result.
