[Resolved] Googlebot cannot access CSS and JS files on http://domain.com.au

By July 29, 2015Services

So quote a few of you will probably have received an email from Google this morning, titled Googlebot cannot access CSS and JS files on http://domain.com.au, most likely on either your WordPress or Joomla website.

This is likely to affect your search rankings if you don’t action this ASAP, as these files are most generally associated with how your website renders… which for those who have been paying attention will know, has become something that Google has been monitoring leading into their April 21st Mobile compatibility algo update.

The solution for WordPress owners is quite simple. Update the below to reflect the correct sitemap URL, then update your website’s robots.txt (in the root of your website directory structure);


Sitemap: http://yourDomain.com.au/sitemap_index.xml
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/plugins/
Disallow: /wp-includes/
Disallow: /comments/feed/
Disallow: /feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: ?comments=*
Disallow: /search?
Disallow: /?p=*
Disallow: /go/
User-agent: Googlebot
Allow: /wp-includes/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /

Then log into your Google Webmaster tools, expand “Crawl” and select “Fetch as Google” and / or “robots.txt Tester” to verify that the change has been picked up correctly and your site renders how it’s supposed to.