The Issue

In the recent weeks, Google has been sending emails to webmasters indicating Googlebot cannot access CSS and JS files on a given website. That email looks something like this:


Screen Shot 2015-07-28 at 1.55.28 PM

It goes on to specify that the problem may reside with the Robots.txt file. Our testing indicates, an addition to the Robots.txt file can resolve this issue.

Resolving the Issue

Here’s how to resolve the issue if you’re using WordPress:

  1. Click the Fetch as Google button in the email.
  2. Click the Fetch button for each option in the drop down (Desktop, Mobile: Smartphone,etc.).
  3. If there are issues you’ll see them indicated in the list like this example: Screen Shot 2015-07-28 at 2.24.53 PM
  4. Now go to your WordPress Dashboard.
  5. From the Dashboard, go to Plugins.
  6. Add the WP Robots Txt plugin
  7. Activate the plugin.
  8. Go into the Settings – Reading section of the Dashboard
  9. You’ll see the Robots.txt file contents shown here. It will look like this:Screen Shot 2015-07-28 at 2.03.21 PM
  10. Add the following entries to the bottom of the Robots.txt file:
    1. User-Agent: Googlebot
      Allow: /*.js*
      Allow: /*.css*
  11. The click the Save Changes button.

Results

The resulting entry will look like this:

Robots.txt after Googlebot CSS and JS fix

Final Test

Then visit your website URL (e.g. www.example.com/robots.txt) to confirm the entries you added are visible. If so, your problem should be solved. Confirm this by Fetching as Google again (click the button in the email), and you can expect to see this: Screen Shot 2015-07-28 at 2.29.14 PM