The Issue
In the recent weeks, Google has been sending emails to webmasters indicating Googlebot cannot access CSS and JS files on a given website. That email looks something like this:
It goes on to specify that the problem may reside with the Robots.txt file. Our testing indicates, an addition to the Robots.txt file can resolve this issue.
Resolving the Issue
Here’s how to resolve the issue if you’re using WordPress:
- Click the Fetch as Google button in the email.
- Click the Fetch button for each option in the drop down (Desktop, Mobile: Smartphone,etc.).
- If there are issues you’ll see them indicated in the list like this example:
- Now go to your WordPress Dashboard.
- From the Dashboard, go to Plugins.
- Add the WP Robots Txt plugin
- Activate the plugin.
- Go into the Settings – Reading section of the Dashboard
- You’ll see the Robots.txt file contents shown here. It will look like this:
- Add the following entries to the bottom of the Robots.txt file:
- User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*
- User-Agent: Googlebot
- The click the Save Changes button.
Results
The resulting entry will look like this:
Final Test
Then visit your website URL (e.g. www.example.com/robots.txt) to confirm the entries you added are visible. If so, your problem should be solved. Confirm this by Fetching as Google again (click the button in the email), and you can expect to see this: