If you want a website that works for both people and search engine spiders that crawl your site for rankings like the GoogleBot you have to spider check your work. It sounds obvious and simple, but if you don’t spider check your work – how do you know it works?
Search engine optimization or SEO is an environment where humans have limited visibility. There is definitely a limit how much human eyeballs alone can see in terms of how the GoogleBot sees your website without actually spider checking your work on Google.
Search engine spiders like GoogleBot are robot software that crawl your website to index it for ranking. Google says compared to humans, “Bots access pattern is completely different” – one of the greatest understatements in Google’s Webmaster Guidelines! People tend to assume Google rankings are much more automatic than they actually are due to this difference between how humans view a website and how robot search engine spiders view it for ranking.
If you’re an SEO and live and breathe Google’s algorithms, many of the problems you deal with on a daily basis is bridging the gap between what works for human visitors to your website and what works for search engine spiders to achieve rankings and bring traffic.
Search engine visitors are the most affordable way of getting free, well targeted traffic to your website. However, one of the biggest SEO mistakes most people make is assuming their web designer is also a search engine optimization expert or SEO.
In most instances, you should not expect the person primarily concerned with the look and feel of your website and its coding to also keep up with the latest in Google’s algorithms that are constantly evolving and updating. While web design and SEO are both at the heart of your website’s functionality, never assume your web designer is also an SEO expert.
SEO is a different knowledge and software set than most web designers can devote time to or should be expected to stay on top of and perform at the highest SEO levels. SEO is also a one strike you are out environment! You can do a hundred things right but get one important element of SEO wrong and it can undo everything else you do.
The question to ask that can often make this point clear is to ask your web designer, “How do you spider check your work?”
Although Google’s Webmaster Guidelines provides tools and guidance as to how to check a website through a spider’s eye view – like looking at the site with a Lynx text based browser or use the URL Inspection Tool in Google Console – in fact, very few web developers have actually read and followed Google’s guidelines.
This is why spider checking your website is so important to see if what you thought you were communicating on the Internet is actually being seen and ranked on Google. If you haven’t spider checked your website, you simply can’t tell with human eyes alone if it is working on search engines. Here are a few easy ways to do that.
Site:Search
First enter into the Google search box:
site:yourdomain.com
No space after the colon. Do not include http:// or https:// in front of either the root domain or wwww. subdomain version of your site.
A site:search is the single most important diagnostic search on all of the major search engines that tells you how that search engine views your website. Because your root domain is the top of your site’s hierarchy, you always want to see your root homepage as the top result of a site:search on Google.
If you don’t see your homepage at the top of a site:search, there may be a problem. Most of your ranking strength is focused in your homepage where the majority of the external links to a site usually point. The lack of appearance of your homepage at the top of a site:search on Google is one of the fastest ways you can see if your site is under a downgrade with some page showing above the homepage – or a penalty. Although a site:search alone is not conclusive evidence of that fact alone.
Note carefully how your homepage displays with a site:search in Google’s and Bing’s listings. 65 characters of your title are displayed in blue text at the top of your search engine listing, 150 characters of meta description appear under it – or a snippet of text from your body text that matches the keywords from a search request.
You can check to see if www. and non www. versions of your pages is appearing in your site, which means you have canonical issues not consolidating these two versions of your site to a single www. or non www. version. It can also show if http and https versions of your site are being indexed. However, again – Do not enter http:// or https:// with a site search or you will get a different set of results.
Click Your “Cached “Link
Next, on Google, Bing and Yahoo move your cursor to the downward arrow to the right of the URL in the search engine listing to make your “Cached” link appear.
If you get the result “Your search – site:yourdomain.com – did not match any documents,” that means either your site is not being crawled and indexed on Google – or you’ve entered the domain incorrectly so check your spelling carefully. It is also possible it has been removed from the index as a result of a penalty in extreme cases, although most of Google penalties don’t result in this extreme an action. Check Google Console to see if manual actions have been applied to the site by Google which will require a reconsideration request to get the penalty lifted.
One thing you do want to look at is if Google is displaying the cached date immediately, or sitting on the results for a while running spam tests before publishing them. If Google is publishing the cache immediately, that’s a good sign. However, bugs have occured with Google’s cache recently, and if you don’t see your cache date being updated run a Google search to see if other people are experiencing issues by entering in the Google search box: Google Cache Not Updating to see if this is a bug on Google others are experiencing.
Click “Text-only version” of Your Google Cache
When you first land on Google’s cache page, note in the upper left corner of the grey header there are three links labeled “Full Version” which is how the site looks to humans, “Text-only version” which is how the site looks to search engine spiders, and “View source” which is the pages html coding.
Click the “Text-only version” link in the upper left corner of the grey Google’s cache to see how it looks to GoogleBot.
This strips the website down to the body text and image alternative text associated with graphic images that Google sees for ranking. After clicking the “Text-only version” link many websites have major portions of their site or even the whole site disappears – meaning Google can’t see your content for rankings.
Clicking back and forth in Google’s cache of your website between the “Full version” and “Text-only version” upper left corner link is how you spider check your work to see if the GoogleBot and humans are seeing the same thing.
Google only gives you rankings for keywords it sees on your website in the “Text-only version”.
Unless you perform this test, you simply can’t tell if spiders are seeing your site properly or not – and very often, they aren’t and the web developer and site owner don’t know it. When Google can’t see important elements of your site as humans do, the result is that you have been hidden rather than promoted on the Internet!
Spider Check Your Keyword Densities
Next, enter a search you want to compete for and find your search result in Google’s rankings or search results. Once again, move your cursor to the arrow to the right of the search engine listing to make the “Cached” link appear and click it. Now you will see exactly how Google sees your keywords that are highlighted in the cache that match the keywords entered in the search.
To get an approximation of your site’s keyword densities that you want in most cases to be between 1-2%, copy and paste Google’s “Text-only version” into Microsoft Word and get a word count of how many words Google sees in the body text and image alternative text of the page. Now do a “Control find” for your keywords to see how many times they are actually mentioned.
1% keyword density is your keywords appearing once in a hundred words – 2% twice in a hundred words. You can compete for many searches with keyword densities outside of this 1-2% average, but you may not compete across as broad a range of searches as each keyword algorithm is very unique. Keep in mind your rankings are also dependent upon your keywords appearing in the link text pointing to your site which you can’t see looking at the webpage because it is an offsite ranking factor.
Other important ways to spider check your work is with Google Console that gives you a wealth of diagnostics about how your site performs on Google, and Google Analytics that shows your traffic and what keywords are actually bringing you visitors to the site.
While spider checking your work on Google takes less than thirty seconds involving three simple steps: 1) Move your cursor to the right of your Google listing to the downward pointing arrow to make the “Cached” link appear, 2) Click your “Cached” link, and 3) Click the “Text-only version” link in the upper right corner – never assume your webmaster has performed this vital three click test.
If after doing this test, important elements humans see on your webpage are not visible to the GoogleBot, your content has been effectively hidden from the Internet for ranking rather than your website promoting you. You need to study Google’s Webmaster Guidelines to diagnose what the problem is – and carefully follow them if you want people to find you at the top of the rankings for a Google search.
Check the Results of Loading Speed on Rankings and Traffic
Another very important element to spider check is the impact of your host’s and site’s loading speed on rankings and traffic.
In Google Analytics, go to left main menu. Click “Behavior” then select “Site Speed”. You are given four choices you can use with increasing specificity. Overview that gives you the overall average loading speeds of the site on any given day. Page Timings breaks the results down on a page by page basis. Speed Suggestion which offers a number of suggestions to speed your site with a recommendation of how much time would be saved to see if the suggestion was implemented to see on a cost benefit analysis if its worth the effort.
Although website loading speed tests like tools.pingdom.com are excellent for giving you a beautiful cascading graph to show the loading speed of each element of your site, you will get very different loading speeds at different times of day due to the server loads on hosts in a shared hosting environment.
Spikes in loading speed are not unusual to see in Google Analytics Site Speed data because on certain days and times of day, the number of requests on the server will vary. If these spikes only occur for a day, it will not impact Google rankings.
However, when loading speed spikes last for more than a day it can diminish rankings. The way to see this is to go into Google Console under “Performance” and select a time period of one to three months. Click “Rankings’ so you see the purple rankings line in the graph. Make a screen shot of this graph and copy it to another document like a Word document or email body.
Next, set the same date range in Google Analytics Site Speed Overview, take a screen image, and place it directly above or below the Google Console graph with the purple line showing so you can see the two graphs side by side.
Note that single loading speed spikes are normal and will not effect rankings, but when these spikes occur two or more days you will see a drop in the purple rankings line. This is how you see the impact of loading speed in a shared hosting environment – which may have nothing to do with activity on your own site – but is negatively impacting your rankings and traffic by slowing the loading speed of your site in Google’s eyes.
Loading speed is a ranking factor, and this is the easiest way to spider check the effect of loading speed on your site by putting these two graphs side by side and seeing if loading speed spikes are occurring for more than one day causing rankings to decline.
Remember, end of the day Googlebot always has the last say whether your website works on search engines or not. Google provides many valuable tools and resources to help you perceive an environment which is largely invisible to the human eye without specialized software and knowledge – to spider check your work and see it like the Googlebot sees it!