The folks at Screaming Frog created an awesome website spidering tool. I have used the Xenu Link checker for years for basic link checking information when reviewing a website or blog.

The Screaming Frog SEO Spider presents more feasible data in a clearer format and I love it.

This desktop program for Mac or PC spiders your website collecting data for up to 500 pages when using the free version. Here is some of the information provided:

1. Links
2. Images
3. CSS
4. Scripts
5. Apps
6. Meta Title – shows the text along with a character count
7. Meta Description– shows the text along with a character count
8. Meta Keyword– shows the text along with a character count
9. H1 and H2 – heading tag text and character count
10. Status codes – Server errors and Redirects (2xx, 5xx, 3xx)
11. Meta Robots – no – index, follow, archive snippet, odp, ydir and more
12. Meta Refresh
13. Canonoical link element
14. In and Out links
15. Anchor Text – for all links and even images that are linked
16. Follow / no follow
17. Images – missing alt text, length of alt text, image file size
18. Custom search can even be added

After the launching the program it is very simple to get started. Just type in the URL and click Start. You can watch the data begin to appear before your very eyes.

The Data is Organized into Tabs

They include:

1. Internal
2. External
3. Response Codes
4. URI
5. Page Titles
6. Meta Description
7. Meta Keywords
8. H1
9. H2
10. Images
11. Meta and Canonical
12. Custom

The expandable bottom pane shows detailed information on any entry above.

These four lower tabs are:

1. URL Info
2. In Links
3. Out Links
4. Image Info

Across the bottom there is information presented to show if the spider is currently active, the average and current URIs and how many pages have been spidered.

How Do I Use It?

When getting acquainted with a new website I like to scan and see what stands out.  Last week I got a call about migrating and redesigning an existing website on a new platform. Using the SEO Spider I can quickly decide whether the site is well build or a mess.

I like to look at the URLs and see whether the directory structure and page names appear to be well thought out.  Sometimes pages and page titles get edited but the URL is not updated.

Status Codes and their status is very important. I like to see a lot of 200 and OK.  Otherwise there is trouble.

Looking over the meta titles, meta descriptions, meta keywords, H1, H2 and the character length of each of these is important.

The first check is do they exist. Missing all of this data is a bad sign.

The Second check is whether they are identical on every page, another bad sign.

The third check is whether they are well written, seem to have good keywords included and are consistent.  If the meta title is about website reviews, the meta description is about shopping centers and H1 is about hair extensions then we have a serious problem. More serious than if they were all blank.

I also like to check the file size and number of inlinks and outlinks to ensure things look reasonable.

An Example Site Review

In this case most of the site data looked reasonable. However, I did find that all meta descriptions were identical and all 187 characters long, which is too long. Interestingly the meta keywords on each page are an exact match for the meta description, almost.  Only 166 characters this time.  This is also too long. And they were all badly worded.

Images and H1 and H2 were completely missing from the website.  Only three meta titles were found across all the pages of the website.

With this quick review taking less than 10 minutes I can see right away that this client will greatly benefit from key word research and populating the meta fields correctly.

Conclusion

While Xenu does produce a nice little webpage to display the data that is easy to email, Screaming Frog will let you export all your data to Excel or as a CSV which is super handy for sorting and focusing.

The only down side of Screaming From SEO Spider is that if you are dealing with a site larger than 500 pages you have to pay an annual fee of (99 pounds) or about $166 for a license. I would kind of like one-time fees better, but heh, it’s a great revenue model.

If you need to check a website or give an assessment of a blog definitely use the Screaming From SEO Spider to quickly gather invaluable information.

What’s your favorite feature on the Screaming Frog SEO Spider? What is missing? Do you have a better SEO Spider to share with us?