Speak to us directly on 781-640-8874

The Invaluable Screaming Frog SEO Site Spider

The folks at Screaming Frog created an awesome website spidering tool. I have used the Xenu Link checker for years for basic link checking information when reviewing a website or blog.

screaming frog seo spider

The Screaming Frog SEO Spider presents more feasible data in a clearer format and I love it.

This desktop program for Mac or PC spiders your website collecting data for up to 500 pages when using the free version. Here is some of the information provided:

  1. Links
  2. Images
  3. CSS
  4. Scripts
  5. Apps
  6. Meta Title – shows the text along with a character count
  7. Meta Description– shows the text along with a character count
  8. Meta Keyword– shows the text along with a character count
  9. H1 and H2 – heading tag text and character count
  10. Status codes – Server errors and Redirects (2xx, 5xx, 3xx)
  11. Meta Robots – no – index, follow, archive snippet, odp, ydir and more
  12. Meta Refresh
  13. Canonoical link element
  14. In and Out links
  15. Anchor Text – for all links and even images that are linked
  16. Follow / no follow
  17. Images – missing alt text, length of alt text, image file size
  18. Custom search can even be added

After the launching the program it is very simple to get started. Just type in the URL and click Start. You can watch the data begin to appear before your very eyes.

The Data is Organized into Tabs

They include:

  1. Internal
  2. External
  3. Response Codes
  4. URI
  5. Page Titles
  6. Meta Description
  7. Meta Keywords
  8. H1
  9. H2
  10. Images
  11. Meta and Canonical
  12. Custom

The expandable bottom pane shows detailed information on any entry above.

These four lower tabs are:

  1. URL Info
  2. In Links
  3. Out Links
  4. Image Info

Across the bottom there is information presented to show if the spider is currently active, the average and current URIs and how many pages have been spidered.

How Do I Use It?

When getting acquainted with a new website I like to scan and see what stands out.  Last week I got a call about migrating and redesigning an existing website on a new platform. Using the SEO Spider I can quickly decide whether the site is well build or a mess.

I like to look at the URLs and see whether the directory structure and page names appear to be well thought out.  Sometimes pages and page titles get edited but the URL is not updated.

Status Codes and their status is very important. I like to see a lot of 200 and OK.  Otherwise there is trouble.

Looking over the meta titles, meta descriptions, meta keywords, H1, H2 and the character length of each of these is important.

The first check is do they exist. Missing all of this data is a bad sign.

The Second check is whether they are identical on every page, another bad sign.

The third check is whether they are well written, seem to have good keywords included and are consistent.  If the meta title is about website reviews, the meta description is about shopping centers and H1 is about hair extensions then we have a serious problem. More serious than if they were all blank.

I also like to check the file size and number of inlinks and outlinks to ensure things look reasonable.

An Example Site Review

In this case most of the site data looked reasonable. However, I did find that all meta descriptions were identical and all 187 characters long, which is too long. Interestingly the meta keywords on each page are an exact match for the meta description, almost.  Only 166 characters this time.  This is also too long. And they were all badly worded.

Images and H1 and H2 were completely missing from the website.  Only three meta titles were found across all the pages of the website.

With this quick review taking less than 10 minutes I can see right away that this client will greatly benefit from key word research and populating the meta fields correctly.


While Xenu does produce a nice little webpage to display the data that is easy to email, Screaming Frog will let you export all your data to Excel or as a CSV which is super handy for sorting and focusing.

The only down side of Screaming From SEO Spider is that if you are dealing with a site larger than 500 pages you have to pay an annual fee of (99 pounds) or about $166 for a license. I would kind of like one-time fees better, but heh, it’s a great revenue model.

If you need to check a website or give an assessment of a blog definitely use the Screaming From SEO Spider to quickly gather invaluable information.

What’s your favorite feature on the Screaming Frog SEO Spider? What is missing? Do you have a better SEO Spider to share with us?



2 Comments on “The Invaluable Screaming Frog SEO Site Spider”

  • Screamingfrog February 6th, 2015 4:17 pm

    I think it is not a disadvantage if you pay $ 99 a year for this extensive software. Other professional SEO tools cost 50, 100 or more $ every month and the Screamingfrog offers for this price incredibly many ways.

  • Dan February 17th, 2016 9:20 am

    “Screaming Frog is my go-to SEO Spider – offering invaluable information during crawls and making my life a lot easier with its numerous features and ways of breaking down data that it finds.