How To Do A Website Analysis With Screaming Frog

In this post, I go over How I run a website analysis to find technical errors on my site with Screaming Frog.

What Is Screaming Frog?

Screaming Frog is, in my opinion, the best crawler available to run an SEO audit of a website. It’s been a tool that I have used for a long time and has always done the job. It crawls a website  to find broken links, missing images, images that are too large, missing meta descriptions and slow loading pages.

First I will crawl the site and see the number of pages. If there are more than 500 files to crawl you will not get all the information with the free version of Screaming Frog. It is generally good enough except for rather large sites.

Fixing Response Code Errors On Website

The first thing I will look at when I crawl the site is for link errors, these can be found in the responce codes section. This gives me a nice readout of the 301, 404 or 500 errors that are on my site.

I can simply select the page that the error is coming from and edit the link on that page for most of these.

I like to fix any internal linking that is going through a redirect to clean up the internal structure. There’s no point in going through a redirect when you can fix in on your site.

Any 404 pages I will go through to see what they’re pointing to and if it’s worth reviving. If this was a popular page in the past and it’s likely to have external links to it as well, a good idea is to build a 301 redirect in your .htaccess file to the most relevant page or an html sitemap.



I will also make sure that any pages that are on a site using https are using the https version on internal links. This can be another small thing if the pages are being redirected. It could also mean bigger problems if they’re not.


Checking The Websites Pages With ScreamingFrog

The next area I will go through goes over everything. I will simply click on the internal html tab. This will take a look at all the html pages. You will notice that the error codes are there as well, I just like to be able to fix that first off, and recrawl until I can see a nice pretty green circle with no errors.

Missing Title Tags

The next tab I will concern myself with during this is the Titles. Are the titles all there, are they long enough, descriptive enough, clickable? Are they under 65 characters? I can go through and fix them so that they meet those requirements and will get more clicks. Titles and meta descriptions are very important for your CTR from the serps.

Missing Meta Descriptions

So the next thing I will look at is, you guessed it, meta descriptions. I want to be sure to include the keywords I’m targeting so that Google will bold those terms for searchers, getting my snippet more attention. There is also a list of action words that can help to improve the click rate from the serps. Words like now, your, or best, there was an entire study and there’s a list somewhere of about 10 words that help CTR. The description will need to be under 155 (this is now 300 for some searches) characters so maximizing the space you have is important. Look for those people using adwords to see how their use of the limited space is going. People running adwords are constantly A/B testing so use their experience.


Analysis Of Heading Tags

I want to be sure if I am using a heading tag on a page it is relevant to the topic. I want to change any generic type heading tags to be more in line with the specific page or at least with the main topic of the site.

I want to be sure that I am using at least 1 H1 tag. You can use more than one ever since html5 came about, although it’s still nice to structure your site as you would a paragraph outline.

A. Main H1 Tag

  1. H2 tag
  2. H2 tag
    • H3 tag
      • H4 tag
      • H4 tag
    • H3 tag
  3. H2 tag
    • H3 tag
    • H3 tag

You can see an example of how I nested these tags in this article as well.

Crawl Depth

I like to see how far someone would have to go through internal linking to get to the desired page. If someone must click more than 2 times to get to any page I should consider fixing that. There would be instances where this may be how the site works for more complicated websites though.

Large Images and Response Time

The last thing I will look at will tbe images and load time. This will part of my site speed cleanup. I want to be sure that any extra large images are sized appropriately and compressed. For this, I use WP Smush, or for a paid tool with better optimization ShortPixel,  if I am on a WordPress site or for other websites. With the images tab I can look to see what the file size of each image is, and make sure to check anything over about 100,000 bytes. This again can vary depending on the website.

I will look again at the html tab for the responce time of each page, clicking on the tab to order them from highest to lowest. I can then look at gtmetrix for those pages to see if I can speed them up a bit. It will generally be a large image or a broken bit of code causing them to load slower than expected.