If you’re serious about enhancing your SEO and improving your organic traffic, you need Screaming Frog.
Don’t let the funny name fool you – Screaming Frog is a widely used SEO tool that looks at the skeleton of your website and points out all aspects of your business website’s technical SEO.
Screaming Frog is an essential part of any SEO strategy. Whether you’re an ecommerce, small business, or corporation, you’re going to need Screaming Frog.
This article is going to show you the most important parts of the tool that any small business owner needs to know without going into the super technical aspects.
What is Screaming Frog?
Screaming Frog (SF) is a tool that uses spiders to crawl your website in a similar manner to Google.
It gives you an understanding of how search engines like Google can see your website.
This means that you can look at one of your pages visually in a browser and compare this to what you see in Screaming Frog.
While the Miles IT homepage portrays what we do visually, we can see more technical details under the hood when we crawl our website.
We can see:
- The page is indexable, meaning there are no issues crawling the page
- Its status code is 200, meaning there are no 404 issues, and it’s not a 301
- The title and description of the page that users will see before they click to this page from SERPs (search engine results pages), as well as how long they are
- The H1 (header 1) and H2 (header 2) text and lengths
- How many words the page has
- How many links from other pages it receives
All of these details give us an overview of the pages of our website. When we start to analyze them together, we get a better picture of the health of our website.
How to Crawl Your Website
To crawl your website, simply download Screaming Frog to your computer, open the program, add your URL to the top navigation bar, and click start.
Crawling your website should follow instructions given in your robots.txt file in the same way Google does.
There are a few options to consider. The most valuable is going to be Spider and List.
The Spider option lets you enter your desired URL and let the crawl bot determine where to go. Like Google’s bot, it will decide where to go from your internal linking structure.
If you choose the list option, you can upload a list of URLs you want to look at. Maybe you want to identify the meta title and other technical information for a few blog or service pages. This option will crawl much quicker than the spider option.
If you have a normal-sized business website, Spider often takes around 5 minutes to complete. If you have an e-commerce website, you probably have thousands of product pages. A spider crawl will take a lot longer in this instance.
We recommend following our automated SEO audit tutorial to speed this process up, so you’re not waiting for crawls.
What Can You Do with Screaming Frog?
Screaming Frog allows you to find advanced technical issues with your website.
This includes finding broken links and identifying which pages have duplicate or missing content (headers, meta, etc.).
The following section will go further in-depth with what you can do in SF.
Before we get there, I want to point out that you can do seemingly infinite things with SF. Most of these will not add tremendous value for small businesses.
Some SF tools sound great but don’t offer the best insight when used.
For example, the Visualise Site Architecture feature sounds amazing. If I could take my website, which has over three hundred pages, and create a visual architecture that updates when my website updates, I’d save myself a ton of time.
In reality, the end result of this feature looks like this.
This image is just wild. It’s a mess and doesn’t tell me anything.
What I’m really looking for is something more like this website architecture outline from Backlinko.
However, SF doesn’t create this for us visually (and from what I’ve seen, no tool out there does).
Instead, SF makes it easy for you to find this information via your URL structure (using “/services” for example) or using meta keywords to categorize your page (discussed further in our Automated SEO Audit article).
Even if you don’t utilize the automated SEO process, you can export any report within SF to Google Sheets where you can analyze the data further with SEO related formulas and filters.
Let’s look at what you should be thinking about when looking at the SF interface.
What are the Most Important Parts of Screaming Frog?
As discussed, there are a thousand things you can do in Screaming Frog.
Meta Titles and Descriptions
Meta information is what Google uses when displaying your website in search engine results pages.
Therefore it is and will always be one of the most important aspects of on-page SEO (factors you can control).
While Google ultimately determines what meta information is displayed, you can make sure that you’re offering the best suggestions possible.
At a glance, you can see which pages have duplicate metas, are missing metas, or have metas over their required lengths.
Meta titles can display up to 60-70 characters in search engines, and meta descriptions can display up to 150-160 characters. These numbers range because all characters are different.
It’s not so much that you have X amount of characters, but rather you have X amount of space. The pipe symbol (“|”) can fit better than W because it is much thinner.
H1s and H2s
Remember that Screaming Frog looks at your website the same way that Google does, not the way that a human does.
To follow best SEO practices, you’ll want to make sure that you have one H1 on each page and label each sub-section of content with H2s.
An example I encountered recently when running through my automated SEO audit was when I saw a few pages that didn’t have H2s. When I went to the website and opened chrome’s inspector, it confirmed that the page did have H2s.
My first thought was that I could ignore this in the SEO audit. Screaming Frog must be wrong!
However, I knew that this problem would keep occurring. Maybe there was some deeper diving I needed to do.
The thing about SF is that it’s a way to check up on your site, but it won’t give you every little detail about each page. That would be way too much data.
What it did tell me was the first instance of an H2. When looking closer at the code, I did indeed have an H2, but above that first H2, there was another. This H2 was empty.
For whatever reason, the code had a blank H2 (this can be due to styling in many websites). But since SF sees it this way, it means Google can see it this way too, so the issue must be fixed.
Finding Orphan Pages, 404s, and 301s
While I admitted earlier I’m not the biggest fan of the directory tree graph in SF for its architecture of your website, I do find it valuable in finding orphan pages.
These are pages that are not linked from anywhere or not linked in the way you hoped. They stand out more when you look at the graph.
I was then able to search for this page by its URL and found that there is one page that this is actually linked from.
I found out last week from Google Search Console that this page no longer exists, so I 301 redirected it. It turns out that it’s still linked from this blog.
301 redirects are important to find on your website for SEO. You want to make sure that any page you’re linking to is directly linked and not 301ing. In this case, I technically have a link to my home page because this page 301s to the home page.
I went into the backend and removed the link completely, thus removing the dependency.
If this page still existed, but it was 301ing (because I changed the URL),I would update the URL to the most recent link rather than deleting it entirely.
You Can Analyze Competitors
You may think that SF can only be used to better understand and structure your own website.
While this is true, you can also use it to explore and spy on competitors.
Everything that you can see about your website, you can see about others. This means you can find what meta information your competitors are using to mine additional keywords and phrases you may be missing.
You can find out which of their pages are more important by how many internal links they’re receiving.
You can also take this one step further with another tool by plugging these URLs into SemRush to see what keywords they’re ranking for.
By clicking on the hyperlinked number under keywords, you can identify what keywords each page is ranking for and where in Google.
While you can do this by typing in your competitor’s URL in SemRush, SF is a great starting point to know which pages are most relevant to your current SEO strategy.
Find Internal Link Opportunities
One of my favorite things you can do in Screaming Frog has to do with finding internal links.
Everyone who works in SEO knows that links are great from external websites.
Fewer people know it’s important to link important pages to each other within your website (internal linking).
And even fewer people than that know how to find these internal links easily.
Take, for example, one of our pillar pages on SEO. A great practice is to make sure that your pillar page is linked to all of the related sub-topics you’ve written about.
After crawling my website, I can search for this pillar page by its URL. Then click the inlinks button at the bottom left of SF. This tells me all of the pages from my website that are linking to this pillar page.
From here, you can make sure that all relevant pages are linked to your pillar page that should be. I also want to make sure that my anchor text is as relevant as it can be.
All of my anchors include SEO in the text. Still, I’ve varied the text to say things like “well-executed SEO” and others that spell out search engine optimization. This helps with LSI (latent semantic indexing) keywords, which is a fancy way of saying we should be using synonyms in our content for SEO.
While SF doesn’t have a fancy tool that easily visualizes what I want to see, I can still use this data to create my own internal linking structure documentation. I’ve visualized it in Google Docs.
This helps me see which blogs should be linked where. I also use it as a pseudo content calendar with my next topic (look, it’s the Screaming Frog Ultimate Guide I’m writing about right now!)
Conclusion
Screaming Frog is an amazing tool that helps you to crawl your website and find all of those technical SEO aspects that might need tweaking.
There are a ton of valuable things you can do in SF, and this only scratches the surface of everything you can do.
However, as a small business owner, it’s important to focus on only what’s essentialwithin the interface so you don’t get lost in the details of what you could be doing for your website.
Not everything you can do to optimize your website is something that you need to do.
Remember to keep it simple and focus on what helps give the most significant boost to your website’s SEO without overcomplicating things.
