SEO crawlers are indispensable tools but they need an SEO pro's insight and experience to determine which warnings to heed or ignore.In many cases, what an SEO crawler marks as a fatal error needs immediate attention – but sometimes, it’s not an error at all.This can happen even with the most popular SEO crawling tools such as Semrush Site Audit, Ahrefs Site Audit, Sitebulb, and Screaming Frog.How can you tell the difference to avoid prioritizing a fix that doesn’t need to be done?Here are a few real-life examples of such warnings and errors together, with explanations as to why they may be an issue for your website.
1. Indexability Issues (Noindex Pages on the Site)Any SEO crawler will highlight and warn you about non-indexable pages on the site. Depending on the crawler type, noindex pages can be marked as warnings, errors, or insights.The Google Search Console Coverage report may also mark non-indexable pages as Errors (if the site has non-indexable pages in the sitemap submitted) or Excluded even though they are not actual issues.The fact that a URL has a “noindex” tag on it does not necessarily mean that this is an error. It only means that the page cannot be indexed by Google and other search engines.The “noindex” tag is one of two possible directives for crawlers, the other one being to index the page.Practically every website contains URLs that should not be indexed by Google.These may include, for example, tag pages (and sometimes category pages as well), login pages, password reset pages, or a thank you page.Your task, as an SEO professional, is to review noindex pages on the site and decide whether they indeed should be blocked from indexing or whether the “noindex” tag could have been added by accident.
2. Meta Description Too Short or EmptySEO crawlers will also check the meta elements of the site, including meta description elements. If the site does not have meta descriptions or they are too short (usually below 110 characters), then the crawler will mark it as an issue.Depending on the size of the site, it is not always possible and/or doable to create unique meta descriptions for all its webpages. You may not need them, either.A good example of a site where it may not make sense is a huge ecommerce site with millions of URLs.In fact, the bigger the site is, the less important this element gets.The content of the meta description element, in contrast to the content of the title tag, is not taken into account by Google and does not influence rankings.Search snippets sometimes use the meta description but are often rewritten by Google.What you as an SEO need to do is keep in mind that each site is different. Use your common SEO sense when deciding whether meta descriptions are indeed an issue for that specific website, or that you can safely ignore the warning.
3. Meta Keywords MissingMeta keywords were used 20+ years ago as a way to indicate to search engines such as Altavista what key phrases a given URL wanted to rank for.This was, however, heavily abused. Meta keywords were a sort of a “spam magnet,” so the majority of search engines dropped support for this element.Screaming Frog always checks if there are meta keywords on the site, by default.Since this is an obsolete SEO element, 99% of sites do not use meta keywords anymore.New SEO pros or clients may get confused thinking that if a crawler marks something as missing, then this element should actually be added to the site. But that’s not the case here!If meta keywords are missing on the site you are auditing, it’s a waste to recommend adding them.
4. Images Over 100 KBIt’s important to optimize and compress images used on the site so that a gigantic PNG logo that weighs 10 MB does not need to be loaded on every webpage.However, it’s not always possible to compress all images to below 100 KB.The fact that the site has images that are over 100 KB does not necessarily mean that the site has issues with image optimization or is very slow.When you see this error, make sure to check the overall site’s speed and performance in Google PageSpeed Insights and the Google Search Console Core Web Vitals report.If the site is doing okay and passes the Core Web Vitals assessment, then usually there is no need to compress the images further.Tip: What you may do with this Screaming Frog report is sort the images by size from the heaviest to the lightest to check if there are some really huge images on specific webpages.
5. Low Content or Low Word Count PagesDepending on the settings of the SEO crawler, most SEO auditing tools will highlight pages that are below 50-100 words as low content pages.Screaming Frog, on the other hand, considers pages below 200 words to be low content pages by default (you can change that setting upon configuring the crawl).Just because a webpage has few words does not mean that it is an issue or error.There are many types of pages that are meant to have a low word count, including some login pages, password reset pages, tag pages, or a contact page.The crawler will mark these pages as low content but this is not an issue that will prevent the site from ranking well in Google.What the tool is trying to tell you is that if you want a given webpage to rank highly in Google and bring a lot of organic traffic, then this webpage may need to be quite detailed and in-depth.This often includes, among others, a high word count. But there are different types of search intents and the content depth is not always what users are looking for to satisfy their needs.When reviewing low word count pages flagged by the crawler, always think about whether these pages are really meant to have a lot of content. In many cases, they are not.
6. Low HTML-Text RatioSemrush Site Audit will also alert you about the pages that have a low text-HTML ratio.This alert is supposed to show you:
- Pages that may have a low word count.
- Pages that are potentially built in a complex way and have a huge HTML code file.
7. XML Sitemap Not Indicated in robots.txtRobots.txt, in addition to being the file with crawler directives, is also the place where you can specify the URL of the XML sitemap so that Google can crawl it and index the content easily.SEO crawlers such as Semrush Site Audit will notify you if the XML sitemap is not indicated in robots.txt.At a glance, this looks like a serious issue even though in most cases it isn’t because:
- Google usually does not have problems crawling and indexing smaller sites (below 10,000 pages).
- Google will not have problems crawling and indexing huge sites if they have a good internal linking structure.
- An XML sitemap does not need to be indicated in robots.txt if it’s correctly submitted in Google Search Console.
- An XML sitemap does not need to be indicated in robots.txt if it’s in the standard location – i.e., /sitemap.xml (in most cases).