The technical SEO checks you can run yourself

Most of the reasons a decent site is not found in search are not competitive. They are mechanical, and they are the kind of fault that produces no error message anywhere a human would look. The page loads perfectly for you. It simply is not in the index.

Six checks catch the large majority of these. They take about twenty minutes together, they need nothing you have to buy, and they should be run in this order, because each one is a precondition for the next mattering.

One: can it be crawled at all

Your robots.txt file is a set of instructions telling crawlers where they may and may not go. It is the first thing a crawler reads, and a single overly broad disallow rule can exclude an entire section of a site.

The failure that matters is inherited rather than intentional. A staging environment is correctly blocked from search while it is being built, that configuration is deployed with the site, and nobody removes it — so the new site launches invisible. Check what your robots.txt actually says, and test the URLs you care about against it rather than reading the rules and assuming.

Two: is it allowed to be indexed

Crawl permission and index permission are two different things, and this trips people up constantly. A noindex directive lets a crawler read the page and tells it not to list the page in results. It is invisible in the browser, it produces no warning, and it is the other half of the staging problem above.

Three: what does a crawler actually see

What you see in a browser and what a crawler receives can differ, particularly on sites that build their content with JavaScript after the page arrives. If the text and links only exist after the browser has run scripts, the crawler may or may not see them depending on how it processes the page.

The check is simple: look at the raw response the way a crawler receives it and confirm your headings, body copy and internal links are present. If the raw response is essentially an empty shell, that is a rendering question worth resolving before anything else on this list.

Four: is there one canonical address

The same page can usually be reached several ways — with and without a trailing slash, with and without the www prefix, with tracking parameters attached, over both protocols. To a search engine those can look like separate pages carrying the same content, which splits whatever standing the page has earned across several addresses.

A canonical tag names the one true address. The two faults are having none, and having one that points somewhere unexpected — a template that hardcodes the homepage as canonical for every page is a real and surprisingly common bug, and it is catastrophic while it lasts.

Five: does the sitemap tell the truth

An XML sitemap is a list of the pages you want found. Its value is as a statement of intent, and its common failure is drifting out of step with the site it describes.

• Pages listed that now redirect elsewhere, or return an error.
• Pages listed that carry a noindex directive, which is a direct contradiction: find this, do not list this.
• Pages missing that you actually care about.
• Last-modified dates that update on every build regardless of whether anything changed, which trains crawlers to ignore them.

The last one is subtle and worth fixing. A date that says everything changed today, every day, carries no information, and a crawler that learns this stops using it as a signal.

Six: what the page says about itself

Only now is it worth looking at titles, descriptions and image alt text — because until the five checks above pass, none of this is being read by anybody.

Titles and descriptions are the page's pitch in the search result, and the faults are consistent: duplicated across many pages, truncated because they are too long, or written to contain keywords rather than to describe what the page offers. Alt text matters for the same reason twice over — it is how the image is understood by search, and it is how the page works for someone using a screen reader, which is not an SEO consideration but is the more important one.

When to run these

After any launch or migration, without exception, and then quarterly. The quarterly pass catches drift; the post-launch pass catches the expensive faults while they are still cheap, which is the same argument as checking a smoke alarm rather than waiting to find out.