SEO Technical Basics: Meta Tags, Sitemaps, and robots.txt

What SiteWatch checks

SiteWatch fetches your homepage and analyzes the technical SEO fundamentals that search engines rely on:

• Title tag: The <title> element — shown in search results
• Meta description: The snippet shown below the title in search results
• Open Graph tags: How your page appears when shared on social media (og:title, og:image)
• Canonical URL: Tells search engines which version of a page is the "official" one
• Noindex directive: Whether you're accidentally blocking search engines
• robots.txt: Rules telling search engines which pages to crawl
• Sitemap.xml: A map of your site's pages for search engines
• Sitemap URL health: Whether the URLs in your sitemap actually work

SiteWatch also tracks changes over time and alerts you when critical SEO elements change unexpectedly.

How to read your results

• OK (green): All essential SEO elements are present and correct.
• Warning (amber): Non-critical issues like missing og:image or broken sitemap URLs.
• Critical (red): Blocking issues like noindex on homepage or Disallow: / in robots.txt.

Critical issues

Homepage has noindex

This is the most serious SEO issue possible. A noindex directive on your homepage tells Google to remove your entire site from search results.

How it happens: - A noindex meta tag was left from a staging environment - A CMS setting was accidentally toggled

Check for it in your HTML:

<!-- This blocks indexing — REMOVE IT -->
<meta name="robots" content="noindex">

Fix: Remove the noindex tag or change it to index, follow. Verify by viewing your page source in the browser.

robots.txt blocks all crawlers

If your robots.txt contains:

User-agent: *
Disallow: /

This tells all search engines to not crawl any page on your site. Your site will be removed from search results.

Fix: Update your robots.txt. A healthy robots.txt looks like:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api

Sitemap: https://yoursite.com/sitemap.xml

Common issues and fixes

Missing title tag

The <title> tag is the single most important on-page SEO element. It appears in: - Browser tabs - Search engine results - Social media shares (as fallback)

Fix: Add a unique, descriptive title to every page:

<title>Your Page Title — Your Site Name</title>

Best practices: - Keep it under 60 characters - Put the main keyword near the beginning - Make each page's title unique

Missing meta description

The meta description doesn't directly affect rankings, but it determines what text appears in search results. A good description increases click-through rate.

<meta name="description" content="A clear, compelling summary of this page in 150-160 characters.">

Missing Open Graph tags

Without Open Graph tags, social media platforms guess what to show when someone shares your link — often with poor results.

<meta property="og:title" content="Your Page Title">
<meta property="og:description" content="A compelling description">
<meta property="og:image" content="https://yoursite.com/social-image.jpg">
<meta property="og:url" content="https://yoursite.com/page">

The og:image should be at least 1200x630 pixels for best display on Facebook and LinkedIn.

No sitemap.xml

A sitemap helps search engines discover all your pages efficiently. Without one, search engines rely on crawling links — which may miss pages.

Create a sitemap:

Most CMS platforms generate sitemaps automatically: - WordPress: Install Yoast SEO or use the built-in sitemap at /wp-sitemap.xml - Static sites: Use a sitemap generator tool

A basic sitemap structure:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yoursite.com/</loc>
    <lastmod>2026-01-15</lastmod>
  </url>
  <url>
    <loc>https://yoursite.com/about</loc>
    <lastmod>2026-01-10</lastmod>
  </url>
</urlset>

Then reference it in your robots.txt:

Sitemap: https://yoursite.com/sitemap.xml

Broken sitemap URLs

SiteWatch samples URLs from your sitemap and checks if they work. Broken URLs in your sitemap tell search engines your site is poorly maintained.

Fix: 1. Remove deleted pages from your sitemap 2. Update URLs that have moved (use the new URL) 3. Regenerate your sitemap if using a CMS

SEO element changed unexpectedly

SiteWatch tracks your title, description, and robots.txt over time. If they change unexpectedly, it may indicate: - A CMS update overwrote your settings - Someone edited the page without realizing the SEO impact - A plugin or theme update reset defaults

Review the change in the check details and revert if needed.