Why Is My Website Not Showing on Google? A Practical Diagnosis Checklist

Real-World Context & Developer Perspective

Last month a boutique roaster in Portland rolled out a new Shopify store. Traffic surged, yet a simple Google search for "Portland roasted beans" returned nothing. The owner freaked out. The SEO lead blamed the copy. The devs blamed the theme. And the owner blamed "bad luck".

That blame game hurts everyone. Small-biz owners lose sales. SEO managers miss targets. Developers waste time hunting ghosts—maybe a stray robots.txt rule, a busted sitemap, or a mis-routed domain.

Most folks think "Google just forgot me" or "I need more backlinks". Those are half-truths. Google won't even try to index a 4xx, a page blocked by robots, or a URL on the wrong host. The fix is a checklist that starts at the network layer and climbs up to content.

We built a website not showing on Google checklist. It isn't a marketing fluff list; it's a terminal-friendly, CI-ready diagnosis. Run it, and you'll spot the break before you start rewriting copy.

Architecture & Technical Prerequisites

Google's crawler is polite. It obeys DNS, HTTP codes, and your directives. Break any of those and it bounces.

DNS – The domain must resolve to the right IP. A typo in an A record or an expired CNAME sends the bot to a dead end. Run dig or nslookup to double-check.

HTTP status – A healthy page returns 200 OK. A 301 is fine if it points to the canonical URL; a 404 or 500 tells Google the page is gone. Don't forget to scan for X-Robots-Tag in the headers.

robots.txt – Lives at https://example.com/robots.txt. One Disallow: / line shuts the whole site down. Only block what truly belongs behind the curtain.

sitemap.xml – A clean XML sitemap is a roadmap. Each <url> needs a <loc>, optional <lastmod>, and a <priority> if you care.

Meta robots – Inside <head> you might see <meta name="robots" c>. One stray tag on a shared template can hide every page that uses it.

Beyond that, enforce a tidy URL scheme—no trailing-slash mismatches, lower-case paths—and correct rel="canonical" tags. A 200 response paired with a wrong canonical sends Google to the wrong page.

Step-by-Step Implementation: Core Setup

Open a terminal. Run the commands in order. Swap example.com for your own domain.

1. Verify DNS

# Using dig
dig +short example.com A

# Or nslookup
nslookup example.com

If you get several IPs, make sure they belong to your host. A stray IP usually means an old record is still cached.

2. Test HTTP response

# Fetch headers only
curl -I https://example.com/

Look for 200 or a proper 301 to the www version. Also scan for X-Robots-Tag: noindex. A 404 means the URL is unreachable.

3. Create a minimal robots.txt

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /admin/

Drop this file at the web root. Then confirm it serves:

curl -I https://example.com/robots.txt

4. Generate a sitemap

WordPress users can grab the Google XML Sitemaps plugin. For static sites, try the CLI tool sitemap-generator-cli:

# Install globally
npm i -g sitemap-generator-cli

# Generate sitemap
sitemap-generator https://example.com -o./public

Upload public/sitemap.xml to the root and verify it's reachable.

5. Submit to Google Search Console

Manual method:

1. Visit Search Console.
2. Select your property.
3. Go to "Sitemaps" and paste https://example.com/sitemap.xml.
4. Hit "Submit".

API method (service account required):

curl -X POST \
 -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
 -H "Content-Type: application/json" \
 -d '{"siteUrl":"sc-domain:example.com","feedpath":"sitemap.xml"}' \
 https://searchconsole.googleapis.com/v1/urlTestingTools/mobileFriendlyTest:run

Swap the endpoint for urlInspection.index:batchUpdate if you need to push many URLs.

6. Quick sanity check

Run Google's URL Inspection on a few key pages. If it reports "URL is not on Google", you've missed something above.

Advanced Configuration & Performance Tuning

With the basics locked down, you can fine-tune crawl-budget signals.

Structured Data

JSON-LD helps Google grasp the page purpose without extra parsing.

<script type="application/ld+json">
{
 "@context": "https://schema.org",
 "@type": "LocalBusiness",
 "name": "Portland Roasters",
 "url": "https://example.com",
 "address": {
 "@type": "PostalAddress",
 "streetAddress": "123 Bean St",
 "addressLocality": "Portland",
 "addressRegion": "OR",
 "postalCode": "97201",
 "addressCountry": "US"
 },
 "telephone": "+1-503-555-0199"
}
</script>

Paste this into the <head> of your homepage. Validate with Google's testing tool.

hreflang for multilingual sites

If you serve the same content in several languages, add link rel="alternate" tags. A missing or wrong hreflang can make Google think you have duplicate pages and throttle crawling.

<link rel="alternate" hreflang="en" href="https://example.com/en/" />
<link rel="alternate" hreflang="es" href="https://example.com/es/" />
<link rel="alternate" hreflang="x-default" href="https://example.com/" />

Lazy-load impact

Lazy-loading images is great for users, but hiding above-the-fold assets behind JavaScript can hide them from the crawler. Use loading="lazy", but keep critical assets in the initial HTML.

HTTP/2 vs HTTP/1.1

Google's bot prefers HTTP/2 because it cuts round-trips. If you're still on HTTP/1.1, you're burning crawl budget on extra handshakes.

# Quick check
curl -I -s -o /dev/null -w "%{http_version}\n" https://example.com

If the output shows 1.1, enable HTTP/2. For Nginx:

server {
 listen 443 ssl http2;
 # …other config…
}

For Apache, add Protocols h2 h2c http/1.1 to httpd.conf.

Caching headers and crawl budget

Set Cache-Control and Expires so Google knows when to revisit. A max-age=0 forces a re-crawl on every hit, draining your budget.

# Nginx example
location ~* \.(js|css|png|jpg|jpeg|svg)$ {
 expires 30d;
 add_header Cache-Control "public, immutable";
}

Run Lighthouse locally to spot issues:

npm i -g lighthouse
lighthouse https://example.com --view --output=json --output-path=report.json

Pay attention to the "Crawl" section. Low scores often trace back to heavy JavaScript bundles or missing caching headers, which shrink the number of pages Google will fetch each day.

Critical Production Pitfalls & Diagnostic Gotchas

Robots.txt, meta robots, sitemap—done. Still, production hides nasty surprises. Miss one and Google pretends you don't exist.

• Accidental noindex – One stray meta name="robots" c in a header file and the whole site vanishes from search. Spot it in Search Console's Coverage report under "Excluded → Page with 'noindex' tag".
• Blocked resources – CSS or JS locked by robots.txt means Google can't render the page. The URL Inspection tool will flash "Blocked by robots.txt". Or run site:example.com and peek at the rendered HTML view.
• Soft 404s – The server returns 200 OK while the page says "Page not found". Google treats that as a soft 404 and drops it. Screaming Frog can filter "Soft 404" and compare the body to your 404 template.
• Duplicate content from staging – If staging.example.com isn't disallowed, Google indexes it alongside production. Run site:staging.example.com and watch the duplicates appear.
• URL parameters as separate pages – Filters, session IDs, tracking tokens explode your URL space. Google may crawl each variant, see thin content, and throttle the whole domain. In Search Console's URL Parameters tool, mark the harmless ones as "No effect".

How do you confirm each issue? Start with Coverage – it tells you why a URL is excluded. Then pull server logs (or fire up GoAccess) and look for 200 responses paired with Googlebot that never surface in the index. Finally, a quick site:yourdomain.com query gives you a sanity check: if you see only a handful of URLs, something's definitely blocking the rest.

Comparison & Decision Matrix

When you decide how deep to dig, you have two main routes: a manual audit with browser tools and Search Console, or a semi-automated crawl with a dedicated SEO crawler. Below is a quick matrix to help you pick the right tool for the job.

If you're a solo developer with a 20-page site, the manual route is fine. As soon as you hit a few hundred URLs, fire up Screaming Frog or Sitebulb. They'll surface hidden noindex tags and blocked resources in minutes, letting you patch them in bulk. Ahrefs shines when you already pay for its backlink suite and need continuous monitoring.

Comprehensive FAQ

Why does Google show "No results found" for my brand?

Google treats a brand name like any other query. If no page matches the exact string—or every match is flagged noindex—Google returns "No results found". Check the Performance report for impressions on your brand keyword. Zero impressions usually mean every page is excluded. Fix the noindex tags, submit a fresh sitemap, and request indexing.

How long does it take after fixing robots.txt?

Google re-crawls robots.txt about once a day. Speed it up with the "Test robots.txt" tool, hit "Submit", then use URL Inspection → "Request indexing". Most pages appear within a few hours to 48 hours; large sites need up to a week for the full crawl to settle.

Can a CDN hide my pages from Google?

Yes. A mis-configured edge cache that returns 403 or 404 for Googlebot will block indexing. Some CDNs serve a stripped-down HTML snippet to unknown agents, which looks like a soft 404. Verify with curl -A "Googlebot" against a few URLs; if the response differs, add an exception for Googlebot.

Does HTTPS affect indexing?

Google prefers HTTPS, but it won't outright ban HTTP. The real danger is mixed content: Chrome blocks insecure scripts, and Google's renderer sees a broken page. That often ends up as a soft 404. Use the Coverage → Mixed content warning to hunt down the offenders.

What does a 301 vs 302 mean for crawlability?

A 301 signals a permanent move; Google transfers link equity and updates the index. A 302 is temporary; Google keeps the original URL in the index and may keep crawling it. Accidentally serving 302 on a page you intend to retire can cause "soft-blocked" behavior and delay indexing. Stick with 301 unless the change truly is short-lived.

Operational Verification Checklist & Next Steps

• ✅ DNS resolves to the correct IP
• ✅ HTTP / HTTPS returns 200 OK for primary pages
• ✅ robots.txt allows / (no disallow for critical paths)
• ✅ Sitemap submitted and status "Success" in Search Console
• ✅ No noindex meta tag on any indexable page
• ✅ Structured data validates (use Rich Results Test)
• ✅ Core Web Vitals score > 80 in PageSpeed Insights
• ✅ Monitoring alerts set for 4xx/5xx spikes in Search Console

Run this list after any major deploy. If everything checks out, you can be fairly confident Google will see your site. If you hit a snag, go back to the "Critical Production Pitfalls" section and hunt for the hidden blocker.

Need a deeper dive? Check out our first part of the guide, or reach out via our contact page for a custom audit.