How to Find Content Ideas Using Google Search Console

Real-World Context & Developer Perspective

Last quarter the tech blog we manage stalled at 2 k clicks a month. The team kept tossing ideas like "maybe we should write about X". I pulled the raw GSC data. A few queries showed 150-300 impressions each, CTR hovering around 1 %. One of them? "how to migrate a WordPress site to Docker". Nobody had covered it yet. We wrote a deep-dive. Two weeks later the page hit #4, earned 1 k clicks, and lifted overall traffic by 12 %.

That's the takeaway: raw GSC numbers are a gold mine. They tell you what people actually type, not what you assume they might care about. Swap intuition for a data-driven list and you stop chasing ghosts.

For developers GSC is just another JSON endpoint. For marketers it's a list of untapped queries. Bridging that gap means turning impressions, clicks, and average position into concrete ideas. The rest of this guide shows how to do that without pulling your hair out.

Architecture & Technical Prerequisites

First, get the right access. You need Owner or Full permissions on the property and the ability to spin up a service account in Google Cloud.

• Service account: Create it in the Cloud Console, download the JSON key, stash it in a secret manager or an encrypted CI file.
• OAuth scopes: https://www.googleapis.com/auth/webmasters.readonly does the job.
• API enablement: Turn on "Search Console API" in the same project.
• Tech stack: Anything that can make HTTPS calls. I usually fire up a tiny Node script or a Python one-liner. Bash works if you're comfortable with curl and jq.

Google caps the API at 600 requests per 100 seconds per project. That's plenty for a daily pull; parallel scrapes need throttling. Data lives for 16 months, so you can backfill historic trends without hitting limits.

In short: one service account, the read-only scope, and a language you trust. No heavy frameworks, no databases—just a CSV at the end.

Step-By-Step Implementation: Core Setup

Below is a minimal Python script that authenticates, fetches the performance report, filters low-CTR, high-impression queries, and writes a CSV. Swap the Python bits for curl + jq if you prefer pure Bash.

# Install deps
pip install google-auth google-auth-httplib2 requests pandas

import json, csv, datetime
from google.oauth2 import service_account
import requests, pandas as pd

# ---- Config ----
SCOPES = ['https://www.googleapis.com/auth/webmasters.readonly']
KEY_FILE = 'gsc-service-account.json'
SITE_URL = 'https://example.com'
START_DATE = (datetime.date.today - datetime.timedelta(days=30)).isoformat
END_DATE = datetime.date.today.isoformat
# ----------------

creds = service_account.Credentials.from_service_account_file(KEY_FILE, scopes=SCOPES)
token = creds.token
headers = {'Authorization': f'Bearer {token}'}

payload = {
 "startDate": START_DATE,
 "endDate": END_DATE,
 "dimensions": ["query"],
 "rowLimit": 25000
}

resp = requests.post(
 f'https://searchconsole.googleapis.com/webmasters/v3/sites/{SITE_URL}/searchAnalytics/query',
 headers=headers,
 json=payload
)

rows = resp.json.get('rows', [])
df = pd.DataFrame(rows)

df['ctr'] = df['clicks'] / df['impressions'] * 100
filtered = df[(df['impressions'] > 100) & (df['ctr'] < 2)]

filtered.to_csv('low_ctr_queries.csv', index=False)
print('Wrote', len(filtered), 'rows')

What's happening?

1. Load the service account and grab an OAuth token.
2. POST to /searchAnalytics/query for the last 30 days, asking for the query dimension.
3. Convert the JSON rows into a pandas DataFrame for easy math.
4. Calculate CTR, filter noise, dump the result.

If you're on Linux and love one-liners, try this Bash alternative:

curl -s -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
 -X POST "https://searchconsole.googleapis.com/webmasters/v3/sites/https%3A%2F%2Fexample.com/searchAnalytics/query" \
 -d '{"startDate":"2023-09-01","endDate":"2023-09-30","dimensions":["query"],"rowLimit":25000}' |
jq -r '.rows[] | select(.impressions > 100 and (.clicks/.impressions) < 0.02) |
 "\(.keys[0]),\(.clicks),\(.impressions),\(.position)"' > low_ctr_queries.csv

Both approaches give you a CSV of queries Google already knows about but you haven't capitalised on. That file is the seed for your next content brainstorm.

Advanced Configuration & Performance Tuning

The simple script works for a single property and a single date window. Real-world pipelines need pagination, multiple slices, and sometimes parallelism.

Pagination & Date Loops

The API returns up to 5 000 rows per request. If you have more, add startRow and loop until the response is empty. Here's a quick async Python example that walks a month-by-month range:

import aiohttp, asyncio

async def fetch(session, site, start, end, start_row=0):
 body = {
 "startDate": start,
 "endDate": end,
 "dimensions": ["query"],
 "rowLimit": 5000,
 "startRow": start_row
 }
 async with session.post(
 f'https://searchconsole.googleapis.com/webmasters/v3/sites/{site}/searchAnalytics/query',
 json=body) as resp:
 return await resp.json

async def gather_month(site, s, e):
 async with aiohttp.ClientSession(headers={'Authorization': f'Bearer {token}'}) as sess:
 rows = []
 row = 0
 while True:
 data = await fetch(sess, site, s, e, row)
 batch = data.get('rows', [])
 if not batch:
 break
 rows.extend(batch)
 row += len(batch)
 return rows

async def main:
 site = 'https%3A%2F%2Fexample.com'
 m ('2023-02-01','2023-02-28')]
 tasks = [gather_month(site, a, b) for a, b in months]
 results = await asyncio.gather(*tasks)
 # flatten & process...

asyncio.run(main)

Each month runs in its own coroutine; pagination is handled internally. Extend months to cover a full year without hitting the 600-request ceiling because the event loop spaces the calls.

Parallel Requests with GNU Parallel

If Bash is your playground, parallel does the heavy lifting. Create a tiny JSON per date range, then fire them off:

#!/usr/bin/env bash
dates=("2023-01-01:2023-01-31" "2023-02-01:2023-02-28")
export TOKEN=$(gcloud auth application-default print-access-token)

run {
 IFS=":" read start end <<< "$1"
 curl -s -H "Authorization: Bearer $TOKEN" -X POST \
 "https://searchconsole.googleapis.com/webmasters/v3/sites/https%3A%2F%2Fexample.com/searchAnalytics/query" \
 -d "{\"startDate\":\"$start\",\"endDate\":\"$end\",\"dimensions\":[\"query\"],\"rowLimit\":5000}" |
 jq -r '.rows[] | "\(.keys[0]),\(.clicks),\(.impressions),\(.position)"'
}
export -f run

printf "%s\n" "${dates[@]}" | parallel -j 4 run > all_queries.csv

Four workers run in parallel, each pulling a month's worth of data. Adjust -j to respect the 600-request window.

Enriching with Trends & Volume

Impressions show interest; CTR shows missed clicks. Add external signals to rank gaps. Google Trends offers a public endpoint; Ahrefs or SEMrush APIs give absolute volumes if you have a subscription.

# Example: fetch a Google Trends interest score
import requests, json

def trends_score(keyword):
 url = ("https://trends.google.com/trends/api/explore?hl=en-US&tz=-120&req="
 f"{{'comparisonItem':[{{'keyword':'{keyword}','geo':'','time':'today 12-m'}}]}}")
 r = requests.get(url)
 # strip the JSONP wrapper
 return json.loads(r.text[5:])['widgets'][0]['token']

After you have low_ctr_queries.csv, run a small loop that appends a trend_score column. Then sort by impressions * trend_score to surface the hottest gaps.

Aggregating by Page

Sometimes a page, not a keyword, is the bottleneck. Group queries by landing page, compute average position drift (today vs. 90 days ago), and flag pages where drift > 5 spots.

import pandas as pd

cur = pd.read_csv('current.csv')
hist = pd.read_csv('historic.csv')

merged = cur.merge(hist, on='query', suffixes=('_now','_old'))
merged['pos_drift'] = merged['position_old'] - merged['position_now']

page_stats = merged.groupby('page_now').agg({
 'pos_drift':'mean',
 'impressions_now':'sum'
}).reset_index

candidates = page_stats[(page_stats['pos_drift'] > 5) &
 (page_stats['impressions_now'] > 500)]
candidates.to_csv('page_opportunity.csv', index=False)

This produces a shortlist of pages that need a content boost or a tighter-targeted article. Pair it with the query list from earlier and you have a full-fledged editorial calendar built straight from GSC data.

Next we'll cover turning those CSVs into briefs, tagging intent, and measuring impact after publishing. Until then, fire up the script, grab the data, and start hunting hidden clicks.

Critical Production Pitfalls & Diagnostic Gotchas

Even after you've nailed the basics, the data pipeline can still bite you. Here are the errors most teams hit and how to squash them.

1. 403 forbidden – wrong scope
A 403 on curl usually means you missed the https://www.googleapis.com/auth/webmasters.readonly scope. Re-authorise the service account, then grab a fresh token.

gcloud auth login
gcloud auth application-default login --scopes=https://www.googleapis.com/auth/webmasters.readonly

After that the API will hand you data instead of a wall.

2. Quota exhaustion
GSC hands out 2 000 queries per day per project. Nightly pulls for every property plus ad-hoc reports eat that fast; you'll see error: {code: 429}. The cure? Batch requests. Slice dateRange into 7-day windows or spread runs across a few service accounts.

3. Mismatched site property URLs
Your property might be https://example.com while the script asks for http://example.com/. The API treats them as distinct, returns nothing. Normalise URLs first; a quick urlparse check catches stray trailing slashes.

4. CSV encoding issues
Dump JSON to CSV and you get "café" turned into gibberish. The culprit is UTF-8 versus Windows-1252. Open the file, verify the BOM. In Python, use encoding='utf-8-sig' to prepend the BOM and keep Excel happy.

Debugging commands you'll love

# Verify auth token
curl -v "https://www.googleapis.com/webmasters/v3/sites/https%3A%2F%2Fexample.com/searchAnalytics/query" \
 -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
 -H "Content-Type: application/json" \
 -d '{"startDate":"2024-01-01","endDate":"2024-01-31","dimensions":["query"]}'

If you see 401 Unauthorized, the token is stale. Refresh and retry.

Logging & alerting
Write every API call to a JSON log (timestamp, property, status). Push those to CloudWatch or Stackdriver. Set an alarm on any entry where status!= 200. That way you'll know instantly when a quota hit or a 403 creeps in, instead of discovering it weeks later in a stale spreadsheet.

Comparison & Decision Matrix

Copy-paste the table into Markdown or Confluence. It lines up the three most common ways to harvest Google Search Console content ideas against the criteria teams care about.

If you already script other data sources, the API route pays off. Need a quick "low-hanging fruit" list for a client meeting? The UI is fine. Prefer not to host cron jobs? A tool like SEMrush saves you headaches.

Comprehensive FAQ

How often should I refresh the query list?

Evergreen content: weekly pulls work. News or product launches: bump to daily. The API lets you set any interval; just watch your quota. A good rule of thumb: pick the smallest window that still gives actionable insight without exhausting your daily limit.

Can I pull data for mobile-only results?

Yes. Add the device dimension and filter on mobile. The response returns impressions, clicks, and average position for mobile users alone. Mobile data can be noisy for low-traffic queries, so apply a minimum impression threshold—say 50—before treating it as a content signal.

What's the best way to combine GSC data with internal analytics?

Export the GSC CSV, then join on page with your Google Analytics pagePath. In BigQuery or a pandas DataFrame you can enrich each query with bounce rate, time on page, and conversions. The sweet spot looks like:

query | page | clicks | impressions | ctr | avg_position | bounce_rate | conversions

That view spots high-click queries that still bounce—a perfect cue for a content refresh.

How do I handle multilingual sites?

Create a separate GSC property for each language sub-directory or sub-domain (e.g., example.com/en/ vs fr.example.com). Pull each property, tag rows with a lang column, then merge. You'll instantly see which language version under-performs for a given query and can prioritise localisation.

Is there a limit to how many rows I can export?

The UI caps at 5 000 rows per export. The API has no hard row limit, but each request returns a max of 5 000 rows. Use rowLimit and startRow to paginate until the rows array is empty, then concatenate the results.

Operational Verification Checklist & Next Steps

Before you go live, run through these items:

1. Verify the service-account JSON key is stored securely and GOOGLE_APPLICATION_CREDENTIALS points to it.
2. Confirm the CSV header order matches your downstream parser (query, page, clicks, impressions, ctr, avg_position).
3. Execute a test pull for a single property and a 7-day window; inspect the first 10 rows manually.
4. Compare those rows against the GSC UI for the same period – numbers should line up within rounding error.
5. Schedule the cron job (e.g., 0 4 * * * for a 4 AM daily run) and watch the logs for a full cycle.
6. Set up an alert email that fires on any non-200 response or when the row count drops below 100.
7. Take the top 10 queries by impressions, map each to a content brief, and assign owners in your project tracker.

That's it. Pick the first insight, write the post, and watch the clicks climb. Need help building the pipeline? Drop us a line or check out our Web Engineering service page for a custom solution.