Automate Keyword Research Using AI: A Workflow With Checks

November 9, 2026

To automate keyword research using AI, schedule a job that collects real query data (Search Console, autocomplete, a keyword API). Then use an LLM only for the narrow jobs it does well: grouping variants, labeling intent and scoring fit with your offer. The output is a short list of page gaps and refresh targets that a person reviews before anything gets written.

Most guides on this topic give you prompts to paste into ChatGPT, or a no-code chain that stops at a spreadsheet of keywords. This one covers the whole pipeline: data sources, API limits, clustering, intent scoring, gap detection, scheduling, cost trade-offs and the checks that keep the output trustworthy.

Want the articles, not the pipeline? SEO Automation finds what your customers type into Google: real searches for your market, with volume and difficulty, refreshed every week. Each one becomes an article on your site. Start for $1 and get your SEO plan in a few minutes.

What you actually automate in keyword research

"Keyword research" is really six separate jobs. Some need data, some need judgment, and only a few need an LLM. If you mix them up, you end up with an AI that makes up search volumes.

Job	Best done by	Input	Risk if it goes wrong
Collect queries your site already shows for	Script + Search Console API	Your property data	Low
Expand into new queries	Script + autocomplete + keyword API	Seed topics	Low
Get volume, difficulty, CPC	Keyword data API	Query list	Low, if the data is real
Group variants into one topic per page	Rules + LLM or embeddings	Query list	High: duplicate articles
Label intent and fit with your offer	LLM with a strict schema	Groups + offer description	Medium: off-target articles
Decide what to write, refresh or skip	Rules, then a human	Scored groups + page map	High: wasted months

The table shows where an LLM belongs. A language model has no access to search data unless you give it some, so any volume or difficulty it states on its own is a guess. The rule is simple: never ask a model for a search volume, fetch it from a data source. Then let the model reason about what the query means.

If you want the bigger picture of which SEO tasks are worth scripting, our guide on how to automate SEO ranks nine of them by risk. This article goes deep on one of them.

The AI keyword research workflow in 6 steps

Here is the full pipeline. Each step writes its output to a table (a database, a JSON file in a repo, or a Google Sheet), so the next step can rerun without starting from scratch.

Pull every Search Console query your site appeared for, with the page that ranked, over the last 90 days.
Expand with Google autocomplete and a competitor gap check, starting from seed topics tied to what you sell.
Enrich each new query with real volume, difficulty and CPC from a keyword data API.
Cluster variants into one group per future page, and map each group to the existing page that already covers it.
Score each group for intent (purchase, comparison, problem, info) and fit with your offer, using an LLM with a fixed output schema.
Output two lists: groups with no page (new articles) and groups whose page sits in positions 4 to 20 (refreshes).

Steps 1 to 3 are plain data plumbing. Steps 4 and 5 are where AI earns its place. Step 6 is rules, plus a human sign off.

Step 1: Pull every Search Console query with its page

Your own Search Console data is the most valuable source, because it shows real queries where Google already considers you relevant. A query where you sit at position 14 is far easier to win than a fresh keyword from a tool.

Use the Search Analytics query method with the dimensions query and page. Two documented limits shape the script:

rowLimit goes from 1 to 25,000, with a default of 1,000 (API reference). If your script never sets it, you only ever see 1,000 rows.
The API exposes at most 50,000 rows per day per search type. To get them all, increase startRow by 25,000 until a response comes back with 0 rows (Getting all your data).

Google's recommendation is to query one day at a time. That keeps each request inside the 50,000 row cap and lets you cache results by date:

from google.oauth2 import service_account
from googleapiclient.discovery import build

creds = service_account.Credentials.from_service_account_file(
    "service-account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"])
gsc = build("searchconsole", "v1", credentials=creds)

def fetch_day(site, day):
    rows, start = [], 0
    while True:
        body = {"startDate": day, "endDate": day,
                "dimensions": ["query", "page"],
                "rowLimit": 25000, "startRow": start}
        res = gsc.searchanalytics().query(siteUrl=site, body=body).execute()
        batch = res.get("rows", [])
        if not batch:
            return rows
        rows += batch
        start += 25000

The service account needs to be added as a user on the Search Console property. Our walkthrough on setting up Search Console API access covers that part.

Quotas and load. The Search Console API limits page sets 1,200 queries per minute per site and per user, and 30,000,000 per day and 40,000 per minute per project. Request counts are rarely the problem. Load is. The same page says queries grouped by page and query string are the most expensive, and a six month range costs far more than a one day range. In practice:

Backfill once. Fetch the last 90 days day by day, store each day, and never fetch them again.
Then fetch one new day per night. Data arrives with a delay of 2 to 3 days (Google), so your nightly job should request the day from three days ago, not yesterday.
Aggregate locally. Build your 90 day view by summing the cached days, rather than sending a 90 day query to the API.

Step 2: Expand with autocomplete and competitor gaps

Search Console only shows queries you already appear for. To find the ones you are missing, add two sources.

Google autocomplete. Start from 10 to 30 seed phrases that describe your product and your buyer's problems. Collect suggestions for each seed, for the seed plus each letter of the alphabet, and for question patterns ("how to", "best", "vs", "for"). Go one level deeper by feeding the best suggestions back in as seeds. The suggestion endpoint is not an official API, so throttle requests (one every few hundred milliseconds is polite) and expect it to change. If you need a contractual source, keyword APIs offer suggestion endpoints too.

Competitor gaps. A keyword data API can list the queries where a competitor ranks on page 1 or 2. Subtract every query where your own site appears, and you have a gap list. Run this step before clustering, so gap queries get grouped and mapped with everything else instead of landing in a separate list.

Keep seeds tight. A seed like "SEO" will flood the table with job listings, courses and definitions. A seed like "automate meta descriptions" stays close to what a buyer would pay to solve.

Step 3: Add real volume and difficulty

Send the deduplicated list of new queries to a keyword data API and store monthly volume, difficulty and CPC next to each one. Make's tutorial does exactly this with the DataForSEO "Get Keyword Ideas" module, an Iterator and Google Sheets, and filters on difficulty and CPC ranges (Make guide).

Two habits save money here:

Only enrich what is new. Keep a table of queries you have already priced and skip them for 30 days.
Treat CPC as an intent signal, not just a cost. Advertisers bid on queries that convert. A query with a real CPC and modest volume is often a better article than a high volume informational one.

Queries that Search Console already gives you do not always need volume data. Impressions over 90 days are a real, site specific measure of demand.

Step 4: Cluster variants into one group per page

This is the step that decides whether your automation helps or hurts. If "automate keyword research with ai" and "ai keyword research automation" end up as two separate topics, you write two articles that compete with each other.

A two pass approach works well:

Deterministic key. Lowercase the query, drop stop words and small modifiers (the, a, for, with, 2026, best), sort the remaining words. Queries with the same key go in the same group.
LLM or embedding merge. For the groups that remain, ask a model (or compare embeddings) which groups mean the same thing to a searcher. Only merge when the answer page would be the same.

STOP = {"the","a","an","to","for","with","of","in","on","how","best","2026","using"}

def group_key(q):
    words = [w for w in q.lower().split() if w not in STOP]
    return " ".join(sorted(set(words)))

groups = {}
for q in queries:
    groups.setdefault(group_key(q), []).append(q)

Then map every group to a page. Use the page Search Console already shows for that group if there is one. If not, check your sitemap for a URL that contains the same words. A group with a page is never a new article. It is either fine as it is or a refresh candidate. This rule stops the pipeline from proposing a second page for a topic you already cover.

A quick way to see if a merge is right: search both queries in Google. If the top results are mostly the same URLs, they belong in one group. You can automate this by comparing the top 10 URLs of each query from a SERP API, and merging when they overlap a lot.

Step 5: Score intent and offer fit with an LLM

Now the model does the work it is good at: reading a group of queries and judging what the searcher wants. Give it your offer in three or four sentences, the list of groups, and a strict output format. Batch a few dozen groups per call.

You classify search queries for a B2B SaaS that automates SEO
(keyword research, articles, publishing, Search Console tracking).
For each group, return JSON only:
{"key": "...", "intent": "purchase|comparison|problem|info|tool|brand|off",
 "fit": 0-10, "reason": "one sentence"}
fit = how likely someone searching this could become a customer.
Score jobs, salaries, courses and definitions without method as 0 to 3.
Score "tool", "brand" and competitor names as page work, not articles.

Three rules keep this step reliable:

Fixed labels. Free text intent labels drift from one run to the next. A closed list does not.
A reason field. One sentence per decision makes the human review in step 6 fast, and it becomes your decision log.
Route tool and brand queries elsewhere. Queries like "keyword research tool" or "keyword tool alternative" belong on your homepage or pricing page, not in a blog post.

Step 6: Output the gap list and the refresh list

Apply simple rules to the scored groups:

Situation	Rule	Action
No page, fit 7 or more, intent not tool/brand/off	New topic	Brief for a new article
Page exists, position 4 to 20, real impressions	Close to page 1	Refresh the existing page
Page exists, top 3	Working	Leave it, monitor
Competitor on page 1 or 2, you absent, fit 7 or more	Competitor gap	Brief for a new article
Tool, brand or competitor name	Commercial page query	Recommendation for a site page
Fit under 5	Off target	Skip, keep in the log

Each new article brief should carry its group's variants, so the writer covers all of them in one page. Refreshes deserve their own process: our article on automating content decay detection with the Search Console API explains how to find and rewrite the pages that are slipping.

How to automate keyword research with AI on a schedule

A pipeline you run by hand once a quarter is not automation. Put it on a timer. If your site lives in GitHub (Next.js, Astro, Hugo), a scheduled GitHub Action is the simplest option:

name: keyword-research
on:
  schedule:
    - cron: "0 3 * * *"   # every night at 03:00 UTC
  workflow_dispatch:
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python pipeline.py
        env:
          GSC_KEY: ${{ secrets.GSC_SERVICE_ACCOUNT }}
          KW_API_KEY: ${{ secrets.KEYWORD_API_KEY }}
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
      - run: |
          git add data/keywords
          git commit -m "keyword map update" || exit 0
          git push

Committing the output to the repo gives you a free history: you can diff what changed in your keyword map from one night to the next. The same script runs just as well from a server cron job, n8n's Schedule trigger, or a scheduled scenario in Make.

Not every step needs to run nightly. A cadence that matches how fast each source changes:

Step	Cadence	Why
Search Console pull (one new day)	Daily	Cheap, and builds history
Autocomplete expansion	Daily, small batch	Rotate seeds and letters so each night adds new queries
Volume and difficulty enrichment	Daily, new queries only	Pay only for what you have not priced
Clustering and intent scoring	Weekly	Groups need enough new queries to be worth regrouping
Gap list and refresh list	Weekly	Matches a weekly content plan
Seeds, offer description, fit rules	Monthly	Strategy decisions, made by a person

No-code chain, script or SaaS: which one fits

There are three ways to build this, and they suit different teams.

	No-code chain (Zapier, Make, n8n)	Your own script + APIs	SaaS that runs the workflow
Setup	Visual, quick to start	A development project	Sign up and connect your site
Search Console depth	Default row limits unless you add pagination	Full: pagination, daily cache, page mapping	Depends on the product
Clustering and page mapping	Hard to do well in modules	Fully under your control	Built in or absent
Cost model	Billed per task or operation, plus data API and LLM usage	Data API and LLM usage, plus your time	Subscription
Maintenance	Can break quietly when a module changes	Yours to maintain	The vendor's
Best for	A marketer testing the idea on one site	A dev team that wants full control	Teams that want articles, not a pipeline

Verdict. If you are a developer who wants to own every rule, build the script. If what you actually want is published articles on the queries your buyers type, a SaaS gets you there without a pipeline to maintain. SEO Automation is built for that second case. Start for $1 and get your SEO plan in a few minutes.

A note on the popular no-code tutorials, which serve a different purpose. Zapier's version starts from a competitor page opened in the browser: the page text is extracted, an AI step lists its main keywords, and the results land in Google Sheets (Zapier guide). That is a good way to map the vocabulary of pages that already rank, for example when you write a brief. The pipeline in this article answers a different question: what people search, how often, and whether you already have a page for it. The two work well together.

For agencies running many client sites, the script route is often the better fit. One codebase, one config file per client (property URL, seeds, offer description, exclusions), and the same nightly job loops through them. The per site quota of 1,200 queries per minute (Google) means you can process many properties in sequence, as long as you cache by day.

If you are weighing these options against hiring, our comparison of automated SEO versus agency cost breaks down what each approach includes.

What to automate and what a human must still check

Automation removes the collecting, sorting and scoring. It does not remove responsibility for the decisions. Here is the split we recommend.

Safe to automate fully:

Pulling Search Console data, autocomplete and API metrics
Deduplication and the deterministic grouping pass
Mapping groups to existing pages
Applying the gap and refresh rules

Automate, then sample by hand:

LLM merges of groups. Each week, open 10 merged groups and check that one page could really answer all of their queries.
Intent and fit scores. Read the "reason" field on the top 20 new topics before they become briefs. A score that sounds wrong usually means your offer description is too vague.

Keep human:

Seeds and exclusions. What you want to rank for is a business decision.
Approving the final list of new articles each week.
Checking that a "gap" is not actually covered by a page the script missed (a page with a different URL slug, or a doc page outside the blog).

Keep a decision log. Store every group with its score, its reason, the decision (write, refresh, skip) and the date. When an article underperforms three months later, you can trace why it was chosen.

Measure the result. For every page created or refreshed from the pipeline, compare clicks, impressions, CTR and average position over the 28 days after it went live against the period before. Leave a few days after publishing before you start counting, since Search Console data lags by 2 to 3 days. Feed the outcome back: if refreshes in positions 15 to 20 rarely move, tighten the refresh range.

Stay inside Google's rules. Automating research is not the issue. Publishing at volume without value is. Google defines scaled content abuse as "when many pages are generated for the primary purpose of manipulating search rankings and not helping users" (Google spam policies). The one group, one page rule and the fit threshold are your first protection: they stop the pipeline from producing a page for every minor variant.

Mistakes we see in AI keyword research workflows

These come up again and again when teams first automate this:

Asking the LLM for volumes. It will answer confidently. The numbers are not real. Always fetch metrics from a data source.
Leaving rowLimit at its default. You see 1,000 rows and miss most of your long tail.
Querying 90 days of page × query in one call. That is the most expensive load pattern Google documents. Fetch day by day and cache.
No page mapping. The pipeline proposes "new" articles for queries you already rank for, and you end up competing with yourself.
One article per variant. "AI keyword research workflow" and "automate keyword research with AI" are one page.
No fit score. The list fills up with definitions, job titles and course queries that will never bring a customer.
Free text intent labels. "Commercial-ish" and "mostly informational" cannot drive rules. Use a closed list.
Running everything nightly. Regrouping and rescoring every night makes the plan unstable. Collect daily, decide weekly.

How SEO Automation handles this

Building the pipeline above is a project, and it needs maintenance. SEO Automation takes the research and the writing off your plate:

Scout reads your site and finds what your customers type into Google: real searches for your market, with volume and difficulty, refreshed every week.
Strategist chooses one article per intent and plans 50 articles a month, easiest wins first.
Writer researches on the web and writes with sources. Editor checks every fact and every link.
Publisher publishes on WordPress, Shopify, Webflow, Wix, Framer, GitHub (Next.js, Astro, Hugo) or by webhook, resubmits your sitemap in Search Console and notifies IndexNow.
Articles on page 2 are reworked, and titles with a low CTR are rewritten.

One plan: $49 per site per month, 50 articles a month, first month at $1, cancel anytime. As with any content, we recommend that a person reads what goes live on your site. Details are on the pricing page.

Start for $1 and get your SEO plan in a few minutes.

Frequently asked questions

Can AI do keyword research on its own?

Not reliably. A model without a data connection has no way to know real search volumes or difficulty, so it guesses. The dependable setup feeds the model real data from Search Console and a keyword API, then asks it to cluster, label intent and score fit. A person still sets the seeds and approves the final list.

Can I use ChatGPT to automate keyword research using AI?

You can use ChatGPT or another LLM as one step of the pipeline, through its API, for clustering and intent labeling. On its own, the chat interface is a manual tool: you paste prompts and copy answers. To automate the work, a scheduled script or a no-code scenario has to collect the data, call the model with a fixed output schema and store the results.

How often should an automated keyword research workflow rerun?

Collect data daily: one new day of Search Console data and a small batch of autocomplete suggestions. Regroup, rescore and produce the gap and refresh lists weekly, to keep the content plan stable. Review seeds, exclusions and your offer description monthly.

What is the difference between keyword clustering and gap analysis?

Clustering groups queries that a single page can answer, so you write one article per intent instead of one per variant. Gap analysis compares those groups with what exists: groups with no page on your site, groups where competitors rank and you do not, and groups where your page sits just outside the top results. You cluster first, then look for gaps.

Is AI keyword research better than traditional research?

It does the same thing on a larger scale and more often. A person can review a few hundred queries in a session. A pipeline can process your full Search Console history every night and flag the handful that matter. The judgment calls, meaning which topics fit your business and which articles get approved, stay with a person.