How to Grade Your AI SEO Rewrites With Search Console

Batch-rewriting old content with AI is one of the easiest things a founder can do this year. An agent can rework a hundred titles and summaries in an afternoon, and it feels like progress. Almost nobody checks whether it worked, because Search Console results lag by weeks and by then you have shipped ten other things.

At Careerkit, a resume and career tool, we rewrote titles and summaries on 130 blog posts on August 18 and 19. When we finally graded them properly, those posts had gone from 876 clicks to 827 over comparable 28-day windows, while blog posts we had not touched grew 20%.

Below: how the rewrites were made, the ledger and the script that grade them, the bug that hid the result for weeks, what the grades say, and how to build your own grader this week. A caveat up front: this does not prove AI rewrites are bad. It is one site with no A/B test, the batches share pages, and new pages went live in the same weeks. But it is a real grade, and most people never get one.

How the rewrites were made

The rewrites came from a Claude Code skill with a mode for click-through work. It works in five steps:

1. A script pulls Search Console data and ranks pages by expected click lift: impressions × (the click rate that position should get − the click rate the page gets).
2. The agent fetches each page's live title and description.
3. It drafts new ones that lead with the highest-impression search, add something specific, and stay under about 60 characters.
4. Everything lands in an approval queue. Nothing is written until a person says yes.
5. Approved changes publish to the CMS, and each one is logged.

Those two days produced three batches: 60 cleanup rewrites on titles, 8 larger title rewrites and 90 summary rewrites. Some posts got more than one, which is how it comes to 130 unique posts.

Step 1: Log every change in a ledger

The one thing we did right was logging. Every change appends one line to a JSONL file at the moment it ships: the page, the field changed, a note on what changed, the before numbers, the window they were measured over, and a date not to check before.

A real row, in plain words: action title_meta_rewrite; note "title -> exact singular phrase best font for cover letter"; before 45,976 impressions, 73 clicks, position 6.1; measured July 19 to August 16; predicted lift 1,746; verify after September 8.

Search Console data lags by about two days, and rankings take weeks to settle, so the verify date is set roughly three weeks out. The point of that date is discipline: it stops you from looking early and calling a blip a win.

Step 2: Write a grader that is one script

The grader goes through the ledger. For every row past its verify date, it queries the Search Console API for that page over a window of the same length as the baseline, ending two days ago. Two details are easy to miss:

• Sum anchor URLs. Search Console reports url#section rows separately, so a page's real total is the canonical URL plus its fragments. Filter by "contains" and then keep only rows that are exactly the URL or the URL with a fragment, because "contains" can also match a longer sibling slug.
• Use read-only credentials. A service account with the webmasters.readonly scope is all a grader needs.

Step 3: Keep the verdicts dumb

Four rules, simple enough to state in one line each:

• WIN: clicks up, and click rate not down.
• REGRESSION: clicks down.
• FLAT: anything else.
• UNDERWATER: at least 2,000 impressions, but a click rate under a quarter of a deliberately low expectation for that position (5% at position 5, 2% at position 10). This overrides everything except a regression.

UNDERWATER was added after the grader called one post a win for gaining 17 clicks, on a page turning 83,968 impressions into 69 clicks. As the comment in the code puts it: "A gate that reports success on a page converting 1 in 1,200 impressions is not a gate." When a page is underwater, the title is not the lever. The page type is wrong for what the searcher wants.

Step 4: Test the grader itself

Here is the embarrassing part. The August rows stored each page as url. The script only read slug. So all 150 of those rows were measured at /blog/undefined, scored zero clicks and were graded FLAT.

A broken grader that says "regression" gets investigated immediately. A broken grader that says "flat" looks like boring news, so nobody questions it. The fix on October 27 was small: read url, page or slug, and re-pull baseline clicks for rows that had only logged impressions. Then the real grades appeared.

The lesson is worth more than the fix: test your grader on one page you know went up. If it says flat, the grader is wrong, not the page.

What the grades said

The same 28-day windows, July 19 to August 16 against September 27 to October 25:

• Posts we did not touch that already existed in July: clicks +20%, impressions +21%. This is the control.
• 60 cleanup rewrites on titles: clicks +25%, roughly the same as doing nothing.
• 8 larger title rewrites: clicks -27%.
• 90 summary rewrites: clicks -39% (195 to 118), impressions -30%.

Across the 130 posts, the verdicts were 13 wins, 76 flat, 35 regressions and 6 underwater. On the summary batch, impressions fell along with clicks while the click rate held. That points to lost ranking rather than unappealing copy: the new summaries were not putting searchers off, Google was showing those pages less.

Why that happened is still an open question. The next step is to look at what those 90 pages lost, query by query, before any more text rewrites.

Why the control changes the whole reading

Look at the cleanup batch without the control: clicks up 25%. On its own that reads like a clear win, the kind of number that goes in a monthly update. Put it next to the posts nobody touched, which grew 20% over the same weeks, and most of that gain disappears. Search traffic was simply rising across the blog.

The same logic runs the other way for the summary batch. A 39% drop looks bad on its own, and it looks worse against a control that grew. Without that comparison you would credit your work for the season in good months and blame the season for your work in bad ones. Measuring a control takes one more query against the same API, so there is no good reason to skip it.

What to do with each verdict

• WIN: keep it, and note what the change was. If a pattern keeps winning, that is your next small batch.
• FLAT: leave it alone. Rewriting a flat page again is how a small waste becomes a big one.
• REGRESSION: look at the query data before you revert anything. Find out which searches the page lost, since that tells you whether the rewrite or something else is to blame.
• UNDERWATER: stop editing the title. Ask whether the page should be a different type of page, such as a tool or a gallery, or whether the search is one Google answers by itself.

Predicted lift is arithmetic, not a forecast

The 8 larger title rewrites logged a predicted lift: 2,895 clicks in total. The actual result was 303 clicks falling to 224, a loss of 79. The largest bet was a post on cover-letter fonts, predicted to gain 1,746 clicks. It gained 4.

The number was never really a forecast. It is impressions multiplied by the gap to an expected click rate, and it assumes the page can reach a normal click rate at all. That fonts post now has 97,552 impressions at position 5.8 and 77 clicks. It is underwater: most likely Google answers the question on the results page, so no title will fix it. Use expected lift to rank a worklist if you like, but never treat it as what you will get.

How to run your next batch so the grade means more

• Hold back a control. Pick similar pages at random and leave them alone. Comparing against "everything else on the blog" is better than nothing, but new posts and uneven topics muddy it.
• Change one thing per batch. Our batches overlapped, so a post that got both a title and a summary rewrite cannot be attributed to either.
• Start small. Ten pages for a month costs little. A hundred pages costs a lot if the direction is wrong, as the summary batch shows.
• Log query-level baselines. Page totals tell you that something moved. Query data tells you which searches you lost, and that is what you need to act on.

Build your own grader this week

1. Log every change in one line when it ships: URL, what changed, clicks, impressions, position, the date window, and a verify date three weeks out.
2. Have your agent write the script: for each row past its date, query the Search Console API for that URL over the same window length, summing fragment URLs.
3. Add four verdict rules, with a minimum impression floor so tiny pages cannot fool you.
4. Measure a control: the same windows for pages you did not touch. Without it, you will give your rewrite credit for the season.
5. Test it on a page you know moved before you believe a single "flat".

Practical takeaways

• AI makes shipping a rewrite almost free. It tells you nothing about whether the rewrite worked.
• You cannot grade a change you did not log. One line per change, written the moment it ships.
• Write verdict rules before you see numbers, and keep them simple enough to fit in four lines.
• "Flat" is the most dangerous verdict from a new grader. Check it against a page you know moved.
• Always compare against pages you did not touch. Our rewritten posts looked merely flat until the control grew 20%.
• Treat predicted lift as a way to rank work, not a forecast.

Conclusion

The rewrites did not do what we expected, and the only reason we know is a ledger line and a short script. That is a good trade: a few hours of building, and every future batch gets a real grade instead of a feeling. October's changes get graded next, on the same rules.

The founder-side version of this story ran in the HelloBuilder newsletter. If you are testing what agents do on your site rather than what Google does, look at Dedica.