Sort support tickets or survey answers into themes with counts you can check
Do it in two passes. First, the agent drafts a codebook of themes from a sample, and you edit it. Then it labels every row against that codebook, one label per row, and counts the labels in code. You check a random sample against your own labels before you trust the counts. Studies and practitioners report that one-shot theme summaries miscount, shift between runs, and silently skip data.
You get a spreadsheet with every ticket or answer labelled, a theme table with counts and example quotes, and an agreement check that shows how far to trust it.
- Difficulty
- Intermediate
- Time
- 1-3 hours for a few thousand rows
- Sources
- 7 · 7 independent
- Updated
- 2026-10-13
- Cost:
- Your agent plan, or API costs if you script the labelling
The prompt
Fill the blanks here, or leave them for your agent to ask. Pick the tab for your agent. The hand-off button gives any agent this page as markdown.
Analyse [file.csv], [N] rows of [support tickets / survey answers to "question text"]. The text is in column [text], the row id in [id]. Before you start, remove or mask names, emails, phone numbers and account numbers. Pass 1: codebook (stop after this pass) 1. With code, take a random sample of 150 rows (fixed seed). Read them and draft a codebook of [8 to 15] themes. For each theme give: short name, one-sentence definition, include and exclude rules, and two example row ids. 2. Add the codes "Other" and "Unclear". 3. Show me the codebook and stop. I will edit it. Pass 2: labelling (only after I approve the codebook) 4. Label every row in batches of 50 with code. For each row output: id, primary theme, secondary theme (optional, only if clearly present), sentiment [positive / neutral / negative], and a quote under 20 words from that row. 5. Coverage check in code: every input id has exactly one primary theme. Print input rows, labelled rows and missing ids. If any id is missing, label it before you go on. 6. Count in code: rows per primary theme, share of total (must sum to 100%), and the same split by [plan / month / channel]. 7. Write labels.csv and a summary with the theme table and three example quotes per theme with their ids. Hard rules - Never count by reading. Every number comes from code over labels.csv. - Do not invent themes that are not in the approved codebook. Put misfits in "Other" and list the 10 most common kinds of "Other" at the end. - Do not upload, share or email the results. Ask me first.
Human checkpoints: the agent stops and asks you
- Edit and approve the codebook before the agent labels the full set.
- Label a random sample yourself and compare before you share any counts.
Nothing is sent or changed, but wrong counts can steer product decisions. Remove names, emails and account details before you upload customer text.
What you need
- The exportdata
One row per ticket or answer, with an id and the text. Add plan, date or channel if you want splits.
- An agent that runs codeapp
Counting must happen in code, not in the chat text.
- Permission to process customer textpermission
Check your privacy terms before you upload support tickets to any assistant.
Steps
Export and clean the text
Export the tickets or answers with ids. Remove signatures, quoted email threads and personal details.
Check: The file has one row per item and no customer names or emails.
Draft the codebook from a sample
Run pass 1. The agent proposes themes with definitions and examples.
Check: You get 8 to 15 themes, each with include and exclude rules.
Edit the codebook
Merge overlapping themes, split vague ones, and rename themes in your team's words. Clear definitions matter. One public project reports that adding category descriptions cut run-to-run variance a lot.
Check: You could give the codebook to a colleague and they would label the same way.
Label every row
Run pass 2. The agent labels in batches and checks that every id has a label.
Check: The coverage check shows zero missing ids.
Check agreement on a sample
Take 50 random rows, label them yourself without looking, and compare with the agent's labels. Note which themes it confuses.
Check: You know the share of rows where you agree, and which themes are weak.
Re-run on a second seed or model
Label the same sample again in a fresh session and compare. Large swings mean the codebook is still too vague.
Check: Theme shares change only a little between the two runs.
Share the theme table
Share counts with the agreement result and example quotes. Say which themes are less reliable.
Check: Every number in the summary traces to labels.csv.
What goes wrong
Theme percentages add up to more than 100 percent, or counts look random.
Fix: Allow one primary theme per row and count in code over a labels file. Ipsos researchers saw totals over 100 percent when their prompt did not ask for one code per case.
Read the lesson →The same prompt gives different themes and shares on different days or models.
Fix: Fix the codebook with definitions and examples, and label against it. Compare two runs before you report.
The agent reads only part of the data, or says information is not in the file.
Fix: Label in batches with a coverage check that lists missing ids, and ask the agent to print columns before answering.
Read the lesson →The model creates fewer, broader themes than people would, and small issues disappear.
Fix: Ask for an "Other" list with the most common misfits, and review it for small but important problems.
Long, wordy answers get more weight than short ones.
Fix: Count rows, not mentions, and check a sample of short answers in your agreement test.
Evidence
Two or more public accounts from different people or sites describe the same job. We summarise each source in our own words. Open the original before you rely on it.
- Web2024-04-17· Custom agentDown the LLM rabbit hole, evaluating LLMs coding open-ended questions
Ipsos researchers coded 270 satisfaction answers with GPT models. The same GPT-4 prompt on three days gave different labels with totals of 107, 106 and 101 percent, which they partly blame on their prompt not asking for one code per case. They conclude themes made sense but results were not consistent and need human review.
Michael Link and Nick Bertoni (Ipsos) · census.gov - Blog2023-05-23· ChatGPTCan ChatGPT replace UX researchers? Comment classifications
On three sets of about 50 complaints, the authors found agreement between ChatGPT runs about as variable as between human coders, and ChatGPT made about half as many themes as humans. They write that each prompt they tried greatly changed the results and that ChatGPT is not deterministic.
MeasuringU (Schiavone, Roberts, Du, Sauro, Lewis) · measuringu.com - Web2025-02-10· Custom agentCluster Faster, Hack for Public Good 2025
The team built a human-in-the-loop clustering tool because staff found LLMs sometimes miscount and output was slow to verify. On about 600 items they report a match with hand sorting of about 90 percent with human input versus 68 percent without, and that category descriptions cut run-to-run variance.
Tan Jie Yin and Darren Ng · hack.gov.sg - Blog2025-10-16· Custom agentExperiences in using LLMs to code open-ended survey responses
The talk compared several models coding about 31,000 open answers against human coding. The speaker reports results close to human coders, with the best model varying by topic, and a possible bias towards the most expressive respondents.
Axel Bruns (liveblog of Paul Pressmann) · snurb.info - Blog2024-01-19· ChatGPTA market researcher's review of ChatGPT's data analyst tool
The reviewer loaded a raw consumer survey CSV and reports that ChatGPT said basic information was not in the data set until he named specific columns. He concludes it was weak at reading raw survey files.
Intuify · intuify.com - Forum2024-12-30· ClaudeFormatting Qualtrics results CSV for GenAI analysis
The poster compared ChatGPT and Claude on Qualtrics exports. They report that one export format needed heavy reformatting for ChatGPT, while the raw data export worked well, and that open-text analysis was where these tools helped most.
TFXR · community.qualtrics.com - Web2026-05-27When models disagree, rethinking LLM evaluation for public comment analysis
The authors had four LLMs code 1,260 public comments from a federal docket. They report that differences between models were larger than differences caused by prompts, and that an expert rubric reduced but did not remove the disagreement.
Najera, Moon, Srinivasan, Veeraraghavan · arxiv.org
- Claude use case, analyze patterns in user feedbackchecked 2026-10-13
- OpenAI docs, code interpreter toolchecked 2026-10-13
Report how it went
Every report makes this page better for the next person and the next agent. Reports are reviewed before they count. Agents can report with the report_run tool.
Loading reports…