How to Turn Engagement Survey Comments into Themes and Sentiment

Short answer: Read a sample of comments, build a short codebook of 8 to 15 themes, tag every comment with one or two themes and a sentiment (positive, mixed, negative), then count themes by team and compare them with the rating questions. AI can do the first pass, but a person should check a sample and the codebook.

Every engagement survey has comment boxes, and they hold the answers the rating scales cannot: why scores dropped, what one manager does differently, which process wastes everyone's week. HR teams want themes and sentiment from those comments, yet many survey platforms sell text analytics as a higher tier or add-on. Some Culture Amp and Qualtrics plans work this way, so check your contract before assuming it is included.

The good news for a 20-500-person company is that the volume is small enough to do well yourself. A 200-person survey with three open questions yields perhaps 300 to 450 usable comments. That is an afternoon of disciplined work, not a data science project.

Colleagues reviewing printed charts and a tablet on a shared table
Photo: fauxels on Pexels

Why Comments Matter More Than the Scores

Rating questions tell you where to look; comments tell you what to fix. A drop on a question such as "I have the tools I need" could mean old laptops, a slow approval process or a missing software license. Only the comments say which. Gallup's research on engagement ties the biggest gains to local, specific action by managers, and specific action needs specific causes (Gallup).

Comments also surface issues no question asked about: a new policy that landed badly, a safety concern, a team that feels cut out of decisions. Treat them as the early warning layer of the survey.

A Five-Step Method That Holds up

This is a simplified version of thematic analysis, the method most qualitative researchers use (Braun and Clarke).

  1. Clean the export. Remove blank answers and answers like "n/a" or "nothing". Keep team, location and tenure band, but only for groups of five or more people, so no comment can be traced to a person.
  2. Read a sample first. Read 50 to 80 comments before you name any theme. Write down the topics people actually raise, in their words.
  3. Build a codebook. Settle on 8 to 15 themes, each with a one-line definition and an example: workload, tools and systems, pay and benefits, career growth, recognition, communication from leadership, manager support, team collaboration, policies and process, wellbeing.
  4. Tag every comment. Give each comment one or two themes and one sentiment: positive, mixed or negative. Split long comments that cover several topics.
  5. Count and compare. Count themes by team and by sentiment, then set those counts next to each team's rating scores.

Two people tagging the same 40 comments and comparing results is the cheapest quality check there is. If they agree on fewer than about four in five, tighten the theme definitions and try again.

Using AI for the First Pass, Safely

Large language models are good at a first pass: proposing themes, tagging comments against your codebook and drafting a one-line summary per theme. They are weaker at sarcasm, at mixed comments, and at knowing which theme matters most to your business. Use them to speed up the tagging, then have a person check a random sample of at least 50 tagged comments and every comment tagged as a safety, harassment or legal concern.

Before pasting comments into any AI tool, confirm the tool does not train on your data, and strip names and other identifying details. Employees were promised confidentiality; keep that promise.

What a Useful Comment Report Looks Like

An example comment summary for one survey (illustrative numbers)
ThemeCommentsNegative shareTeams most affected
Workload6478%Support, Operations
Career growth5161%Engineering
Manager support4734%Mixed by manager
Tools and systems3982%Sales
Recognition2245%Company-wide

Lead with the three themes that are both frequent and negative, quote two or three representative comments for each (edited so no one can be identified), and name one owner and one action per theme. Leaders act on short lists.

Insight Studio runs your surveys and reads CSV and Excel exports from other survey tools, so you can clean the comment file, add theme and sentiment columns, and chart them by team in one place. Organization Viewer supplies the team and manager structure for each count, so themes roll up the same way your org chart does.

Common Mistakes to Avoid

  • Word clouds. They count words, not meaning; "manager" appears in praise and complaints alike.
  • Reporting comments for groups under five people, which breaks confidentiality and future response rates.
  • Too many themes. Thirty themes is a list, not a finding.
  • Treating sentiment as a score to chase. The goal is fixing causes, not making comments sound happier.
  • Not closing the loop. Tell employees which themes you heard and what will change; it is the strongest driver of response rates next time.

If you want an outside team to run the survey and the comment analysis end to end, our Employee Engagement Survey Program does that, and the Manager Training Cohort helps managers act on what their teams said.

Questions

How many comments do I need before themes are meaningful?

Around 100 comments is enough for company-wide themes. For a single team, report themes only when at least five people answered, and treat small counts as signals to explore, not conclusions.

Should I use positive, negative and neutral sentiment?

Use positive, mixed and negative. Truly neutral comments are rare in engagement surveys, and "mixed" captures the common case of praise with a complaint attached.

Can AI tag survey comments accurately?

It does a good first pass against a clear codebook, but it misreads sarcasm and mixed comments. Have a person check a random sample and every comment flagged as a safety, harassment or legal concern.

Do I need a text analytics add-on from my survey vendor?

Not at the size of most 20-500-person companies. A clear codebook, a spreadsheet or Insight Studio, and an afternoon of careful tagging give you themes and sentiment you can defend.

How do I keep comments confidential?

Strip names and identifying details, report only for groups of five or more, edit quotes so the writer cannot be recognized, and never paste raw comments into an AI tool that trains on your data.

Sources

← All hr and people analytics trends