Most People Have No Idea Their Surveys Are Bad
And no one’s pushing them to do better.
Let me paint you a picture.
I’m meeting with a client—not to talk about survey design, but big-picture evaluation strategy. We’re looking at their past data, their goals, and what tools they’ve been using. At some point, a survey gets pulled up on the screen. It’s familiar. It’s been used for years. They’re proud of it. Or, at least, not concerned.
Me, on the other hand? I’m panicked. Within 15 seconds, I know that survey is a major problem for our work.
We’re going to have to craft analyses for data that—at best—came from a tool that wasn’t designed to give them what they need. And at worst? Is total garbage.
This is the hidden cost of bad survey design: distorted narratives. Shaky conclusions that sound definitive. And hours spent on the back end trying to make flawed data look like real insight.
But they don’t know that. To them, the questions feel solid. Validated. Vetted.
I take no pleasure in bursting their bubble. But because I love and respect my clients—and because I don’t want them to build narratives, strategies, or stories on bad information—I have to say something.
It’s hard to fix a problem you don’t know you have.
When I first got my dog Ken, I walked him with a harness and a leash—just like most people do. Just like how I walked his brother, Calix.
It felt normal. No one told me I was doing it wrong. And honestly, I didn’t feel like I was doing it wrong.
But then we started working with a trainer who suggested using two leashes—front and back steering—to help with control and comfort.
It was a game-changer.
And I remember thinking: I never would’ve known to do this unless someone showed me.
That’s exactly how survey design works.
Most people are doing what feels normal. What looks standard. What they’ve seen other people do. But just because something seems fine doesn’t mean it’s actually working for you.
Sometimes, you need someone to step in and say: There’s a better way to walk this dog.
So how does bad design happen?
It’s almost never intentional. In fact, most of the time, people genuinely believe they’re doing a great job—or at least a good enough one. After all, they’re using the data.
Here are four common types of survey designers I’ve encountered over the years:
The Loyalist
“We’ve used this survey for years. We have benchmark data.”
They trust the legacy. They don’t want to lose the trend line. And they assume that if it worked before, it must still be working now. But familiarity doesn’t equal quality. And even data that is benchmarked can still be bad.
The Stat Worshiper
“This is a validated scale. It has good reliability.”
They trust the numbers. They lean on validation. But validation isn’t a force field—it just means the questions perform consistently. Even poorly written, confusing questions can produce clean-looking stats. Spurious correlations are real. Reliability is not the same thing as usefulness.
The Overwhelmed
“It’s not perfect, but we had to launch.”
They’re juggling competing priorities. They know it’s not ideal—but they need something fast. Data collection is just one task in a never-ending list. And when no one’s holding them accountable for survey quality, the rushed version becomes the final version.
The Optimist
“It’s just a survey. How hard can it be?”
They think good intentions are enough. After all, people ask questions all day—how different could a survey be? But surveys have their own logic. Their own structure. Their own traps. And skipping that complexity means missing what matters most.
None of these people are villains. None of these people are stupid.
They’re thoughtful, capable, mission-driven professionals doing important work.
But…most of them were never trained in what good survey design actually looks like. So, they default to what’s worked in the past, what looks good, what they have time for, and what feels right.
And that’s where the trouble starts.
Survey design mistakes aren’t just common. They’re invisible.
Not because people are careless.
But because no one ever taught them how to spot what’s broken.
And until that changes, bad questions will keep looking fine—and shaping decisions they shouldn’t.
Surveys are built into so many jobs: HR, education, evaluation, grant making, community engagement, event planning. People are expected to write them, distribute them, interpret them—often without ever being taught how.
It reminds me of something I used to hear in education policy: Everyone thinks they can teach because they went to school.
Everyone thinks they can write a survey because they know how to ask a question.
Neither of those things are true in practice. Teaching is a craft. So is survey design.
And without training or guidance, most people just do what they know:
They reuse old questions
They borrow from validated tools
They trust their instincts
And here’s the kicker:
Once those responses get dropped into a paragraph, chart, or polished report—no one questions it. Not the client. Not the board. Not the funder. And definitely not the person who wrote the survey.
The Real Problem? No One’s Asking Us to Do Better.
This isn’t just an individual oversight—it’s a structural problem.
If no one teaches you how to spot bad survey design, and no one around you questions the data, there’s no reason to change. You can keep using the same flawed questions, generating misleading results, and making decisions based on shaky ground—without ever getting called out.
Because the truth is: most people don’t know there’s a problem.
And if you don’t know something’s broken—and no one around you is pointing it out—there’s no pressure to fix it.
We don’t ask who wrote the questions.
We don’t ask what was left out—or who.
We don’t examine how the data was collected, or what assumptions were baked in from the start.
We definitely don’t look at the individual questions and consider how they were worded or what answer choices were (or were not) included.
So the cycle repeats.
Bad surveys stay in use.
Flawed data shapes strategy.
And no one feels the urgency to stop it.
Worse? It’s not just that no one notices. It’s that bad data can still look impressive.
Wrap it in a chart. Add a well-written paragraph. Drop it in a slide deck. Suddenly, it feels solid. Professional. Legitimate.
But appearances can be deceiving.
That’s the problem:
We’ve been trained to treat data as objective.
We see percentages and bar graphs and assume rigor.
We mistake polish for proof.
Even flawed questions can generate clean stats.
Even misaligned surveys can produce “insights.”
And when no one’s asking where the data came from, or how the questions were built, it’s dangerously easy to take those results at face value.
What does this look like in practice?
A nonprofit asks staff, “How satisfied are you with your workload and support?”
It sounds reasonable. It’s a good thing to ask.
But it’s also two questions crammed into one—and you have no way of knowing if satisfaction (or disatisfaction) is with the workload, the support, or both. You also don’t know if people are disatisfied because they have too much work (or support) or not enough. Still, the bar chart gets made. The report goes out. And leadership makes decisions based on results that don’t actually mean what they think they mean.
A district survey offers “neutral” as a response option on every scale. But half the respondents are just clicking it to opt out of the question. No one digs into the patterns. The neutrality gets interpreted as indifference. And now, a program that actually matters is on the chopping block.
A funder asks: “How much funding do you need to make up for government budget cuts?” The options are: $1–5 million, $5–10 million, and More than $10 million. The answer choices overlap so it’s unclear where to categorize $5 million responses. Still, the funder moves forward. They’ve budgeted $100 million and decide to give $5 million each to 20 organizations—because most respondents selected the $1–5 million range. But in reality, most organizations only needed $1 million. That decision cost them the chance to fund 80 more initiatives. Still, the headline sounds great: “Funder Fills Budget Gap for 20 Nonprofits.” It looks like a win. No one asks about the categories. And everyone moves on.
These stories are everywhere.
Because when you don’t know what to look for, survey mistakes feel invisible.
And when no one’s questioning the data, they get treated like truth.
When I review those surveys, I see the same things over and over:
Double-barreled questions that confuse the results.
Neutral response options that don’t actually mean anything.
Answer choices that overlap or leave people stuck.
Scales that technically “work” but don’t reflect how people actually think.
To someone outside the field, those issues don’t jump out.
To me, they’re flashing neon signs.
That’s the thing about survey design:
It feels like common sense—until you learn how to really see it.
People see clean charts and assume the question worked.
They compare responses to last year’s and feel reassured by the consistency.
They plug flawed data into decision-making processes—not knowing that the foundation was already cracked.
Most survey mistakes don’t feel like mistakes.
They don’t set off alarms.
They don’t get caught in the report.
They just keep being used—year after year—because they’ve never been examined closely.
So What Can You Do?
You don’t need a PhD.
You don’t need to become a data scientist.
But you do need awareness, and a system.
You need a checklist. A process. A reminder to pause and ask:
Am I asking one thing, or two?
Will these answers actually tell me something useful?
Could someone misinterpret what I’m asking?
Would I feel confident making a decision from this?
Because the real danger in survey design isn’t doing it badly.
It’s not realizing you’re doing it badly.
And when flawed data looks clean—when misleading questions produce polished charts—it becomes easier to keep moving forward.
Easier to make confident decisions on broken foundations.
Easier to mistake the noise for insight.
That’s what makes bad surveys dangerous:
They don’t just fail quietly.
They succeed loudly.
And if no one is questioning them?
They’ll keep shaping strategy, policy, funding, outcomes.
Not because they’re good.
But because they look good.