Survey research sounds really easy. Type up the questions, maybe put it on Google Forms, get a whole bunch of people to answer your questions (pester them if you have to) then bingo, you have a bunch of data. Use this data to whip up a few charts saying things like “22% of people said they had a high degree of confidence”. Maybe you can even get fancy and do a pie chart or a bar graph or a cross tab (or go extra crazy and run a regression or two)
Here’s the problem: There are a lot of bad survey questions.
It’s a major problem. I’m the strange person who loves to take surveys, who gleefully snapped a picture of my long form census taker when he asked if he could come in the house to do the face-to-face interview. I take phone surveys, surveys about politics. I don’t even mind it when the survey takes a little longer than it is supposed to. I’m pro-survey research.
But most of the time, survey questions are written SO BADLY. They’re badly worded questions that go on for way too long, they’re questions asking an opinion about something I (the questioner) care nothing about, they are questions that assume knowledge I do not have, they are questions that don’t take into account the categories I consider true or my opinions are too complex to fit into “acceptable, somewhat acceptable, Neutral, somewhat unacceptable or unacceptable”. I could rail against all the reasons the questions are so bad, but there is a solution that I would like to offer. It’s simple, but not easy.
When you’re creating a survey, think about what kind of chart, analysis or outcome you want. What do you want to be able to say? What chart would come from asking these questions? Does this question measure the outcome I want to create? Create questions to construct those analyses.