I have written somewhere between a zillion and a gazillion survey questions in my career. I am approaching 3,000 projects managed or overseen and I have been the primary questionnaire author on at least 1,000 of them.  Doing the math, if an average questionnaire is 35 questions long, it means I have written or overseen 35,000+ survey questions. That is 25 questions a week for 26 years!

More importantly, I’ve had to analyze the results of these questions, which is where one really starts to understand if they worked or not.

I started in the (land line) telephone research days. Back then, it was common practice for questionnaire authors to step into the phone center to conduct interviews during the pre-test or first interview day.  While I disliked doing this, the experience served as the single best education on how to write a survey question I could have had.  I quickly understood if a question was working, was understood by the respondent, etc. It was a trial by fire and in addition to discovering that I don’t have what it takes to be a telephone interviewer I quickly learned what was and wasn’t working with the questions I was writing.

Something in this learning process is lost in today’s online research world. We never really experience first-hand the struggles a respondent has with our questions and thus don’t get to apply this to the next study.  For this reason I am thankful I started in the halcyon days of telephone research. Today’s young researchers don’t have the opportunity to develop these skills in the same way.

There are many guides to writing survey questions out there that cover the basics. Here I thought I’d take a broader view and list some of the top things to keep in mind when writing survey questions.  These are things I wish I had discovered far earlier!

1. Begin with the end in mind. This concept is straight out of the 7 Habits of Highly Effective People and is central to questionnaire design.  Good questionnaire writers are thinking ahead to how they will analyze the resulting data.  In fact, I have found that if this is done well, writing the research report becomes straightforward.  I have also discovered that when training junior research staff it is always better to help them develop their report writing skills first and then move to questionnaire development.  Once you are an apt report writer questionnaire writing flows naturally because it begins with the end in mind.  It is also a reason why most good analysts/writers run from situations where they have to write a report from a questionnaire someone else has written.
2. Start with an objective list. We start with a clear objective list the client has signed off on. Every question should be tied to the objective list or it doesn’t make it in the questionnaire. This is an excellent way to manage clients who might have multiple people providing input. It helps them prioritize. Most projects that end up not fully satisfying clients are ones where the objectives weren’t clear or agreed upon at the onset.
3. Keep it simple – ridiculously simple. One of the most fortuitous things that happened to me in my career is that for a few years I exclusively wrote questionnaires that were intended for young respondents.  When I went back to writing “adult” survey questions I didn’t change a thing as I realized that what works for a 3^rd grader is short, clear, unambiguous questions with one possible outcome.  The same thing is true for adults.
4. Begin with a questionnaire outline. Outlines are easier to work through with clients than questionnaires. The outlines keep the focus on the types of questions we are asking and keep us from dwelling on the precise wording or scales. Writing the outline is actually more difficult than writing the questionnaire.
5. Use consistent scales. Try not to use more than 2-3 scale types on the same questionnaire as it is confusing to the respondents.
6. Don’t write long questions. There is evidence that respondents don’t read them. You are better off being more wordy in the answer choices than in the question itself, as online many respondents just look at the answer choices and don’t  even read the question you spent hours tweaking the wording on.
7. Don’t get cute. We have a software system that allows us to do all sorts of sexy things, like drag-and-drop, slider scales, etc.  We rarely use them, as there is evidence that the bells and whistles are distracting and good old fashion pick lists and radio buttons provide more reliable measures.
8. Consider mobile. From major research panels, the percentage of respondents answering on mobile devices is just 15% or so currently, but that is rapidly changing. Not only does your questionnaire have to work on the limited screen real estate of a mobile device, but it also is increasingly less likely to be answered by someone tethered to a desktop and laptop screen in a situation where you have their attention.  Your questionnaires are soon going to be answered by people multitasking, walking the dog, hanging with friends, etc.  This context needs to be appreciated.
9. Ask the question you are getting paid to ask. Too many times I see questionnaires that dance around the main issue of the study without ever directly asking the respondent the central question. While it is nice to back into some issues with good data analysis skills, there is no substitute to simply asking direct questions. We also see questionnaires that allow too many “not sure/no opinion” type options. You are getting paid to find out what the target audience’s opinion is, so if this seems like a frequent response you have probably not phrased the question well.
10. Think like a respondent and not a client. This is perhaps the most important advice I can give. The respondent doesn’t live and breathe the product or service you are researching like you client does. Survey writers must appreciate this context and ask questions that can be answered. There is a saying that if you “ask a question you will get an answer” – but that is no indication that the respondent understood your question or viewed it in the same context as you client.

Anecdotally, I have found that staff with the strongest data analytics skills and training can be some of the poorest questionnaire writers. I think that is because they can deploy their statistical skills on the back end to make up for their questionnaire writing deficiencies. But, across 3,000 projects I would say less than 100 of them truly required statistical skills beyond what you might learn in the second stats course you take in college. It really isn’t about statistical skills; it is more about translating study objectives into language a target audience can embrace.

Good questionnaire writing is not rocket science (but it is brain surgery). Above all, seek to simplify and not to complicate.