There has been a critical trend happening in market research data collection that is getting little attention. If you are gathering data in online surveys and polls, chances are that most of your respondents are now answering your questionnaires on mobile devices.

This trend snuck up on us. Just three years ago we were advising clients that we were noticing that about 25% of respondents were answering on mobile devices. Of the last 10 projects we have completed, that percentage is now between 75% and 80%. (Our firm conducts a lot of research with younger respondents which likely skews this higher for us than other firms, but it remains the case mobile response has become the norm.)

Survey response tools have evolved considerably. Respondents initially used either the mail or provided responses to an interviewer on the other end of a clipboard. Then, people primarily answered surveys from a tethered land-line phone. The internet revolution made it possible to move data collection to a (stationary) computer. Now, respondents are choosing to answer on a device that is always with them and when and where they choose.

There are always “mode” effects in surveys – whereby the mode itself can influence results. However, the mode effects involved in mobile data collection has not been well-studied. We will sometimes compare mobile versus non-mobile respondents on a specific project, but in our data this is not a fair comparison because there is a self-selection that occurs. Our respondents can choose to respond either on a mobile device or on a desktop/laptop. If we see differences across modes it could simply be due to the nature of the choice respondents make and have little to do with the mode itself.

To study this properly, an experimental design would be needed – where respondents are randomly assigned to a mobile or desktop mode. After searching and asking around to the major panel companies, I wasn’t able to find any such studies that have been conducted.

That is a bit crazy – our respondents are providing data in a new and interesting fashion, and our industry has done little to study how that might influence the usefulness of the information we collect.

Here is what we do know. First, questionnaires do not look the same on mobile devices as they do on laptops. Most types of questions look similar, but grid-style questions look completely different.  Typically, on a mobile device respondents will see one item at a time and on a desktop they will see the entire list. This will create a greater response-set type bias on the desktop version. I’d say that this implies that a mode effect likely does occur and that it doesn’t vary in the same way across all types of questions you are asking.

Second, the limited real estate of a mobile device makes wordy questions and responses look terrible. Depending on the survey system you are using, a lengthy question can require both horizontal and vertical scrolling, almost guaranteeing that respondents won’t attend to it.

Our own anecdotal information suggests that mobile respondents will complete a questionnaire faster, are more likely to suspend the survey part-way, and provide less rich open-ended responses.

So, how can we guard against these mode effects? Well, in the absence of research-on-research that outlines their nature, we have a few suggestions:

• First and foremost, we need to develop a “mobile-first” mentality when designing questionnaires. Design your questionnaire for mobile and adapt it as necessary for the desktop. This is likely opposite to what you are currently doing.
• Mobile-first means minimizing wording and avoiding large grid-type questions. If you must use grids, use fewer scale points and keep the number of items to a minimum.
• Visuals are tough … remember that you have a 5 or 6 inch display to work with when showing images. You are limited here.
• Don’t expect much from open-ended questions. Open-ends on mobile have to be precisely worded and not vague. We often find that clients expect too much from open-ended responses.
• Test the questionnaire on mobile. Most researchers who are designing and testing questionnaires are looking at a desktop/laptop screen all day long, and our natural tendency is to only test on a desktop. Start your testing on mobile and then move to the desktop.
• Shorten your questionnaires. It seems likely that respondents will have more patience for lengthy surveys when they are taking them on stationary devices as opposed to devices that are with them at all (sometimes distracting) times.
• Finally, educate respondents not to answer these surveys when they themselves are “mobile.” With the millions of invitations and questionnaires our industry is fulfilling, we need to be sure we aren’t distracting respondents while they are driving.

In the long run, as even more respondents choose mobile this won’t be a big issue. But, if you have a tracking study in place you should wonder if the movement to mobile is affecting your data in ways you aren’t anticipating.