Back to the Future: What Paper Surveys Can Teach Us About Online Survey Design

Today, for the first time in my professional career, I had to design a paper survey. This was an interesting experience, because I’m used to using software like Survey Monkey or Zoomerang where I just choose the type of question and then create the text components. Sure I probably could have created the survey in one of these programs and then printed it out, but I decided to do it the old fashioned way.

What I took away from this experience was that this was a great exercise in survey design. Ultimately, while online surveys allow us many freedoms and are in theory easier to compose, we should stick to the principles of paper surveys, even when doing them online.

Here are the challenges/principles that I faced:

1. Brevity – There’s only a limited number of questions you can fit on an 8x11 sheet of paper, even back and front. This meant that I had to really hone in on what it was I was trying to get at, and I could not boil the ocean. And even online, who wants to spend 20 minutes filling out a survey?

2. Usability – How can I make this survey an easy and positive experience for respondents? What font size and spacing are most appealing? How should I lay out that likert scale?

3. Design – How do I make sure that the survey is visually designed in a way that represents my client’s brand and looks professional enough to deliver to their clients.

4. Prototype – How does all of this come together in the end? Once I finished mocking up my survey in word, I wanted to see how it would translate to paper. I found myself printing the survey with multiple font sizes, and I caught errors in grammar and spelling. It is always a best practice to test your surveys online or off on yourself and probably a few others.

Can you think of more? I’d love to hear them!

Part 2 - Context: What primary, qualitative research provides, that no other means can

This is where lab settings and quantitative methods fail. Sure there are focus groups, and other qualitative and quantitative methods that can be interesting in a lab setting, but they are only telling part of the story, and the accuracy of that story can be questionable (I’m not going to debate focus groups in this post). What is missing is context, which illustrates what people do, not just what they say they do as well as all of the things that go on around what they are doing. This knowledge can lead to insights and innovations that could only be gained by in-context observations.

In-context research reveals many more options than the ones that you bring to a focus group, or that your participants can think up or rationalize on the spot. It shows us things that maybe aren’t recognized as important, but really are.

Who cares what your product is supposed to do. If you don’t understand how it actually fits into people’s real lives then you will never be able to truly design a good product or experience and then market it properly.

By understanding the context that surrounds products and services, we are able to suggest innovations, improvements and communications that truly resonate with people. The upfront costs of this are definitely more expensive, but the ROI on this investment will definitely have positive ROI if done right.

Part 1 - Inherent Understanding: What primary, qualitative research provides, that no other means can

When we talk about ethnography, we often talk about what it is going to do for design or marketing communication. What is often unrecognized, (always reality once you’ve done it) is that it is a powerful means of discovery and business immersion for us, the agency, on behalf of our clients - of course in addition to the plethora of design and communication insights that will be garnered.

Yes, as anthropologists, consultants and marketers, it is our job to immerse ourselves in our client’s business. However, there is only so much that can be gained from stakeholder interviews and existing research immersion. I’d suggest that this can get us 80% there. But, it’s that other 20% that can really make business-changing differences.

By actually meeting with current and future customers/users, we get an understanding that gets us closer to an etic (insider) understanding, which can only lend to us being more successful on behalf of our clients. The process of designing, conducting, analyzing, and synthesizing custom research provides a level of insights and empathy that cannot be gained through other means.

Furthermore, there is a clear difference in the caliber of insights, innovation, and design that is produced from people who truly understand the business, the marketplace AND consumers, when compared to those who don’t.

While a 12-week, global ethnographic study need not be necessary, some form of primary, immersive customer research is always worth fighting the good fight for.