【沃頓商學院】商業分析 全套課程(客戶、運營、人力資源、會計) P6 - 6.[P006]04_descriptive-data-collection-s

財富導航
16粉絲
2026年4月25日
Now what I want to do to kind of conclude this part of survey design
is talk about some steps in survey design and some do's and don'ts and best practices.
So let me lay it out there what are the typical steps that are out there and we'll go through
each of these steps in more detail. I think step zero which is perhaps the most important one
is to make sure your results are generalizable to an appropriate population.
What does that mean? What that means is typically when you're doing a survey
maybe about a thousand people, two thousand people, three thousand people,
whatever the number might be, you're obviously interested in what they have to say.
But what you're more interested in is how does the thing that you're capturing there,
for example if it's customer satisfaction, how does it generalize to your entire customer base?
So that's what I mean by step zero and we'll talk a little bit about how you do that.
Step one, develop a detailed listing of what you want to capture.
Capture all of these things, write the draft questions, design the flow and the layout
appropriately, evaluate, test, retest, pilot test, redraft. This is a circular process in the sense
that you first set it up, you do some pilot testing, go back to the drawing board, clean up
some questions that come about, test it again and so on till you're comfortable going ahead and
implementing that survey. Step number six perhaps is very important from implementation point of
view, which is if you're working within your company or if you're a consultant working for
another company, get approval from all the parties. Extremely important of course if you have to
implement the survey with customers. And finally you go ahead, do one more test in terms of figuring
out whether the survey is good, look at the final copy and implement the survey. So let me go into
more detail about each of these steps. As I said, step zero perhaps the most important one. It
ensures that the data that you're collecting from the survey will be useful. A couple of things you
have to think about. First, the population has been defined correctly. The sample is representative
of the population. So for instance, suppose you're a company that targets let's say 25 to 30 year old
males. What you want to make sure is you've defined that population correctly, that is indeed your
target market. Then you say if that's your target market, you look at the sample that you've
collected for the survey that is representative of the target market. Why is that? Because then
you can be more confident that the data that you collect from the survey is indeed representative
of what might happen in your target market. Then you have to of course ensure that respondents
selected to be interviewed are available, they understand the question, have the appropriate
knowledge to answer the questions and you're of course motivating them with some incentives
to provide information to the service. And here the big issue of course are response rates.
You might not be surprised or perhaps you are, the typical response rates in surveys is about five
percent. So what that means is of the hundred people you send out a survey to, only five of
them will respond. So clearly it's very low. So to think about appropriate ways in which you can
incentivize respondents to in fact respond to your survey. A couple of ways to think about this.
One is how do you handle this non-response? First you want to collect data as much as possible
on people who don't respond. Why? Because you want to make sure that the people who respond
and people who don't are not systematically different from each other. If they are, that
might not be a very good way of generalizing the results from your sample to the population.
Let me give you an example. If you're interested in capturing people's responses to the prices
that you're charging for your product. And let's say there is systematic difference between people
who respond and people who don't respond in terms of their income. That's clearly not very good
because then what you're capturing is not a representation of the true target market.
And why is income an important metric? Because what you're interested in is how do they respond
to the prices that you're paying. So clearly income would be a very important indicator of
how price sensitive people are. So if there is a difference in income levels of people who
respond and people who don't, that clearly suggests that there is a bias in your sample
in terms of its representativeness to the population. What can one do? Well, one can
follow up with the non-respondents. Try to convert them into responding. Typically what happens is
you send them reminders. On the pro side, you would get people who respond. What's the con?
The con could be if they're just giving you bad data after repeated requests.
So what is typically done? You typically compare each wave of respondents to make sure they're
similar. So for instance, you might have the first wave. These are the initial set of responders.
Let's say about 50 percent of people respond. You collect that wave one. Then after a week or so,
or two weeks, you send people reminders. These are people who have not responded. Let's say 20 to 30
percent of them respond. Rather than just aggregating or collating that data, you would
want to make sure that the 50 percent who responded in week one are not systematically different from
the 30 percent who responded in week two. If that is the case, you can convert all the data together.
If that's not the case, I would advise you to look at the data separately. Now let's go back
to other steps. Step one, what bits of information to collect? I think here what you have to make
sure is, you translate whatever research objectives you have into information requirements.
One way to think about it is, is thinking in terms of question relevancy.
What would you do with the answer if you knew it? So can I work backwards? So think about
the ideal situation in which you knew the answer. Then think about how your managerial decisions
would be different. Once you put that perspective in mind, it becomes easy to understand how should
you ask those questions. Then the question is, how do you format those questions? There are two
big ways of thinking about it. One are what are called as open response questions. Why do you shop
at Gennady's for example? Another is closed response questions. How often do you shop at
Gennady's? Very often to not at all. So what you notice here, open response questions basically
ask respondents to come up with their answers. They're not really given any feedback per se.
On the other hand, in closed response questions, you typically have a particular question in mind
and you have a scale for example from very often to not at all in this question. Let's talk a
little bit about what are the advantages and disadvantages of open-ended and closed-ended
questions. In terms of advantages, there are quite a few of open-ended questions. First,
respondents can give a general reaction to questions such as, why do you see a brand x is
better? Responses are given in their own words, in their own language. It becomes easier for people
to respond. It can also help interpret closed-ended data. For example, if people are thinking about
a particular color, you can follow up to kind of say, why was that particular color important to
you? And it might also suggest what other questions to ask. Lots of advantages. At the same time,
there's some drawbacks as well. First of all, it's not very good for self-administered surveys.
Why? Because sometimes survey fatigue might come in. So if you think about respondents, they're
going through a lot of questions, you keep asking them open-ended questions. After a little while,
they might have some fatigue and they might not be as articulate. And of course, all the answers that
you get depends upon how articulate a person is. If they're able to understand what they're
thinking about, they're very articulate, open-ended questions are great. Why? Because they're responding
in their own language, in their own verbiage. If they're not as articulate, you don't get a
lot of information. And finally, in terms of analyzing the data, post-coding can be quite tedious
because it's open-ended, it's in their own verbiage. It might be very difficult to understand
all the different things that one has to quote. So now let's flip back and kind of talk about
closed-ended questions. Closed-ended questions, like I said before, if you think back to the
example, how often do you shop at Gennady's? The respondents provided with the pre-determined
descriptions and select one or more of them. What are the advantages here? Quite clear, it's easy to
use, it's less threatening for the respondents, it's very simple to code and much cheaper to administer.
On the flip side, it requires a lot of pre-testing. You want to make sure that the questions you're
asking, the categories that you're giving people to tick or select are clear, they understand what
it means and in fact it presumes that the list of responses is complete. A lot more effort but I
think in terms of coding, it becomes much easier. So again, when you think about what type of
questions you want to ask, I think you have to keep the end goal in mind. If the end goal is
thinking more about what consumers are thinking about, open-ended might be great. On the other
hand, if the end goal is you have a very good set of questions you want to ask them, you have a good
idea of what responses you're looking at or what you want to see, closed-ended might be better.
In terms of best practice, you often do exploratory research using open-ended questions
and use this for codes in the quantitative survey with closed-ended. So this gives you a sense of
how one would design a survey. Typically, you start with open-ended, get a sense of all the
different issues that come up, you get a sense of what customers are thinking about and then you
design the closed-ended questions. Now let's move on to step three, drafting the questions. I think
it's very very important here to use simple conventional language. I'll give you an example
in the next slide. Avoid ambiguity, be specific as possible and don't have long questions and start
broad and then narrow down. Keep in mind that when respondents are answering surveys, they typically
don't have a lot of time. So if you make it very long, you ask long questions, it's very ambiguous,
you might see a lot of dropouts. People who start a survey and then drop out because it's becoming
very very tedious to answer those questions. Here's an example. Consider your target sample
and use familiar language. Did you notice any malfunctions at the time of purchase versus
did you notice anything wrong with it when you bought it? What you see immediately is that the
latter question is a little bit more easy to understand for the general audience. The first
question, did you notice any malfunctions, uses specific language, specific verbiage that might be
difficult for the common respondent to answer. So again, keep your respondent in mind, understand
what they're comfortable with and design the questions in that sense. And finally, think about
sequencing. Think about the layout guidelines. Here are some best practices. Open the survey with an
easy non-threatening question. You want people to ease into the survey. So for instance, if you want
information about their income or where they live, those might not be very good questions to start
out with. These are questions that in fact ask quite a lot of personal detail. If you do want to
ask those questions, it's probably better to ask them later on when they've kind of eased into the
survey. It should have a smooth and logical flow from very general to kind of specific and there
can be a lot of order biases. I'll give you an example in the next slide what I mean by it.
A typical common bias is survey fatigue. Now what's the way around this kind of bias? Randomize
ordering of your questions as much as possible. Let me give you an example of an order bias.
Here are four examples. What I'm trying to look at here is what is the percentage of respondents
who said they're very much interested in buying a new product. That's what you see on the right
column. In the first row, when no questions were asked before asking that question, 2.8 percent
people were interested in buying this product. When you ask people only about the advantages of
the product, what you see is that that number jumps up to 16.7 percent. Of course, if you only
ask people about disadvantages of the product, nobody wants to buy it. And finally, when you
ask people about both advantages and disadvantages, it's a number between 2 percent and 16 percent is
about 5 percent. What do we see here? How we order the questions, where is it being asked in the
overall survey is extremely important. So typically a rule of thumb is if you think about a question
that's important to you, make sure to randomize its order. Sometimes some people will get it in
the beginning, some people might get it towards the end, some people might get it in the middle.
What you want to see and make sure is that there is no order bias. No matter where they get it,
you're getting a similar answer. So let me conclude with step five and seven, pre-testing,
correcting problems. You can keep in mind step six, if you remember, was about just making
sure that all the people who are interested in doing the survey, you have a buyout with them.
So it's got little to do with survey design per se, but making sure that everybody who's
interested in doing the survey is on board. So having said that, let's go back to step five and
seven. Here the idea is to make sure that before you implement the survey with your final target
sample, just clean out the survey. Make sure all the questions that you do have are meaningful.
People who are interested in answering those surveys, they understand what the questions are.
You're designing the questions with the respondent in mind. What that means is,
you make sure that you're using verbiage or language with the respondent would be familiar
with. You're asking questions which go from more general to more specific. Always lead off with
questions which are non-threatening, so that consumers or respondents who are answering the
survey can easily get into the survey. Once all of that is done, hopefully what you will have as
an end result is a survey that is easy to understand, easy to implement, respondents
are comfortable answering it, and you'll have good data from the survey. So let me conclude
with what we've done in this whole session about survey design. One is, you think about and
carefully look at a survey as a mini market research. What that means is, you think carefully
about every question. You think about how to design those questions. What are the best questions to
ask? Should it be open-ended or closed-ended? Well, it depends upon what your objective is.
What kind of language should be used? Again, it depends upon what the target market is.
So think carefully about when you're designing a survey, each question is very important,
and how the questions are ordered together, even more important. Typically what you want to do is,
if you're asking some specific kinds of questions, for instance about customer satisfaction,
there's a lot of research that has already happened in terms of what might be the best
ways to answer that question and ask that question. So what I mean by that is, search for and use
proven questions. So that way you don't have to do the heavy lifting, somebody else has already
done the heavy lifting for you. And finally, before you go ahead and implement the questionnaire,
always pre-test it. Maybe start out with a sample of 20 to 30 people, to make sure that they
understand each of the questions, to make sure that the answer that you're getting makes sense,
before you go ahead and implement that survey with a much larger sample.
So hopefully with this particular part about survey design, you have a good understanding
of the do's and don'ts of survey design, what are the things that you should keep track of.
Now you also have a good understanding of a particular type of question, the net promoter
score, that is extremely popular in the marketplace. What I mean by good understanding is,
you know the pros, but you also know the cons. So what I want you to take away from all of this is,
that when you start designing a questionnaire or a survey, always keep every question in mind,
think about the end goal and work backwards, whether that particular question you're asking
is going to help you answer the end goal.
