DESIGNING AND USING QUESTIONNAIRES
by David S. Walonick
Excerpts from: Survival Statistics
Copyright: David S. Walonick, 1997 - 2000 - All Rights Reserved: ISBN
0-918733-11-1
Published by: StatPac, Inc., 4425 Thomas Ave. S., Minneapolis, MN 55410, Tel:
(612) 925-0159 Fax: (612) 925-0851
Email: statbook-sales@statpac.com
This is the information age. More information has been published in the last decade than in all previous history. Everyone uses information to make decisions about the future. If our information is accurate, we have a high probability of making a good decision. If our information is inaccurate, our ability to make a correct decision is diminished. Better information usually leads to better decisions.
Ways to Get Information
There are six common ways to get information. These are: literature searches,
talking with people, focus groups, personal interviews, telephone surveys, and
mail surveys.
A literature search involves reviewing all readily available materials.
These materials can include internal company information, relevant trade publications,
newspapers, magazines, annual reports, company literature, on-line data bases,
and any other published materials. It is a very inexpensive method of gathering
information, although it generally does not yield timely information. Literature
searches take between one and eight weeks.
Talking with people is a good way to get information during the initial
stages of a research project. It can be used to gather information that is not
publicly available, or that is too new to be found in the literature. Examples
might include meetings with prospects, customers, suppliers, and other types
of business conversations at trade shows, seminars, and association meetings.
Although often valuable, the information has questionable validity because it
is highly subjective and might not be representative of the population.
A focus group is used as a preliminary research technique to explore
peoples ideas and attitudes. It is often used to test new approaches (such as
products or advertising), and to discover customer concerns. A group of 6 to
20 people meet in a conference-room-like setting with a trained moderator. The
room usually contains a one-way mirror for viewing, including audio and video
capabilities. The moderator leads the group's discussion and keeps the focus
on the areas you want to explore. Focus groups can be conducted within a couple
of weeks and cost between two and three thousand dollars. Their disadvantage
is that the sample is small and may not be representative of the population
in general.
Personal interviews are a way to get in-depth and comprehensive information.
They involve one person interviewing another person for personal or detailed
information. Personal interviews are very expensive because of the one-to-one
nature of the interview ($50+ per interview). Typically, an interviewer will
ask questions from a written questionnaire and record the answers verbatim.
Sometimes, the questionnaire is simply a list of topics that the research wants
to discuss with an industry expert. Personal interviews (because of their expense)
are generally used only when subjects are not likely to respond to other survey
methods.
Telephone surveys are the fastest method of gathering information from
a relatively large sample (100-400 respondents). The interviewer follows a prepared
script that is essentially the same as a written questionnaire. However, unlike
a mail survey, the telephone survey allows the opportunity for some opinion
probing. Telephone surveys generally last less than ten minutes. Typical costs
are between four and six thousand dollars, and they can be completed in two
to four weeks.
Mail surveys are a cost effective method of gathering information. They
are ideal for large sample sizes, or when the sample comes from a wide geographic
area. They cost a little less than telephone interviews, however, they take
over twice as long to complete (eight to twelve weeks). Because there is no
interviewer, there is no possibility of interviewer bias. The main disadvantage
is the inability to probe respondents for more detailed information.
Questionnaire Research Flow Chart
Questionnaire research design proceeds in an orderly and specific manner.
Each item in the flow chart depends upon the successful completion of all the
previous items. Therefore, it is important not to skip a single step. Notice
that there are two feedback loops in the flow chart to allow revisions to the
methodology and instruments.
Design Methodology
hi
Determine Feasibility
i
Develop Instruments
i
Select Sample
i
Conduct Pilot Test
hi
Revise Instruments
i
Conduct Research
i
Analyze Data
i
Prepare Report
Time Considerations
Many researchers underestimate the time required to complete a research
project. The following form may be used as an initial checklist in developing
time estimates. The best advice is to be generous with your time estimates.
Things almost always take longer than we think they should.
This checklist contains two time estimates for each task.
The first one (Hours) is your best estimate of the actual number of hours required
to complete the task.
The second one (Duration) is the amount of time that will pass until the task
is completed.
Sometimes these are the same and sometimes they are different. Most researchers
and business-people have to divide their time among many projects. They simply
cannot give all their time to any one project. For example, my estimate of goal
clarification may be four hours, but other commitments allow me to spend only
two hours a day on this study. My "hours" estimate is four hours,
and my "duration" estimate is two days.
To arrive at your final time estimates, add the individual estimates.
The hours estimate is used for budget planning and the duration estimate is
used to develop a project time line.
Hours Duration
1. Goal clarification ________ ________
2. Overall study design ________ ________
3. Selecting the sample ________ ________
4. Designing the questionnaire and cover letter ________ ________
5. Conduct pilot test ________ ________
6. Revise questionnaire (if necessary) ________ ________
7. Printing time ________ ________
8. Locating the sample (if necessary) ________ ________
9. Time in the mail & response time ________ ________
10. Attempts to get non-respondents ________ ________
11. Editing the data and coding open-ended questions ________ ________
12. Data entry and verification ________ ________
13. Analyzing the data ________ ________
14. Preparing the report ________ ________
15. Printing & distribution of the report ________ ________
Cost Considerations
Both beginning and experienced researchers often underestimate the cost
of doing questionnaire research.
Some of the most common costs are:
Proposal typing and editing. ________
Cover letter and questionnaire typing. ________
Addressing mailing envelopes. ________
Following up on non-respondents. ________
Mailing list cost (if necessary). ________
Artwork and keylining. ________
Cover letter and survey printing costs. ________
Envelope costs (both ways + more). ________
Postage costs (both ways + more). ________
Incentives. ________
Data entry and verification. ________
Statistical analysis programmer. ________
Distribution of the final report. ________
Advantages of Written Questionnaires
Questionnaires are very cost effective when compared to face-to-face
interviews. This is especially true for studies involving large sample sizes
and large geographic areas. Written questionnaires become even more cost effective
as the number of research questions increases.
Questionnaires are easy to analyze. Data entry and tabulation for nearly all
surveys can be easily done with many computer software packages.
Questionnaires are familiar to most people. Nearly everyone has had some experience
completing questionnaires and they generally do not make people apprehensive.
Questionnaires reduce bias. There is uniform question presentation and no middle-man
bias. The researcher's own opinions will not influence the respondent to answer
questions in a certain manner. There are no verbal or visual clues to influence
the respondent.
Questionnaires are less intrusive than telephone or face-to-face surveys. When
a respondent receives a questionnaire in the mail, he is free to complete the
questionnaire on his own time-table. Unlike other research methods, the respondent
is not interrupted by the research instrument.
Disadvantages Of Written Questionnaires
One major disadvantage of written questionnaires is the possibility of
low response rates. Low response is the curse of statistical analysis. It can
dramatically lower our confidence in the results. Response rates vary widely
from one questionnaire to another (10% - 90%), however, well-designed studies
consistently produce high response rates.
Another disadvantage of questionnaires is the inability to probe responses.
Questionnaires are structured instruments. They allow little flexibility to
the respondent with respect to response format. In essence, they often lose
the "flavor of the response" (i.e., respondents often want to qualify
their answers). By allowing frequent space for comments, the researcher can
partially overcome this disadvantage. Comments are among the most helpful of
all the information on the questionnaire, and they usually provide insightful
information that would have otherwise been lost.
Nearly ninety percent of all communication is visual. Gestures and other visual
cues are not available with written questionnaires. The lack of personal contact
will have different effects depending on the type of information being requested.
A questionnaire requesting factual information will probably not be affected
by the lack of personal contact. A questionnaire probing sensitive issues or
attitudes may be severely affected.
When returned questionnaires arrive in the mail, it's natural to assume that
the respondent is the same person you sent the questionnaire to. This may not
actually be the case. Many times business questionnaires get handed to other
employees for completion. Housewives sometimes respond for their husbands. Kids
respond as a prank. For a variety of reasons, the respondent may not be who
you think it is. It is a confounding error inherent in questionnaires.
Finally, questionnaires are simply not suited for some people. For example,
a written survey to a group of poorly educated people might not work because
of reading skill problems. More frequently, people are turned off by written
questionnaires because of misuse.
Questionnaire Design - General Considerations
Most problems with questionnaire analysis can be traced back to the design
phase of the project. Well-defined goals are the best way to assure a good questionnaire
design. When the goals of a study can be expressed in a few clear and concise
sentences, the design of the questionnaire becomes considerably easier. The
questionnaire is developed to directly address the goals of the study.
One of the best ways to clarify your study goals is to decide how you intend
to use the information. Do this before you begin designing the study. This sounds
obvious, but many researchers neglect this task. Why do research if the results
will not be used?
Be sure to commit the study goals to writing. Whenever you are unsure of a question,
refer to the study goals and a solution will become clear. Ask only questions
that directly address the study goals. Avoid the temptation to ask questions
because it would be "interesting to know".
As a general rule, with only a few exceptions, long questionnaires get less
response than short questionnaires. Keep your questionnaire short. In fact,
the shorter the better. Response rate is the single most important indicator
of how much confidence you can place in the results. A low response rate can
be devastating to a study. Therefore, you must do everything possible to maximize
the response rate. One of the most effective methods of maximizing response
is to shorten the questionnaire.
If your survey is over a few pages, try to eliminate questions. Many people
have difficulty knowing which questions could be eliminated. For the elimination
round, read each question and ask, "How am I going to use this information?"
If the information will be used in a decision-making process, then keep the
question... it's important. If not, throw it out.
One important way to assure a successful survey is to include other experts
and relevant decision-makers in the questionnaire design process. Their suggestions
will improve the questionnaire and they will subsequently have more confidence
in the results.
Formulate a plan for doing the statistical analysis during the design stage
of the project. Know how every question will be analyzed and be prepared to
handle missing data. If you cannot specify how you intend to analyze a question
or use the information, do not use it in the survey.
Make the envelope unique. We all know how important first impressions are. The
same holds true for questionnaires. The respondent's first impression of the
study usually comes from the envelope containing the survey. The best envelopes
(i.e., the ones that make you want to see what's inside) are colored, hand-addressed
and use a commemorative postage stamp. Envelopes with bulk mail permits or gummed
labels are perceived as unimportant. This will generally be reflected in a lower
response rate.
Provide a well-written cover letter. The respondent's next impression comes
from the cover letter. The importance of the cover letter should not be underestimated.
It provides your best chance to persuade the respondent to complete the survey.
Give your questionnaire a title that is short and meaningful to the respondent.
A questionnaire with a title is generally perceived to be more credible than
one without.
Include clear and concise instructions on how to complete the questionnaire.
These must be very easy to understand, so use short sentences and basic vocabulary.
Be sure to print the return address on the questionnaire itself (since questionnaires
often get separated from the reply envelopes).
Begin with a few non-threatening and interesting items. If the first items are
too threatening or "boring", there is little chance that the person
will complete the questionnaire. People generally look at the first few questions
before deciding whether or not to complete the questionnaire. Make them want
to continue by putting interesting questions first.
Use simple and direct language. The questions must be clearly understood by
the respondent. The wording of a question should be simple and to the point.
Do not use uncommon words or long sentences. Make items as brief as possible.
This will reduce misunderstandings and make the questionnaire appear easier
to complete. One way to eliminate misunderstandings is to emphasize crucial
words in each item by using bold, italics or underlining.
Leave adequate space for respondents to make comments. One criticism of questionnaires
is their inability to retain the "flavor" of a response. Leaving space
for comments will provide valuable information not captured by the response
categories. Leaving white space also makes the questionnaire look easier and
this increases response.
Place the most important items in the first half of the questionnaire. Respondents
often send back partially completed questionnaires. By putting the most important
items near the beginning, the partially completed questionnaires will still
contain important information.
Hold the respondent's interest. We want the respondent to complete our questionnaire.
One way to keep a questionnaire interesting is to provide variety in the type
of items used. Varying the questioning format will also prevent respondents
from falling into "response sets". At the same time, it is important
to group items into coherent categories. All items should flow smoothly from
one to the next.
If a questionnaire is more than a few pages and is held together by a staple,
include some identifying data on each page (such as a respondent ID number).
Pages often accidentally separate.
Provide incentives as a motivation for a properly completed questionnaire. What
does the respondent get for completing your questionnaire? Altruism is rarely
an effective motivator. Attaching a dollar bill to the questionnaire works well.
If the information you are collecting is of interest to the respondent, offering
a free summary report is also an excellent motivator. Whatever you choose, it
must make the respondent want to complete the questionnaire.
Use professional production methods for the questionnaire--either desktop publishing
or typesetting and keylining. Be creative. Try different colored inks and paper.
The object is to make your questionnaire stand out from all the others the respondent
receives.
Make it convenient. The easier it is for the respondent to complete the questionnaire
the better. Always include a self-addressed postage-paid envelope. Envelopes
with postage stamps get better response than business reply envelopes (although
they are more expensive since you also pay for the non-respondents).
The final test of a questionnaire is to try it on representatives of the target
audience. If there are problems with the questionnaire, they almost always show
up here. If possible, be present while a respondent is completing the questionnaire
and tell her that it is okay to ask you for clarification of any item. The questions
she asks are indicative of problems in the questionnaire (i.e., the questions
on the questionnaire must be without any ambiguity because there will be no
chance to clarify a question when the survey is mailed).
Qualities of a Good Question
There are good and bad questions.
The qualities of a good question are as follows:
1. Evokes the truth.
Questions must be non-threatening. When a respondent is concerned about
the consequences of answering a question in a particular manner, there is a
good possibility that the answer will not be truthful. Anonymous questionnaires
that contain no identifying information are more likely to produce honest responses
than those identifying the respondent. If your questionnaire does contain sensitive
items, be sure to clearly state your policy on confidentiality.
2. Asks for an answer on only one dimension.
The purpose of a survey is to find out information. A question that asks
for a response on more than one dimension will not provide the information you
are seeking. For example, a researcher investigating a new food snack asks "Do
you like the texture and flavor of the snack?" If a respondent answers
"no", then the researcher will not know if the respondent dislikes
the texture or the flavor, or both. Another questionnaire asks, "Were you
satisfied with the quality of our food and service?" Again, if the respondent
answers "no", there is no way to know whether the quality of the food,
service, or both were unsatisfactory. A good question asks for only one "bit"
of information.
3. Can accommodate all possible answers.
Multiple choice items are the most popular type of survey questions because
they are generally the easiest for a respondent to answer and the easiest to
analyze. Asking a question that does not accommodate all possible responses
can confuse and frustrate the respondent.
For example, consider the question:
What brand of computer do you own? __
A. IBM PC
B. Apple
Clearly, there are many problems with this question.
What if the respondent doesn't own a microcomputer?
What if he owns a different brand of computer?
What if he owns both an IBM PC and an Apple?
There are two ways to correct this kind of problem.
The first way is to make each response a separate dichotomous item on the questionnaire.
For example:
Do you own an IBM PC? (circle: Yes or No)
Do you own an Apple computer? (circle: Yes or No)
Another way to correct the problem is to add the necessary response categories
and allow multiple responses.
This is the preferable method because it provides more information than the
previous method.
What brand of computer do you own? (Check all that apply)
__ Do not own a computer
__ IBM PC
__ Apple
__ Other
4. Has mutually exclusive options.
A good question leaves no ambiguity in the mind of the respondent. There
should be only one correct or appropriate choice for the respondent to make.
An obvious example is:
Where did you grow up? __
A. country
B. farm
C. city
A person who grew up on a farm in the country would not know whether to select
choice A or B. This question would not provide meaningful information. Worse
than that, it could frustrate the respondent and the questionnaire might find
its way to the trash.
5. Produces variability of responses.
When a question produces no variability in responses, we are left with considerable
uncertainty about why we asked the question and what we learned from the information.
If a question does not produce variability in responses, it will not be possible
to perform any statistical analyses on the item. For example:
What do you think about this report? __
A. It's the worst report I've read
B. It's somewhere between the worst and best
C. It's the best report I've read
Since almost all responses would be choice B, very little information is learned.
Design your questions so they are sensitive to differences between respondents.
As another example:
Are you against drug abuse? (circle: Yes or No)
Again, there would be very little variability in responses and we'd be left
wondering why we asked the question in the first place.
6. Follows comfortably from the previous question.
Writing a questionnaire is similar to writing anything else. Transitions
between questions should be smooth. Grouping questions that are similar will
make the questionnaire easier to complete, and the respondent will feel more
comfortable. Questionnaires that jump from one unrelated topic to another feel
disjointed and are not likely to produce high response rates.
7. Does not presuppose a certain state of affairs.
Among the most subtle mistakes in questionnaire design are questions that
make an unwarranted assumption. An example of this type of mistake is:
Are you satisfied with your current auto insurance? (Yes or No)
This question will present a problem for someone who does not currently have
auto insurance. Write your questions so they apply to everyone. This often means
simply adding an additional response category.
Are you satisfied with your current auto insurance?
___ Yes
___ No
___ Don't have auto insurance
One of the most common mistaken assumptions is that the respondent knows the
correct answer to the question. Industry surveys often contain very specific
questions that the respondent may not know the answer to. For example:
What percent of your budget do you spend on
direct mail advertising? ____
Very few people would know the answer to this question without looking it up,
and very few respondents will take the time and effort to look it up. If you
ask a question similar to this, it is important to understand that the responses
are rough estimates and there is a strong likelihood of error.
It is important to look at each question and decide if all respondents will
be able to answer it. Be careful not to assume anything. For example, the following
question assumes the respondent knows what Proposition 13 is about.
Are you in favor of Proposition 13 ?
___ Yes
___ No
___ Undecided
If there is any possibility that the respondent may not know the answer to your
question, include a "don't know" response category.
8. Does not imply a desired answer.
The wording of a question is extremely important. We are striving for objectivity
in our surveys and, therefore, must be careful not to lead the respondent into
giving the answer we would like to receive. Leading questions are usually easily
spotted because they use negative phraseology. As examples:
Wouldn't you like to receive our free brochure?
Don't you think the Congress is spending too much money?
9. Does not use emotionally loaded or vaguely defined words.
This is one of the areas overlooked by both beginners and experienced researchers.
Quantifying adjectives (e.g., most, least, majority) are frequently used in
questions. It is important to understand that these adjectives mean different
things to different people.
10. Does not use unfamiliar words or abbreviations.
Remember who your audience is and write your questionnaire for them. Do
not use uncommon words or compound sentences. Write short sentences. Abbreviations
are okay if you are absolutely certain that every single respondent will understand
their meanings. If there is any doubt at all, do not use the abbreviation. The
following question might be okay if all the respondents are accountants, but
it would not be a good question for the general public.
What was your AGI last year? ______
11. Is not dependent on responses to previous questions.
Branching in written questionnaires should be avoided. While branching can
be used as an effective probing technique in telephone and face-to-face interviews,
it should not be used in written questionnaires because it sometimes confuses
respondents. An example of branching is:
1. Do you currently have a life insurance policy ? (Yes or No)
If no, go to question 3
2. How much is your annual life insurance premium ? _________
These questions could easily be rewritten as one question that applies to everyone:
1. How much did you spend last year for life insurance ? ______
12. Does not ask the respondent to order or rank a series of more than five
items.
Questions asking respondents to rank items by importance should be avoided.
This becomes increasingly difficult as the number of items increases, and the
answers become less reliable. This becomes especially problematic when asking
respondents to assign a percentage to a series of items. In order to successfully
complete this task, the respondent must mentally continue to re-adjust his answers
until they total one hundred percent. Limiting the number of items to five will
make it easier for the respondent to answer.
Pre-notification Letters
Many researchers have studied pre-notification letters to determine if
they increase response rate. A meta-analysis of these studies revealed an aggregate
increase in response rate of 7.7 percent. Pre-notification letters might help
to establish the legitimacy of a survey, thereby contributing to a respondent's
trust. Another possibility is that a pre-notification letter builds expectation
and reduces the possibility that a potential respondent might disregard the
survey when it arrives.
Pre-letters are seldom used in marketing research surveys. They are an excellent
(but expensive) way to increase response. The researcher needs to weigh the
additional cost of sending out a pre-letter against the probability of a lower
response rate. When sample sizes are small, every response really counts and
a pre-letter is highly recommended.
1. Briefly describe why the study is being done and identify the sponsors. This
is impressive and lends credibility to the study.
2. Explain why the person receiving the pre-letter was chosen to receive the
questionnaire.
3. Justify why the respondent should complete the questionnaire. The justification
must be something that will benefit the respondent. For most people, altruism
is not sufficient justification. If an incentive will be included with the questionnaire,
mention the inclusion of a free gift without specifically telling what it will
be.
4. Explain how the results will be used.
Cover Letters
The cover letter is an essential part of the survey. To a large degree,
the cover letter will affect whether or not the respondent completes the questionnaire.
It is important to maintain a friendly tone and keep it as short as possible.
The importance of the cover letter should not be underestimated. It provides
an opportunity to persuade the respondent to complete the survey. If the questionnaire
can be completed in less than five minutes, the response rate can be increased
by mentioning this in the cover letter.
Flattering the respondent in the cover letter does not seem to affect response.
Altruism or an appeal to the social utility of a study has occasionally been
found to increase response, but more often, it is not an effective motivator.
There are no definitive answers whether or not to personalize cover letters
(i.e., the respondents name appears on the cover letter). Some researchers have
found that personalized cover letters can be detrimental to response when anonymity
or confidentiality are important to the respondent.
The literature regarding personalization are mixed. Some researchers have found
that personalized cover letters with hand-written signatures helped response
rates. Other investigators, however, have reported that personalization has
no effect on response.
The signature of the person signing the cover letter has been investigated by
several researchers. Ethnic sounding names and the status of the researcher
(professor or graduate student) do not affect response. One investigator found
that a cover letter signed by the owner of a marina produced better response
than one signed by the sales manager. The literature is mixed regarding whether
a hand-written signature works better than one that is mimeographed. Two researchers
reported that mimeographed signatures worked as well as a hand-written one,
while another reported that hand-written signatures produced better response.
Another investigator found that cover letters signed with green ink increased
response by over 10 percent.
It is commonly believed that a handwritten postscript (P.S.) in the cover letter
might increase response. One older study did find an increase in response, however,
more recent studies found no significant difference.
1. Describe why the study is being done (briefly) and identify the sponsors.
2. Mention the incentive. (A good incentive is a copy of the results).
3. Mention inclusion of a stamped, self-addressed return envelope.
4. Encourage prompt response without using deadlines.
5. Describe your "confidentiality/anonymity" policy.
6. Give the name and phone number of someone they can call with questions.
Response Rate and Following up on Nonrespondents
Response rate is the single most important indicator of how much confidence
can be placed in the results of a survey. A low response rate can be devastating
to the reliability of a study.
One of the most powerful tool for increasing response is to use follow-ups or
reminders. Traditionally, between 10 and 60 percent of those sent questionnaires
respond without follow-up reminders. These rates are too low to yield confident
results, so the need to follow up on nonrespondents is clear.
Researchers can increase the response from follow-up attempts by including another
copy of the questionnaire. When designing the follow-up procedure, it is important
for the researcher to keep in mind the unique characteristics of the people
in the sample. The most successful follow-ups have been achieved by phone calls.
Many researchers have examined whether postcard follow-ups are effective in
increasing response. The vast majority of these studies show that a follow-up
postcard slightly increases response rate, and a meta-analysis revealed an aggregate
gain of 3.5 percent. The postcard serves as a reminder for subjects who have
forgotten to complete the survey.
Nonresponse Bias
Many studies have attempted to determine if there is a difference between
respondents and nonrespondents. Some researchers have reported that people who
respond to surveys answer questions differently than those who do not. Others
have found that late responders answer differently than early responders, and
that the differences may be due to the different levels of interest in the subject
matter. One researcher, who examined a volunteer organization, reported that
those more actively involved in the organization were more likely to respond.
Demographic characteristics of nonrespondents have been investigated by many
researchers. Most studies have found that nonresponse is associated with low
education. However, one researcher reported that demographic characteristics
such as age, education, and employment status were the same for respondents
and nonrespondents. Another study found that nonrespondents were more often
single males.
Most researchers view nonresponse bias as a continuum, ranging from fast responders
to slow responders (with nonresponders defining the end of the continuum). In
fact, one study used extrapolation to estimate the magnitude of bias created
by nonresponse. Another group of researchers argue that nonresponse should not
be viewed as a continuum, and that late respondents do not provide a suitable
basis for estimating the characteristics of nonrespondents.
The Order of the Questions
Items on a questionnaire should be grouped into logically coherent sections.
Grouping questions that are similar will make the questionnaire easier to complete,
and the respondent will feel more comfortable. Questions that use the same response
formats, or those that cover a specific topic, should appear together.
Each question should follow comfortably from the previous question. Writing
a questionnaire is similar to writing anything else. Transitions between questions
should be smooth. Questionnaires that jump from one unrelated topic to another
feel disjointed and are not likely to produce high response rates.
Most investigators have found that the order in which questions are presented
can affect the way that people respond. One study reported that questions in
the latter half of a questionnaire were more likely to be omitted, and contained
fewer extreme responses. Some researchers have suggested that it may be necessary
to present general questions before specific ones in order to avoid response
contamination. Other researchers have reported that when specific questions
were asked before general questions, respondents tended to exhibit greater interest
in the general questions.
It is not clear whether or not question-order affects response. A few researchers
have reported that question-order does not effect responses, while others have
reported that it does. Generally, it is believed that question-order effects
exist in interviews, but not in written surveys.
Anonymity and Confidentiality
An anonymous study is one in which nobody (not even the researcher) can
identify who provided data. It is difficult to conduct an anonymous questionnaire
through the mail because of the need to follow-up on nonresponders. The only
way to do a follow-up is to mail another survey or reminder postcard to the
entire sample. However, it is possible to guarantee confidentiality, where those
conducting the study promise not to reveal the information to anyone. For the
purpose of follow-up, identifying numbers on questionnaires are generally preferred
to using respondents' names. It is important, however, to explain why the number
is there and what it will be used for.
Some studies have shown that response rate is affected by the anonymity/confidentiality
policy of a study. Others have reported that responses became more distorted
when subjects felt threatened that their identities would become known. Others
have found that anonymity and confidentiality issues do not affect response
rates or responses.
The Length of a Questionnaire
As a general rule, long questionnaires get less response than short questionnaires.
However, some studies have shown that the length of a questionnaire does not
necessarily affect response. More important than length is question content.
A subject is more likely to respond if they are involved and interested in the
research topic. Questions should be meaningful and interesting to the respondent.
Incentives
Many researchers have examined the effect of providing a variety of nonmonetary
incentives to subjects. These include token gifts such as small packages of
coffee, ball-point pens, postage stamps, key rings, trading stamps, participation
in a raffle or lottery, or a donation to a charity in the respondent's name.
Generally (although not consistently), nonmonetary incentives have resulted
in an increased response. A meta-analysis of 38 studies that used some form
of an incentive revealed that monetary and nonmonetary incentives were effective
only when enclosed with the survey. The promise of an incentive for a returned
questionnaire was not effective in increasing response. The average increase
in response rate for monetary and nonmonetary incentives was 19.1 percent and
7.9 percent, respectively.
Most researchers have found that higher monetary incentives generally work better
than smaller ones. One researcher proposed a diminishing return model, where
increasing the amount of the incentive would have a decreasing effect on response
rate. A meta-analysis of fifteen studies showed that an incentive of 25¢
increased the response rate by an average of 16 percent, and $1 increased the
response by 31 percent.
Notification of a Cutoff Date
Several researchers have examined the effect of giving subjects a deadline
for responding. While a deadline will usually reduce the time from the mailing
until the returns begin arriving, it appears that it does not increase response,
and may even reduce the response. One possible explanation is that a cutoff
date might dissuade procrastinators from completing the questionnaire after
the deadline has past.
Reply Envelopes and Postage
A good questionnaire makes it convenient for the respondent to reply.
Mail surveys that include a self-addressed stamped reply envelope get better
response than business reply envelopes. Some investigators have suggested that
people might feel obligated to complete the questionnaire because of the guilt
associated with throwing away money--that is, the postage stamp. Others have
pointed out that using a business reply permit might suggest advertising to
some people. Another possibility is that a business reply envelope might be
perceived as less personal.
A meta-analysis on 34 studies comparing stamped versus business reply postage
showed that stamped reply envelopes had a 9 percent greater aggregate effect
than business reply envelopes. In another meta-analysis on nine studies, an
aggregate effect of 6.2 percent was found.
The Outgoing Envelope and Postage
There have been several researchers that examined whether there is a
difference in response between first class postage versus bulk rate. A meta-analysis
of these studies revealed a small, but significant, aggregate difference of
1.8 percent. Envelopes with bulk mail permits might be perceived as "junk
mail", unimportant, or less personal, and thus will be reflected in a lower
response rates.
A few researchers have also examined whether metered mail or stamps work better
on the outgoing envelope. The results of these studies suggest a small increase
in response favoring a stamped envelope. A meta-analysis of these studies revealed
that the aggregate difference was slightly less than one percent.
Many researchers have reported increased response rates by using registered,
certified, or special delivery mail to send the questionnaire. The wisdom of
using these techniques must be weighed against the consequences of angering
respondents that make a special trip to the post office, only to find a questionnaire.
It is not clear whether a typed or hand-addressed envelope affects response.
One study, conducted at the University of Minnesota, reported that students
responded better to hand-addressed postcards, while professors responded better
to typed addresses.
This writer could find no studies that examined whether gummed labels would
have a deleterious effect on response rate, although we might predict that response
rate would be less for gummed labels because they have the appearance of less
personalization.
This writer could also find no studies that examined whether the color of the
envelope affects response rate. First impressions are important, and the respondent's
first impression of the study usually comes from the envelope containing the
survey. Therefore, we might predict that color would have a positive impact
on response because of its uniqueness.
The "Don't Know", "Undecided", and
"Neutral" Response Options
Response categories are developed for questions in order to facilitate
the process of coding and analysis. Many studies have looked at the effects
of presenting a "don't know" option in attitudinal questions. The
"don't know" option allows respondents to state that they have no
opinion or have not thought about a particular issue.
The physical placement of the "undecided" category (at the midpoint
of the scale, or separated from the scale) can change response patterns. Respondents
are more likely to choose the "undecided" category when it was off
to the side of the scale. There are also different response patterns depending
on whether the midpoint is labeled "undecided" or "neutral".
Several researchers have found that the physical location of the middle alternative
can make a difference in responses, and that placing the middle option at the
last position in the question increases the percentage of respondents who select
it by over 9 percent. Frequently, offering respondents a middle alternative
in a survey question will make a difference in the conclusions that would be
drawn from the data. The middle option of an attitudinal scale attracts a substantial
number of respondents who might be unsure of their opinion.
Researcher have also studied the "don't know" option for factual questions.
Unlike attitude questions, respondents might legitimately not know the answer
to a factual question. Surprisingly, the research suggests that the "don't
know" option should not be included in factual questions. Questions that
exclude the "don't know" option produce a greater volume of accurate
data. Furthermore, there is generally no difference in response rate depending
on the inclusion or exclusion of the "don't know" option. There is
still a controversy surrounding the "don't know" response category.
Many researchers advocate including a "don't know" response category
when there is any possibility that the respondent may not know the answer to
a question. The best advice is probably to use a "don't know" option
for factual questions, but not for attitude questions.
Question Wording
The wording of a question is extremely important. Researchers strive
for objectivity in surveys and, therefore, must be careful not to lead the respondent
into giving a desired answer. Unfortunately, the effects of question wording
are one of the least understood areas of questionnaire research.
Many investigators have confirmed that slight changes in the way questions are
worded can have a significant impact on how people respond. Several authors
have reported that minor changes in question wording can produce more than a
25 percent difference in people's opinions.
Several investigators have looked at the effects of modifying adjectives and
adverbs. Words like usually, often, sometimes, occasionally, seldom, and rarely
are "commonly" used in questionnaires, although it is clear that they
do not mean the same thing to all people. Some adjectives have high variability
and others have low variability. The following adjectives have highly variable
meanings and should be avoided in surveys: a clear mandate, most, numerous,
a substantial majority, a minority of, a large proportion of, a significant
number of, many, a considerable number of, and several. Other adjectives produce
less variability and generally have more shared meaning. These are: lots, almost
all, virtually all, nearly all, a majority of, a consensus of, a small number
of, not very many of, almost none, hardly any, a couple, and a few.
Sponsorship
There have been several studies to determine if the sponsor of a survey
might affect response rate. The overwhelming majority of these studies have
clearly demonstrated that university sponsorship is the most effective. A meta-analysis
of these studies revealed an aggregate increase in response rate of 8.9 percent.
This may be due to the past benefits that the respondent has received from the
university. Another possibility is that a business sponsor implies advertising
or sales to potential respondents.
Sampling
It is incumbent on the researcher to clearly define the target population.
There are no strict rules to follow, and the researcher must rely on logic and
judgment. The population is defined in keeping with the objectives of the study.
Sometimes, the entire population will be sufficiently small, and the researcher
can include the entire population in the study. This type of research is called
a census study because data is gathered on every member of the population.
Usually, the population is too large for the researcher to attempt to survey
all of its members. A small, but carefully chosen sample can be used to represent
the population. The sample reflects the characteristics of the population from
which it is drawn.
Sampling methods are classified as either probability or nonprobability. In
probability samples, each member of the population has a known non-zero probability
of being selected. Probability methods include random sampling, systematic sampling,
and stratified sampling. In nonprobability sampling, members are selected from
the population in some nonrandom manner. These include convenience sampling,
judgment sampling, quota sampling, and snowball sampling. The advantage of probability
sampling is that sampling error can be calculated. Sampling error is the degree
to which a sample might differ from the population. When inferring to the population,
results are reported plus or minus the sampling error. In nonprobability sampling,
the degree to which the sample differs from the population remains unknown.
Random sampling is the purest form of probability sampling. Each member
of the population has an equal and known chance of being selected. When there
are very large populations, it is often difficult or impossible to identify
every member of the population, so the pool of available subjects becomes biased.
Systematic sampling is often used instead of random sampling. It is also
called an Nth name selection technique. After the required sample size has been
calculated, every Nth record is selected from a list of population members.
As long as the list does not contain any hidden order, this sampling method
is as good as the random sampling method. Its only advantage over the random
sampling technique is simplicity. Systematic sampling is frequently used to
select a specified number of records from a computer file.
Stratified sampling is commonly used probability method that is superior
to random sampling because it reduces sampling error. A stratum is a subset
of the population that share at least one common characteristic. The researcher
first identifies the relevant stratums and their actual representation in the
population. Random sampling is then used to select subjects from each stratum
until the number of subjects in that stratum is proportional to its frequency
in the population. Stratified sampling is often used when one or more of the
stratums in the population have a low incidence relative to the other stratums.
Convenience sampling is used in exploratory research where the researcher
is interested in getting an inexpensive approximation of the truth. As the name
implies, the sample is selected because they are convenient. This nonprobability
method is often used during preliminary research efforts to get a gross estimate
of the results, without incurring the cost or time required to select a random
sample.
Judgment sampling is a common nonprobability method. The researcher selects
the sample based on judgment. This is usually and extension of convenience sampling.
For example, a researcher may decide to draw the entire sample from one "representative"
city, even though the population includes all cities. When using this method,
the researcher must be confident that the chosen sample is truly representative
of the entire population.
Quota sampling is the nonprobability equivalent of stratified sampling.
Like stratified sampling, the researcher first identifies the stratums and their
proportions as they are represented in the population. Then convenience or judgment
sampling is used to select the required number of subjects from each stratum.
This differs from stratified sampling, where the stratums are filled by random
sampling.
Snowball sampling is a special nonprobability method used when the desired
sample characteristic is rare. It may be extremely difficult or cost prohibitive
to locate respondents in these situations. Snowball sampling relies on referrals
from initial subjects to generate additional subjects. While this technique
can dramatically lower search costs, it comes at the expense of introducing
bias because the technique itself reduces the likelihood that the sample will
represent a good cross section from the population.
Significance
What does significance really mean?
Many researchers get very excited when they have discovered a "significant"
finding, without really understanding what it means. When a statistic is significant,
it simply means that you are very sure that the statistic is reliable. It doesn't
mean the finding is important.
For example, suppose we give 1,000 people an IQ test, and we ask if there is
a significant difference between male and female scores. The mean score for
males is 98 and the mean score for females is 100. We use an independent groups
t-test and find that the difference is significant at the .001 level. The big
question is, "So what?". The difference between 98 and 100 on an IQ
test is a very small difference...so small, in fact, that its not even important.
Then why did the t-statistic come out significant? Because there was a large
sample size. When you have a large sample size, very small differences will
be detected as significant. This means that you are very sure that the difference
is real (i.e., it didn't happen by fluke). It doesn't mean that the difference
is large or important. If we had only given the IQ test to 25 people instead
of 1,000, the two-point difference between males and females would not have
been significant.
Significance is a statistical term that tells how sure you are that a difference
or relationship exists. To say that a significant difference or relationship
exists only tells half the story. We might be very sure that a relationship
exists, but is it a strong, moderate, or weak relationship? After finding a
significant relationship, it is important to evaluate its strength. Significant
relationships can be strong or weak. Significant differences can be large or
small. It just depends on your sample size.
One-Tailed and Two-Tailed Significance Tests
One important concept in significance testing is whether you use a one-tailed
or two-tailed test of significance. The answer is that it depends on your hypothesis.
When your research hypothesis states the direction of the difference or relationship,
then you use a one-tailed probability. For example, a one-tailed test would
be used to test these null hypotheses: Females will not score significantly
higher than males on an IQ test. Blue collar workers are will not buy significantly
more product than white collar workers. Superman is not significantly stronger
than the average person. In each case, the null hypothesis (indirectly) predicts
the direction of the difference. A two-tailed test would be used to test these
null hypotheses: There will be no significant difference in IQ scores between
males and females. There will be no significant difference in the amount of
product purchased between blue collar and white collar workers. There is no
significant difference in strength between Superman and the average person.
The one-tailed probability is exactly half the value of the two-tailed probability.
There is a raging controversy (for about the last hundred years) on whether
or not it is ever appropriate to use a one-tailed test. The rationale is that
if you already know the direction of the difference, why bother doing any statistical
tests. While it is generally safest to use a two-tailed tests, there are situations
where a one-tailed test seems more appropriate. The bottom line is that it is
the choice of the researcher whether to use one-tailed or two-tailed research
questions.
Procedure Used to Test for Significance
Whenever we perform a significance test, it involves comparing a test
value that we have calculated to some critical value for the statistic. It doesn't
matter what type of statistic we are calculating (e.g., a t-statistic, a chi-square
statistic, an F-statistic, etc.), the procedure to test for significance is
the same.
1. Decide on the critical alpha level you will use (i.e., the error rate you
are willing to accept).
2. Conduct the research.
3. Calculate the statistic.
4. Compare the statistic to a critical value obtained from a table.
If your statistic is higher than the critical value from the table:
Your finding is significant.
You reject the null hypothesis.
The probability is small that the difference or relationship happened by chance,
and p is less than the critical alpha level (p < a ).
If your statistic is lower than the critical value from the table:
Your finding is not significant.
You fail to reject the null hypothesis.
The probability is high that the difference or relationship happened by chance,
and p is greater than the critical alpha level (p > a ).
Modern computer software can calculate exact probabilities for most test statistics.
If you have an exact probability from computer software, simply compare it to
your critical alpha level. If the exact probability is less than the critical
alpha level, your finding is significant, and if the exact probability is greater
than your critical alpha level, your finding is not significant. Using a table
is not necessary when you have the exact probability for a statistic.
Want to see more?
Download the entire copy of Survival Statistics for only $29.95.
Set your browser to:
http://www.statpac.com/site-map.htm
Order by phone: (612) 925-0159
Order by mail: StatPac Inc.
4425 Thomas Ave. S.
Minneapolis, MN 55410
http://www.statpac.com