The objectives of this project are to plan, design, and develop an online survey to assess students’ opinions of three podcasted lectures used this spring semester of 2007 in General Microbiology.
I have begun by consulting educational literature (e.g. The Survey Research Handbook by Pamela L. Alreck and Robert B. Settle) about the creation of a representative survey tool. I have read chapter 4 of this book about composing questions. The following is an outline of this chapter:
Basic Attributes of Questions
1) Focus, brevity and clarity
a) Focus - every question should focus on a single, specific issue. Ask precisely what it is that you (or the sponsor) want to know.
b) Brevity - short questions are less prone to error, more likely to be concise.
c) Clarity - the meaning of the question must be clear to all respondents (not open to multiple interpretations)
2) Expressing the question
a) Vocabulary - Use words that are in the core vocabulary of virtually all respondents. Don’t use fancy vocab. because the ultimate measure of sophistication of a survey tool is generating reliable data.
b) Grammar - break compound sentences into simple sentences. Eliminate superfluous words.
3) Instrumentation Bias and Error
a) Be certain that criteria are simply stated.
b) The question must be applicable to all respondents.
c) Avoid examples that are among those that a respondent might list. This will likely lead the respondents to list these examples to the exclusion of others.
d) Don’t pressure your respondents to recall difficult things. (e.g. How many times per week did you wash your toilet in 1982?)
e) Avoid overgeneralizations and overspecificity (things people couldn’t remember in detail)
f) Avoid using wording that is overemphatic - this will likely draw a particular response.
g) Check and recheck to be sure that the words used in the survey have a common meaning to everyone.
h) No double-barreled questions - 2 questions within one item. Ask yourself whether part of the question might be true and part false?
i) Leading questions create bias.
j) Watch for loaded questions (e.g. making it sound as though if you answer no to this question you don’t value human life.)
k) Be certain the question is as free of threat as is possible.
4) Sources of Response Bias
a) Social Desirability - respondents are likely to report what’s socially acceptable, rather than their true answers. Wording can be changed in such a way that the question is less likely to evoke the socially desirable response (e.g. provide alternatives without stating the norm!)
1)) If a respondent feels the researcher is looking for a certain response, they may well provide it!
How to avoid this?
- assure respondents that an honest answer is more valuable.
- don’t make one response appear more positive than another.
2)) Avoid yes or no or positive / negative responses. Some people have a tendency to always agree and others to always disagree. This can also show the respondent an overall negative/positive bias.
3)) Avoid questions that invite respondents to seek prestige (e.g. On a scale of 1 to 10, rate yourself in terms of self-reliance). Try to design questions that show respect to everyone.
4)) Reduce threat (e.g. instead of asking: What would you do if you lost your spouse?, ask: How might people might cope with the loss of a spouse?)
5)) Avoid questions that evoke hostility but in the event that that can’t be done, be sure to give the respondent ample lease to express his/her feelings. That way, he/she will be less likely to carry hostile feelings throughout the survey.
6)) It may not be necessary to tell respondents who is sponsoring a survey as that may generate bias.
7)) If you set a frame of reference in one question (e.g. in the past five years) respondents may be confused in the next question, if no new frame of reference is set.
8)) The order of questions may affect the response:
initiation, routine and fatigue
*List less important questions first
*Vary questions and make them unique to guard against routine.
*Keep the list short
9)) Extremity bias - difficult to distinguish shades of gray on a 1 to 10 scale so people may tend to choose the extremes (1 or 10). Use only as many points as are necessary (pertinent).
5) Question Format
a) Unstructured (open-end) questions
1)) Do not give dimension (range) upon which respondents can answer.
2)) Respondents rarely answer in an expected way
3)) Often produce data that aren’t directly comparable. - Must group data when editing and or postcoding which is laborious and open to interpretation of the investigator.
4)) More prone to recording error.
5)) likely to be lower response rate, more missing data, and fewer random responses.
b) Structured - provide responses
1)) Take time and effort to frame these questions BUT THEY ARE BETTER!
2)) May loose richness and variety of response but impossible to keep this anyway in a mass survey. So conduct a focus group first to get a feel for variety and richness.
3)) Choosing categories for structured questions.
a)) An all-inclusive list
Every conceivable answer must fit into a category.
Advisable to include on open “other” category.
b)) a mutually exclusive list
No overlapping categories
c)) Meaningful clusters
Don’t have too much variation in meaning within a category.
d)) Size and number of categories
Upper limit of six to eight categories.
“If there is a doubt about how precise or fine-grained the data must be, it’s best to use the larger number of more narrow categories.” because they can be easily combined later.
4)) Verbal and numeric categories
a)) If the answer can be in number form than that is better. How many years of formal education have you completed?
b)) numeric data can easily be categorized later.
