The HERI College Senior Survey

The Higher Education Research Institute (HERI) at UCLA executes a national survey every year to better understand the college experience so that higher education can be improved. They use a meticulously laid out questionnaire and this year’s version is available at: GSEIS. Christian and Dillman (2004, p. 78) found that the amount of space and how it is apportioned can affect response. The HERI questionnaire has evenly apportioned and pleasing spaced formatting.
They exclusively use close-ended questions with a mix of dichotomous, multiple choice, and mostly scaled response. While their questionnaire is highly organized, it does not always reflect the principles of questionnaire design espoused by marketing research. Applying these principles would improve their instrument.

McDaniel and Gates (2008, p. 298) note that dichotomous questions are often subject to measurement error because they offer only black and white choices when many times shades of gray are needed. This is not the case with the HERI questionnaire. Question 6 is the only set of dichotomous choices and they are truly yes/no.

The first three pages of the HERI instrument are mostly scaled response questions. They not only allow seniors to express an opinion about a subject but also calibrate the intensity of that feeling (p. 299). A potential problem with scaled response, remembering category options, is avoided in this questionnaire by its careful design.

The fourth page introduces multiple choice questions. McDaniel and Gates (p. 299) warn that the choices may not cover all alternatives but this may be mitigated by offering an ‘other’ choice. HERI does not do this. They should to improve the questionnaire over time. In addition, Question 27 may have positioning bias. The two positive answers about colleges are positioned where they are most easily seen.

The next question reveals another common problem in the questionnaire. McDaniel and Gates (p. 301) say that words should have the same meaning to all respondents and that, additionally, words used in questions should have a precise meaning. In Question 28 both flaming liberals and draconian conservatives could indicate themselves as moderate. This same issue occurs in Question 13 about drinking frequently or occasionally. What I consider occasional, someone else may consider frequent. Finally, in Question 5, what does the word regularly mean in “regularly communicated with professors?” Specific guidance about how many for frequent, occasional and regular is needed.

Sensitive or embarrassing questions are handled by HERI in a robotic manner. In Question 13 students are directly asked about mental depression, the need for professional counseling, and alcoholic drinking habits.

Question 19 asks about marijuana, same sex marriage, denial of services to undocumented immigrants, and affirmative action. Many may be sensitive to these issues and McDaniel and Gates (p. 327) recommend using one of two techniques in such cases: 1.) Counterbiasing; or 2.) 3rd person voice. HERI does not. This is not an anonymous survey.

McDaniel and Gates (p. 301) also admonish us not to ask questions the respondents could not answer correctly. In Question 14, HERI asks students to guess what quintile they are in for various aptitudes or domain knowledges. Ambrose and Anstey (2007, p. 28) state that an important topic category in a survey is measuring the knowledge of the population. By this they mean discovering the levels of understanding. However, in the HERI survey, the students would have no frame of reference for knowing the answers to the various parts of Question 14 and so McDaniel and Gates apply with full force.

McDaniel and Gates (p. 302) also recommend that time periods be kept recent. They highlight a question as poorly worded because its time period is a year. In Question 9 of the HERI instrument, the time period of a year is used in twenty parts to that question. As McDaniel and Gates ask, unless the students kept an accurate diary of each of the twenty activities, how would they know? A better approach is to ask how much the respondent has done in the past two weeks and then if that is more or less than average (p. 302).

All in all, I found the HERI questionnaire to be well done but a few questions could be fine tuned to get more complete and more accurate results.


References
Ambrose, David and John Anstey (March 2007). Better Suvey Design is Stick for an Answer. ABA Bank Marketing. Retieved from http://www.imc.wvu.edu/ on July 4, 2009.


Christian, L and D DIllman (Spring 2004). THE INFLUENCE OF GRAPHICAL AND SYMBOLIC LANGUAGE MANIPULATIONS ON RESPONSES. Retieved from http://www.imc.wvu.edu/ on July 4, 2009.


McDaniels, C and R Gates (2008). Marketing Research Essentials. John Wiley.