Reliability and Validity

Always in the forefront of the minds of those planning to conduct a survey should be the question of the quality of data that will be produced. The two traditional measures of survey quality are:

The American Drug and Alcohol Survey (ADAS) has proven to be very reliable when used in the general population and with five major ethnic groups in the United States. The survey uses multi-item scales (3 to 6 items per scale) to measure involvement with ten different drugs. Another four drugs are assessed with 2 items each. Most other drug surveys use only single items.

Statisticians use Cronbach’s alpha reliability test to determine how reliable a multi-item scale may be for a given population. The table below uses a large population of 6^th-12^th graders across the U.S. It includes a substantial (though not precisely representative) proportion of students from each major ethnic group. It shows that the Cronbach alpha reliabilities for fourteen drug-use scales on the ADAS range from .72 to .94, with the majority in the high .80 to .90 range. Generally, alpha reliabilities above .70 are considered good when measures are used in large surveys, while alpha reliabilities in the range from .80 to .99 are considered to be exceptionally high. The strong alpha reliabilities of the drug use scales on the American Drug and Alcohol Survey show that the students are responding consistently to the survey items.

Cronbach Alpha Reliabilities for Scales Assessing Drug Involvement American Drug and Alcohol Survey (6-12^th Grades)
	Both Genders	Males Only	Females Only
Drug Involvement scales (# items in scale)	n = 108,249	n = 51,819	N = 54,562

Alcohol (6)	.93	.94	.93
Drunk (5)	.88	.88	.87
Marijuana (4)	.90	.90	.89
Cocaine (5)	.87	.87	.86
Inhalants (3)	.89	.88	.89
Uppers (5)	.81	.81	.80
Downers (3)	.82	.82	.82
Heroin (3)	.90	.90	.89
LSD/other psychedelics (6)	.88	.89	.86
PCP (3)	.85	.85	.87
Ritalin (2)	.78	.79	.76
Narcotics other than heroin (2)	.78	.77	.80
Ketamine (2)	.75	.77	.72
Ecstasy (2)	.80	.82	.78

In addition to the statistical methods for insuring reliability, the ADAS checks for exaggeration and inconsistent responses to screen every survey before it is used as part of a school report.

Exaggerated drug use could be a very critical problem, and one that could lead to skepticism about the survey results. We do, however, have evidence that not many students are exaggerating their drug use on the ADAS.

First, we have a fake drug on the survey. We ask students if they have used a drug that does not exist. If a student says he or she uses this "drug," we would suspect there might be exaggeration on the use of other drugs as well. We find, however, that on average, only 1.2% of 12th graders and 3.6% of 8th graders indicate they use this fake "drug."

Over the years, we have analyzed surveys from students who are obviously exaggerating their drug use. We have analyzed these surveys individually and developed computer code for detecting them. The computer checks every survey and eliminates the obvious exaggerators from the analysis. Their responses are not included in the report. As just one example of how this is done, we "flag" anyone who says they have used heroin but have never used marijuana. While this is certainly a possible drug use pattern, it is extremely unlikely, and may be the result of an attempt to fake the survey. Using twenty different checks for exaggeration we have to exclude less than 3% of all surveys when calculating the results.

For a variety of reasons, some students may be inconsistent in how they respond to the survey questions. They may mark their answers randomly, just to get the survey over with. They may have reading problems and be unable to understand the questions. They may give false responses just for fun, to try to "screw up" the survey. If this happens they might, for example, indicate on one question that they have never used marijuana, and on another question they might answer that they’ve used marijuana 10 times or more in the last month. A total of 53 different checks are made for inconsistencies. While any student might just mark an answer wrong in a survey this long, if it happens more than twice on one survey, the survey is deleted from the report process.

Establishing survey validity is not as straightforward as reliability and usually requires evidence from several sources.

If the ADAS is actually measuring rates of drug use, we expect to find concurrent validity, which means that the ADAS results are consistent with other surveys that also measure drug use. Concurrent validity for the ADAS has been established by demonstrating that similar results are obtained when ADAS results are compared with the findings from other well-designed and established surveys over time. The National Institute on Drug Abuse, for example, supports the University of Michigan’s National Monitoring the Future Survey (www.monitoringthefuture.org). That survey tracks drug use nationally and reports national results every year. When we compare all of the schools surveyed in a given year with the ADAS with those national results, we find that they are essentially the same. The similarity in annual results exists not only for the overall levels of drug use in a given year, but also for the decrease seen from 1981 to 1989, the increasing use rates from 1990 to 1997, and the leveling off and decline between 1997 and 2000. While it is conceivable that students in one small location may "collude" in the faking of survey results, it is hard to imagine a national conspiracy on the part of students across the country over a two decade span! Some very real changes in the patterns of youth drug use have occurred in this time, and well designed surveys like the ADAS have been able to track these changes.

If the survey is actually measuring drug use, we also expect to find construct validity. This means that we expect to find out that the results of the survey make sense: Drug use, as measured by the survey, should relate to other things that we would expect drug use to be associated with.

For example the following table shows the percentage of students who scored in the "risk" range on each of a few scales measuring problems that we might expect to be related to drug use. (These scales are included on the Prevention Planning Survey, which can be used with the ADAS.) The table shows that students who are highly drug involved are much more likely to score in the "risk" range on these scales than are students who have moderate drug-involvement. Similarly, the moderately drug-involved students are more likely to score in the "risk" range than students who have low or no drug use. (The table also shows the alpha reliability scores for the scales on the PPS used to examine these risk factors).

Cronbach Alpha Reliabilities and Validity Indicators from the Prevention Planning Survey with the American Drug & Alcohol Survey
	Alpha >reliability of scales from PPS for 6-12^th graders n=25,541	Percent of 6-12^th graders who are "at risk" on each scale-construct (grouped by level of drug use as measured by The American Drug & Alcohol Survey)
		High drug use	Moderate drug use	Low drug use
Low family caring (3)	.77	21.8%	16.3%	7.9%
Low parental monitoring (4)	.77	75.0%	63.5%	49.9%
Family fights a lot (2)	.79	20.3%	17.7%	9.2%
Family doesn’t support school goals (4)	.75	19.9%	15.7%	8.9%
Student has poor school adjustment (6)	.82	24.6%	15.8%	6.8%
Student believes school rules are unfair (2)	.74	14.9%	7.6%	4.3%
Friends have poor school adjustment (5)	.82	21.4%	13.8%	6.6%
Friends fight a lot (2)	.75	17.4%	11.5%	4.6%
Tolerance of deviance (4)	.83	25.8%	18.0%	6.6%
Lack of empathy (3)	.79	6.2%	3.6%	1.9%
High level of anger (5)	.88	28.9%	21.9%	11.4%
Feels help is unavailable (5)	.84	13.7%	9.7%	5.0%

A high quality survey should also be useful for scientific research. When an article is to be published in a refereed scientific journal, it has to be accepted as good science not only by the journal editors, but also by several different external reviewers. Those reviewers are prominent scientists, chosen by the editor for their knowledge of the field. Their names remain unknown to the authors of an article, so that they can honestly critique the methods used in the study. The American Drug and Alcohol Survey has been used in over 45 publications in peer reviewed journals. (Click here for a list of references). The fact that there are so many published articles on studies that have used the American Drug and Alcohol Survey to measure drug use means that the survey has been accepted by other scientists as a good measure of drug use. Furthermore, many of the studies described in these articles also provide further evidence of construct validity, because the findings of the studies reveal the ways in which drug use, as measured by this survey, is meaningfully related to other predictable behaviors, attitudes, and circumstances.