ED470604 2002-12-00 Scientifically Based Research: What Does It Mean for Counselors? ERIC Digest.

ERIC Identifier: ED470604
Publication Date: 2002-12-00
Author: Walz, Garry R. - Jeanne C. Bleuer
Source: ERIC Clearinghouse on Counseling and Student Services Greensboro NC.

Scientifically Based Research: What Does It Mean for Counselors? ERIC Digest.

THIS DIGEST WAS CREATED BY ERIC, THE EDUCATIONAL RESOURCES INFORMATION CENTER. FOR MORE INFORMATION ABOUT ERIC, CONTACT ACCESS ERIC 1-800-LET-ERIC

INTRODUCTION

A core component of many new educational programs funded under Bush administration's No Child Be Left Behind Act of 2001 is that they be based on scientifically based research. Further emphasizing the important role that this type of research will play in future U.S. Department of Education priorities has been the establishment of the What Works Clearinghouse. Under the leadership of Dr. Grover Whitehurst, this new clearinghouse will "...establish standards for research and then determine...which studies meet those standards" (Taub, 2002).

Like others in the educational community, counselors and counseling researchers welcome the opportunity to empirically test the validity of their practices and then, just as important, demonstrate that what they do makes a positive difference in students' lives. But they also have many concerns about the practical feasibility of implementing scientifically based studies. The purpose of this Digest is to provide an overview of the characteristics of scientifically based research, compare it to other types of research, and address some of the challenges and issues that are particularly relevant to conducting scientifically based studies of counseling interventions. To accomplish this, the authors have included selected portions of the ERIC/CASS publication, Research in Counseling & Therapy (Loesch & Vacc, 1997).

WHAT IS SCIENTIFICALLY BASED RESEARCH?

In Times article (November 10, 2002), James Taub pointed out that "Journalists using the most exacting method available to social science - that is, counting - have determined that the phrase 'scientifically based research' occurs more than 100 times in the No Child Left Behind Act of 2001." Other terms that are being used more frequently now by educators and policy makers include empirical evidence, empirical research, evidence based education and randomized control trials (or RCTs). Whatever term is used, the intent is to refer to experimental research studies that employ two fundamental procedures - the use of control groups and the random assignment of subjects to the different treatment groups. Typically, experimental research is undertaken when the research question involves a question of causality, i.e. whether changes in one variable cause changes in another variable (Hadley & Mitchell, 1995). It is precisely this type of research, which will provide evidence of what works (or what works better).

C EXPERIMENTAL RESEARCH METHODS

Isaac and Michael (1981) identified the purpose of experimental research as investigating "possible cause-and-effect relationships by exposing one or more experimental groups to one or more treatment conditions and comparing the results to one or more control groups not receiving the treatment" (p.52). They also identified seven characteristics of experimental research implied in their definition: (a) management of the predictor and criterion variables along with the conditions in which the investigation is conducted; (b) use of a control group; (c) attempting to control variance among the predictor and criterion variables: (d) internal validity; (e) external validity; (f) ability to manage multiple predictor, criterion, and extraneous variables; and (g) exercise of control which makes experimental research powerful (but also somewhat artificial) when applied to human subjects.

PREDICTOR AND CRITERION VARIABLES

There primary types of variables in experimental research (Hadley & Mitchell, 1995). The independent (aka treatment) variable is manipulated, managed, or administered by the researcher. The result of the manipulation is the measured or observed change in the dependent variable. While the terms independent and dependent variables have been used traditionally, the terms predictor and criterion variables are better descriptors, particularly in the context of experimental research. Predictor variables must be carefully chosen or designed to maximize differences due to their effects. Reliable and valid criterion variables must be selected or designed to accurately measure change caused by the predictor variables (Pickering, 1997).

CONTROLLING VARIANCE

Kerlinger (1973) offered an often-cited mnemonic to define what is meant by variance control in experimental research. MAXMINCON refers to MAXimizing the variance associated with the relationship between the predictor and criterion variables, MINimizing the error variance associated with measurement of the criterion variables, and CONtrolling extraneous variance attributable to other variables not included in the investigation. According to Kerlinger, maximizing the variance related to the interaction of the predictor and criterion variables requires designing the levels of the predictor variables to be as different from each other as possible. Minimizing the error variance is accomplished by controlling the conditions in which the investigation is conducted and choosing reliable measures of the criterion variables. Controlling extraneous variance may involve any of a variety of procedures, such as selecting a group of subjects who are homogeneous on the variable, randomly selecting and assigning subjects to groups, or perhaps adding the variable to the investigation as another predictor variable.

INTERNAL VALIDITY

Internal validity refers to the level of confidence that the predictor variable(s), rather than an extraneous variable, produced the change found in the criterion variable(s). Hadley and Mitchell (1995), Heppner, Kivlighan, and Wampold (1992), and Isaac and Michael (1981) listed a variety of threats to internal validity including group composition, experimental mortality, history, maturation, practice effects, placebo effects, the Hawthorne effect, the John Henry effect, experimenter bias, demand characteristics, rater and observer effects, instrumentation, and statistical regression. For example, when subjects are selected because of group membership rather than being randomly selected, when subjects know they are part of an experiment and merely respond to receiving "special" treatment, or when the measuring instruments are not reliable and valid, it is doubtful whether the treatment variable caused the change in the criterion variable or whether it was caused by some other extraneous factor(s) (Pickering, 1997).

EXTERNAL VALIDITY

External validity which the results can be generalized to other populations. Hadley and Mitchell (1995), Heppner, et al. (1992), and Isaac and Michael (1981) described a variety of threats to external validity including initial population-sample differences, mortality, artificial research arrangements, pretest influence, and multiple-treatment influences. For example, generalization to other populations probably will be limited if the sample chosen is not actually representative of the intended population, if subjects who leave the investigation differ in some way from those who remain, if subjects studied in laboratory settings perform differently than they do in naturally occurring situations, if pretesting sensitizes subjects to the treatment, or if multiple treatments are administered to each subject. Managing threats to external validity involves attempting to insure that both the subjects and the context in which the investigation is conducted are appropriately representative.

RIGOR VERSUS RELEVANCE

Experimental almost always results in procedural compromise because control of one type of variance may cause problems in attempting to control another type of variance. Gelso (1979) labeled this "the bubble hypothesis," referring to the difficulty which arises when someone attempts to place a decal on a window and a bubble appears. When the bubble is depressed in one area, it arises somewhere else. Gelso also discussed how experimental rigor is related to internal validity. Threats to internal validity are most easily managed in controlled laboratory conditions, but human behavior rarely occurs in tightly controlled laboratory situations and thus generalization is limited.

ISSUES IN DESIGNING AND IMPLEMENTING RESEARCH ON COUNSELING

Experimental methods are often touted as the sine qua non of research in counseling. However, they are not always the most appropriate to answer questions. Heppner, et al. (1992) stated that just as there is no uniform method of counseling, there is no uniform method for conducting research. They offered five guidelines to determine whether experimental methods should be used: (a) does the professional literature support the use of experimental methods or is a less rigorous approach more appropriate? (b) does the literature include a variety of research methods or have one or two methods dominated? (c) are sufficient resources available to support the type of research method desired? (d) can rigor and relevance be balanced to answer the research question? and (e) can responses to the previous four guidelines be balanced with each other? Whenever experimental methods are preferred or recommended to answer the research question, the issues presented should be considered carefully in designing the investigation.

CONCLUSION

In reviewing the characteristics of experimental research presented above, it is apparent that some aspects of counseling are not readily amenable to the application of scientific research methods. This is not to say that randomized control trials cannot or should not be undertaken when feasible. However, due to the policies of some organizations that preclude use of no treatment control groups and/or parental groups opposed to the involvement of their children in research, these studies will be limited. Given the many potential threats to validity, consumers of the results will need to carefully evaluate the quality of the evidence as well as its relevance to other counseling situations. Of particular interest to counselors is the comparison of the effectiveness of different counseling approaches in obtaining desired outcomes. Thus, an alternative to the treatment-no treatment design is a Treatment A vs. Treatment B design. However, even this design can produce flawed results due to differences in the implementation of treatments by individual counselors. Thus, in assessing the usefulness of results obtained from experimental studies, one must always consider which is better - to base actions on poor experimental research (i.e., research which has not fully met the rigors of controlling all of the relevant variables) or to base actions on good qualitative research (i.e., research that applies intensive and extensive analysis from a number of perspectives and sources).

At the present time, it appears that those who teach and practice counseling will have to rely heavily upon a careful evaluation of the results of all types of research, well conducted meta-analyses of counseling research (e.g., Sexton, et.al., 1987), and best human judgments. In fact, in a speech delivered at a seminar where leading experts in the fields of education and science discussed the meaning of scientifically based research hosted by the Assistant Secretary for Elementary and Secondary Education, Feuer (2002) stated that, "...what scientists themselves often acknowledge is that there is a dimension of human judgment that can be missed with an overzealous focus on the rigor of scientific method."

REFERENCES

Campbell, D. C., Experimental and quasi-experimental designs for research. Chicago, IL: Rand-McNally.

Feuer, M. (2002). The logic and the basic principles of scientific based research. Paper presented at the Scientifically Based Research - U.S. Department of Education seminar, February 6, 2002. (www.ed.gov/offices/OESE/esea/research/feuer-towne.html)

Gelso, C. J. (1979). Research in counseling: Methodological and professional issues. The Counseling Psychologist, 8 (3), 7-35.

Hadley, R. G., & Mitchell, L. K. (1995). Counseling research and program evaluation. Pacific Grove, CA: Brooks/Cole.

Heppner, P. P., Kivlighan, D. M., Jr., & Wampold, B. E. (1992). Research design in counseling. Pacific Grove, CA: Brooks/Cole.

Huitema, B. E. (1980). The analysis of covariance and alternatives. New York, NY: Wiley.

Isaac, S., & Michael, W. B. (1981). Handbook in research and evaluation (2nd ed.). San Diego, CA: EdITS.

Kerlinger, F. N. (1973). Foundations of behavioral research (2nd ed.). New York: Holt, Rinehart and Winston. Loesch, L. C., & Vacc, N. A. (Eds.) (1997). Research in counseling & therapy. Greensboro, NC: ERIC Counseling and Student Services Clearinghouse.

Sexton, T. L., Whiston, S. C., Bleuer, J. C., & Walz, G. R. (1997). Integrating outcome research into counseling practice and training. Alexandria, VA: American Counseling Association. Taub, J. (2002). Does it work? The New York Times, November 10, 2002.

-----

Garry R. Walz, Ph.D., and Jeanne C. Bleuer, Ph.D., are Co-Directors of the ERIC Clearinghouse on Counseling and Student Services at the University of North Carolina at Greensboro.

ERIC Digests are in the public domain and may be freely reproduced and disseminated. This publication was funded by the U.S. Department of Education, Office of Educational Research and Improvement, Contract No. ED-99-CO-0014. Opinions expressed in this report do not necessarily reflect the position of the U.S. Department of Education, OERI, or ERIC/CASS.

Title: Scientifically Based Research: What ERIC Digest.
Document Type: Information Analyses---ERIC Information
Available From: ERIC Counseling and Student Services North Carolina at Greensboro, 201 Ferguson Building, P.O. Box 26171, Greensboro, NC 27402-6171. Tel: 336-334-4114; Tel: 800-414-9769 (Toll Free); Fax: 336-334-4116; e-mail: Descriptors: Accountability, Counseling, Counseling Counselors, Scientific Methodology, Scientific Research
Identifiers: ERIC Digests, No Child Left Behind Act 2001

###

Target Audience: Counselors, Practitioners Clearinghouse, University of data-cfemail="0267706b616163717142776c61652c676677">[email protected]; Web site: http://ericcass.uncg.edu. Effectiveness, School src="http:\\www.ericfacility.net\icons\up.gif" alt="">[Return to ERIC Digest Search Page]