Student Learning Outcomes
At the completion of this unit of instruction students will be able to:
Describe an example illustrating the importance of ethical considerations when conducting research and the appropriate procedures to be followed
Literature Reviews and Research Synthesis
Whatever kind of research you choose to pursue you will be expected to complete a literature review. As you remember from earlier discussions the literature review provides information about what has been done before, and may be helpful in formulating a future direction to pursue.
In conducting a literature review you attempt to synthesize the results of previously conducted research. This is often a tough assignment because you will inevitably be faced with conflicting findings from studies that included many different procedures. Making sense of these findings and attempting to draw conclusions is not easy under any circumstances.
Thomas and Nelson note that sometimes a literature review, rather than being just part (chapter 2) of a study, can become a research study itself. These types of studies they refer to as research synthesis and they emphasize that these studies follow similar patterns of logical and systematic examination that we expect from any research investigation.
Meta-Analysis
One of the main problems in traditional literature reviews is the reliance on the author’s interpretation of the literature. Although the researcher tries to be logical, systematic, and unbiased, the fact remains that any conclusions are subject to personal biases. You are never quite sure what the author chose to exclude when making comparisons and drawing conclusions. We have a methodology problem.
An technique designed to overcome some of the limitations inherent in traditional literature reviews is known as meta-analysis. This is a relatively new technique proposed by Glass in 1976. Simply stated, meta-analysis is designed as a quantitative method to objectively review completed research.
Procedures of Meta-Analysis
Two procedures are involved that are lacking in traditional literature reviews:
1. Methodology is clearly presented so that others can see how decisions were made.
2. Results are statistically analyzed.
Methodology: You will remember that we earlier noted that scientific research differed in several ways from the kind of haphazard "research" we might do to find the best chocolate chip cookie recipe. Traditional literature reviews are often rather haphazard because the researchers fail to report the procedural steps they took, and therefore what decisions they may have taken regarding inclusion, exclusion, or interpretation of a paper.
Meta-analysis as proposed by Glass has seven clearly defined procedural steps the main purpose of which is to translate the findings of different research to a common parameter. It is a bit like taking apples, oranges, and bananas and in order to compare the taste of each translating our measurements of taste onto a common scale we call "fruit." In meta-analysis this common scale is called "Effect Size (ES)." You should review the procedures described in your text for determining an ES.
Statistical Analysis: Once the results of many different studies have been translated in the single common denominator of Effect Size, it become possible to use statistical techniques to compare findings. We are therefore able to use a quantitative method for analyzing the findings of these different studies.
Examples of Meta-Analyses
To get a better idea of how this technique works in practice you should review the examples provided in the text. You can see that using meta-analysis researchers were able to compare the findings of a large number of studies and attempt to identify commonalties.
Value of Meta-Analysis
As you can imagine, in many areas a great deal of research has been conducted over the years yielding many conflicting, contradictory findings. Meta-analysis offers a method to reexamine completed studies and subjecting them to statistical analysis.
I was especially intrigued by the report of a meta-analysis conducted in 1981 by Hyde on the studies reviewed by Maccoby and Jacklin in 1974. Maccoby and Jacklin’s report on gender differences received widespread attention. In their study it was concluded that gender differences were well established in three areas:
1. Boys were superior to girls in math.
2. Boys were superior to girls in visual-spatial abilities.
3. Girls were superior to boys in verbal abilities.
These findings were widely publicized and probably influenced the thinking if not the behaviors of educators and parents in the expectations they held for males and females. In 1981 Hyde reexamined the studies that Maccoby and Jacklin had reviewed in formulating their conclusions. Subjected to a meta-analysis it was found that gender differences in these three areas were very small - certainly not worthy of the attention they attracted in Maccoby and Jacklin’s study.
Concerns About Meta-Analysis
As noted in your text, some of the techniques used in meta-analysis have been questioned. The very act of attempting to compare the findings of studies that used different measurements, subjects, and designs is open to criticism. (Can we really compare apples, oranges and bananas?). As a relatively new research technique it is probably understandable that refinements will continue to be made to address the concerns that have been raised.
(Revised 2/10/99)