Student Learning Outcomes

At the completion of this unit of instruction students will be able to:

1. Explain how the nine threats to internal validity can impact the results of research studies and how they can or cannot be controlled. (You should know these nine threats and be able to give practical examples. Random assignment is critical.)
2. Explain how the four threats to external validity can impact the results of research studies and how they can or cannot be controlled. (You should know these nine threats and be able to give practical examples. Random selection is critical.)
3. Explain through the use of practical examples, the limitations of preexperimental designs. (Lack of randomization makes findings about the treatment inconclusive.)
4. Explain through the use of practical examples, the advantages and disadvantages of each of the three experimental designs. (Randomized groups, pretest/post test, and Solomon designs are all true experimental designs but allow different comparisons.)
5. Explain through the use of a practical example, the advantages and disadvantages of using a quasi-experimental design. (Can use preexisting groups, lack of randomization, etc.)

Introduction

Q. In experimental research what are we primarily interested in establishing?

A. Cause and effect. Remember however, that statistics alone don't do this. Need to examine all aspects of the experimental arrangements to determine alternative explanations.

Q. What might be possible alternative explanations?

A. Incorrect statistics, poor control of variables, inappropriate subjects, incorrect interpretation of results, etc. Also - we have concerns about the experimental design which affects the internal and external validity of experiments (Q. What are these two types of validity?)

Q. In striving for high internal and external validity we are forced to compromise. Why?

A. The more we control the variables in an experiment the less we are able to generalize outside of this setting. It's a dilemma and source of disagreement in some areas where researchers seem to be obsessed with one or other orientation.

These threats to IV and EV were closely examined more than 30 years ago by Campbell and Stanley and resulted in a long research article that was later published as a small book (show?) and has become the bible of design for researchers.

Eight Threats to Internal Validity

So let's look at the threats C&S identified. There were eight identified threats to IV:

History - events occurring other than part of the experiment (including weather, season, etc.)

Maturation - passage of time

Testing*- repeat effects of testing, e.g. usually people do better on second attempt at any test because of familiarity, etc.

Instrumentation*- calibration, differences between observers

Statistical regression - danger if groups are selected on extreme scores

Selection biases - if groups are not random and start different

Experimental mortality*- loss of subjects due to non random reasons

Selection x maturation interaction - affects nonequivalent (non random) groups where the passage of time might affect one more than the other.

Rosenthal added a ninth threat which he entitled expectancy - when researchers have a bias toward some type of result and are in a position to influence findings. This occurs when the researchers have the opportunity to record data based on their subjective impressions.

You should know these nine threats to IV. I use mnemonic MIS RM SMITHE. (Show how this helps by drawing on board.)

These factors affect IV - which is what?

Q. How do you think we might control many of these threats to IV?

A. By random assignment of subjects to the experimental groups. In fact only three of the threats remain if random assignment occurs (see asterisks). In other words randomization does not control for these factors influencing both groups evenly.

Q. Placebos, blind and double blind arrangements are also ways to minimize threats to IV. What are they?

A. Placebos are used to see if any changes are real or due to other reasons. In blind experiments the subject does not know if he or she is in the control or experimental groups. In double blind neither the subject not the researcher know the subject's group. These techniques help to control expectancy effects.

External Validity

What about EV? C&S identified 4 additional factors that could influence our ability to generalize:

Reactive or interactive effects of testing - a pretest often sensitizes a person in a direction that in effect changes the person. For example, if you are told you have high cholesterol level, or shown the tar in cigarettes and the blackened sick looking lung tissue

Interaction of selection biases and experimental treatment - sometimes treatments are only effective with certain subjects. For example, you use students in your conditioning class to test your ideas on swim training, or swimmers to test your ideas on conditioning. In either case to what population is it reasonable to generalize your findings? (Maw study for injured runners used regular students as subjects)

Reactive effects of experimental arrangements - it is often difficult to generalize from lab settings. (Hawthorn effect? Subjects performance changes when attention is paid to them.)

Multiple treatment interference - it is difficult to generalize if subjects are exposed to more than one level of treatment. For example, if you use the same subjects to evaluate the effect of different training programs there will inevitably some carryover.

Q. What is the key to minimizing threats to EV? (Think about it - what is your concern and how can you increase your ability to generalize?)

A. By random selection. Solomon 4 group design allows examination of the effects of testing. Multiple treatment effects can be controlled by matching.

Types of Experimental Designs

Attempting to control many of these threats involves thinking about your experimental design. Lots of different designs but text divides them into three categories

1. preexperimental

2. experimental

3. quasi experimental

Q. Can you distinguish?

A.

1. preexperimental - little control and no random assignt.

2. experimental - random assignt.

3. quasi experimental - non random groups but attempts made to control threats.

1. One shot study - difficult to conclude anything - can't really attribute the performance to the treatment.

2. One group pretest-post test - can now see improvement but you don't know why (could be maturation, testing, history etc.)

3. Static group comparison - problem is that the groups are not equivalent at the beginning so hard to conclude the reason for any observed differences.

4. Randomized groups design - is like #3 but with random groups. Controls for many of the threats to IV

5. Pretest - post test randomized group design - Useful if interested in the amount of change as a result of treatment. Possible reactive effects of testing is a threat to IV but should be evened out between groups. However reactive effects of testing remain a threat to EV.

6. Solomon four group design - can now evaluate the effects of testing in EV. But you need lots of subjects!

7. Time series designs - can compare rates of changes over time, e.g. between O1 - O4 and O5 - O7 as well as differences between O4 - O5.

8. Reversal design - similar to above, allows insertion of treatment then withdrawal, then treatment.

9. Non-equivalent control groups - we attempt to find similar groups and use as controls. Is similar to design #5 but without the randomization.

10. Ex Post Facto design - Is like #3 but with no control over the treatment the groups received. For example, we could go back and look at records of students in several school districts and perform a statistical analysis.

Other Types of Quasi-Designs

The three additional types described in this section illustrate that experimental design is not static but changes as the result of applying innovative and creative thought to research questions. For example, we will be having class presentations on single subject designs when we discuss case studies. These designs have already been used in many research studies conducted in sport psychology.

Knowledge Check
Think about a possible problem in your area of interest that could be examined using an experimental design. Using the notations (R, O, T etc.) contained in the text draw out the design. Be sure you can explain it.

(Revised 2/3/99)