Research Supplement

I. The Goals of Science

  1. Describe: The accurate representation of variables, as in scale development; e.g., measurement of intelligence. Naming and measuring an event on relevant variables.
  2. Explain: A relation between two variables is suggested. Cause is typically implied though may not have been demonstrated. E.g., "He got wet and it made him angry" implies that being angry is an effect of getting wet. "He got angry and kicked the dog" implies that anger is a cause. Explanation speaks of events that occurred in the past.
  3. Predict: Given a causal event, being able to describe the effect.
  4. Control: Arranging causal events to produce desired effects. The certainty that you can create a particular effect by applying a particular treatment.

II. Asking and answering questions.

A. Forming a research question: Using words to describe the kinds of relations you are attempting to establish between variables.
    1. Hypothesis testing: A hypothesis describes either the supposed relation or the null hypothesis. E.g., DISTAR produces higher reading scores on the Woodcock Reading Mastery Test than does GINN. or (and this is the null hypothesis which one is attempting to disprove): There will be no difference in scores on the Woodcock Reading Mastery Test produced by the DISTAR and GINN reading methods.
    2. Research questions: A research question asks a question about the relation. E.g., Will students taught using the DISTAR reading approach have higher reading scores on the Woodcock Reading Mastery Test than those taught using the GINN reading approach?

B. Identifying variables of interest

1. Alterable variables: Variables which can be manipulated by the experimenter, e.g., warmth of the classroom.

2. Unalterable variables: Variables which cannot be manipulated by the experimenter, e.g., gender, ethnicity.

III. Different Types of Studies: There are three dimensions along with studies vary

  1. Design
    1. Experimental vs. non-experimental
    2. True experiments vs. quasi-experiments
    3. Correlation vs. Descriptive
  1. Setting
    1. Lab
    2. Field
  1. Data Collection
    1. Observation
    2. Self-Report

IV. Descriptive Research

  1. Variables of interest are quantified.
  2. Case studies describe interventions and variables of interest are described and sometimes quantified.
  1. Correlational Reseach

A. The degree and direction of relation between variables is established. The degree to which knowing one allows prediction of another.

B. Selecting variables

1. At least two variables must be related in a correlational study, but often more than two are compared.

2. Unlike in experimental research, variables in correlational research are not differentiated as independent and dependent.

C. Determining the degree and direction of relation between two variables.

1. Degree of the relation is reflected by a number between 0 and |1|.

a. The closer the number is to |l|, the stronger the relation.

b. Numbers close to 0 show little or no relation.

c. Numbers close to .5 are equivocal.

2. Direction of the relation is reflected by the sign (either + or -).

a. A positive correlation indicates a direct relation between two variables.

b. A negative correlation indicates an inverse relation between two variables.

D. Interpretation of correlations

1. Correlation doesn't explain anything about cause.

2. If two variables are highly correlated, one MAY cause the other or both may be influenced by some third variable.

3. Correlation is a necessary but not sufficient condition to demonstrate causation.

E. Getting a picture of correlation: The scattergram!

1. Scattergrams provide an interesting picture of the degree of relation between two variables.

2. A scattergram is constructed by plotting pairs of numbers each of which represents the value of one of the two variables for a single subject.

3. When one become good at reading a scattergram, one can come very close to guessing the actual correlation. For our purposes, we will become good at guessing whether a scattergram shows high positive, high negative, moderate positive, moderate negative, or no correlation by looking at examples in class.

VI. Quasi-Experimental Research

  1. In this research, participants are separated according to some unalterable characteristic, e.g., gender, race.
  1. The participants are then observed or have an opportunity to behave with respect to some variable.
  1. Participant’s results are compared across the separated groups, e.g., males vs. females; Blacks vs. Asians, etc.
  1. The separated groups are referred to as the independent variable and the behavior that is measured or observed is referred to as the dependent variable.

VII. Experimental (Causal) Research

  1. Experimental research attempts to determine the degree to which one variable causes another.

B. One variable is manipulated and the effect of this manipulation on another variable is observed or measured.

C. Experimental control is established in one of three ways.

1. Participants are randomly assigned to groups or to conditions.

2. Groups are counter balanced.

3. A variable is held constant.

D. Two major subgroups of experimental research

1. Single subject design

2. Group design

E. Two kinds of variables are identified in the research design.

1. Independent variables (Sometimes called treatment variables): The variable in research that is manipulated by the experimenter. The independent variable is a characteristic which varies and it can vary across different LEVELS. The levels are selected by creating different qualities (e.g., impulsive vs. reflective) of a variable (personality), different quantities (frequencies, intensities, and/or duration; e.g., different numbers of minutes) of a variable (length of timeout), or the presence or absence of a variable (e.g., adult observer in the room or not).

2. Dependent variables: The behaviors or ratings that are measured. In the personality example above, we might be measuring reaction time on a motor task to see if impulsive vs. reflective personality types differ on this dependent variable. For the second, we might be measuring the number of instances of time out. In the third, we might be measuring the number of times one preschooler hits another with and without an adult observer present.

F. Extraneous variables are those that need to be controlled, that is, prevented from having an effect on behavior that could be confused with the effects of the independent variable. They are only a problem is they predictably and systematically vary with the independent variable, if they are "locked" to levels of the independent variable in which case they are referred to as confounding variables.

G. Confederates

1. A confederate is a person who cooperates with the experimenter to create a treatment condition.

2. The confederate appears to others to be an uninformed observer or a routine participant in the study.

VII. Experimental Group Designs

  1. Three major types
    1. Between subjects (most common)
    2. Within subjects
    3. Mixed designs
  1. Number of independent variables
    1. At least one
    2. Can be many
      1. Results are harder to interpret as the number of independent variables increase
      2. Research may be more socially valid as number of independent variables increase.
  1. Number of dependent variable
    1. At least one
    2. Can be many
  1. The logic of hypothesis testing: Experiments produce groups of scores that must be examined for DIFFERENCES. The logic of experiments is one that attempts to create differences between measurements of behavior in different experimental groups.
    1. There will almost certainly be differences of SOME size between groups.
    2. The differences can be explained as the effects of chance or as the result of the experimental treatment.
    3. Differences between groups are NORMALLY DISTRIBUTED.
    4. If the treatment(s) in an experiment truly made no difference, the difference(s) between group means will be zero, on the average.
    5. But there may be differences that aren't zero, even when the treatments have had no effect. Since these differences are normally distributed, we can estimate the chance likelihood of differences of a given size.
    6. Null Hypothesis/ Alternative Hypothesis: Correct and incorrect conclusions can be associated with each type of decision.
  1. Errors in hypothesis testing
    1. Type I error -- Wrongly rejecting the null hypothesis. Seeing a"mirage" where nothing exists. A false alarm. We try to control this type of error because its consequences may be more severe than type II error. .05 is conventional level. Type I errors may be overrepresented in the published literature.
    2. Type II error -- Wrongly accepting the null hypothesis. Overlooking a true effect,
  1. Analysis: The point of a group design is to look for differences in means on the dependent variable between different treatments.
    1. Group designs use a statistical analysis, often a t-test or an F-test, to derive a number that can be compared to a chart to determine if group differences are significant..
    2. Level of significance is reported as a probability that the two (or more) groups are the same, so the larger the differences between groups, the smaller this probability will be.
    3. Probabilities at or below .05 are the conventional requirement to suggest significance.
    4. Between groups variablility vs. within groups variability = SEPARATION.
      1. More separation = greater certainty of a treatment effect. Greater certainty of a treatment effect = Less chance of Type I error. Larger t and F indicate greater separation between groups, so larger t or F mean less chance of Type I error.
      2. More separation (higher t or F) is required from small groups because of the greater likelihood that sampling error has distorted group averages.

G. Getting a Picture of Differences
    1. Main Effects
    2. Interaction Effects

VIII. Single subject designs

  1. There are four major designs
    1. Reversal Design
    2. Multiple Baseline Design
    3. Alternating Treatment Design
    4. Changing Criterion Desing
  1. Results of single subject designs are evaluated according to visual changes in performance between baseline and treatment on three dimensions:
    1. Level
    2. Trend
    3. Variability
  2. Visual inspections of data are enhanced when social validity of results is demonstrated.

IX. Questions of validity and reliability of a research study

A. Reliability refers to the degree to which the procedure used and the data are accurate reflections of the study protocol.

1. Procedural reliability

2. Data Collection reliability

B. Validity refers to the degree to which changes in the dependent variable are due solely to the manipulation of the independent variable. Issues of both internal and external validity are addressed on the attached handout.

1. Internal validity: The degree to which the conduct of the study was consistent with its description.

    1. History
    2. Maturation
    3. Testing
    4. Instrument Decay
    5. Statistical Regression
    6. Non-Equivalent groups (Assignment error)
    7. Experimental mortality

2. External validity: The degree to which the results extend to other environments.

    1. Accessible population vs. target population
    2. Personal variables interact with treatment
    3. Incomplete description of experimental setting
    4. Interference of previous treatments
    5. Prestest sensitization
    6. Posttest sensitization
    7. Hawthorne effects (demand characteristics)
    8. Experimenter effects (bias, expectancies)
    9. Time of measurement of dependent variable
    10. Novelty and disruption effects
    11. Measurement of dependent variable.

X. Questions to ask in evaluating a research study (Woolfolk, 1990)

A. Were the groups to be studied reasonably equal before the experiment?

B. Were all the variables except the independent variable controlled so that the only real difference in the treatment of each group was the change in the independent variable?

C. Were the measurement procedures applied consistently to each group?

D. Are the results of the study due to the experimental procedures rather than to the novelty of the situation?

E. Has the investigator who designed the study biased the results in any way?

F. Is it reasonably certain that the results did not occur simply by chance?

G. Will the findings in this particular study be likely to fit other, similar situations?

H. Has the study been replicated?