Quasi-Experimental Designs

Ø used when true randomization of subject and extraneous variables cannot be achieved

Ø used when random assignment of participants to different levels of the IV is not possible

Ø includes potential for presence of confounding variables

Ø requires careful consideration of threats to internal validity

Comparison to “Real Experiment”

True experiments:

· Control & manipulate one or more IV

· Group equivalence (subject assignment to groups)

· Control over extraneous variables

Ethical restriction

· can’t provide proper control groups

· can’t withhold treatment

· can’t administer harmful treatments; use “nature’s experiments”

Social psychology examples

· response to catastrophic events

· sudden wealth (win lottery)

· factors causing social change (use of internet)

Subject variables

· age, gender, education, personality, SES, IQ, drug use, etc.

Quasi-experimental designs select subjects, rather than randomly assign to a treatment

Quasi-experimental designs: vulnerability to selection bias

· sample of subjects may not be representative

· sample may have other features correlated with the target that account for observed group differences

· potential to have many other confounding variables influencing the groups

Problem of Confounding

1. Because subjects are selected based on some target characteristic, many other variables may correlate with the target characteristic and be responsible for observed group differences.

2. In many instances, the nature and number of potential confounding variables in a group selected on a “criterion” may not be known.

3. When groups are selected on the basis of extreme scores on some scale (e.g., “high” performers and “low” performers, the measures will be subject to confounding from regression to the mean (the same subjects tested again will likely score closer to the mean of the population).

Threats to Validity in Quasi-experimental Designs:

1. History – refers to unintended differences in prior experience of selected subjects, or unintended or unavoidable differences in subjects’ experience or outside influences when a manipulated variable (or a “natural treatment”) is the target of investigation.

2. Maturation - this could be maturation, like (a) aging; i.e., during longitudinal studies which we normally think of as maturation, people change in different ways. (b) But also this can refer to any shorter-term internal physical or psychological changes in subjects.

3. Statistical regression [regression toward the mean] - when subjects are assigned to conditions on the basis of extreme scores on some measure, a common phenomenon is regression toward the mean without any treatment, just retest - extreme scores tend to be less reliable than average scores

4. Subject mortality [attrition] - in many experiments, subjects are selected for a characteristic and then tested more than once throughout the course of an experiment (repeated measures). The longer and more involved an experiment, the more likely it is that some subjects will drop out of the experiment. The reason why a particular subject dropped out and another did not is a form of self-selection, and may depend on the original selection criterion. Thus, you may have experimental results that reflect only those subjects who persevered or did not dropout for some reason.

5. Selection bias - whenever a procedure other than random selection or random assignment to groups is used to assign subjects to treatment groups [i.e., the subjects are selected non-randomly for participation in a group], there is always a chance that some unintentional bias may have occurred; e.g., if decided to use two different classes to test the effects of two types of lecture methods, the classes could differ in many ways other than the lecture method.

6. Selection interaction - when subjects selected rather than randomly assigned to groups, are differentially affected by some other variable or component of the study, so that the result could be an interaction [i.e., the effect of the subject characteristic of interest changes as a function of the level of the other variable]

Non-experimental Designs (“X” = treatment; “O” = observation)

· One-Group Posttest-Only: [X ® O] (very weak)

· Posttest-Only with Non-Equivalent Groups (chosen using different selection mechanism, often after-the-fact, matching some known or suspected key variable:

X ® O

-- ® O

(difference cannot be attributed with certainty to treatment)

· One-Group Pretest-Posttest: O₁ ® X ® O₂

Quasi-experimental Designs

· Pretest-Posttest with Nonequivalent Control Group

O₁ ® X ® O₂

O₁ ® -- ® O₂

Evaluate comparability of groups based on pretest scores

Evaluate pretest-posttest scores of control group for history & maturation threats

Most serious threat is selection interaction

Natural Treatments; A-B-A variant: observation-natural (or imposed) event-observation (e.g., introduction of a new method of teaching or advising in an academic setting). Often, a comparison group may be a “non-equivalent control. Subjects are not randomly assigned in advance, so may be threatened by selection bias.

Interrupted Time-Series: a variation of a “pre-post” design, where observations are made, in a natural context, before and after some target event (either imposed or naturally occurring), measuring change over time before and after the event (e.g., dental health measures before and after introduction of fluoridated water).

Designs Including Subject Variables as Factors (common)

-controls may include matching strategies

Developmental Studies

¨ Age as a variable (not a “true” variable, just a dimension that has many things correlated with it)

¨ Cross-sectional designs—different subjects at different ages; confounded by generation and contemporary experience, but especially useful when interactions of age with a specific variable of interest are predicted

¨ Longitudinal designs—same subjects at different ages; confounded by concurrent changes in the external world

¨ Cross-sequential designs—testing two or more age groups at two or more times; can make both cross-sectional comparisons and longitudinal comparisons