To overcome the limitations of simple before-and-after comparisons, the experimental-design approach, using the concepts of equivalent control and experimental groups and preprogram and postprogram measurements, has been adopted. The experimental-design approach typically makes comparisons among individuals in randomly selected groups, some of whom are served by the program and some of whom are not served by it or are served in some other way. Comparison groups must be specified before the program implementation, and the groups are made similar through random selection and assignment.
The evaluator identifies program objectives and corresponding evaluation criteria and takes the following steps:
- Selects control and experimental (treatment or target) groups, usually by random assignment or probability sample. Measures the preprogram status (conditions) of each group using the selected evaluation criteria.
- Applies the program to the target group, but not to the control group. Monitors the operation of the program to prevent outside events from having a distorting impact. Makes adjustments to eliminate or reduce outside influences as necessary.
- Measures target -and control- group statuses after the program has had an opportunity to effect changes. Compares these to preprogram status levels. If changes exists in the target group but not in the control group, and if, after careful examination, no outside factors are found to have caused the change it is assumed that the program did in fact account for the observed change in the target group.
The basic pretest, posttest, control-group experimental design is diagrammed in Table 1. Before-program and after-program conditions are shown for both control and treatment groups, with Ts and Cs indicating observations or measurements for the treatment and control groups, respectively, and the subscripts 1 and 2 indicating, respectively, preprogram and postprogram measurements. Random selection is used to assign participants to both the treatment and control groups. If the program has an effect, this can be detected in postprogram differences between treatment- and control-group scores.
Table 1: Pretest, Posttest, and Control-Group Evaluation Design
| Indicators | ||
| Before-Program Status | After-Program Status | |
| Treatment Group | T1 | T2 |
| Control Group | C1 | C2 |
| Key: T1 = value of indicator for treatment group before program is implementedT2 = value of indicator for treatment group after program is implementedC1 = value of indicator for control group before program is implemented
C2 = value of indicator for control group after program is implemented |
||
Literature and Internet Links
King, Gary, Emmanuela Gakidou, Nirmala Ravishankar, Ryan T. Moore, Jason Lakin, Manett Vargas, Martha María Téllez-Rojo, Juan Eugenio Hernández Ávila, Mauricio Hernández Ávila, and Héctor Hernández Llamas. 2007. A “politically robust” experimental design for public policy evaluation, with application to the Mexican universal health insurance program. Journal of Policy Analysis and Management 26(3): 479-506. http://dash.harvard.edu/handle/1/4215039