Friday, April 24, 2015

Letter to SLI from American Institute of Research


                             

 

 

                                                                                                      April 18, 2011

 

John Simmons

Strategic Learning Initiatives

954 W. Washington Blvd.  Suite 600

Chicago, IL  60607

 

Dear John:                                                                                     

 

Thank you for sharing your most recent report using the 2010 Illinois test score data on the eight Chicago public schools in which Strategic Learning Initiatives has been working. 

 

This latest report builds on the data that served as the basis for our report “Validating the Impact of Strategic Learning Initiatives’ Focused Instruction Process (FIP Model)” (Steven Leinwand and Sarah Edwards, American Institutes for Research, July, 2009).

 

In the 2009 report, we concluded that “the results that SLI has achieved, and that AIR has validated, are very impressive and suggest that well before decisions are made to reconstitute schools under the mandates of NCLB, school districts would be wise to consider far less drastic, but clearly powerful, interventions such as the Focused Instruction Process.”

 

The updated 2010 ISAT data that you have provided show that the schools supported by Strategic Learning continue to close achievement gaps and accelerate student learning.  SLI continues to make a real difference in the lives of students and it is clear that its successes can, and should be, broadly adopted elsewhere.

 

 

Sincerely,

Steven Leinwand

Principal Research Analyst

 

 

Monday, April 20, 2015

American Institutes of Research: Validating the Impact of SLI's FIP Model


 
American Institutes for Research®
 
 
Validating the Impact of Strategic Learning Initiatives’
Focused Instruction Process (FIP)
Model
 
 
 Steven Leinwand
Sarah Edwards
 
 
 
Submitted to:
Strategic Learning Initiatives
954 W. Washington Blvd.  Suite 600
Chicago, Illinois 60607-2224
 
Submitted by:
American Institutes for Research
1000 Thomas Jefferson St., N.W.
Washington, DC 20007-3835
 
 
 
July 29, 2009
 
 
 
“American Institutes for Research” is a registered trademark. All other brand, product, or company names are trademarks or registered trademarks of their respective owners.
 
1000 THOMAS JEFFERSON ST, NW|WASHINGTON, DC  20007-3835|
TEL 202 342 5000|FAX 202 342 5020|WEBSITE WWW.AIR.ORG
 
 

Introduction 

This report arises from discussions between John Simmons, President of Strategic Learning Initiatives (SLI) and the American Institutes for Research (AIR) to provide an external validation of the impact data in ten Chicago Public Schools elementary schools that participated in SLI’s Focused Instruction Process between 2006 and 2008.   

AIR agreed to complete two tasks:
 

·         Rerun and validate the ISAT-Reading data for the 10 FIP schools; and

·         Compare the ISAT scores between the 10 FIP schools and a matching set of 10 non-FIP schools.

 

The results that SLI has achieved, and that AIR has validated, are very impressive and suggest that well before decisions are made to reconstitute schools under the mandates of NCLB, school districts would be wise to consider far less drastic, but clearly powerful, interventions such as the Focused Instruction Process.
 

Findings
 

1.      How well did the 10 FIP schools do during the first two years of intervention when compared with the Chicago city average?
 

Figure 1 shows that for the two-year period from 2006 to 2008 all but two of the 10 FIP schools had ISAT Reading gains in the percent of students at or above proficient that exceeded the Chicago city average and that as a group, the gains in the FIP schools were nearly twice the city average (11.4 percentage points vs. 6.3 percentage points).
 

Figure 1

Two-Year Gains in Percent of Students Meeting/Exceeding Reading Proficiency for 10 FIP Schools During FIP (2006-08)

 

School
   Gains
Cather
16.1
Faraday
14.2
Goldblatt
10.8
Morton
2.1
Tilton
14.2
Cardenas
18.7
Castellanos
6.0
Finkl
15.3
Gary
8.3
Kanoon
8.5
 
 
10 FIP Schools
11.4
 
 
Chicago City Average
6.3

Figure 2 shows the annual average gains in the FIP schools during the two-year period (2006-2008) during which the intervention was in place when compared with the annual average gains for the four-year period (2001-2005) prior to the intervention. Once again, the gains made during the period of the intervention are impressive relative to the gains prior to the invention and relative to non-FIP schools.

 

Figure 2

Annual Average Gains in Percent of Students Meeting/Exceeding Reading Proficiency Before (2001-2005) and During (2006-2008) the Intervention

 

 
Before FIP (2001-2005)
During FIP (2006-2008)
5 Area 7 FIP schools
0.8
5.7
5 Area 10 FIP schools
3.6
5.7
10 FIP schools
2.2
5.7
Non-FIP schools
2.1
3.2

 

 

2.      How well did the 10 FIP schools do when compared with a set of matching schools?
 

AIR asked SLI to identity a set of ten matching schools to the ten FIP schools. SLI used the following process to make these matches:

1.        Data from publicly available data bases from Chicago Public Schools and Illinois State Department of Education was extracted for the 10 FIP schools and all other elementary schools in Chicago, Illinois.  This data included:

a.       Percent low income as indicated by participation in free or reduced price lunch program.

b.      Percent of enrolled students who were identified as being in any of six specific racial/ethnic groups:  White, Black, American Indian, Asian, Hispanic, Multi-Racial.

c.       ISAT Percent Meeting or Exceeding Proficiency in Reading for grade level in the years 2001-2008.  It should be noted that standards of proficiency were altered between academic years ending in 2005 and 2006.

d.      From the ISAT data, Average Annual Gains/Losses in Percent Proficient were calculated for the 2001 to 2005 (4 year) difference and separately for the 2006-2008 (2 year) period.

2.        A computer program was written to initially screen for matches to the 10 FIP schools.  The program ignored a criterion that was originally considered:  No consideration was given to whether the schools matched were in the same city neighborhood (“Area”).

3.        The program initially searched for schools that had a Low Income Percent within 1.0 percent of the Low Income Percent of the FIP school to which they were to be matched.  In some cases, this resulted in short list of schools which would be difficult to match to the FIP school using the additional criteria to be applied.  Accordingly, a second run of the program was performed to select schools that would match within 2.0 percent of the Low Income Percent of the FIP schools.  This resulted in a longer list.  In subsequent matching, preference was given to schools that appeared on the “within 1.0 percent” match list.

4.        Using the schools that matched on the Low Income criteria, a statistic designated as “stress” was calculated consisting of the sum of squared differences in racial/ethnic percentages between the FIP school and each potential matching school.

5.        The list of matching schools for each of FIP schools was sorted low to high on this statistic.  The lower the “stress” statistic, the closer the match on racial/ethnic composition.

6.        Further matching was performed manually with the following criteria:

a.       Percent Meeting or Exceeding Proficiency in 2005 was roughly equal in FIP school and matched school.  This criterion was applied because improvement due to interventions will depend upon starting point prior to the intervention.

b.      Average Annual Percent Gain/Loss for the period 2001 to 2005, prior to FIP intervention, was roughly equal.

7.        Finally, some schools which would otherwise be considered as possible matches were excluded because they had participated in other Strategic Learning Initiatives programs.

 

Figure 3 shows the results of this matched pair analysis of FIP vs. non-FIP schools prior to and during the intervention.  As would be expected given the selection process, before the intervention there is a very small difference in the average annual increase in the percent of students meeting or exceeding reading proficiency between the FIP and non-FIP schools (2.24 percentage points vs. 2.45 percentage points respectively for the period 2001-2005).  However, during the intervention there is a significant difference in the average annual increase – 5.71 percentage points for the FIP schools vs. 1.14 percentage points for the non-FIP schools – for the period 2006-2008.  (See Appendix, Tables 1, 2, and 3 for summary statistics and p-values for the t tests.)  Before the intervention, the mean difference in the average annual increase of percent of students meeting or exceeding reading proficiency between matched FIP and non-FIP schools was -0.21.  The mean difference in average annual increase between matched FIP and non-FIP schools during the intervention (4.57) was significantly greater.  (See Appendix, Tables 4 and 5 for summary statistics and p-value for the t-test.)

 

Figure 3

Average Annual Increases and Differences in Test Scores

for Matched Pair FIP and Non-FIP Schools Before and During FIP

Pair
FIP School
Non-FIP School
FIP School: Average Annual Increase Before FIP (2001-2005)
Non-FIP School: Average Annual Increase Before FIP (2001-2005)
FIP School: Average Annual Increase During FIP (2006-2008)
Non-FIP School: Average Annual Increase During FIP (2006-2008)
Difference in Average Annual Increase in Matched FIP and Non-FIP School Before FIP
Difference in Average Annual Increase in Matched FIP and Non-FIP School During FIP
1
CATHER
DEPRIEST
2.78
2.90
8.05
2.40
-0.13
5.65
2
FARADAY
O'TOOLE
1.25
2.23
7.10
-1.40
-0.98
8.50
3
GOLDBLATT
HERZL
-0.13
0.45
5.40
3.70
-0.58
1.70
4
MORTON
O'KEEFFE
0.10
0.05
1.05
0.25
0.05
0.80
5
TILTON
OWENS
0.20
-0.93
7.10
1.65
1.13
5.45
6
CARDENAS
LARA
2.78
2.45
9.35
2.50
0.33
6.85
7
CASTELLANOS
COOPER
7.30
3.65
3.00
-0.55
3.65
3.55
8
FINKL
STOWE
3.15
4.33
7.65
2.10
-1.18
5.55
9
GARY
MCCORMICK
2.90
4.55
4.15
-2.15
-1.65
6.30
10
KANOON
WHITNEY
2.08
4.85
4.25
2.90
-2.78
1.35
 
All FIP schools
Matched non-FIP schools
2.24
2.45
5.71
1.14
-0.21
4.57

 

Figure 4 shows the average change in the percent of students meeting or exceeding reading proficiency between each year in the 10 FIP schools and the 10 matched non-FIP schools for the entire 2001-2008 period and reveals graphically the apparent impact of the FIP intervention.  

 

 

Figure 4
 

 

Note: The test format changed significantly from 2005 to 2006 so the 2005 to 2006 test score change is not included

 

Conclusion

 

It is clear, on the basis of the ISAT Reading scores for the percent of students meeting or exceeding proficiency in ten Chicago elementary schools for the period 2001-2008, that the Focused Instruction Process intervention has had a positive and significant impact on student achievement in the cohort of ten schools that participated in the FIP model from 2006 to 2008.  Whether compared to pre-intervention achievement, or to the entire set of Chicago elementary schools, or to a carefully selected set of matched schools, the data suggest that FIP has resulted in gains that are very unlikely to have occurred without the intervention.  

                       


APPENDIX

 

 

Table 1: Mean Average Annual Increase in Scores Before and During FIP

 for 10 FIP Schools

Variable
N
Mean
Std Dev
Minimum
Maximum
Average annual increase
before FIP (2001-2005)
10
2.24
2.18
-0.13
7.30
Average annual increase
during FIP (2006-2008)
10
5.71
2.58
1.05
9.35

 

 

Table 2: Mean Average Annual Increase in Scores Before and During FIP

for 10 Matched Non-FIP Schools

Variable
N
Mean
Std Dev
Minimum
Maximum
Average annual increase
before FIP (2001-2005)
10
2.45
2.01
-0.93
4.85
Average annual increase
during FIP (2006-2008)
10
1.14
1.98
-2.15
3.70

 

 

Table 3: t-test for Mean Average Annual Increase in Scores Before and During FIP

for 10 FIP Schools and 10 Matched Non-FIP Schools

Statistics
 
 
 
 
 
 
 
 
 
 
 
Variable
FIP
N
Lower CL
Mean
Upper CL
Lower CL
Std Dev
Upper CL
Std Err
Minimum
Maximum
 
 
 
Mean
 
Mean
Std Dev
 
Std Dev
 
 
 
beforeave
0
10
1.01
2.45
3.89
1.39
2.01
3.68
0.64
-0.93
4.85
beforeave
1
10
0.68
2.24
3.80
1.50
2.18
3.98
0.69
-0.13
7.30
beforeave
Diff (1-2)
 
-1.76
0.21
2.18
1.59
2.10
3.10
0.94
 
 
afterave
0
10
-0.28
1.14
2.55
1.36
1.98
3.61
0.63
-2.15
3.70
afterave
1
10
3.86
5.71
7.56
1.78
2.58
4.72
0.82
1.05
9.35
afterave
Diff (1-2)
 
-6.73
-4.57
-2.41
1.74
2.30
3.40
1.03
 
 

 

 

T-Tests
 
 
 
 
 
Variable
Method
Variances
DF
t Value
Pr > |t|
beforeave
Pooled
Equal
18
0.23
0.8235
afterave
Pooled
Equal
18
-4.44
0.0003

 

 

Table 4: Mean Difference in Average Annual Increase

in Matched Pair FIP and Non-FIP Schools Before and During FIP

Variable
N
Mean
Std Dev
Minimum
Maximum
Difference in Average Annual Increase in Matched FIP and Non-FIP School Before FIP
10
-0.21
1.74
-2.78
3.65
Difference in Average Annual Increase in Matched FIP and Non-FIP School During FIP
10
4.57
2.59
0.80
8.50

 

 

 

Table 5: Paired t-test for Mean Difference in Average Annual Increase

in Matched Pair FIP and Non-FIP Schools Before and During FIP

Statistics
 
 
 
 
 
 
 
 
 
 
Difference
N
Lower CL
Mean
Upper CL
Lower CL
Std Dev
Upper CL
Std Err
Minimum
Maximum
 
 
Mean
 
Mean
Std Dev
 
Std Dev
 
 
 
afterdiff - beforediff
10
2.56
4.78
7.01
2.14
3.11
5.67
0.98
-0.10
9.48

 

T-Tests
 
 
 
Difference
DF
t Value
Pr > |t|
afterdiff - beforediff
9
4.87
0.0009