About Catalpa

Experience

Program and Policy Evaluation

Chicago Public Education Fund and The Spencer Foundation

Introduction
Practice-Based Inquiry is well tested as a methodology for studying the performance of a single professional-service institution.

But since it is a methodology of inquiry about practice, it has considerable potential for other research settings. Two Chicago foundations, both with an interest in finding new ways to generate information that would help them evaluate grants and provide implementation advice to funded projects, supported two program and policy evaluation applications of Practice-Based Inquiry, partly to test the value of Practice-Based Inquiry as a program and policy evaluation methodology.

The Policy Study of National Board Certification Program
The Chicago Public Education Fund had provided support for several years, along with other Chicago foundations, to training and support programs run by three different agencies to support Chicago teachers who sought National Board Certification as master teachers. Demands for greatly increasing the numbers of Board Certified Teachers in Chicago raised questions for the Fund about what had been learned about effective program practices and whether Board of Education policies that shaped these programs were adequate.

In March 2005, The Fund engaged Catalpa to design and carry out a study that utilized Practice-Based Inquiryfor a team of local and national experts on National Board Certification.
The study addressed these issues:

• How to make judgments about the effectiveness of program components;
• How to understand the constraints and possibilities present in the political, institutional and funding context surrounding the programs;
• How to decide whether program components served as steps for the candidate to climb up or stimulated the candidate to learn how to effectively improve how he actually taught in the classroom.

Practice-Based Inquiry ^® did provide a solid methodological base and technology for this study and the reported findings. The report of this Catalpa Study, "Strengthening National Board Certification in Chicago Public Schools," is copyrighted and distributed by the Chicago Public Education Fund.

Program Evaluation Study of the Academy of Urban School Leadership
The Chicago Public Education Fund and The Spencer Foundation asked Catalpa to take the lead in the design and conduct of this uniquely designed program-evaluation study. The Center on Research in Education Policy at the University of Memphis is the fourth collaborator in this study. 

For three years, The Academy of Urban School Leadership (AUSL) had run an innovative teacher training program to train and educate people with prior careers to teach successfully in Chicago public schools. AUSL ran two public schools to provide controlled sites for year-long residencies for the new teachers in training.

In fall 2004, AUSL launched a new initiative aimed at demonstrating the impact of its approach on the teaching and learning in Chicago neighborhood schools that had been designated "in need of improvement." Large cohorts of AUSL graduates were hired at four neighborhood schools as part of that the faculty at each of these schools.

• How much and what kind of influence has AUSL had on the four neighborhood schools in the first year of the cohort initiative?
• Did learning and teaching improve?
• Did AUSL ideas spread to influence the regular faculty of the school?

The design of this study provided an extensive test of the value of Practice-Based Inquiry as a program evaluation methodology. First, one of the two sources of data for the study was a special Practice-Based Inquiry visit. The team of thirteen experienced Chicago teachers visited and wrote conclusions about each of the four neighborhood schools and the two training schools. Second, data was also collected about learning and teaching at the six schools through more conventional social science instruments including surveys, observations and interviews. The Center on Research in Education Policy (CREP) at the University of Memphis carried out this data collection. Both the Practice-Based Inquiry visit team and CREP produced school based reports of findings about the central study questions. A final Overview report is now under preparation that combines the findings from the two sources of data.

While the AUSL Practice-Based Inquiry visit protocol met the guidelines for a Practice-Based Inquiry protocol in terms of being consistent with Practice-Based Inquiry principles and procedures, it did stretch the complexity and volume of the team's task beyond other Practice-Based Inquiry protocols. The amount of work the AUSL visit required of the team (six days instead of five), the number of institutional settings it had to understand at some level (six instead of one) and the number of conclusions the team had to prepare (45 instead of 18) were all more than previous protocols required. In addition to conclusions about the functioning of a single organization, the AUSL protocol required teams to form cross-institution conclusions about the impact of AUSL on the learning and teaching at each school.

Catalpa carefully monitored the AUSL visit to identify design flaws that could be fixed in future program evaluations using Practice-Based Inquiry .

The AUSL visit and this report demonstrate that Practice-Based Inquiry has great potential as an effective program evaluation methodology. While there were several aspects of the visit that would be refined, Practice-Based Inquiry proved to have great potential for program evaluation.

Next:
Public Speaking and Workshops »

Back:
Protocol Revision for Accreditation Agencies »

ˆ back to top