Carnegie Commons

A place to come together to exchange ideas about teaching and learning

Thursday

17

July 2014

How to Spur Improvement Activity in Networks

Written by , Posted in What We Are Learning

Over the past five years, the Carnegie Foundation for the Advancement of Teaching has launched a set of three Networked Improvement Communities (NICs). We have played roles in the launch and support of two NICs in particular, the Building a Teaching Effectiveness Network and the Student Agency Improvement Community.

NICs are scientific learning communities distinguished by four essential characteristics. They are: (1) focused on a well specified common aim; (2) guided by a deep understanding of the problem and the system that produces it, and a shared theory of how to improve it; (3) disciplined by the rigor of improvement research; and (4) coordinated to accelerate the development, testing, and refinement of interventions and their rapid diffusion out into the field, as well as their effective integration into varied educational contexts. These characteristics create conditions under which organizations can learn from their own practices and benefit from innovations from both within and outside of their organization to systematically address high-leverage problems.

Through the initiation and development of several NICs, Carnegie has gained some insight into what it takes to spur improvement activity in networks:

Start with a high-leverage problem of practice
A high-leverage problem of practice is an issue that, if addressed, can disrupt status quo practices in an organization and render improvements throughout the system. This is a compelling problem area that, if solved, will propel the organization toward achieving its core mission. Long-term leadership and stakeholder commitment to solving this problem is critical to the success of the NIC, and the determination to work on a recognized high-leverage problem (as perhaps one that the organization has struggled with for a while) can do much to evoke that commitment and will. Moreover, the process for selecting and messaging the NIC’s high-leverage problem needs to be transparent and evidence-based.

Build on work already being done
With more rigorous standards, evolving policy demands, and tight budgets, school personnel are striving to realize increasingly ambitious objectives with limited resources. The desire to apply improvement science in networks often outstrips school personnel’s capacity to conduct such work. To introduce improvement work into a district, we would recommend beginning the improvement activity in an existing team or learning community. Infusing improvement methods into existing collaborative structures (e.g., partnerships, meeting structures, conceptual framings) adds capacity to work already being done as opposed to adding “one more thing” to the work of educators who are already operating at full capacity.

Assemble a diverse team
We have found it efficient to start improvement work in existing collaborative structures, but it is also true that solving high-leverage problems in schools will require a range of perspectives and levels of expertise. Given the interdependencies of processes in complex systems, often the processes of improvement will uncover previously overlooked drivers of the problem. It is not uncommon for new team-members to be added as the improvement work evolves. Organizations can be hierarchical, but in NICs, each member brings an essential perspective to solving the problem at hand. In fact, in improvement work, it is often the case that those workers closest to the “front line” are those with the best ideas about how to solve the problem. NICs can foster a spirit of co-development by demonstrating openness to feedback and rapidly integrating and testing member ideas in the network.

Provide access to improvement guidance
Improvement science offers a set of new frameworks and methods for approaching work. As with most newly acquired skills, users will struggle in the process of integrating this approach into real-life contexts. It is imperative to ensure that just-in-time feedback and support is provided reliably in order to scaffold learning and help members see the value in the improvement work. Early wins will also help to build will for the work in the organization.

Balance in-person and virtual communication
Launching a network is best done in an in-person convening of network members and stakeholders. Often convenings can galvanize enthusiasm around solving the high-leverage problem and build momentum for the work. However, it is often the case that that enthusiasm can wane when members return to face the challenges of their daily routines and momentum often flags. A collaborative online platform can foster continued communication to build upon the sense of community garnered at the first convening. It can also provide for the sharing and spread of what is learned through ongoing improvement efforts.

Certain conditions are obvious prerequisites for seeing improvement work gain traction in practice: building the improvement capabilities of professionals through training and ongoing coaching, for example, or creating the infrastructural capacity by establishing a supporting Hub that provides supports for improvement science, collaborative work, knowledge management, etc. The items introduced here have emerged in our work as more particular issues that require attentions if improvement work is to be pursued in a manner that is deep, widespread, and enduring.

Wednesday

28

May 2014

Iowa Mentoring Program Targets Needs of Beginning Teachers

Written by , Posted in What We Are Learning

As Carnegie Senior Associate Susan Headden writes in her recent report “Beginners in the Classroom,” public education loses a lot of new teachers to attrition, upwards of 750,000 a year, and pays a heavy price in talent and treasure. They leave for many reasons, Headden reports, but at the top of the list are concerns related to lack of support, such as limited professional development, little helpful feedback on performance that supports improvement, and feeling isolated from colleagues.[1]

Mentoring programs for new teachers may help address these issues. Effective mentoring programs, research suggests, promote new teachers’ sense of professionalism and hence their satisfaction and retention. Such programs can improve teachers’ instructional abilities and thereby increase their students’ achievement.[2]

Of course, not all mentoring programs are created equal. Among the success stories is Iowa’s Grant Wood Area Education Agency (AEA). In 2000, Iowa passed a law requiring that every new teacher have a mentor. Today’s iteration of the program benefits from knowledge gleaned from early mistakes. In the beginning, mentor teachers were given a stipend, but no training or release time from their own classroom duties in order to meet with mentees. The program had no oversight, the mentors were accountable to no outcomes, and no data were collected on implementation or results. Turnover among new teachers remained high.

Currently, however, mentors are released from their classrooms for three years; they are full-time mentors who stay with the same group of mentees for two years. Mentor selection is rigorous. Each applicant is interviewed multiple times, illustrates their ability to create model lessons, provides assessments of student work, and writes essays to show evidence of his or her capacity for reflection, a necessary skill for mentor success. Perhaps the most impressive component of the program is the training provided to Iowa mentors through the New Teacher Center, a non-profit organization that helps train new teachers. Sessions are differentiated for both new and expert mentors, and there are options for administrators as well. During training, mentors complete assignments in conjunction with their beginning teachers as well as reflecting on their own assignments.

Collectively, the Grant Wood AEA’s model includes what beginning teachers need in order to feel supported: instructional guidance, frequent and actionable feedback, and meaningful relationships within the school. Careful selection of mentors, as well as the ongoing training they receive, sets mentors up to succeed, giving them the instructional and reflective tools they need to meet their mentees wherever they are in their practice. Mentors must periodically submit evidence of their meetings with mentees, ensuring that new teachers are in fact getting individualized and ongoing feedback from their mentors to help improve their practice.[3] Providing full release time to mentors means that they can spend a significant amount of time with each mentee, forming meaningful relationships based on trust and support. This last piece is perhaps the most important, since these relationships help to tie new teachers to their schools, sustain them in the difficult work of beginning teaching, and keep high-performers in the profession.

Though it is too early to determine the long-term effects of the program, feedback from teachers, mentors, and principals has been overwhelmingly positive. Officials are collecting data on the implementation and impact of the program, including information on which skills mentors are helping their mentees develop. They have discovered that, early in the school year, new teachers’ primary concerns are classroom management and instructional planning—valuable insight that can help schools target future professional development efforts. Data collected thus far show that all mentors are spending 60-90 minutes per week with each mentee. And, critically, beginning teacher attrition is low; of the 33 new teachers who mentors worked with in the 2012-2013 school year, only two have left.[4] Time will tell if Grant Wood AEA’s mentoring program has a lasting effect on teacher quality or teacher turnover, but based on initial results and feedback, it seems that it is providing beginning teachers with the support and sense of belonging they need in order to improve and to stay in the profession.


[1] Beginners in the Classroom, pg. 5.

[2] Richard Ingersoll and Michael Strong, “The Impact of Induction and Mentoring Programs for Beginning Teachers: A Critical Review of the Research,” Review of Education Research. Vol. 81, 2 (2011): 201-233. Retrieved from: http://repository.upenn.edu/gse_pubs/127

[3] Grant Wood AEA, “Mentoring and Induction Program.” Accessed April 28, 2014. http://www.aea10.k12.ia.us/leadership/mentoracademy/

[4] Beginners in the Classroom, pg. 22.

Friday

2

May 2014

Is a Networked Improvement Community Design-Based Implementation Research?

Written by , Posted in What We Are Learning

A new NSSE yearbook chapter, co-authored by Jon Dolle, Louis Gomez, Jenn Russell, and Tony Bryk sheds light on Carnegie’s approach to building professional communities as Networked Improvement Communities and its relationship to design-based implementation research (DBIR). This post summarizes and builds upon that chapter.

For the last five years, Networked Improvement Communities (or NICs) have been at the center of Carnegie’s work. Many observers have been uncertain how to categorize NICs within the field of education research. Design-based implementation research (DBIR), in particular, bears a family resemblance to a portion of the work done by NICs. But NICs are not a research approach, and their raison d’être is not theory building. Here is a brief exploration of similarities, differences, and the productive relationship that can exist between the two.

Similarities
The umbrella of DBIR covers research that generally adheres to four principles:

(1)   “A focus on persistent problems of practice from multiple stakeholders’ perspectives;

(2)   “A commitment to iterative, collaborative design;

(3)   “A concern with developing  theory and knowledge related to both classroom learning and implementation through systematic inquiry;

(4)   “A concern with developing capacity for sustaining change in systems.”

When broadly interpreted, these principles characterize many of the activities in which NICs engage. Carnegie’s Community College Pathways NIC, for example, is organized around the instructional challenges of diverse community college faculty (Principle 1). Its improvement work is conducted through rapid Plan, Do, Study, Act (PDSA) cycles and supported by a variety of analytic approaches (Principles 2 and 3). As Pathways members and leadership test new change ideas and learn more effective implementation strategies, this knowledge gets represented in many different forms, including revised driver diagrams (our theory of change), updated change packages (a mechanism for sharing changes), as well as published reports and white papers (Principle 3). And all of the Pathways work is focused on capacity building with the goal of systems change (Principle 4).

Given these similarities, the temptation to classify NICs as a form of DBIR is understandable.

Differences
Carnegie’s resistance to categorizing NICs as a research approach can be stated succinctly: a NIC is a professional community structured around the accomplishment of a shared improvement aim. It is not an approach to research, though NICs use research as an essential aspect of their work and, on occasion, engage in research themselves. Just as it would be odd to categorize a network of hospitals as an approach to clinical research, the DBIR label fits some NIC activities but it is not their reason for being. In both cases, networks use and sometimes engage in research, but they are not research networks. On its own, producing new and better knowledge is rarely sufficient to affect system-level improvement. NICs are a mechanism for making new knowledge a live resource within a system.

Beyond this fundamental difference in purpose, there is another reason to distinguish NICs from DBIR. As improvement-oriented social organizations, NICs prioritize practical “know how” over theoretical “knowledge that” something might improve a system. The only way to bridge the evidentiary gap between “knowledge that” and “know how” is to learn through the process of actively changing a system. NICs learn about practice by actively trying to improve it. All the elements of a NIC (its membership, its aim, its theory of action, its core capacities, etc.) are organized around enabling the kind of system learning necessary for effective and reliable improvement at scale.

We posit that there are at least four network capacities that can enable distributed improvement work:

  • A rapid analytics infrastructure is a core capacity of the hub that helps collect, manage, analyze, and share data across the network.
  • Common tools and routines that enable disciplined inquiry are critical to coordinating member activities across a dispersed professional network. They facilitate network learning and engagement that is essential to scaling improvement within an education system.
  • Innovation conduits are the way promising ideas inside or outside of the network are identified, tested, refined, and scaled.
  • A culture that embraces a collaborative science of improvement supports the development of professionals committed to collaborative inquiry around a shared problem.

A Productive Relationship
Because DBIR is a research approach, its primary knowledge products are familiar: new, empirically grounded theories and explanations of social phenomena. DBIR recommends developing these theories and explanations in close partnership with practitioners, as well as “developing the capacity of the entire system to implement, scale, and sustain innovations” (Fishman et al, p. 145). However, research typically doesn’t develop capacity on its own. (If it did, academic journal subscriptions would likely exceed those of major newspapers and pop culture magazines!) Consequently, DBIR needs a coordinating entity with the capacity for intelligent integration of the knowledge that it produces into a system. NICs are one such coordinating mechanism.

The confusion over the relationship between NICs and DBIR arises because NICs do, in part, engage in inquiry that can fit under the umbrella of DBIR, and also because the different aims of this inquiry are easily confused. The knowledge that results from academic theory building may or may not develop capacity within a system. Academic theory often plays an important role in improvement efforts, especially as a resource for testing and innovation: it can help improvers understand problems of practice, guide the development of practical theories, and generate change ideas for testing. Research is conducted as a means of making progress towards an improvement aim, but the ends of a NIC—what the community agrees to hold itself collectively accountable for—is the improvement of practice at scale. Theory building is a priority only to the extent that it advances this aim or our collective capacity to pursue such aims.

As the body of knowledge produced by DBIR grows, NICs are a natural mechanism for making these theories a vital resource for improvement within and across educational systems. NICs will also likely contribute to this body of knowledge, but only in so far as it advances shared improvement aims or enhances the collective capacity to improve.

###

To learn more about DBIR, check out the other chapters in the NSSE volume, as well as two excellent articles by Bill Penuel, Barry Fishman, and colleagues.

Monday

14

April 2014

Building a High-Quality Feedback System that Supports Beginning Teachers

Written by , Posted in What We Are Learning

In 2011-2012, nearly a quarter of the teachers in the U.S. had five or fewer years of experience and nearly 7 percent were brand new to the profession.[1]  While novice teachers bring new skills, perspectives, and energy to their schools, they also tend to leave the profession at high rates, with nearly half leaving the classroom in their first five years.[2],[3]

At a recent event hosted by Carnegie’s Washington, D.C., office, this statistic was brought to life as three beginning teachers reflected on their futures in the classroom: one was committed to staying, one was committed to leaving, and the third was looking beyond the classroom to an administrative role where he hoped he might “have an even bigger impact” on students’ lives.

Invited to respond to “Beginners in the Classroom,” a report on the condition of beginning teachers by Carnegie Senior Associate Susan Headden, these teachers spoke candidly about their experiences as novice teachers. And though the contexts in which they entered the teaching profession differed—Lauren Phillips cut her teeth as a New York City Teaching Fellow, Rene Rodriguez as a Capital Teaching Resident in Washington, D.C., and Diana Chao as a university-trained teacher in Montgomery County, Md.—all three agreed that their first year might have been improved by more frequent and more actionable feedback from the instructional leaders at their schools.

Research suggests feedback might do more than simply improve teachers’ early experiences and performance in the classroom; it might help convince them to stay, too. In a 2012 study by TNTP, top-performing teachers who experienced supportive, critical feedback and recognition from school leadership stayed in their schools for up to six years longer than top-performers who did not receive such attention.[4]

Likewise, in a survey of 580 teachers in the Baltimore City Public School system, researchers from Carnegie’s Building a Teaching Effectiveness Network (BTEN) found that teachers who felt engaged in their schools and were made to feel confident about their classroom contributions were significantly more likely to stay at their schools. Among the 25 percent who felt least confident and least engaged in their school communities, fewer than half were likely to stay the following year.

Despite growing evidence that high-quality feedback—feedback that builds trust and leads to improvements in teaching and learning—may be a crucial lever for increasing teaching quality and retention rates, providing such feedback has proven a significant challenge in America’s school systems. Even in districts that have made instructional improvement a priority, the feedback teachers receive is often infrequent, inactionable, and incoherent.

feedback-components

Components of a Prototypic Feedback Process

Carnegie addresses this challenge in its latest publication, Developing an Effective Feedback System, which aims to help districts rethink feedback not simply as a series of isolated conversations between principals and teachers, but rather as a complex system of many interconnected factors at the district, school, and classroom level—all of which shape the nature of feedback teachers receive.

Drawing on scholarly research and in-depth interviews with expert practitioners, the report provides a framework of key drivers—processes, norms, and structures—that should be in place at each level for a district to maintain a coherent, high-quality feedback system that can drive improvement in teaching quality and contribute to the retention of teachers who are successful.  A clear instructional framework, training and support for feedback providers, coherent and coordinated feedback, and a trusting culture committed to continuous learning are among the key drivers explored in greater depth.

To provide even greater direction for school-based educators working with new teachers, the paper also outlines components of a model feedback process, including concrete steps principals and coaches can take to coordinate and improve the interactions they have before, during, and after feedback conversations with novice teachers.  These are conversations that, according to the panelists on Carnegie’s recent panel, tend to lack substance, if they occur at all. And they are conversations that, if done well, have the potential to improve new teachers’ practice and, hopefully, keep them in the classroom for the long haul.


[1] NCES Schools and Staffing Survey, 2011-12.

[2] Matthew Ronfeldt, Susanna Loeb, and James Wyckoff, “How Teacher Turnover Harms Student Achievement,” American Educational Research Journal 50, no. 1 (2013): 4–36. Retrieved from: http://aer.sagepub.com/content/50/1/4.

[3] Richard Ingersoll and Lisa Merrill. Seven Trends: The Transformation of the Teaching Force. Consortium for Education Policy Research. (2013) Retrieved from: http://www.cpre.org/sites/default/files/workingpapers/1506_7trendsapril2014.pdf.

[4] TNTP. “The Irreplaceables” (2012).

Tuesday

25

March 2014

How to Change Things When Change is Hard

Written by , Posted in What We Are Learning

Dan Heath, author of Switch: How to Change Things When Change is Hard, speaking at Carnegie’s Summit on Improvement in Education, acknowledged to those working toward positive change in education that a new approach might be in order. He said that instead of change being about interventions, as we usually approach it now, it is instead about the “long game” of changing direction through motivation.

elephant-riderHeath framed his talk around the compelling elephant-rider analogy—an explanation of the two (often at odds) sides of human nature—borrowed from University of Virginia psychologist Jonathan Haidt. The analogy suggests that everyone has two sides—a rider and an elephant. The rider represents the rational thinker, the analytical planner, the evidence-based decision-maker. The elephant, on the other hand, is an emotional player, full of energy, sympathy and loyalty, who stays put, backs away, or rears up based on feelings and instincts. The elephant is often on automatic pilot. It is the part of the brain that tells us to go ahead and eat the ice cream, after the rider has decided to put us on a diet. Although the rider holds the reins and appears to lead the elephant, the six-ton elephant can, at any time, overpower the rider and the rider, although he may not know this, can’t force the elephant to go anywhere unless he appeals to him and motivates him in some sustainable way. “In order to change the elephant, we have to appeal to a felt need,” Heath said. “Sparks come from emotion, not information.”

Nowhere is the elephant-rider dilemma clearer than in education reform. Policymakers are classic riders, pointing straight ahead and asserting all the while that “this is the right way, the clearest best path … I’ve got this beast under control.” Researchers, too, often act as riders, captivated by their carefully collected data and certain that their objective findings will prove compelling. Meanwhile, educational systems—slow, strong, passionate elephants that they are—plod along, sometimes responding to the switch of the rider and other times arching their backs in resistance. These elephants may try out a few different trails, lumber up a few small hills, but can buck that rider off at any point. After all, they’ve been living on this land for years. Riders come and go.

Don’t ignore the elephant, Heath urges us. The rider can’t just rely on a carefully charted, smartest, best path. He also must appeal to the elephant’s motivations. Good teachers understand this—they don’t get their students to read Shakespeare by telling them that it’s part of a canon that they need to know to be an educated person and be “college- and career-ready.” They show them Shakespeare is about love and hate and all of the raw emotional experiences they’re having in their own lives. And, he said, Carnegie understands this, citing our Productive Persistence interventions that reinforce a student’s sense of belonging as necessary to move the needle on student success in developmental mathematics.

The elephant also needs a well-directed rider, one who can see clearly beyond the trees and steer through the fog to what Heath calls “bright spots”. These spots are not the schools with 100 percent high achievers, but the places where small changes are making big differences. A good rider can lead an elephant to the bright spots since, as Heath explains, what looks like resistance is usually just a lack of clarity.

In the end, Heath leads us to a simple but important lesson for educational change: We need to get our riders and elephants in sync. That means finding smart, evidence-based paths to improve education, and finding those paths of least resistance. He again cited Carnegie’s efforts in developmental mathematics, noting instead of merely changing the course materials, Carnegie developed pathways that would get students through a college-credit math course in one year, instead of the current multi-year path where students often drop out between quarters or semesters.

“For change to succeed,” Heath concluded, “there are three ingredients. We need paths shaped for clear and easy passage. We need riders who know where to go and can see the bright spots. And, perhaps above all, we need enthusiastic elephants.”

Monday

17

March 2014

Performance Assessment for Teachers and of Teachers: Combining the Development of Teaching with Teacher Evaluation

Written by , Posted in What We Are Learning

Editor’s Note: In a previous post by Lee Nordstrom, he warns that we should not conflate evaluation and improvement processes, but he also points out that systems of evaluation and improvement are not mutually exclusive. This post explores the conditions under which these two systems might be coherently integrated.

The national push to revamp systems of teacher evaluation has spurred a growing call for the need to attend to teacher development, not just evaluation. But can school leaders effectively evaluate teachers and simultaneously support their growth? Are these goals too contradictory to combine, or can a single system support both efforts?

In a recent 90-Day Cycle[i] conducted at the Carnegie Foundation, we explored the question of if teacher evaluation and teacher development efforts can and should be combined as aspects of a single system.  From 20 expert scholars and practitioners in education and several key pieces of literature, we heard an emphatic “yes.” These experts argued not only that is it possible to address both goals in a blended way, but it is preferable for these two efforts to be woven together. One important caveat was that the larger school cultural context within which these processes occur matters greatly. If the school culture is focused on professional growth, the potential success of combining evaluation with improvement is much greater than if such a development-oriented culture does not exist.

Combining Formative and Summative: Reconciling Characteristics of Assessment Across Purposes

The scholarly conversation about whether and how formative and summative assessments can be combined has been ongoing for decades in the field of education. There is a widely shared understanding among experts that the differing aims of formative and summative assessment lend themselves to different characteristics of an assessment system. Some of the more prominently discussed characteristics include:

  • the grain-size of the data—formative: specific and actionable; summative: broad and global;
  • the frequency of assessments and feedback—formative: frequent; summative: infrequent;
  • the importance of reliability of data—formative: less critical because context-specificity is valuable; summative: important because valid global and uniform conclusions depend on high reliability;
  • the criteria by which to make judgments about students’ learning—formative: dependent on context and the individual’s own past performance; summative: criterion- or norm-referenced to enable uniform judgments across learners.

While these characteristics may appear contrary across the two purposes of assessments, some scholars assert that these differences are not mutually exclusive.  In “Systems of Coherence and Resonance: Assessment for Education and Assessment of Education,” authors Paul LeMahieu and Elizabeth Reilly point out that some characteristics are necessary for a particular purpose, while others are common but not required. They give the example that frequent feedback is necessary for formative assessment, but summative assessments do not require infrequency—they can also be frequent. In the same vein, in “Assessment and Learning: Differences and Relationships between Formative and Summative Assessment”  Wynne Harlen and Mary James assert that detailed, context-specific data are requirements for formative assessment, but that these data can be aggregated over time to produce a holistic perspective and more reliable data for summative purposes. Summative assessments do not require strictly general and non-specific evidence—even if they are often informed by such data.

What we must differentiate when formative and summative assessments are combined is the lens through which judgments are made. Formative assessments should depend on learners’ own past performance and the particular context of assessment, while summative assessments should be judged against external standards or norm-referenced criteria, so that uniform judgments are made across all learners. The critical point for this discussion is that with thoughtful operationalization, the evidence and the mode of data collection that serves formative purposes can also function for summative purposes with the aggregation of fine-grained and frequent data.  The differentiation comes when making inferences and determining next steps, which require different lenses, but do not necessitate entirely separate systems.

Practitioners Call for Combining Improvement and Evaluation Efforts

In addition to technical characteristics of assessment systems, there is a set of issues articulated by the individuals who experience and utilize processes of assessment and feedback. In a study, “Seeking Balance Between Assessment and Support,” of 83 teachers in six high-poverty urban schools, Stefanie Reinhorn found that most teachers said they want to be evaluated and that the evaluation should be connected to support in the same process. These teachers explained that the combination of evaluation and support led to a professionalization of their work, holding all teachers to clear and high standards. Other experts have also found that teachers prefer to be evaluated by someone who knows their practice well and who has seen their growth over time, rather than an evaluator who visits their classroom infrequently.

On the other side of the feedback relationship, feedback providers also described a preference for combining support and evaluation. These experts explained that teachers are more likely to take feedback seriously and to make changes in their practice when the feedback is connected to evaluation. This is especially the case when the feedback includes critiques of the teacher’s current practice. Brian Yusko and Sharon Feiman-Nemser make this point in their study of two induction programs, “Embracing Contraries,” describing how the feedback from Consulting Teachers (CTs) in Cincinnati had “teeth,” since there were consequences if teachers did not act on the CTs’ feedback.

Experts also discussed some unintended negative consequences of a system where evaluation and development are separated. In such a system, teachers are left to their own devices to “connect the dots” between the multiple sources of feedback. Especially for early career teachers, this may prove to be challenging, leaving teachers feeling overwhelmed or confused.  When coaches and evaluators are not able to align their feedback for teachers, they are also prevented from combining and coordinating their strengths. In a system with a firewall, feedback providers with specific expertise cannot easily enhance the work of their colleagues who lack this expertise through a team-based approach to providing feedback.

Building Trust in the Presence of Evaluation

Trust between teachers and feedback providers is essential for transparency of practice, communication, and the uptake of recommendations that can lead to the improvement of teaching. A reason often given for separating development and evaluation efforts is that teachers will feel more comfortable sharing their practice with someone who is not also responsible for evaluating them. The experts we consulted agreed with the importance of trust to promote transparency and growth, but they argued that whether teachers trust their feedback providers does not depend on whether she does or does not also evaluate.  Instead, they explained that trust depends on whether teachers see the feedback providers as effective aides to their professional growth who are genuinely committed to supporting them. Yusko and Feiman-Nemser found this to be the case for CTs in Cincinnati, who both evaluate and support teachers’ development.  CTs reported that their relationships with early career teachers usually developed trust over time, even though they evaluate the teachers.

Next Steps

The experts whom we consulted laid out a strong set of arguments that it is possible and even preferable for efforts of teaching development to be combined with teacher evaluation. There is research that supports these assertions. But this says little of how school leaders should combine these efforts in their day-to-day practice. CTs in PAR programs offer one powerful example, and we should leverage what we can learn from their work.  However, this is one model, and there is also a need to document and explore other examples in other contexts that can serve as practical guidance for school leaders. Collecting and condensing the wisdom from the field about how, concretely, to combine efforts of evaluation with efforts of teaching improvement should be a next step in this line of inquiry.  Then, taking an improvement science approach, school leaders interested in moving towards an effective model of combining evaluation with development can test these practices in their contexts to ultimately serve the goal of improved teaching and learning in their schools.


[i] 90-Day Cycles are a disciplined and structured form of inquiry adapted from the work of the Institute for Healthcare Improvement (IHI).  90-Day Cycles aim to:

  • prototype an innovation, broadly defined to include knowledge frameworks, tools, processes, etc.;
  • leverage and integrate knowledge from scholars and practitioners;
  • leverage knowledge of those within and outside of the field associated with the topic; and
  • include initial “testing” of a prototype.

Thursday

30

January 2014

Building the Capacity for Districts to Continuously Improve

Written by , Posted in What We Are Learning

Continuous improvement organizations are not common in the field of education, but a few organizations have initiated improvement methodologies borrowed from methods that have been used in other fields and found them to align well with efforts to improve teaching and learning outcomes.  The vibrancy and dynamism that characterize continuous improvement stand in stark opposition to the approach to improvement in many schools today.  Standards-based reform and teacher accountability systems typify the field’s approach to improvement – large-scale, top-down interventions largely devoid of clearly articulated pathways for improvement.

Continuous improvement presents a new approach to work that is not about being “soft.”  It brings accountability to the system and offers a tested framework and rigorous methodology for advancing the work of teaching and learning.  It taps the wisdom of the members of the entire organization and empowers them to systematically move toward better learning outcomes for all students. In particular, continuous improvement empowers members of the system to grapple with the underlying complexity in schools to undertake the steady work required to catalyze and sustain improvements.

Carnegie’s Advancing Teaching-Improving Learning program hosted a convening of experts in continuous improvement methodology to understand what it takes to approach improvement efforts in the K-12 space in this way. We identified four essential organizational conditions for continuous improvement to take root and thrive: constancy of purpose, a culture of improvement, standard work, and quality improvement methodology.

(1) Constancy of purpose: describe and maintain coherent, student-focused vision and consistent action over time

Continuous improvement requires a widespread and unwavering commitment to improvement that is robust to the vicissitudes of leadership changes, policy shifts, and short-term incentives. Continuous improvement is a disciplined approach to sustained, iterative innovation that aggregates over time into system-wide transformation. Such transformation depends on a clearly articulated organizational purpose and a relentless commitment to this purpose. A long-term purpose fosters a culture of stability within which small-scale innovations can be tried, tested, and spread. It is the primary obligation of leadership to generate, promote, and pursue this purpose.

Example practices

a. Define organizational values. Create a set of values and communicate values system-wide.

b. Define district goals that embody these values, and then identify a subset of high-leverage goals to turn into long-term S.M.A.R.T. aims (specific, measurable, attainable, relevant, and time-bound).

c. Focus on critically important initiatives. Reduce the number of initiatives to manageable level for each department and campus. As a rule of thumb, halve the number of initiatives at each site.

d. Create a measurement system to actively measure and monitor progress toward the aims.

e. Align district and campus resources (leadership effort, staff time, money, political capital, etc.) to support the aims.

 

(2) Culture of improvement: Build and support shared responsibility for improvement toward the organization’s purpose throughout the organization

Improvement requires change, but psychologically, embracing change is hard. Research calls the human tendency to resist change a status quo bias, a cognitive preference for ways things have always been. Systematic barriers carry a great deal of institutional inertia, and changes to tackle pernicious problems at the heart of teaching and learning require deep will and effort. But such change requires us to release the familiar, opening ourselves up to ambiguity and vulnerability. Embedding continuous improvement into an organization requires a profound culture shift. In contrast to a culture of top-down mandates, in a culture of improvement all members of an organization are committed to advancing the best interests of its students, and leadership ensures the conditions that enable all members to make improvements at all levels of the organization to achieve that end. Such a culture requires a shared sense of mission, transparency, trust, innovation, collaboration, and an orientation to learn from failure, not punish it.

Example practices

a. Assemble a leadership system to launch and sustain cultural transformation toward the organization’s purpose.

  • Secure the support of the school board and elements of the business community to accept that systems, not individuals, are responsible for the vast majority of problems observed.
  • Ensure senior leaders model their improvement orientation in their own work including sharing data about improvements based on their own leadership.
  • Utilize social media to “market” the process improvements and positive changes in the organization.

b.  Empower members at all levels of the organization — from classroom to board room to iteratively improve.

  • Anticipate early resistance to this new approach — identify and enlist the support of building-level leaders.
  • Generate ideas for process changes close to where they occur. Workers engaged in the processes are best suited to identify opportunities for improving their day-to-day work.  Recognize the pride workers take in doing their job well.
  • Capitalize on members’ intrinsic motivation, desire to learn, creativity, and joy in accomplishment.
  • Include process improvement as a part of all job descriptions.
  • Hire for values and do not accept behaviors inconsistent with values. Human resource functions are aligned to the new culture change, including hiring, on-boarding, training, evaluation, succession planning, and reward and compensation plans.
  • Design learning plans for all staff to build up capabilities to engage in improvement work.

c. Design for collaboration.

  • Provide structured opportunities for members to work together to improve common processes.
  • Share team reports widely throughout the organization.

d. Recognize and celebrate improvement and lessons learned.

  • Document and spread improvement in work processes.
  • Make improvement work, inclusive of both data and student stories, visible throughout the organization (e.g., in classrooms, hallways, on websites).
  • Distill and disseminate learning from both successful and ineffective improvement efforts.

 

 (3) Standard work: Define the organization as a system of interrelated evidence-based processes
The work of schooling involves a complex set of funding streams, departments, schools, programs, services, and initiatives, all with their own set of coordinated and interrelated processes. Each system component typically has its own aims, which may or may not be well-aligned to support the district’s aims. Frequently, district members, both at the district and school levels, work in isolation and do not understand the relationship of their work to the work of others. One of the first steps a district can make toward continuous improvement is to draw a process map, or flow diagram, to show how each component depends on others. Indeed, the act of process improvement presupposes the ability to map, or “see”, processes and the relationships between multiple work processes.

Documenting processes reveals interdependent processes, but also serves as the baseline for continuous improvement. Standard work 1) precisely describes the most efficient and effective way known to perform a particular task or process to achieve the desired end result, 2) becomes the norm for how the task or process is executed throughout the organization, and 3) is expected to be continually improved. After a particular process is improved, the new process becomes the baseline for further improvements.

Example practices

a. Map the processes for core district work – both the learning and the related support services.

b. Create and implement methods and tools to gather stakeholder feedback on the mapped processes, include feedback from students, faculty, parents, community members, etc.

c. Engage in explicit alignment activities in order to codify best practice in key operational functions.

d. Develop a process to train others on the district’s standard work.

e. Collect stories of what works and codify that into knowledge products — “changes that work”. Post artifacts and best practices on a common website location available to all district personnel.

f. Use coaching support to adapt best practices to local context.

g. Execute well-articulated and well-understood processes consistently.

 

(4) Quality improvement principles and methods: Sustain a disciplined approach to daily data collection, theory development, and hypothesis testing to improve processes

Often the least well-understood aspect of continuous improvement is the methodological rigor it demands. Continuous improvement provides a set of tools and methods that, if used with integrity, can accelerate a district’s capacity to improve. In contrast to large-scale reform approaches, changes are informed by a clearly defined theory of action, and designed and tested over time. Importantly, the changes are thoroughly tested to determine whether they indeed represent improvements. Additional tests refine and build confidence in the changes before they are applied on a wider scale.

A hallmark of continuous improvement organizations is the improvement project. In recognition of the interdependencies of processes within complex systems, continuous improvement organizations identify a process in need of improvement and endorse an improvement team (i.e., members from different roles who are familiar with the different aspects of that process) to work together toward this end. Each improvement team has an executive sponsor who takes responsibility for the success of the improvement project.

Example practices

a. Develop and execute an improvement capability plan throughout the organization.

  • Define quality and continuous improvement in terms that make sense to all stakeholders.
  • Develop a common improvement methodology and vocabulary system wide.
  • Train workers at all levels of the organization in the selected improvement approach.
  • Implement improvement coaching as a core part of adult learning support.
  • Articulate core district aims and a process for setting local project aims that are aligned.
  • Identify and map core processes associated with important project goals. Develop and use process measures that show how teaching and learning is being done.

b. Test changes in improvement teams.

  • Carefully select initial sites for testing ideas, learning, and sharing what works to create high likelihood of early success (e.g., choose test sites with broadly trusted campus or administrative leaders).
  • Create formal project charters with clear intent, aims, and deliverables.
  • Identify improvement team members to fill specific roles (sponsor, process owner, team lead, team members, and subject matter experts).
  • Train team members on appropriate roles and tools for improvement.
  • Allocate resources to achieve the aims of each team charter.
  • Document tests using tools of improvement (e.g., PDSA documentation, run charts) that support discerning if a change is an improvement.
  • Schedule system level reviews of improvement projects.

c. Design a system to spread ideas that work.

  • Articulate a formal testing framework to implement initially successful changes at new sites.
  • Document adaptations required to make change successful in new sites.
  • Develop a package of successful adaptations to document the important aspects of context sensitivity.

 

Wednesday

11

December 2013

Year Two Success in Math Pathways Illustrates the Power of Networks

Written by , Posted in What We Are Learning

In its second year, Carnegie’s Community College Pathways (CCP) program sustained its high level of student success while also experiencing a growth in the number of students enrolled and the number of campuses teaching Pathways courses. The Community College Pathways: 2012-2013 Descriptive Report provides detailed student outcomes as well as insights into the challenges and improvements in the second year of Pathways implementation.

The Pathways were developed to address the alarming failure rates of students in developmental math in community colleges. Over 60 percent of the nation’s 14 million community college students are required to take at least one developmental mathematics class before they are eligible to enroll in college-credit courses (Achieving the Dream, 2006; Bailey, Jeong, and Cho 2010). However, 80 percent of the students who place into developmental math do not successfully complete any college-level mathematics courses within three years (Bailey, Jeong, and Cho, 2010). Instead, many students spend long periods of time repeating courses and leave college without a credential. This means that millions of students each year fail to acquire essential mathematics skills and are unable to progress toward their career and life goals.

Working through a Networked Improvement Community (NIC), Carnegie put into the field two new mathematics pathways, Statway® and Quantway®. Both aim to simplify the path through the development mathematics sequence for students. Rather than a seeming random walk through a maze of possible course options (Zeidenberg and Scott, 2011), Statway and Quantway reduce the number of courses required while improving the content and pedagogy for developmental mathematics.

Carnegie believes that working in NICs and focusing on continuous improvement in classroom practice were key reasons for the repeated positives outcomes for students in Year 2.

The NIC realized the following results in its second year of operation:

Statway
  • 52 percent of the 853 Statway community college students successfully completed the year-long pathway (received a grade of C or better in the final term). This is consistent with the results of 49 percent in Year 1 (2011-2012).
  • Statway expanded to two additional colleges within the California State University (CSU) system adding a total of 204 students.
  • 75 percent of CSU Statway students successfully completed the pathway, comparable with 74 percent in Year 1.
Quantway
  • The number of students enrolled in Quantway 1 tripled from Year 1 for a total of 1,402 enrolled.
  • Quantway 2, the second semester of the pathway, was launched for the first time at three community colleges with 49 students.
  • 52 percent of students successfully completed Quantway 1, demonstrating continued positive outcomes with 56 percent in Year 1.
  • In its first semester, 68 percent of students successfully completed Quantway 2.

With considerably more students enrolled, the Year 2 data indicate that the Pathways offer the opportunity for students in a variety of contexts to gain key mathematical skills and reach their academic goals.

For more information on the Year 2 results, download Community College Pathways: 2012-2013 Descriptive Report.

 

Wednesday

4

December 2013

ROI Study Shows Colleges Gain Revenue from Implementing Carnegie’s Math Pathways

Written by , Posted in What We Are Learning

Carnegie’s two pathways — Quantway® and Statway® — for students placing into development mathematics have had notable success in their first implementation. Over 51 percent of Statway students successfully completed the entire pathway, including earning a college-level credit in statistics in one year, compared to 6 percent of a baseline comparison group attempting the traditional sequence in a year. Even when expanding the timeframe for the traditional group to three years, only 20 percent of the entering cohort achieved transfer-level success, again compared to the Statway rate of 51 percent. These results represent a 250 percent increase in one year success over the conventional approach extended to three years. For the eight colleges starting Quantway in spring 2012, results were just as heartening.

Achieving results like these generated a good deal of interest from institutions and faculty members. But as Rob Johnstone, with the National Center for Inquiry and Improvement, writes in a Carnegie-commissioned Return on Investment (ROI) study, “While programs such as Statway and Quantway may very well demonstrate an impressive improvement in student outcomes, a significant concern still emerges from a college standpoint: Can we afford to do this at scale?” Johnstone finds that Statway and Quantway very well may make money for an institution. For a relatively modest initial investment of upfront costs that very often is less than the costs of boutique programs, Statway and Quantway can be implemented at scale for the entire range of students needing such an alternate approach rather than for the handful of students served by most boutique programs.

Generally these returns on investment are due to the increases in realized tuition and state apportionment funding from subsequent course taking, and Johnstone employs an ROI model that uses a number of inputs that allow a college to customize it to their local situation, including:

  • Salaried personnel costs
  • Hourly personnel costs
  • Other incremental fixed costs for Statway and/or Quantway implementation (e.g., Carnegie subscription fees, travel to a summer forum and winter regional meetings, supplies, recruiting, etc.)
  • College tuition
  • State-level apportionment funding per FTEs
  • Number of students in Statway and/or Quantway at the college, either as a pilot or at scale
  • Predicted increase in FTEs from Statway and/or Quantway

When Johnstone studied the six Pathways colleges who were selected to  participate in the ROI modeling exercise, he found that even with relatively conservative estimates on downstream FTEs generated by successful Statway and Quantway students, the model generated a positive return on investment with an associated positive net revenue figure every time.

Figure 1 provides the modeled net revenue figures over a three-year period for each of the six colleges associated with a single Statway or Quantway cohort, ranging from $6,744 to $177,905.

figure1

After calculating these incremental net revenue figures using the model noted in Figure 1, Johnstone also calculated a classic ROI rate. In Figure 2, ROI is calculated by taking the incremental net revenue associated with a program and dividing it by the total incremental program costs.

figure2

The ROI rates produced by the model, even with reasonably conservative estimates, were all positive, and ranged from 12 percent to 334 percent.  Note that even the 12 percent ROI indicates that the college would generate net revenue that is 12 percent higher than the incremental program costs. From industry standards, ROI rates that are above 50 percent (achieved in three of the six sampled colleges) are considered quite high and those over 100 percent (as observed in three of the six) are extremely high.

Taken together, the results of these ROI models are quite positive, Johnstone notes, concluding that in addition to the more important outcome of getting 2.5 times as many students to succeed in transfer-level mathematics in one year as a traditional group did in three years, the ROI models suggest that Statway and Quantway will likely be able to achieve this outcome at a net revenue gain to these colleges.

In addition, Johnstone also modeled other fiscal impacts such as cost per completer, student tuition and books savings, and student wage gains, finding in each case that a successful Statway / Quantway program would positively impact these key fiscal factors. The study found that “if Statway and Quantway can help get students through their community college pathway one year sooner, these students are estimated to register a nearly immediate $24,600 wage gain. Clearly, the catalytic effect of this gain to transform our students’ lives and accelerate their momentum upward on their career path is remarkable.”

Read the NCII report and download the Excel model >

Tuesday

3

December 2013

Evaluation and Improvement: Let’s Agree to Not Conflate

Written by , Posted in What We Are Learning

Teacher evaluation has evolved markedly over the past four years.  In 2013, it is more common than not that teacher performance evaluation is determined, at least in part, by student achievement, whereas this was not the case just four years ago. This rapid evolution has been catalyzed in no small part by federal Race to the Top and Teacher Incentive Fund monies, as well as a national discourse that asserts that quality teaching must, by definition, raise student achievement. Unsurprisingly, consequent proliferation of evaluation systems has also yielded a great deal of variation in terms of system design, structure, and coherence.

This is the essence of the latest State of the States report published last month by the National Council on Teacher Quality (NCTQ): while most states now include measures of student achievement in teacher evaluations, not all states are connecting evaluation policies with other personnel decisions of consequence.  The report compares states’ teacher evaluation policies and assesses the extent to which they are linked explicitly via legislation to 11 consequential decisions (i.e., tenure, professional development, improvement plans, public reporting of aggregate ratings, compensation, dismissal, layoffs, licensure advancement, licensure reciprocity, student teaching placements, and teacher preparation program accountability). According to Sandi Jacobs, vice president of NCTQ, “[S]ome states have a lot more details sorted out than others do” with regard to linking teacher evaluation – based mostly on student achievement – to these decisions.

There are a number of implicit assumptions in this report that warrant exploration, not necessarily in response to the report itself, but rather in response to state action that is generally moving in this direction. Indeed, Louisiana has connected teacher evaluation policies to nine of the specified decisions (all but licensure reciprocity and student teaching placements), while legislation in Colorado, Delaware, Florida, Illinois, Michigan, Rhode Island, and Tennessee has connected evaluations to more than half of the decisions. The discussion here recommends that states use caution when “connecting the dots” between teacher evaluation and personnel decisions.

First, it is worthwhile to note that NCTQ is making an a priori assumption that tying teacher evaluation to the 11 personnel decisions specified in the report is a meaningful, productive, and efficient way to organize a teacher evaluation system.  By extension, other systems not connected in this way are counterproductive and inefficient. The report does not offer evidence to support this claim, either in terms of increased student achievement or other measures of productivity (e.g., decreased teacher turnover). Related to this, state legislation is not synonymous with an evaluation system, a point recently made elsewhere by Rick Hess. A set of statutes may constitute a set of facilitating conditions under which an evaluation system is created, but the NCTQ report does not discuss how a state might do this. In this respect, the theory of change suggested by the NCTQ report is quite lengthy: from “connected” state legislation to a coherent evaluation system that elicits information about teachers that could inform personnel decisions, which may positively impact the teaching workforce, thereby enhancing the quality of teaching, which in turn may improve student outcomes.

Second, the report assumes – based on other influential findings, such as those reported by the Measures of Effective Teaching Project – that teacher evaluation results should be composed of multiple measures, with student achievement as the chief determinant. But outside of a passing reference to one state (Rhode Island), there is no indication in the NCTQ report that the way in which these measures are combined matters at all.  In fact, it does matter a lot, according to a recent brief by Carnegie Knowledge Network (CKN) panelist Doug Harris. Harris shows that while many states employ a weighted average approach to multiple measures resulting in a singular index of effectiveness, this approach is likely to be more costly than other approaches (i.e., all measures are needed for all teachers) and may diminish valuable nuance in the data. Further, he maintains that we should ultimately be concerned with the outcomes of the personnel decisions (i.e., teachers’ reactions to them) rather than the particular method by which they are made. However, Harris notes that this requires a “different kind of evidence” than is currently collected – that is, evidence is needed on the implementation of consequential decisions and measures are necessary to ascertain unintended and averse side-effects.

Third, the part of the NCTQ report most emphasized in a recent U.S. News article is the use of evaluation data to identify high-quality teacher preparation programs. The assumptions here are that: (1) some programs are more effective than others, and (2) teacher evaluation can elicit useful information against which teacher prep programs might be held accountable. Another Carnegie brief by Dan Goldhaber discusses issues related to the latter. According to Goldhaber, comparing individual teacher prep programs is tricky because “we cannot disentangle the value of a candidate’s selection to a program from her experience there.” In other words, some programs benefit from more rigorous selection criteria. Critically, Goldhaber argues that smaller teacher training programs are unlikely (due to standard errors and confidence intervals) to be statistically distinguishable from average. This would incentivize small programs to stay small unless they can drastically improve, and induce large programs to reduce the number of graduates unless they also can drastically improve. But because research into the aspects of teacher training that matter most for producing quality graduates is limited, drastic improvement would be difficult.

Fourth, and arguably most critically, the NCTQ report assumes that systems of evaluation and systems of improvement are one and the same, and that information elicited by the former can be used for the latter. This is not inherently true, though neither are the two types of systems mutually exclusive. A Carnegie policy brief on post-observation feedback for teachers by Jeannie Myung and Krissia Martinez illustrate this tension. Myung and Martinez find that while many districts have committed to collecting a large quantity of observational data, “the field still has a lot to learn about how best to use data to support the improvement of teaching.” Perceived threats in post-observation evaluative conversations (e.g., unclear expectations, a sense of disempowerment, the absence of helpful information) can impair attempts at improvement. In contrast, the authors found that feedback facilitates the improvement of teacher practice when improvement-oriented strategies (i.e., scaffolded listening strategies, a predictable format, addressing the teacher’s concerns, and the co-development of next steps) are purposefully incorporated in the feedback protocol.

A small and final point is the not insignificant costs associated with more “ambitious evaluation systems.” The NCTQ report appears to assume that there is an abundance of both time and money allocated to evaluation and that, consequently, there will be no trade-offs or additional real or opportunity costs associated with “ambitious” evaluation. But since time and money is not something most districts have in abundance, trade-offs are likely. Moreover, the final price tag is unknown for many states. For example, researchers priced Minnesota’s new teacher evaluation system somewhere between $80 million and $289 million. The Carnegie Foundation has developed a Cost Calculator that can help district leaders estimate both the time and financial resources involved in teacher personnel evaluation. This tool can certainly aid in enumerating the costs of evaluation, and districts will have to supplement the data generated with opportunity costs.

Teacher evaluation is certainly here to stay; there is no one who disagrees with the assertion that the demanding and important work of teaching – like that of other professions – should be evaluated. But that does not preclude caution or a healthy dose of skepticism when “connecting the dots” between teacher evaluation and consequential personnel decisions.  Neither does it suggest that “connecting the dots” is a simple matter, that there are no trade-offs, or that there are no complex issues to confront. Our work at the Foundation suggests such caution is warranted. But most of all, we should not mistakenly assume that teacher evaluation alone will lead to the improvement of teaching. Our system of education would be the worse for it.