Carnegie Commons

A place to come together to exchange ideas about teaching and learning



October 2014

The Standardization Paradox

Written by , Posted in What We Are Learning

In the field of healthcare, there persists a belief that the practice of medicine is artisanal in nature. Doctors are seen as the ultimate authority, and in truth, a patient’s illness is often an experiential learning ground where doctors try out various forms of treatment in an effort to find the one that works. Practice and learning often look like, “You have an infection, let’s try this antibiotic. If it doesn’t work then we will try a different one.” Or an oncologist might prescribe one regimen of chemo therapy and when it doesn’t produce the desired result may try a different regimen including different drugs and the addition of radiation therapy.

During the balance of their careers, these doctors learn (or don’t) to select treatments that will work for most people, tailoring their recommendations based on small variations they see in individual patients. Teachers do the same. Over time they learn (or don’t) what style produces the best learning outcomes in the majority of their students and they tailor their pedagogy to meet the needs of individuals in their classrooms. Where a system based on this style of mastery fails is in placing the burden of performance onto the shoulders of the practitioner, whether doctor or teacher. And it fails because each practitioner is left to a journey of self-discovery about what does and does not work. With each new clinician and teachers comes a journey repeating the same steps and missteps for every individual. In short, it fails to consider the broader system at work and what might be leveraged to anticipate needs, provide structure, and accelerate learning. In healthcare this mentality is starting to change. A realization is dawning that standardization, in training, in process, and, to a degree, in treatment can actually improve health outcomes while freeing clinicians to apply patient specific recommendations as needed. Standardization is creating freedom in the practice of medicine.

There seems to be a deep aversion to the idea of standardization in education. I sense it is because when educators think of standardization they think of examples such as “scripted curriculums” and “teacher proofing.” Their reaction may stem from two common beliefs: each student is unique, and each teacher is a craftsman. This seems to parallel what has been experienced in healthcare, each patient is unique and each clinician is a craftsman. Seen in this way, teaching and learning are artisanal activities. Teaching is a craft learned by trial and error, refined over time, and ultimately perfected by the educator within the confines of their classroom. While to an extent this will always be true, as an improvement advisor, my perspective is standardization can be less about “teacher proofing” and more akin to the concept of standard work. Defined as those day-to-day activities in which all educators engage but which display so much variation in practice. When applied thoughtfully standard work can improve outcomes while creating freedom to meet individual needs when those needs vary from the majority.

People like Avendis Donabedian, Paul Batalden, and Donald Berwick, in an effort to transform the quality of healthcare, led the way in applying tools first leveraged in manufacturing to assist in creating a systems perspective for their field. They did this while preserving their appreciation for healthcare as both unique and more complex than assembly lines. What these thinkers realized was they could not achieve the outcomes they desired for patients until they could first see the system (those common processes and practices) that was producing those outcomes. They also knew a core component of systems thinking would be the ability to measure what they did—or did not—achieve. They needed to move the focus from artisanal knowledge, where all expertise is tacit, contained in a singular practitioner’s mind, to a system, where much knowledge of practice is explicit.

Since 1991, when Don Berwick and the Institute for Healthcare Improvement began their work, healthcare has struggled to find its way into pragmatic standard practices where appropriate. Pragmatic, in that there is recognition that each patient is unique, that each patient will require local adaptation of treatment to achieve improved health outcomes but that there are also processes and practices that are common to all patients.  There are now known standard work practices in medicine for the assessment, diagnosis, and routine treatment of patients. However, the clinician is never reduced to a robotic assembly-line worker. Instead, standard practices free practitioners to leverage all of their learning, experiences, and tacit knowledge to focus on the uniqueness of each patient they treat. Within standard work processes the clinician can now adjust, leverage new information, adapt to emerging circumstances, all in an effort to customize treatment for the individual. The paradox emerges; it is through standardization that the optimal opportunity for individual customization is realized.

The realization of a visible system, one with known processes, known data, providing some level of predictable outcomes for teachers and students is what education needs now. Teachers should have the discernment and expertise of an artisan, but also the ability to leverage their expertise within standard work processes when the circumstances of the student, the classroom, or the school demand it. They should be able to recognize when a standard process for assessment, diagnosis, or an intervention works for the majority of cases and when accessing and utilizing tacit knowledge of teaching and learning will provide for improved outcomes for students. As in healthcare, the tension between standardization and customization can be resolved to provide teachers the greatest freedom to address the needs of their students. It recognizes the most successful systems are systems that are defined and once so defined are adapted locally, reflecting the unique circumstances, knowledge and experience the teacher has developed over the course of their career.

I recently had the opportunity to experience just how freeing a piece of standard work can be. While coaching a group of principals focused on improving their instructional feedback to new teachers, one principal created a feedback conversation protocol to use during each interaction with his new teachers. The initial reaction to this standard piece of work by the other principals was immediate rejection, “this protocol will take too long, does not reflect the circumstances of my school or in my district, etc.” But the principals’ changed their minds when they observed the protocol in practice between the initiating principal and one of his teachers. It was in that moment they realized the standard work gave numerous opportunities for the principal to apply his subject matter expertise, tailoring the conversation in a productive way while adhering to the intent of the protocol he had developed.

Ultimately the principals involved had a breakthrough regarding the efficacy and applicability of standard work in their settings. When education as a whole can embrace the paradox of standardization as a means of freeing practitioners to apply their expert knowledge, then, I believe, we will be best positioned to describe our systems, to measure them, and to pursue the continuous improvement we need to achieve the results we want to see.



September 2014

The Role of Policy in Improvement Science

Written by and , Posted in What We Are Learning

In the world of education reform, policy is often looked to as an effective method to generate change. While clearly a potentially powerful tool, “policy-making is like going into surgery with a sword instead of a scalpel,” said Michael Cohen of Achieve at Carnegie’s most recent convening, Using Evidence to Advance Teaching: The Promise of Improvement Science in Networks. The purpose of this convening was twofold: to examine the process of quality improvement through the real-life case study and to discuss the role of policy in furthering the science of improvement. The case study focused on Carnegie’s three year partnership with a network of schools including the Austin (TX) Independent School District (AISD) that addressed issues of beginning teacher efficacy and retention by using improvement science.

This collaboration with the AISD has highlighted how change and improvement can start with one principal and expand across a substantial network. Since the results are compelling and enthusiasm is growing for this work, there is a push to scale this type of improvement quickly, with some suggesting policy as a means of doing so. The question of how to create a supportive policy environment—one that both promotes policies to initiate and sustain improvement science work and eliminates policies that impede it—was the focus of the convening’s closing panel “Policy Implications: Discussion of the Consequences of Improvement Science for Policymakers.” Experienced policymakers and political advisors Brad Jupp, Senior Program Advisor at the US Department of Education, Bethany Little of the consulting firm Education Counsel, and Michael Cohen, president of the nonprofit organization Achieve responded to questions from Tom Toch of the Carnegie Foundation, as well as from the audience.

policy panelThe panelists began by explaining their perceptions of what policy-making is and can do. Jupp listed three main activities in which policy engages: setting external aspirations; allocating resources in order to achieve those aspirations; and providing guidance on what localities can and cannot do as they attempt to reach those aspirations. A persistent concern for the limits of policy was an idea that was often woven into the comments of the panelists throughout the session. All three were cautious about how far policymakers should go in terms of furthering improvement science. Jupp said that “the worst idea would be [for policy] to require” schools, districts, or states to use improvement science.

But all of the panelists suggested that the time is right for policymakers to talk about improvement science. The US education system has shifted in recent years from a focus on minimum standards to more ambitious goals, including high expectations for college and career readiness. To reach these goals, policy needs to change from what Little referred to as the “command and control” mindset to one that allows for learning and best practices to be shared between states, districts, and practitioners. The potential to increase the sharing and development of local knowledge is why there is growing interest in supporting improvement science.

There are significant challenges that stand in the way of such a change, chief among them the inherent instability of leadership in the realm of policy. As Little noted, “policy is driven by politics,” and politicians experience constant turnover. This unpredictability is incompatible with the work of improvement science, which requires sustained commitment and attention over time. Improvement science is also inherently risky, in the sense that one can expect that a number of change efforts will fail or not work perfectly until an iterative process of learning yields consistent, positive results. This too can spook politicians, who rely on quick wins and the promise of silver bullets for continued public support.  And yet, if one agrees that real innovation requires some failure, then this risk is a necessary aspect of true innovation.

Cohen’s response to these challenges was to encourage policymakers to focus on building leaders on the ground who can make improvement work happen in a sustained way. Little suggested that, since policymakers have the power to direct resources, they should invest in what works—in her view, improvement science. And Jupp focused his recommendations on building an “ethic of improvement” among policymakers. Policymakers not only need to learn about what works from practitioners, but they need to learn how to learn from practitioners—a process that can be informed by the sharing strategies utilized by networked improvement communities, intentionally structured groups of individuals working together on a shared problem.

Overall, the panelists agreed that improvement science has the potential to dramatically change how schools improve to reach their desired outcomes. Despite the barriers that currently exist in using policy to advance the field, all three participants recognize that the time is right for policy action to support districts and schools in their efforts to implement improvement science.



September 2014

Creating Opportunities for Students to Become Flexible Experts

Written by , Posted in What We Are Learning

As students and teachers head back into the classrooms, they are likely thinking about all they will have to learn and teach, respectively. For math in particular, students may be dreading learning new formulas and procedures. What students do not often recognize is that math is not simply about memorizing, but about being able to improvise and see what could work in a given situation. Jim Stigler, a psychology professor at UCLA and Carnegie Senior Fellow, describes this as having “flexible expertise.” According to Stigler, flexible expertise provides students the ability to adapt to new situations and solve problems by drawing on previous learning and experiences.

Jim Stigler photo

Jim Stigler at the Pathways National Forum, July 2014.

Stigler said that for students to gain comfort with this type of approach to mathematics and problem solving generally, teachers need to create recurring and sustained exposure to three key learning opportunities: productive struggle, explicit connections, and deliberate practice. At the Community College Pathways National Forum this summer, Stigler introduced these learning opportunities as essential components in the instructional system that make up Carnegie’s two mathematics pathways. Employed together they can aid students in developing rich knowledge and a focus on problem solving.

Productive Struggle

Stigler explained that while not all struggle is productive, studies have shown that students learn and retain more over time when learning is difficult. Confusion over directions or what question is being asked is simply struggle and not productive. But if teachers can make learning a challenge, students may feel as though they are not simply struggling, but in fact that they are learning deeply. On the other hand, if learning is easy, students may not internalize the material long-term.

Stigler described a study conducted by Roediger and Karpicke in which two groups were given a passage to read. One group had four, five-minute sessions to read and reread the text and the other group had one, five-minute session to read the text and then engage in a form of productive struggle in which they had three, five-minute recall sessions, where they could not look at the text, but were told to write down everything they could remember. At the conclusion, the group that was able to reread multiple times felt they had learned from the passage, while the recall group felt they had not successfully learned it. Although five minutes after the sessions the reread group did slightly better on the content quiz, a week later the group that participated in the recall sessions did significantly better than the group that only reread the text. By making learning an active challenge for students, they are more likely to take the knowledge with them and be comfortable applying it later.

Explicit Connections

Often topics seem discrete and disconnected to students as they move from lesson to lesson, but by highlighting the relationships between these ideas, teachers can help structure what students should be thinking about when they encounter unfamiliar problems. Stigler explained that teachers know what they want their students to do, but by focusing on what students should be thinking about, teachers can help students make connections across what they are learning in different lessons. If students have many connections between what they learned, they will have greater ability to retrieve the knowledge they need for any given problem. Using a video from one of University of Michigan Professor Deborah Ball’s mathematics lessons in an elementary school class, Stigler illustrates that leading students in a discussion of all of their answers can provide teachers with the opportunity to draw out the key concepts and make the connections between them.

Deliberate Practice

Often students simply repeat the same type of problems over and over again. Although this is, of course, practice, deliberate practice is different in that students should be challenged to practice analyzing the problem situation and identifying how to solve it, not just repeating the use of procedures and formulas. Relating this type of practice to learning to play the guitar, Stigler recalls how his guitar teacher said that the best guitarists “never practice, but they play all the time.” It is this difference between playing and practice that demonstrates the different between repetitive practice and deliberate practice. Presenting students with problems that give them the opportunity to determine how to approach them and which steps they should take to solve the problem allows students to practice one of the key skills of flexible expertise that they can use when they encounter new, unfamiliar problems.

Stigler says that introducing the three learning opportunities into pedagogy need not be prescriptive. There are countless ways to incorporate these ideas of productive struggle, explicit connections, and deliberate practice into teaching and learning and each situation is different and will require varied employment of the learning opportunities. It is simply important that teachers keep these pedagogical tools in mind as they teach and look for ways to incorporate them into lessons.

Stigler acknowledged the considerable challenges that introducing new practices into teaching presents to teachers, but suggested working with other educators to share ideas and using curriculum and assessments that support this type of learning and teaching. He concluded that “we need to think about [teaching] as not needing to invent it, but working to improve it.”



September 2014

How Carnegie Is Using Technology to Enable Collaboration in Networks

Written by , Posted in What We Are Learning

As the Carnegie Foundation continues to serve as an innovator in education by sharing our thinking about improvement science, we are also exploring new ways to seamlessly integrate technology into that work to better leverage the power of networks to accelerate improvement. A question central to this pursuit is: How might we design an online workspace that would accelerate improvement?

The Foundation’s strategy for tackling this question begins with Doug Engelbart’s theory of the Collective Intelligence. Engelbart described Collective Intelligence as how people work together in response to a shared challenge and how they can leverage their collective memory, perception, planning, reasoning, foresight, and experience into applicable knowledge—with the joint capabilities being more than the sum of the individual elements. Engelbart had a vision of people using technology to improve the Collective IQ of organizations and to “build a collaborative community of knowledge workers.” The real power of computers, he believed, lies not simply in automating work processes, but in “augmenting human intellect” to address social problems.

While we are able to leverage the Foundation’s ability to convene experts and host large events to launch collective work on a shared problem (we call these groups networked improvement communities or NICs), we lack the crucial mechanisms and tools necessary to support rapid data sharing and learning across the networks.

In response, CarnegieHub is our online workspace and serves as the primary access point for all network members to Carnegie resources and provides multiple opportunities and avenues for engaging in work and collaboration.

Four Reasons Why CarnegieHub Is So Powerful in Strengthening Networks

A main component of ensuring deeper and richer learning in our NICs is the Foundation’s capability to provide a technology infrastructure and support for strengthening and sustaining the collaboration and participation across the network. With the implementation of CarnegieHub, our process in building a robust network workspace is focused on design features that afford the particular opportunities and benefits for maximizing the potential of rich collaboration in our NICs.

Here are the four design principles of CarnegieHub that have improved traditional ways of collaborating and sharing learning across the network.

1. Shared Creation of Content for Collaborative Learning

google docs picImagine a group of faculty tasked with coming up with the best ways to increase students’ classroom attendance. Rather than waiting for one faculty member to create a document locally on their own computer, then email it to everyone to edit and make changes—which often results in multiple versions of the same document—what if all the faculty members shared one document in the cloud and each had the ability to edit simultaneously in real time with everyone else? With the Google Drive (Google docs, spreadsheets, slides, forms, etc.) members can participate in the concurrent development of an artifact, give feedback through comments, and engage in chat while working visibly across distances and time zones. This type of running activity tracks revisions, authors, viewers, and editors. This translates to a sense of collaborative learning because contributions to the work are distributed across multiple network members.

2. Centralized File Storage with Search Capability for Improving Knowledge Management and Discovery

What if that same group of faculty members all wanted to reference an old file, but no one could find it because no one could remember whose computer contained the file. With Google Drive, all files and folders are located in one single, web-based location. Having a centralized location to store all files not only allows people to work off one version of each document, it also minimizes the amount of effort and time it takes to find a desired file through a robust search engine. As files become better managed, more discoverable and accessible, partners in the NIC can focus on the content of their work together.

3. Asynchronous Curated Conversations to Bolster Work Across Distance and Time

On CarnegieHub, a faculty member can easily use the discussion board feature to see what others in various classroom contexts have already discussed. The benefit of having asynchronous discussions is the flexibility of activity. Members can respond whenever they feel appropriate and their posts are publicly displayed. Conversations then become a catalyst for innovation. In addition, these discussion threads are documented, providing new members of the community with a running history of the conversation. 

4. The Combination of These Features Supports the Creation and Maintenance of Social Ties

A defining aspect of NICs is a commitment to a shared problem as a community of practice. Rather than working in isolation and feeling alone in making progress, CarnegieHub provides a space for peer support and social interaction. Participants now have the ability to search for resources, find the contact information of the author or forum poster, use the directory to locate that research or practice expert, and initiate a dialogue. Participants can also reflect on their experiences by contributing blog posts to further enhance their social ties with one another. As they interact with peers and engage in networking, members keep each other accountable and engage in collective sense making.

Conclusion and Future Implications

When we ask for a commitment from our network members to accept a new pedagogy, new tools, a new work process, and a new knowledge collection sharing mechanism, we are cultivating a living system of improvers, innovators, and collaborators. Ultimately, CarnegieHub is digitally altering the online space to one that coordinates and supports network-wide learning. It is our hope that the democratizing impact of these new technologies will indeed deliver the augmentation of network learning that Doug Engelbert describes. We also look to the faculty leaders across the NIC to leverage these tools in ways we have not yet imagined to build and share knowledge across institutions. Together, we are exploring new collaborative technology horizons of accelerating improvement in our networks.



August 2014

Revisiting the Purposes of Practical Measurement for Improvement: Learning from the BTEN Measurement System

Written by , Posted in What We Are Learning

“The failure of educational systems to integrate research evidence productively into practice impedes progress towards making schools and colleges more effective ….” So state the authors of Practical Measurement, a Carnegie Foundation paper describing tenets of measurement and research that support the improvement of practice. For the past three years, the participants of Building a Teaching Effectiveness Network (BTEN) have sought to build the type of integrated system of measurement described in Practical Measurement that is so often lacking in our educational systems—one that hews closely to the processes occurring at the ground level of schools, and that supports the day-to-day efforts of practitioners to improve their work.

The BTEN measurement system, designed to support the improvement of processes by which new teachers receive feedback on their instructional practice from school leaders, consists of varying grain sizes of measures to support and spur improvement efforts. The measurement system spans the conceptual space from the micro-level process being improved to outcomes of this process to the ultimate aim of the effort, teacher development and retention.

As a result of the experiences gained through using measures in BTEN, we can take next steps with the ideas originally discussed in Practical Measurement. One of these ideas concerns the purposes of measurement in the improvement context, which we have revisited, augmented, and reorganized, and now propose anew.

Measures in the improvement context are used for:

  • Learning About Your System
  • Priority Setting
  • Testing the Practical Theory of Improvement
  • Tailoring Interventions to Individual Participants’ Needs
  • Developing Social and Psychological Stances Necessary for Improvement

Leaning About Your System

Enacting widespread and sustained improvements in a system requires knowledge of that system. Often, practitioners are steeped in their context – their department, their classroom, the role that they play – but they do not see the larger system within which their work is embedded. Knowledge of a system precedes improving that system, and practical measurement can support this type of learning. Furthermore, improvement teams can gain important knowledge about enhancing performance from successful members.

BTEN Example
BTEN improvement teams collected information about the frequency with which new teachers receive feedback. In one of the partnering districts, improvement teams discovered that 42 percent of new teachers responding to the survey indicated that they had not received any feedback from their principal, assistant principal, or mentor from September to October. This baseline knowledge provided a reference point from which the improvement team could gauge their success in increasing the frequency and regularity of feedback on instruction.

Priority Setting

Tackling any complex problem requires improvement teams to narrow their focus, to have a place to commit their improvement efforts and resources. Measures in the improvement context can enable practitioners to choose a priority area. These areas may be those of greatest weakness, the “low hanging fruit” for which improvements are likely to be quick and easy, or the area of greatest collective interest among the improvement team members.

BTEN Example
When the improvement team at one BTEN school surveyed their new teachers about their feedback experience, they discovered that some teachers in the school felt that the feedback they received from different feedback providers was not consistent. The team decided to focus their improvement efforts on “coordination meetings” where feedback providers met to discuss the new teachers to align their support efforts. In the next survey administration, they saw an improvement in teachers’ responses about the degree of feedback consistency.

Testing the Practical Theory of Improvement 

A measurement system supporting improvement work provides data against which improvement teams can test their theory of practice improvement. As practitioners make changes in their work processes to spur improved outcomes, they can use data from the measurement system to see if the changes are actually happening and if they are resulting in the outcomes they hoped to see.

BTEN Example
The BTEN improvement teams improved their feedback processes by making feedback more frequent, giving feedback providers a conversation protocol to guide the feedback conversation with the new teacher, and establishing coordination meetings among the feedback providers. They tracked these changes with tools that allowed them to see whether these changes were happening. They were able to consider this information in conjunction with teachers’ survey responses to see if these changes led to improved teacher perceptions of their feedback experiences.

Tailoring Interventions to Individual Participants’ Needs

As improvement teams work on improving a process, they are likely to find that the intended beneficiaries of the improvement differ in their starting points, their needs, and how they respond to the changes being tested. Certain individuals or subgroups may be at higher risk of failure than others, or may otherwise need a specific intervention to achieve the outcome sought. Improvement teams can use measures to identify these individuals or subgroups, ascertain their areas of struggle, and craft targeted interventions.

BTEN Example
At one of the BTEN schools districts, data from a survey of new teachers provided evidence that teachers entering the system through alternative certification routes were more likely to show signs of disengagement and burnout than teachers who entered through traditional certification routes. This prompted school leaders to consider how they might partner with the alternative certification providers to better support these teachers.

Developing Social and Psychological Stances Necessary for Improvement

When improvement teams collectively engage with data relevant to their improvement work, they can develop a psychological approach and a social dynamic that supports their efforts. These cultural and mindset shifts include the development of a shared language and understanding about the problem being addressed and the theory of improvement; increased will and engagement with the improvement work; and a sense of internal accountability, both personally and within a team, to enact changes.

BTEN Example
On this last point about a culture of internal accountability, one BTEN principal explained:
“[The BTEN data] gave us a reason to not make excuses about why we weren’t doing it or how other things got in the way. I guess it sort of prioritized the work for us so that we made a commitment to a certain timeline, we made a commitment to each other that we would stay on that timeline …. I think in that way it kept us accountable.”

As Carnegie moves forward in developing a robust methodology of measurement for improvement, we will no doubt be revisiting and revising this list.



August 2014

Testing Stress Impact on Students

Written by , Posted in What We Are Learning

Editor’s Note: Jeremy Jamieson is an assistant professor of psychology at the University of Rochester where his primary research interests focus on emotion regulation and how stress impacts decisions, emotions, and performance. Aaron Altose is an assistant professor in mathematics at Cuyahoga Community College in Cleveland, OH. He is also a Quantway instructor within the Carnegie Foundation’s Community College Pathways. They are working as a team as part of one of Carnegie’s Alpha Labs.


“The greatest weapon against stress is our ability to choose one thought over another.”
— William James, philosopher and psychologist


Every math teacher in a developmental-level class knows about the challenges of test anxiety. It is understood that in an environment where success is primarily measured by achievement on a proctored assessment, promoting student success means devoting some effort to test preparation and other activities that reduce test anxiety. Carnegie’s Quantway and Statway Pathways are designed and implemented in a way that focuses more on the students’ learning (over the instructors’ teaching) and are increasingly based on what research has demonstrated to be effective learning structures and processes. So, naturally, embracing the challenge of test anxiety follows a similar approach: focus more on what the students understand about their own anxiety as we build off of the insights of research.

In an effort to promote research-based practice, Carnegie has developed Alpha Labs. Within each Alpha Lab there are researchers who partner with a community college faculty member to test interventions that address specific instructional routines, skills development approaches, or mindsets interventions necessary to promote success in the classroom. A growing body of evidence indicates that teaching people to reinterpret stress arousal as a potentially useful coping resource that can help improve outcomes during stressful evaluative situations, such as interviewing for a job, speaking in public, or performing in an academic setting (Crum, Salovey, & Achor, 2013; Jamieson et al., 2010; 2013; Jamieson, Nock, & Mendes, 2012; 2013, Woods, 2014). Given this, one Carnegie Alpha Lab has tested arousal reappraisal routines to develop students’ ability to cope with the stress and anxiety of testing situations. Arousal reappraisal instructs individuals that the physiological arousal experienced during stress is not harmful, but rather can be conceived of as a coping resource that aids performance. This perspective builds directly on reappraisal research from the emotion regulation literature (Gross, 1998; 2002).

Aaron Altose explains: “In my Quantway class, I’ve tried many approaches over the years to help students confront their negative feelings around taking exams. We talk about the proper amount of time to spend studying, finding the right environment, eating and sleeping well, and not cramming before a test. I’ve tried guiding students through progressive muscle relaxation before tests. I had a constantly evolving PowerPoint presentation comprised of information and articles I found online about understanding the ‘fight or flight’ response and high-pressure performers like Air Force pilots and emergency room doctors. But students sitting down to take a math test seemed to never take this information to heart. Negative interpretations of bodily signs of stress persisted. However, this material clearly demonstrated to students that they would have to overcome their performance anxiety to succeed.”

There are methods specifically designed to help people improve their performance in stressful situations by re-interpreting the meaning of bodily signs of stress arousal, such as sweaty palms or racing hearts. Rather than seeking to reduce or eliminate stress or emotional intensity to improve outcomes, this new arousal reappraisal approach seeks to change “bad stress” into “good stress.” The Alpha Lab developed and tested interventions were developed for just that purpose.

When students participate in an Alpha Lab activity to test the efficacy of methods to reverse the negative effects of stress, they complete very specific inventories of their feelings and emotions immediately before they take an exam. The arousal reappraisal intervention involves having students read summaries of scientific articles highlighting the adaptive benefits of stress. After reading the materials, students are asked to respond to two multiple choice questions that ensures they read the materials and to encourage them to accept the information provided.

While the purpose of the experiment is not shared with students, students have said that the activity helped them better understand what they were feeling. “I know before a test, I just feel bad, but maybe what I really feel is determined,” one student said, after participating in the inventories as part of the experiment. Seeing students take a more empowered approach to understanding their stress, just as they take more control of their mathematics learning, has been powerful for faculty as well.

Although data collection is still ongoing, preliminary results are promising. Upon completion of the current project Pathways faculty and researchers will further develop the reappraisal intervention for more widespread dissemination and testing.

By testing interventions like these, Alpha Lab faculty and researchers working together are able to help students overcome key hurdles, stress and anxiety, and to maximize their success in the classroom.

To continue this discussion, Carnegie will be hosting a webinar with Jeremy Jamieson this fall. Join our mailing list to be the first to hear when registration opens. 



July 2014

Designing a Collective Learning System

Written by , Posted in What We Are Learning

Editor’s Note: This blog is based on the author’s doctoral capstone done while she was in residence at the Carnegie Foundation. The full capstone is titled, “Walking the Talk, Teaching the Talk: Building a Collective Learning System at the Carnegie Foundation for the Advancement of Teaching” and was submitted to the Harvard Graduate School of Education.

Carnegie’s work rests on the assumption that we need to increase the rate of learning in order to reach the high aspirations we have for the education system–to provide quality education to the millions of children across the country. In our Networked Improvement Communities (NICs), we aim to “learn our way into better systems.” A key component of that vision is being able to build on the learning of others. We are all familiar with the lament of “not wanting to reinvent the wheel,” yet if you have been around education long enough, you have probably seen efforts that seemingly “reinvent the flat tire.” Despite living in a time of connectivity and open access to information, we still struggle with getting the right information to the right people at the right time. We are “information rich, but knowledge poor.” At Carnegie we’ve been studying knowledge management in an effort to address this problem, both within our organization and in the NICs we support.

Knowledge management has a natural fit within NICs because they aspire to produce knowledge that can improve practice and expedite the spread and use of that knowledge. In his original conception of NICs, Doug Engelbart argued that the collective IQ, or “how well people can work together to solve important problems,” depends on their ability to engage in “concurrent development, integration, and application of knowledge.” His description prefigures how the American Productivity and Quality Center, a leading voice in the field, defines knowledge management. They define it as, “a systematic effort to enable information and knowledge to grow, flow, and create value. The discipline is about creating and managing the process to get the right knowledge to the right people at the right time and help people share and act on information in order to improve organizational performance” (O’Dell & Hubbert, 2011, p. 2). Both of these definitions sounded promising, and over the course of the last year we began to explore the knowledge management needs for Carnegie and for our NICs. We also studied what has been learned from previous efforts in other organizations.

The first lesson that came through loud and clear was that many organizations have made the mistake of thinking of knowledge as a set of assets to be managed. This view leads to an over reliance on tools like repositories and lessons learned databases. The theory being that if we can get everything codified and ordered, then the knowledge will travel. This has proven to be insufficient for many reasons. One of the most significant reasons is that such an approach does little to account for how people learn and use knowledge. Here is where our own expertise as educators offered a particularly useful lens. We knew we needed to consider how adults learn, and in particular how they learn in the workplace, if we were to be successful in our efforts. We also know quite a bit about the human processes that structure the use of new knowledge. All these considerations needed to be accommodated in a truly effective knowledge management system.

Design Principles for a Collective Learning System

Rather than design a knowledge management system for Carnegie, I proposed that we need what I called a Collective Learning System (CLS). I define it as a set of interrelated social practices (routines, norms, roles, and processes) supported by technology tools that facilitate the learning necessary to achieve a group’s mission. It provides a means to increase capacity in an organization or in a multi-organizational structure such as a NIC. The name and the definition show a disposition towards social practices and learning which is more in line with the highly collaborative work of NICs. My current thinking about what Carnegie’s Collective Learning System should include is outlined in the seven principles that follow:

1. Attend to practical knowledge and appropriate processes for learning it.
Different types of knowledge call for different types of learning experiences. Given its work, Carnegie’s knowledge needs tend more towards practical knowledge, or the know-how needed to enact improvement. Efforts to teach and learn this practical knowledge need to follow what is known about adult learning.

2. Provide guidance on the social structures needed.
It is not enough to say that people should share knowledge; a CLS should articulate the social structures needed to support learning. Social structures include, but are not limited to, protocols, norms, physical setting, facilitation, and documentation.

3. Attend to psychological safety and lower barriers to entry.
Efforts should be made to create a space wherein people feel safe to learn and fail. A CLS will include conversations with people across different levels of the organization and of unequal power. Social practices should allow the person with the relevant expertise or insight to express it regardless of their status. Designs should allow for people to enter the learning process in various ways since different people learn and express themselves in different ways.

4. Embed the learning processes in the work.
Whenever possible, learning should be embedded in the actual work of practitioners. This will retain the context that generated the knowledge and where it will need to be applied, maximizing the depth of understanding and utility of the learning.

5. Take a system perspective.
Processes should aim to focus attention to all relevant components of the system. And, in crafting social processes, input should be sought from all relevant parties in the system.

6. Integrate learning from across the organization.
Knowledge is being created across the boundaries of units inside the Foundation and outside it. A CLS needs to serve as an integrator across these boundaries. While the work will continue to occur across boundaries, and it would be ineffective to have everyone involved in all the work, collective learning requires integration in order to maximize learning.

7. Take advantage of technology to the extent appropriate.
Technology, especially that which facilitates real-time collaboration, should be utilized when appropriate. However, technology should not aim to eliminate the human learning experience.

Although this is an ambitious list, in the sprit of improvement science, we’ve started small by iterating on the design of specific processes like after action reviews and failure analyses. As with all work at Carnegie, we will continue to learn our way into this problem.



July 2014

Creating a Classroom Culture for Student Success

Written by , Posted in What We Are Learning

When students walk into developmental math classes they are most likely carrying something weightier than their backpacks, something much more insidious. They bring with them negative mindsets that they can’t do math or that they aren’t a math person, reinforced by the history of past math classes where they experienced failures. And many bring with them the threat of stereotypes, some math-based and others defined by gender or ethnicity.

In designing two alternative mathematics pathways for students who place into college developmental math classes, Carnegie has acknowledged this student baggage as one of the key drivers that must be addressed in order to fully support student success. And they have embedded interventions into the instructional design of the two Pathways—Statway in statistics and Quantway in quantitative reasoning—to address these drivers.

At the annual Community College Pathways National Forum, Claude Steele, the leading expert on stereotype threat, and David Yeager, whose work on transforming student mindsets has been incorporated into the Pathways instructional system since the initial design, suggested approaches that in other settings had been shown to reverse the roles these threats play in negatively affecting the motivation and engagement of students, and thereby their educational outcomes and performance. As Steele explained, these influences are “powerful but not determinative.”

steeleSteele provided an example of stereotype threat especially relevant to Carnegie’s work. Female and male students who excelled at math at the University of Michigan were administered the half hour section of the graduate math exam. The premise was that the gender stereotype that females weren’t as good at math as males would suppress the performance of the female students, not allowing them to do the necessary cognitive work on the exam that they were clearly capable of doing. In this case, it held true. The female students significantly underperformed compared to the male students on the same test. Although both men and women were stressed merely by having to take a test, women experienced the additional pressure of the stereotype.

To mitigate the stereotype—“the preoccupying presence” as Steele puts it—students equally as gifted as the first tested cohort were told beforehand that this particular test was one on which women always did well. Under these conditions, the female students’ performance increased to match that of men. Similar impacts have been observed for various racial/ethnic groups as well.

Steele suggests remedies for stereotype threat. These include changing the cues that educators send to students. Changing the language in a classroom can create relationships between students, advisors, and teachers that tell a student that there is no presumption that he/she lacks something needed to succeed. Instead, there is a presumption that the student can succeed.

And the idea that ability is malleable is a tremendous relief to kids, and a liberating idea, Steele said. “It significantly reduces stereotype judgment.”

Carnegie is introducing the idea of malleability in the Pathways. One of the exercises in the Pathways Starting Strong package is an exercise where students read an article explaining that neuroscience shows that the brain is like a muscle and that with enough effort they can grow their brain.

Through the Starting Strong package, students who come to the Pathways thinking that they aren’t “math people,” or they don’t belong because they aren’t smart enough to succeed are supported in developing a “growth mindset.” In addition to learning from the article that intelligence is not fixed, students are given strategies to support persistence through the course, and the encouragement from the start and throughout the course that gives them the courage to use the strategies to succeed.

yeagerYeager said that in a randomized control trial the introduction of this one article on the concept of brain growth has been shown to have a significant effect on student persistence and success. There is evidence from selected Pathways classrooms that indicate that the effect has been replicated through the Starting Strong activities as well.

The Pathways—which subsume rigorous materials, new and more engaging pedagogies, the productive persistence interventions like the use of this exercise, and the behavior and speech that support it—have produced amazing results. Students have tripled their success rates in half the time and Carnegie has been able to maintain this level of student accomplishment, even as the initiative has grown to include new colleges, new faculty, and many more students over the past three years.

Yeager offered some specific recommendations to those attending this year’s Forum and just beginning to teach the Pathways. He said to create a class culture that supports success, not one that implies expectations of failure. He said to provide praise after accomplishment (not disassociated from effort and accomplishment), encouragement often, and continuous feedback—in class, during office hours, through emails.

brain-workoutHe said to continue to remind students that the brain is analogous to a muscle, that “the more you use it, the better it works.” Or, “the more you practice, the smarter you become.” When students seem to get discouraged, give them a boost—indeed, there are “booster” activities included in the pedagogy that Pathway faculty use. Use phrases like: “other students say that when you come to the difficult part where you have to struggle, it is a particularly helpful and productive part of the learning process” or “when you struggle, then you’re growing.” Yeager said that the really wonderful thing about what he had discovered in his research is that the lowest achievers change the most and become some of our highest achievers.

Yeager concluded with a challenge: “Students have theories about their success,” Yeager said. “It is up to us to shift those theories” to more positive and productive ones.



July 2014

How to Spur Improvement Activity in Networks

Written by , Posted in What We Are Learning

Over the past five years, the Carnegie Foundation for the Advancement of Teaching has launched a set of three Networked Improvement Communities (NICs). We have played roles in the launch and support of two NICs in particular, the Building a Teaching Effectiveness Network and the Student Agency Improvement Community.

NICs are scientific learning communities distinguished by four essential characteristics. They are: (1) focused on a well specified common aim; (2) guided by a deep understanding of the problem and the system that produces it, and a shared theory of how to improve it; (3) disciplined by the rigor of improvement research; and (4) coordinated to accelerate the development, testing, and refinement of interventions and their rapid diffusion out into the field, as well as their effective integration into varied educational contexts. These characteristics create conditions under which organizations can learn from their own practices and benefit from innovations from both within and outside of their organization to systematically address high-leverage problems.

Through the initiation and development of several NICs, Carnegie has gained some insight into what it takes to spur improvement activity in networks:

Start with a high-leverage problem of practice
A high-leverage problem of practice is an issue that, if addressed, can disrupt status quo practices in an organization and render improvements throughout the system. This is a compelling problem area that, if solved, will propel the organization toward achieving its core mission. Long-term leadership and stakeholder commitment to solving this problem is critical to the success of the NIC, and the determination to work on a recognized high-leverage problem (as perhaps one that the organization has struggled with for a while) can do much to evoke that commitment and will. Moreover, the process for selecting and messaging the NIC’s high-leverage problem needs to be transparent and evidence-based.

Build on work already being done
With more rigorous standards, evolving policy demands, and tight budgets, school personnel are striving to realize increasingly ambitious objectives with limited resources. The desire to apply improvement science in networks often outstrips school personnel’s capacity to conduct such work. To introduce improvement work into a district, we would recommend beginning the improvement activity in an existing team or learning community. Infusing improvement methods into existing collaborative structures (e.g., partnerships, meeting structures, conceptual framings) adds capacity to work already being done as opposed to adding “one more thing” to the work of educators who are already operating at full capacity.

Assemble a diverse team
We have found it efficient to start improvement work in existing collaborative structures, but it is also true that solving high-leverage problems in schools will require a range of perspectives and levels of expertise. Given the interdependencies of processes in complex systems, often the processes of improvement will uncover previously overlooked drivers of the problem. It is not uncommon for new team-members to be added as the improvement work evolves. Organizations can be hierarchical, but in NICs, each member brings an essential perspective to solving the problem at hand. In fact, in improvement work, it is often the case that those workers closest to the “front line” are those with the best ideas about how to solve the problem. NICs can foster a spirit of co-development by demonstrating openness to feedback and rapidly integrating and testing member ideas in the network.

Provide access to improvement guidance
Improvement science offers a set of new frameworks and methods for approaching work. As with most newly acquired skills, users will struggle in the process of integrating this approach into real-life contexts. It is imperative to ensure that just-in-time feedback and support is provided reliably in order to scaffold learning and help members see the value in the improvement work. Early wins will also help to build will for the work in the organization.

Balance in-person and virtual communication
Launching a network is best done in an in-person convening of network members and stakeholders. Often convenings can galvanize enthusiasm around solving the high-leverage problem and build momentum for the work. However, it is often the case that that enthusiasm can wane when members return to face the challenges of their daily routines and momentum often flags. A collaborative online platform can foster continued communication to build upon the sense of community garnered at the first convening. It can also provide for the sharing and spread of what is learned through ongoing improvement efforts.

Certain conditions are obvious prerequisites for seeing improvement work gain traction in practice: building the improvement capabilities of professionals through training and ongoing coaching, for example, or creating the infrastructural capacity by establishing a supporting Hub that provides supports for improvement science, collaborative work, knowledge management, etc. The items introduced here have emerged in our work as more particular issues that require attentions if improvement work is to be pursued in a manner that is deep, widespread, and enduring.



May 2014

Iowa Mentoring Program Targets Needs of Beginning Teachers

Written by , Posted in What We Are Learning

As Carnegie Senior Associate Susan Headden writes in her recent report “Beginners in the Classroom,” public education loses a lot of new teachers to attrition, upwards of 750,000 a year, and pays a heavy price in talent and treasure. They leave for many reasons, Headden reports, but at the top of the list are concerns related to lack of support, such as limited professional development, little helpful feedback on performance that supports improvement, and feeling isolated from colleagues.[1]

Mentoring programs for new teachers may help address these issues. Effective mentoring programs, research suggests, promote new teachers’ sense of professionalism and hence their satisfaction and retention. Such programs can improve teachers’ instructional abilities and thereby increase their students’ achievement.[2]

Of course, not all mentoring programs are created equal. Among the success stories is Iowa’s Grant Wood Area Education Agency (AEA). In 2000, Iowa passed a law requiring that every new teacher have a mentor. Today’s iteration of the program benefits from knowledge gleaned from early mistakes. In the beginning, mentor teachers were given a stipend, but no training or release time from their own classroom duties in order to meet with mentees. The program had no oversight, the mentors were accountable to no outcomes, and no data were collected on implementation or results. Turnover among new teachers remained high.

Currently, however, mentors are released from their classrooms for three years; they are full-time mentors who stay with the same group of mentees for two years. Mentor selection is rigorous. Each applicant is interviewed multiple times, illustrates their ability to create model lessons, provides assessments of student work, and writes essays to show evidence of his or her capacity for reflection, a necessary skill for mentor success. Perhaps the most impressive component of the program is the training provided to Iowa mentors through the New Teacher Center, a non-profit organization that helps train new teachers. Sessions are differentiated for both new and expert mentors, and there are options for administrators as well. During training, mentors complete assignments in conjunction with their beginning teachers as well as reflecting on their own assignments.

Collectively, the Grant Wood AEA’s model includes what beginning teachers need in order to feel supported: instructional guidance, frequent and actionable feedback, and meaningful relationships within the school. Careful selection of mentors, as well as the ongoing training they receive, sets mentors up to succeed, giving them the instructional and reflective tools they need to meet their mentees wherever they are in their practice. Mentors must periodically submit evidence of their meetings with mentees, ensuring that new teachers are in fact getting individualized and ongoing feedback from their mentors to help improve their practice.[3] Providing full release time to mentors means that they can spend a significant amount of time with each mentee, forming meaningful relationships based on trust and support. This last piece is perhaps the most important, since these relationships help to tie new teachers to their schools, sustain them in the difficult work of beginning teaching, and keep high-performers in the profession.

Though it is too early to determine the long-term effects of the program, feedback from teachers, mentors, and principals has been overwhelmingly positive. Officials are collecting data on the implementation and impact of the program, including information on which skills mentors are helping their mentees develop. They have discovered that, early in the school year, new teachers’ primary concerns are classroom management and instructional planning—valuable insight that can help schools target future professional development efforts. Data collected thus far show that all mentors are spending 60-90 minutes per week with each mentee. And, critically, beginning teacher attrition is low; of the 33 new teachers who mentors worked with in the 2012-2013 school year, only two have left.[4] Time will tell if Grant Wood AEA’s mentoring program has a lasting effect on teacher quality or teacher turnover, but based on initial results and feedback, it seems that it is providing beginning teachers with the support and sense of belonging they need in order to improve and to stay in the profession.

[1] Beginners in the Classroom, pg. 5.

[2] Richard Ingersoll and Michael Strong, “The Impact of Induction and Mentoring Programs for Beginning Teachers: A Critical Review of the Research,” Review of Education Research. Vol. 81, 2 (2011): 201-233. Retrieved from:

[3] Grant Wood AEA, “Mentoring and Induction Program.” Accessed April 28, 2014.

[4] Beginners in the Classroom, pg. 22.