Background
In July 2018 the EdQ Center was selected to participate in an Improvement Research Fellowship designed to build the improvement capacity of participating teams and support improvement research on teacher preparation. We are utilizing the fellowship to study data use for program improvement within the CSU educator preparation system. The fellowship is supported by the S.D. Bechtel, Jr. Foundation and led by the WestEd improvement science team. This opportunity grew from our Center's work as part of the New Generation of Educators Initiative (NGEI) that sparked the development of the EdQ DataView dashboard and EdQ's new vision for supporting data use that drives continuous improvement in educator preparation programs. Our team, along with teams from Fresno, Bakersfield and San Luis Obispo was awarded fellowships and embarked on a 13-month improvement journey using the tools of improvement science.
Improvement Science
Improvement science is an organizational learning theory with associated tools, practices and mindsets that brings together research knowledge about what works, with professional knowledge about organizations, and improvement knowledge about complicated systems. W. Edwards Deming is credited as the father of this school of thought with his book
The Profound Theory of Knowledge first published in 1986. In recent decades, the Institute for Healthcare Improvement (IHI) has made improvement science a common practice and a way of making quick adjustments to improve patient care and outcomes. In education, K-12 and higher education leaders have begun to embrace and use these principles and practices to solve tough problems thanks in large part to the work of the Carnegie Foundation's Center for the Advancement of Teaching. Fellowship coaches from WestEd are trained facilitators of improvement science and have been supporting each of the fellowship teams along the way.
How is EdQ Using Improvement Science?
The improvement process begins with focusing in on what you want to see improved in very specific ways. An organization starts with asking, "What is the gap between our vision as an organization and the reality on the ground?" Once you have zeroed in on a problem area through various types of data collection and analysis, the task is to engage in cycles of inquiry to better understand what is causing the problem and what changes might make a difference.
Problem Focus
For EdQ, this meant first examining the EdQ DataView Dashboard through the lens of the user. After launching the dashboard in December 2017, we introduced it to campuses across the system, ran trainings and webinars, and provided online resources and coaching. What we wondered was, how many campuses are now using the dashboard regularly to make program improvements? The answer was discouraging. After analyzing user data, we learned that only a small number of campuses were using the dashboard with regularity. While most of those said they used it for improvement, little was known about what exactly that meant to them. In general, this made us wonder about the prevalence of data use for improvement (in any form) as a part of campus culture in educator preparation programs across the CSU system. This became the problem focus.
Analysis and Investigations
The next step for our EdQ team was to research potential obstacles or root causes for this problem. EdQ Improvement Coach (Melissa White, Senior Researcher with WestEd), worked with the team to do a root cause analysis exercise called a "fishbone diagram." The fishbone diagram helped us map the problem within the larger system of CSU teacher preparation. Seeing the system that exists around a problem helps better identify critical points of leverage for potential action. We next visited CSU Bakersfield and observed one of their all-hands data meetings and conducted "empathy interviews" with faculty, staff, and administrators. Empathy interviews help to reveal the "experience" of someone who is involved in the problem area. The Bakersfield interviews helped us see the importance of creating a safe space for data review where teachers can view data as a helpful tool for improvement rather than a tool for compliance, evaluation or judgment. Bakersfield had been successful in creating this type of culture.
Identifying Gaps and Barriers
Our Team has also worked with campus deans to get their perspective on the biggest barriers to regular data use for program improvement on their campuses. In June, EdQ's data scientist, Sarah Kolbe, facilitated a gap analysis exercise with the deans. This exercise revealed some expected and unexpected results. While EdQ expected to learn that the lack of coherent data systems across campuses was still a major barrier, they also found that deans perceived faculty's lack of engagement with data for improvement purposes to be a major barrier as well. In the most recent deans' meeting, EdQ's Director Paul Tuss, and new Data Coach Ginger Simon conducted a root cause exercise with the group to further explore faculty engagement with data. Ideas were shared such as an over-abundance of data for compliance taking the focus off of more targeted data for improvement.
Testing Ideas For Change
With earlier findings in hand, we had already begun to test an idea for change. We are examining whether using a diagnostic rubric to assess the quality of data use on campuses may be helpful for helping campuses begin a conversation about their current state of data use, and track their growth in data use as they work on improving it.
Improvement Science emphasizes rapid cycles of testing changes before going to scale. The most common tool for this is the Plan-Do-Study-Act cycle (PDSA). To date, we have conducted two PDSAs by creating a draft rubric adapted from an existing rubric that emphasizes collaborative cultures and continuous improvement, sharing the draft rubric with two campus leaders, incorporating their feedback on the usefulness of the tool, and adjusting their vision for how and with whom the tool would be used and facilitated.
What's Next?
Shortly, EdQ will begin working with volunteer campuses to test the newly improved and adapted rubric through facilitated sessions with educator preparation program teams. The goal is to ultimately engage with five campuses this year to use the rubric and to continue to work with those campuses to help build capacity for improvement work.