August 21, 2008

Challenges

U.S. public education faces numerous challenges, including international competition, stagnant U.S. achievement, and the achievement gaps between groups of U.S. students. For example, on the latest TIMSS (Trends in International Mathematics and Science Study), U.S. eighth graders performed below their peers in 20 of the participating 45 countries in mathematics and 13 countries in science. Five Asian countries (Chinese Taipei, Hong Kong SAR, Japan, Korea, and Singapore) along with Estonia and Hungary outperformed the U.S. in both mathematics and science at the eighth-grade level. Comparable groups of students were used across countries (i.e., the general U.S. student population was not compared to elite groups of students in other countries).[1] U.S. student achievement has not improved significantly over the last several decades, despite dramatic increases in educational expenditures, and despite the growing advantages of increased educational attainment.[2] For example, in Utah, the percentage of students scoring at a proficient (or passing) level or higher in fourth-grade reading has increased only slightly, from 30% in 1992 to 34% in 2007 (see Figure 1), while the majority of students continue to perform below the proficient level (which is usually considered the target or “passing” level). There is no statistical difference in the average fourth-grade reading score for 1992 and 2007.

In addition to increased international competition coupled with stagnant U.S. achievement, public schools also face a significant and persistent achievement gap between whites and minorities and between wealthy and poor students. 2007 NAEP data show that African American, Hispanic and poor students are already two to three years behind other students in reading and math by the end of fourth grade. On the 2007 NAEP, Utah’s achievement gap between students who are not poor and poor students was about two grade levels for eighth grade math (see Figure 2). For whites and Hispanics, the gap was about three grade levels (see Figure 3).

In considering these educational challenges, many reformers naturally focus on teachers and teacher quality. Teacher salaries represent the largest chunk of educational expenditures. In addition, most education experts argue that teacher quality is the most important schooling factor influencing student achievement.[3] There is also reason to be concerned that teacher quality has declined over the last 50 years, due to declining teacher salaries relative to other professions and expanded labor opportunities for women.[4] There is also substantial evidence that teacher quality, as measured in a number of different ways, varies systematically by demographic characteristics of student populations. Based on evidence from three states, researchers with the Education Trust, a nonprofit organization advocating for better educational opportunities for poor and minority students, find that low-income and minority students are assigned to teachers who have less classroom experience and who are less likely to have a college minor or major in the subject they teach.[5] These patterns in the distribution of teacher quality in the U.S. have important implications in terms of equal educational opportunity and the achievement gap. The large disparities in teacher quality can partly be explained by the fact that teachers prefer to work in schools with higher achieving, higher income, non-minority students.[6]

There is also concern that the imminent retirement of many teachers, increased enrollment (the “echo boom”), and opportunities in non-teacher labor markets will result in serious teacher shortages.[7] In Utah, at the same time that student enrollment is growing, the number of college students studying to become teachers is decreasing and the rate of teacher attrition is rising, all contributing to a shortage of teachers in Utah’s public schools. Already, a majority of Utah public school districts are reporting problems with finding and hiring teachers.[8]

Defining & Measuring Teacher Quality

Most everyone agrees that student achievement is a key goal in education. Although student achievement on standardized tests is now widely accepted by the public and especially policymakers as a legitimate measure of educational output, this method of measurement continues to have critics. Critics argue that standardized tests have questionable validity, since they measure only particular skills for certain content areas in a possibly biased manner that is divorced from the relevant real-world context. In addition, when teachers know that outputs are measured according to standardized tests, they have a powerful incentive to teach students to take the test well, rather than to teach students the skills and abilities that the tests are intended to measure. Nonetheless, standardized tests represent the most objective, widespread tool for measuring student achievement. Measurements of teacher quality are therefore often based on student achievement on standardized tests. However, it is important to keep in mind that in addition to high test scores, Americans may also expect schools to foster such qualities as creativity, civic virtue and social skills, which are generally difficult to quantify.

The problem of measuring teacher quality in terms of student outcomes is complicated by the fact that many factors that are outside of the control of teachers also affect student outcomes. Many researchers and policymakers are therefore very excited about value-added modeling (VAM), “a collection of statistical techniques that uses multiple years of student test score data to estimate the effects of individual schools or teachers.”[9] While VAM has great potential for identifying effective teachers, and is already being used by several states to describe the performance of schools and school districts, VAM does have technical issues that make it susceptible to bias. Potential statistical problems include: teachers with small classes (resulting in too small sample sizes), the use of observational data rather than data from a controlled experiment (in which students would be randomly assigned to different teachers), student mobility resulting in missing data, lack of comparability of tests across years, and uncontrolled effects from the characteristics of school, districts, or prior teachers.[10]

A 2004 RAND review concluded that VAM could be used to identify extremely effective or ineffective teachers, but should not be used to rank teachers (because of uncertainty in estimates) or be used for high-stakes decisions. RAND did recommend that states develop databases that can support VAM estimation of teacher effects. The National Council on Teacher Quality concludes that states need a longitudinal data system with three types of data (student identifier system, teacher identifier system, assessment system that matches student test records over time) to implement a value-added model. Utah has all three.[11]

In recent years, several researchers have attempted to synthesize the numerous studies of teacher quality. These studies generally utilize student performance on standardized tests as a measure of teacher quality. The conclusions of four of these literature reviews of teacher quality research are discussed below.

Eric A. Hanushek and Steven G. Rivkin, 2003[12]
Hanushek’s and Rivkin’s review of the teacher quality research was prepared for Brookings Papers on Education Policy in order to relate existing evidence to policy initiatives. Both Hanushek and Rivkin are prominent experts in the economics of education. Using “high quality estimates” (value-added estimation conducted within a single state) from studies through 1994 in the U.S., Hanushek and Rivkin find no support for a relationship between teacher education level (such as a master’s degree) and student performance, some (but not very strong) evidence of a positive relationship between teacher experience and student achievement, and no strong evidence that teacher salaries are a good measure of teacher quality. Hanushek and Rivkin report that teacher scores on achievement tests have more frequently been correlated with student outcomes than teacher education, experience or salary, but that the evidence is “far from overwhelming.”

Finally, Hanushek and Rivkin conclude that the magnitude of total teacher effects is “impressive.” They cite research that the difference between a good teacher and a bad teacher can be a full grade level of student learning in a single academic year, and that a string of good teachers could effectively close the achievement gap between lower income and higher income students. Hanushek and Rivkin conclude that “the currently available data give us little reason to believe that we know enough about good teachers to set appropriate training and hiring standards.” They also state that principals and superintendents need to be held accountable for their management decisions in order to motivate them to make “difficult and uncomfortable” decisions instead of taking the “path of least resistance” with respect to teacher tenure.

National Council on Teacher Quality, 2004[13]
“Increasing the Odds: How Good Policies Can Yield Better Teachers” was written and published by the National Council on Teacher Quality (NCTQ), after being reviewed by education experts Eric Hanushek, Michael Podgursky, Richard Murnane, and Dan Goldhaber. The research synthesis concludes that advanced degrees have no impact on teacher effectiveness and that education courses taken before teaching have little impact. They find that a few years of experience makes a teacher more effective while strong training in secondary subject matter (as opposed to training in pedagogy) adds “significant value.” The publication also concludes that more effective teachers score relatively higher on literacy tests and attend more selective colleges. Also, the report states that traditional routes have no advantage over alternative routes, and that there is limited evidence for a positive effect from matching teacher and student race. In addition, they claim that the “teacher attributes that matter most,” such as the ability to motivate students, are the hardest to measure. While acknowledging the nuances and complexities of the educational research, the National Council on Teacher Quality recommends that policymakers reconsider basing teacher salary on accruing credits and experience, allow flexible certification systems, and pay attention to potential teachers’ academic credentials.

The Future of Children (Princeton University & Brookings Institution), 2007[14]
The Spring 2007 edition of the publication The Future of Children was entitled “Excellence in the Classroom” and included research reviews by numerous education researchers. Among the findings of the participating researchers: difficult working conditions may drive teacher transfers and turnover, performance-based pay for teachers is problematic but could be effective if the process were continually improved, neither graduate course work nor most activities considered professional development boost teacher quality, and telecommunication and technology could potentially offset some of the drawbacks associated with teaching in rural areas.

Based on the body of research presented, Ron Haskins and Susanna Loeb propose loosening (but not eliminating) certification requirements in order to increase the pool of teachers; identifying effective teachers through value-added models in combination with more traditional methods of evaluation; granting tenure to only effective teachers; using differential pay to attract teachers into problem schools and difficult-to-staff fields; and designing professional development that lasts several days, focuses on subject-matter instruction, is aligned with the goals and curriculum materials of the teachers’ school system, and utilizes mentoring arrangements.

Laura Goe and Leslie M. Stickler, National Comprehensive Center for Teacher Quality, 2007
Goe’s research synthesis was prepared for the National Comprehensive Center for Teacher Quality, a collaborative effort of the Education Commission of the States, Educational Testing Services, and Vanderbilt University with funds from the U.S. Department of Education. Goe and Stickler also produced a policy brief for educational leaders and policymakers based on the research review. Goe utilized studies published between 2000 and 2007 along with earlier landmark studies to examine four categories of teacher quality indicators: teacher qualifications (paper credentials), teacher characteristics (attitudes and attributes), teacher practices (instructional strategies and interactions with students), and teacher effectiveness (a teacher’s contribution to students’ learning).

Regarding teacher qualifications, across studies, Goe consistently found strong, positive effects for teachers’ knowledge of mathematics (whether measured by courses, degree, or certification), particularly at the secondary level, as well as teacher’s experience during the first five years of teaching. Several studies found that “professional development that is sustained, aligned with the curriculum and focused on instruction” positively influences math and science achievement at all levels. With respect to teacher characteristics, Goe reports no clear consensus on any of the measured characteristics, but does report positive school-level (rather than individual-level) effects from teacher collaboration. Although statistically significant and convincing evidence is lacking for the effect of teacher practices or instructional quality, studies do suggest that “best practices” include alignment of instruction and assessments, clear objectives and expectations, intellectual challenge, frequent assessments during learning with feedback to students, as well as interactive and hands-on teaching. In addition, several studies find that expert observers and principals can successfully evaluate teacher quality.

Teacher effectiveness measures teacher quality based on growth in student learning (most often through value-added methods). Goe concludes that empirical studies demonstrate the existence of differences in teacher effectiveness and attribute the majority of this variation to “unobserved variables.” Goe’s and Stickler’s policy brief urges leaders and policymakers to remember that the importance of context (different subject matter, grade levels, student populations) suggests that “it does not make sense to demand a uniform set of qualifications for all teachers.” They also urge readers to remember the limitations of using standardized tests to measure teacher quality and to consider teacher contributions to other outcomes, such as “self-esteem, student attendance, teacher collaboration and collegiality, and school culture.”

State Teacher Policies

Teacher Preparation and Certification

Reformers disagree whether teacher preparation and certification requirements (such as completion of coursework, performance on assessments, and student teaching) should be tightened or loosened. No Child Left Behind and the U.S. Department of Education advocate more rigorous teacher preparation program standards and certification requirements in order to improve teacher quality. Others, especially researchers who find little or no connection between certification and student achievement, caution that entry requirements may decrease rather than increase teacher quality since little is known regarding “which aspects of certification improve teaching and student achievement and which aspects so reduce the pool of teachers as to worsen student outcomes.”[15]

According to Education Week’s Quality Counts 2008, Utah does not require substantial coursework (a major or the equivalent) in the subject area taught for teacher certification while 27 other states do.[16] However, the Director of Educator Quality and Licensing for the Utah State Office of Education (USOE) reports that all of Utah’s teacher preparation institutions have rigorous content paths (as required by their accreditation bodies), and that Utah requires a major or major equivalent for licensure which also ensures a high level of content preparation.[17]

In 2007-2008, Utah joined 41 other states in requiring teachers to pass a subject-matter knowledge assessment for initial licensure. Six states also require a subject-matter pedagogy assessment as well. Thirty-nine states require an assessment of basic skills. Four states require only a basic skills assessment for initial licensure. Thirty-five other states require the basic skills assessment in addition to one or both subject-matter assessments. Utah has neither a subject-matter pedagogy test nor a basic skills test. Three states require no assessments at all for initial licensure.

According to the U.S. Department of Education’s 2005 Report on Teacher Quality, Utah and 46 other states have alternative routes to teacher certification. Alternative certification programs generally provide shortened training focusing on pedagogy to potential teachers who already possess subject matter mastery. For the 2004-2005 school year, 5.3% of Utah teachers were not fully certified versus 2.5% of teachers nationally. For the 2006-2007 school year, 3.8% of Utah teachers were not fully certified (national data not yet available).[18]

According to Education Week’s Quality Counts 2008, Utah does not hold teacher-preparation programs accountable for graduates’ performance in classroom settings (as 18 states do). Utah does provide assessment pass rates (as required by the U.S. Department of Education) for teacher preparation programs, but does not rank institutions (as 30 states do). According to TQ Source (a database of state teacher policy compiled by the Education Commission of the States), for undergraduate teacher preparation programs, 33 states have state intervention or closure policies for poorly performing programs, and 38 states have data collection requirements for their programs. Utah has neither.

Teacher Evaluation and Tenure

Because many reformers believe that teacher qualifications and characteristics (such as years of experience) account for very little of the variation in teacher quality, they emphasize the need for more rigorous evaluation of teachers on the job. And because collective bargaining agreements make it difficult for a teacher to be fired once he or she has achieved permanent status, or tenure, some reformers have suggested postponing tenure so that a teacher can be more easily terminated if he or she does not show sufficient competence or progress during the probationary years. The National Council on Teacher Quality (NCTQ), for example, recommends formal evaluations of all teachers annually and extending the probationary period to a minimum of five years. According to NCTQ, only 14 states require annual evaluations. Utah requires the annual evaluation of provisional and probationary teachers, but leaves the frequency of evaluations for permanent teachers to the districts. In addition, Education Week reports that Utah does not require evaluators to receive formal training (26 states do). Nationally, teachers generally achieve tenure after a probationary period of one to five years. Currently, 35 states (including Utah) have probationary periods of three years. In 8 states teachers achieve tenure in less than three years, and in 6 states teachers achieve tenure in four or five years.[19]

Teacher Salary

Across-the-board salary increases can help attract more people into teaching and increase teacher retention, which could help eliminate teacher shortages and decrease the number of inexperienced teachers in classrooms. However, many education experts argue that such salary increases are inefficient and, due to the difficulty of identifying high quality teachers during the hiring process, do not necessarily result in higher teacher quality. Educational researchers tend to favor differentiated salaries for teachers as a more effective use of public funds to improve teacher quality. There are several forms of differentiated pay, including pay based on demonstration of knowledge and skills (through national board certification, for example), working conditions, subject area, and student performance. Many experts caution against over-reliance on students’ test scores to determine teacher quality.

Education Week reports that 10 states have “teacher-salary parity,” meaning that public school teacher salaries are comparable to occupations with similar skill demands (accountant, nurse, etc.). Utah is described as having teacher salaries equal to 87% of the salaries of comparable occupations, which puts Utah in the bottom third of the states for this indicator.[20] However, Utah has significantly increased teacher salaries in recent years, including an across-the-board $2,500 increase in teacher salary for 2007-2008 and an additional $1,700 budgeted for 2008-2009. Because this is an across-the-board dollar amount raise (rather than a percent increase in salaries), beginning teachers salaries will jump by the largest percentage, which lawmakers hope will attract more people into the profession.[21]

According to TQ Source, 11 states have state-level differentiated teacher compensation programs, defined as programs funded by the state and in state statute that institute a pay structure different from the traditional method of teacher compensation (which is based on course credits and experience). The pay structure must also be partially tied to teacher job performance, often measured through growth in student achievement. The programs can be mandatory or voluntary. Examples include Minnesota’s Q-comp which includes a pay-for-performance component and allows districts to voluntarily apply for state funds in order to offer it to the teachers in their district. While TQ Source does not currently describe Utah as having a state-level differentiated teacher compensation program, the 2008 Utah legislature did approve funding for one-time performance-based compensation. The Utah State Office of Education is in the process of evaluating charter schools’ and districts’ plans for the experimental pay-for-performance program.[22]

In addition, Utah has neither differentiated pay for rural schools (as 11 states do) nor differentiated pay for hard-to-staff schools (as 9 states do). However, Utah does have differentiated pay for high-needs subjects, along with 11 other states. The statutes instituting differentiated pay in Utah were passed during the 2008 legislative session: the Teacher Salary Supplement Program funds teacher salary supplements for secondary math, integrated science in grade 7 or 8, chemistry, or physics; and The Utah Science Technology and Research Initiative (USTAR) Centers Program potentially awards increased compensation for math and science teachers. Finally, 30 states provide a wage premium or bonus for national board certified teachers. Thirteen additional states (including Utah) will pay the certification fee or provide tuition support for national board certification, but offer no additional financial incentive.[23]

Mentoring and Professional Development

States have long relied on professional development to inform teachers of new state and federal policies as well as to train teachers on new curriculum and performance standards. Now, professional development is viewed as vital to school success due to the complex challenges and rigorous academic standards of today’s classrooms. The traditional one-shot workshop taught by an outside consultant has been widely criticized by experts. Instead, researchers have identified model professional development programs as being collaborative, sustained and coherent, engaged in practical tasks, and involving feedback and follow-up activities.[24] Most of the responsibility for professional development falls upon districts, but states have adopted policies to guide districts, regarding professional development standards and minimum time for professional development, for example.[25]

Utah and 23 other states finance professional development for all districts. Thirty states (including Utah) require districts to align professional development with local priorities and goals.[26] In Utah, state law requires the local school board to review schools’ professional development plans to ensure their compatibility with the district plan.[27]

Many teachers begin professional development as soon as they enter teaching through induction or mentoring programs. Of the 42 states with mentor programs, 25 states, including Utah, require teachers to participate in a state-funded mentoring program.[28] Utah’s mentoring program for new teachers lasts 3 years, which is the most that any state requires (more than half of the states with mentor programs require only one year).

Conclusion

There is some consensus that teaching experience (up to 5 years), subject matter mastery for secondary subjects (particularly science and math), academic ability (teacher scores on achievement tests, quality of undergraduate institution), and well-designed professional development positively impact teacher quality. Surprisingly, there is little evidence that teaching experience beyond five years and graduate courses, the two determinants of teacher salary schedules, affect student achievement. Also, because research shows that certification requirements affect the teaching pool, and because certification has not been conclusively tied to student achievement, it is important to carefully consider certification requirements. Researchers also conclude that teachers are distributed inequitably, in part because teachers tend to prefer working in high-performing schools with fewer disadvantaged students. Because the information available to schools and districts when hiring teachers explains very little of the differences in teacher effectiveness, many experts favor a longer probationary period or more rigorous evaluations of teachers on-the-job to enable the firing of ineffective teachers. Although value-added models hold great promise for isolating the effect of individual teachers on student achievement, experts also caution against overreliance on VAM and test scores in general when evaluating teachers. However, it may be useful in identifying the very best and very worst teachers, which could be useful for weeding out bad teachers.

States are experimenting with many ways to improve teacher quality. Utah is utilizing some of these methods, including differential pay for math and science teachers as well as experimenting with performance pay. Utah should consider additional promising reforms to improve teacher quality, such as additional measures to hold teacher preparation programs accountable for quality, more rigorous evaluation of teachers on the job, and incentives to draw teachers to hard-to-staff schools.

Endnotes

This research brief was written by Utah Foundation Education Consultant Elizabeth Escandon. Foundation President Stephen Kroes may be reached at (801) 355-1400 or by email at: steve@utahfoundation.org. For more information about Utah Foundation, please visit our website: www.utahfoundation.org.