A pro-charter-school paper by Thomas Kane at the Harvard Ed School has been going around and reached us here in Sudbury. I looked into it a bit and wrote what I think is a reasoned response, which I have circulated locally and on the MaDemsForum. Today with the help of some prayer and fasting, I have converted it to html, and I’m posting it here. I hope it will be of some use in the discussion of Question #2.
Thomas J. Kane’s Analysis of Charter Schools in Massachusetts
Carl D. Offner
1. Introduction
Thomas J. Kane is a professor at the Harvard Graduate School of Education. By
profession he is actually an economist. He has recently put out a paper,
“Let the Numbers Have Their Say”,
which purports to use rigorous statistical data to show that charter schools, at
least in Massachusetts, are really quite good and should be supported. The
conclusion of the report is that people should vote Yes on Question #2 on the
November ballot, opening the door to 12 new charter schools in Massachusetts
each year.
His paper has been publicized and passed around quite a bit, and I think it’s
important to look at it critically. And I intend to do that here, although
really the paper touches on so many topics that a complete consideration of it
would be a massive amount of work, more than I have time for. However, I have
treated some of the main points he raises, enough to cast serious doubt on his
entire approach.
This critique contains several parts. First, I need to make some general
comments on the role of statistics and measurement in education. In large part
they are based on my own experience. I was a Junior High School and High School
teacher for 16 years, and for the last 25 years or so I have been moonlighting
at UMass Boston teaching one Computer Science course a term, mainly to advanced
undergraduates and graduate students.
Next, I consider three key points that Kane addresses in his paper.
And in conclusion I consider some related issues involving charter schools
and Question #2.
2. The difficulty of measurement in education
2.1 Statistics in education
Statistics is a messy business in any case, but when it comes to education,
it’s particularly difficult. I have seen statistics used to arrive at
conclusions that were silly or destructive or both. Here are two things I
remember from my own teaching career:
- When I was a junior high school teacher, I subscribed to a journal of
“research in mathematics education”. While I don’t remember many details at
this point, I did stop subscribing after 2 years because I found it completely
useless to me as a practitioner. Here’s one thing I recall (and again, the
details elude me, but this is the way I remember it): there were two articles
published within a few months of each other—each on different ways of
teaching something like adding fractions—which came to exactly opposite
conclusions; each “within the .05 level of significance”. What could that
possibly mean?Well, what it means is that there were lots of variables that were not taken
account of and quite likely could not have been. And so the results were
actually meaningless. - When I was teaching, administrators loved to tell us that large classes
were fine because studies had shown that class size made no difference. We
heard this story for years. And there were in fact such studies.
Every teacher, and pretty much every parent, knew that these results were
bogus, but they were relied on heavily by “decision makers”. It was only
much later that more careful studies were done, that took into account such
matters as family income. And those studies showed that class size was in
fact very significant. So you don’t hear about this any more. But a lot of
damage was done in the meantime.
2.2 Measurement of student achievement
This is notoriously difficult and really impossible in any exact sense. What
students get out of school, or out of any particular class or from any specific
teacher, is not only hard to measure, but may not in fact be evident for many
years. Further, the success of a school is due in large part to the support
that it receives.
I taught Junior High for 13 years. It was in a regional school district that
was not a prestige system, but was respectable and in which there were many
teachers who I would have been happy to have teach my own children. Junior High
students are notoriously confusing. They are growing rapidly, at vastly
different rates. Some have gone through puberty; others won’t even start for
some years. They are still trying to figure out at a very basic level who they
are. As one of my colleagues said, “Some of these kids are going to be brain
surgeons. And others will wind up with honest jobs on the town road crew. And
you can’t tell at this point who is who.” We saw our job as “getting them
through”. And by and large, we did.
Then Proposition 2 1/2 was passed. 25% of the teachers were laid off that
year. Class sizes increased significantly. Much more of our time was spent in
what is euphemistically called “classroom management”. And many students who
needed some extra hand-holding didn’t get it. And many of these students
started slipping through the cracks.
For the most part, parents did not understand the causes of this. They
couldn’t see the class size increases, and they couldn’t see the difference in
how classes had to be conducted. They tended to blame either the teachers or
their own children—I saw both.
2.3 Evaluation of teachers and administrators
Finally, evaluation of teachers and other education professionals is
extremely difficult. Do you remember Chris Gabrieli? He ran for the Democratic
nomination for Governor in 2006, the year Deval Patrick won. I remember
Gabrieli saying in a primary debate on education: “You measure the kids at the
beginning of the year, and you measure them at the end. And that’s the way you
evaluate teachers.” It’s an industrial model (and Mr. Gabrieli in fact comes
from an industrial background), and if children were refrigerators being put
together on an assembly line, perhaps it might make some sense. But they’re
not. Every teacher has had classes that from year to year are wildly different,
and students who change significantly from month to month. From time to time we
are asked to quantify the gains our students have made. We do this because we
have to, but no one believes any of it. It’s make-work and it contributes
nothing to our ability to be better teachers.
Chris Gabrieli is now an adjunct lecturer at the Harvard Graduate School of
Education.
The best principal I ever had—and this was an opinion shared
universally by the teaching staff (and I believe by the parents)—was a
person who would have failed any standard way of evaluating principals. Just as
one example: new textbooks got ordered late and often weren’t available in
September. But he was remarkable in his intuitive ability to size up and
hire—and keep—good people, and he maintained a faculty that was
proud to teach in his school and who were given the freedom to teach to their
strengths. He cared a lot about the teachers and about the students. In
addition to having high standards, he had a remarkable ability to relate to
people. Once I saw a furious parent storm into the building, demanding that a
teacher be fired. The principal asked the parent to come into his office to
discuss the matter. An hour later, the parent left, with a smile on his face,
and to my knowledge the teacher involved was not even aware of this.
Things like this can’t really be measured.
I know, by the way, of another school system in which new teachers have been
told, “Watch out—every student comes with a parent, and every parent comes
with a lawyer.” And the administration is terrified of this, and it has a
corrosive effect on the system. These things matter.
3. Kane’s paper
Now let me specifically address Kane’s paper; in particular the data he
refers to and the uses he makes of it. I can’t really address
everything—the paper is quite long, and refers to other long papers. But
I think I can address some of his most significant points.
3.1 Discipline and student retention
Kane responds to a statement from the Massachusetts Teachers Association that
“Charter Schools’ discipline policies and high suspension rates lead to
push-outs and poor outcomes for some students. …” There are actually two
things at issue here:
- discipline policies
- push-outs and poor outcomes
and it may well be that the MTA erred in linking them so closely. In any case,
Kane wants to respond to both of them. Here’s how he does it:
Kane gives statistics indicating that the differences in numbers of students
disciplined between charter and traditional schools are small at the high school
level. He does agree that at elementary and middle-school levels the difference
is significant.
He then says: “Despite the higher discipline rates in middle schools and, to
a lesser degree, elementary schools, there is no evidence that students are
fleeing charter middle schools at higher rates.”
Now this could be for a number of reasons. In the first place, his use of
the term “fleeing” is strange. From everything I’ve heard, students don’t
“flee” a charter school—they are encouraged to leave. And this happens
largely at the high school level, where there is a great emphasis on showing
good performance on standardized tests such as the MCAS and SAT tests. So
middle school leaving rates are likely beside the point.
He also notes that “Charter ninth-grade students were about 10 percentage
points less likely to attend the same school in 10th grade [relative to a mean
of 86%] and 16 points less likely to attend the same school in 12th grade.” The
figures I’ve seen the past were greater, but these are still pretty significant,
as Kane agrees, and says he does not understand:
seem to go in the opposite direction from the differences in discipline rates.
Relative to traditional public schools discipline rates are particularly high
in charter middle schools. And yet it seems that the proportion of students
switching school is lower in charter middle schools than in traditional public
schools. Moreover, despite the fact that discipline rates are essentially
equivalent in charter and traditional high school, the school retention rates
are lower for charter high schools.
It’s not at all clear to me, based in what I mentioned above, that this is
really anything to be puzzled about at all.
As far as discipline itself goes, here are some other figures, compiled by
the Lawyer’s Committee for Civil Rights and Economic Justice in a
report “Not
Measuring Up: The State of School Discipline in Massachusetts”. This report
reflects the most recent numbers available from the Massachusetts Department of
Eduction, from the 2012-2013 school year. It’s possible that things may have
changed somewhat since then, but there is no indication that they have. The
report itself is not concerned primarily with charter schools per se, but it
does include this paragraph:
out-of-school suspension rates. While charter schools enrolled only 3% of
students in Massachusetts in 2012-13, charters accounted for 6% of all
disciplinary removals. On average, charter schools in the state had a 10.7%
discipline rate. And while only 4% of Massachusetts’ public schools are
charter schools, they accounted for nearly 14% of the schools with discipline
rates over 20%. Additionally, charter schools in the city of Boston had an
average discipline rate of 17.3%, and rates well over 20% were not
uncommon. Roxbury Preparatory Charter suspended 59.8% of its students
out-of-school at least once, for example. By comparison, Boston Public Schools
had an average discipline rate of only 6.6% and its non-charter middle and
high schools, including disciplinary alternative schools, had a discipline
rate of 11.1%. This indicates that for a similar student body, Boston-area
charter schools were much more likely to use exclusionary discipline,
particularly in response to minor student behavior violations (the
“non-violent, non-criminal, non-drug” disciplinary incidents discussed
below).
Based on these figures, there really does seem to be a significant difference
between serious disciplinary actions at charter schools compared to traditional
public schools, even allowing for the widespread assumption that charter school
students are at least somewhat self-selected.
3.2 Cream-skimming
There is a persistent belief that charter schools engage in cream-skimming
when selecting their students. Kane goes to some lengths to debunk this (“ELL”
in the following refers to “English language learner”):
Charter schools have made similar progress in attracting ELLs. In 2004, the
percentage of charter students who were English learners in the entry grades
for middle schools and high schools was one third the percentage in BPS: 4%
versus 12%. By 2014, the proportion of ELL students in BPS sectors grew as a
result of federal pressure to appropriately categorize language
minorities. However, the gap between charter schools and BPS has largely
closed. Among fourth- and fifth-grade students, the percentage of charter
applicants who were ELLs (24%) was the same as in BPS overall. Among
eighth-grade students, the percentage of charter applicants who were
categorized as ELL was 27%, compared to 30% among BPS students.
Setren used the lottery data to estimate impacts of charter school
admission for ELL and special education students. These students saw similar
gains from charter admission as other students—over .26 standard
deviations per year in math and .19 standard deviations in English. Setren
also found that admission to charter schools doubled the likelihood that a
student in special education at the time of the lottery would be reclassified
out of special education the following year and tripled the likelihood that
ELL students would be reclassified out of ELL status. The reclassifications
help to explain why special needs students and ELL students appear
underrepresented in charters
Setren (referred to in this quote) is or was an MIT graduate student in
economics. Her work quoted in Kane’s paper is in a paper that I was unable to
retrieve from its website at MIT. “Lottery data” refers to admission procedures
conducted by charter schools.
These statistics in any case are misleading, because the range of abilities of
English language learners can be quite large. So if charter schools take a
significant number of English language learners, but those students really have
enough English competence that they can be placed in ordinary classes with
minimal additional help, while students requiring more serious English help are
left in the traditional public schools, then the actual statistics really don’t
mean very much. And that is indeed what seems to be happening. Here are the
facts of the matter (again from the Lawyers’ Committee for Civil Rights and
Economic Justice):
“Boston-area charters continue to enroll English Language Learners at half
the rate of Boston Public Schools, and fail to enroll those who most need help
learning English,” said Roger Rice of META, Inc., another of the students’
counsel. Citing results of the state’s 2015 English proficiency exam, Rice
noted that only 7% of English learners at Boston charters needed the most
intensive supports, compared to 25% in Boston Public Schools.
“Not only do Boston’s traditional public schools serve significantly more
students with disabilities than the charters serving BPS kids, but they serve
students with the full range of disabilities—from mild to
severe—not just those with less severe disabilities who are most able to
be educated in the regular education classroom with limited supportive
services,” stated Kathleen Boundy, Co-Director of the Center for Law and
Education and another counsel for the amici. Students with disabilities
comprise 20% of Boston Public Schools’ enrollment, compared to 16% at
Boston-area charters this school year—with the 5 lowest-serving charters
enrolling less than 10% of students with disabilities.
3.3 Outcomes
Do charter schools provide superior education? Here’s what Kane says:
Those offered a slot at one of the charter schools were considerably more
likely than applicants who did not attend a charter school to score high
enough on their 10th-grade assessment to subsequently satisfy the MCAS
competency standard and to have scores sufficiently high to qualify for the
state’s Adams Scholarship, which waives public university tuition.
In addition, enrolling in a charter high school doubled applicants’ chances
of sitting for an Advanced Placement exam and scoring sufficiently high to
receive college credit. Although charter enrollment did not improve students’
chances of taking the SAT, those offered a slot at a charter subsequently did
have higher SAT scores. The estimated impact of charter admission on any
postsecondary enrollment was positive, but not statistically
significant. However, those offered a slot at a charter school were more
likely to attend a four-year and less likely to attend a two-year college
after graduation. Students were not yet old enough to reveal impacts on degree
attainment, but given the high non-completion rates at two-year colleges, the
shift from two-year to four-year is a positive predictor of their likelihood
of completing higher degrees.
Actually, this is much less significant than it might appear. Charter
schools are notorious for concentrating heavily on “test prep”—preparing
students to do well on the MCAS and SAT tests. So—even ignoring the
cream-skimming—it’s not so surprising that the SAT scores from students at
charter schools are somewhat higher than at traditional public schools.
On the other hand, for some years now, more and more colleges have been
discounting SAT scores, or even ignoring them completely. The reason is that
they have found that SAT scores are not actually a very good predictor of
college success. And in fact, this does seem to be the case
here. The
Globe reported on January 20, 2015 that
Graduates of city-run high schools in Boston are having better success
completing college than their peers from charter schools, according to a new
report.
Fifty percent of students who graduated from Boston public high schools in
2007 and enrolled in college had earned a degree or another kind of
postsecondary credential within six years, according to the report by the
Boston Opportunity Agenda, a partnership between the city and philanthropic
organizations.
By contrast, 42 percent of 2007 graduates from Boston charter schools had
completed their postsecondary schooling within six years.
So there’s a lot less here than meets the eye, and in fact, it’s really an
indication of a rather impoverished form of instruction in charter schools.
In my own experience as a college teacher—and I am aware that this is
by no means unique to me—I have noticed there are a number of
students (not all, by a long shot, but more than one would hope)
entering college who are still at a rote-learning stage. This makes it
extremely difficult for them to do well in college-level work, and we have to go
to great lengths to try to bring them up to speed.
Schools that get somewhat higher scores by concentrating on test prep just
make this matter worse.
4. Structural issues
Charter schools nationally are well-known for high teacher turnover. Average
teacher experience in many cases is 3 years or less. This has been reported in
many places, including
a New
York Times article from 2013 about this. I don’t know what the actual
statistics are in Massachusetts, but there is plenty of anecdotal evidence that
the situation is no different here.
This is a real problem. When I was a public school teacher, no one started
out as a good teacher. We weren’t necessarily terrible, but we were pretty raw;
we made all the beginning teacher mistakes; and we got a lot better over the
years. And we kept getting better. And in that process, most of what I learned
about teaching I learned from older and more experienced teachers. A school
without such teachers is simply not a good school.
In fact, I think the only way you can take a bunch of raw new teachers and
have them staff a school is to impose a rigid curriculum and insist that they
follow it. And this does actually fit with the drill-and-kill test-prep model
that we have heard about so often. These young teachers are massively
overworked, massively underpaid, massively overmanaged, and chewed up and spit
out after just a few years.
5. Are charter schools a civil rights issue?
Charter schools were originally put forward as a place to experiment with
educational innovation. The idea was that traditional public schools made such
experimentation difficult or impossible. (Usually some anti-union dog-whistling
appeared at this point.) These innovations were then supposed to be adopted by
traditional public schools.
Well, no educational innovation that I have heard of has ever appeared from
any charter school. And they are not now being promoted using that line of
argument: that justification has been abandoned. Rather, they are being
promoted purely as a better place for our young people to get an education.
And in particular, they are being justified as a civil rights
measure—charter schools are being characterized as the last best chance of
African-American students in bad city schools. And white suburban voters are
being told that voting against Question #2 is tantamount to racism.
However, charter schools are opposed by the NAACP and by the Black Lives
Matter movement. This is a difficult matter for the Question #2 proponents to
skirt around. They do it by maintaining that Massachusetts charter schools are
different from those in other states. And that is true to some extent. But the
fact is that the NAACP in Massachusetts also opposes Question #2, as does the
Black Lives Matter movement here. And one of the most impressive leaders of the
Black community in Boston—Tito Jackson—is absolutely opposed to this
question. If in fact his constituents favored it massively, as the supporters
of Question #2 assert, it’s hard to see how he could get away with this. As it
is, he convinced his colleagues on the Boston City Council to vote (by 11-2) to
oppose Question #2. I don’t believe this is a matter of civil rights.
6. Who is behind Question #2? And why?
Finally, without going into any great detail, I would point you to a report
“The
Hidden Money Behind Great Schools: Strategic Grant Partners” by Maurice
Cunningham, a political scientist at UMass Boston who provides a detailed
account of the dark money behind Question #2. The money behind this campaign is
not only not raised from a popular movement—it is almost entirely
“dark” money that is largely untraceable by design and not reportable to any
government agency.
We do know some things. We know, for instance, that the Walton Foundation is
one of the sponsors. We know that there are a number of astroturf groups, like
the so-called “Democrats for Education Reform” who evidently are really a group
of hedge-fund managers who are pouring immense amounts of money into charter
school expansion initiatives all over the country, including here.
It’s not hard to find that under the surface, these dark-money sponsors are
really aiming for the privatization of public education; the destruction of
teachers unions and with that, the degradation of the teaching profession; and
the establishment of a two-tiered system of education. But going into that is a
report in itself. Diane Ravitch’s blog
is a good place to start.
It’s not clear to me what you are quoting from and what’s your own thoughts. Maybe try block quotes from the excerpts in the original paper you are quoting from and links to that paper?
In the broader sense, this vote is all about the money. It’s not a referendum on charters, it’s a referendum on whether adding an additions 12 a year is fiscally sustainable for the districts affected. My answer to that question is No, and I think charter proponents had the opportunity to fix the funding disparity and chose not to.
The material taken from other sources is either in explicit quotes or in indented blocks with the sources evident. If there is something unclear, I’d like to know, and I’d be happy to clear it up.
The article was originally composed in LaTeX and turned into PDF. I took the original source and by hand added html markup. It actually looked quite good when I viewed it as a file in my browser. It doesn’t come through quite as well, unfortunately on whatever is being used here. But I think it’s clear enough.
One of the reasons I did write it was that — other than the need to unravel what Kane was saying — it seemed to me that people opposing Question 2 were essentially saying that “sure, charter schools are good, but they take away money from the rest of us.” That’s only partly true. There may be some good charter schools. But evidently not many, and I don’t think it’s a good idea to promote that notion. I have tried to give some reasons here that go in the opposite direction.
Might have been my mobile. I really need to do my BMGing on my laptop when I can help it.
I miss using a proper document creation tool.
the first requirement for understanding quantitative research.
Most of us lack the expertise, and almost all of us lack the time, to truly understand the metrics involved. Think tanks have the money to push out research to the press and release it to the press without peer review or even public release. The media doesn’t want to get into the weeds and reporters lack the prerequisite skills.
Significance tests are now being reconsidered in economics because readers, and often researchers, think they mean research is significant, when, in fact, it means that the statistics produced have a .05 chance of being random.
Many or most of these results end up being irreproducible, which tends to indicate that much of the result is due to chance, mistakes, or skullduggery.
This is a somewhat-recognized crisis in science.
Is this OK with Professor Kane?
…to quote at length for the purposes of engaging an argument, as long as it is appropriately credited.
on a Christmas card?
Cripes, Carl cited it. Linked to it. Are people not going to read his article because Carl quoted him?
I didn’t realize this was campaign literature, I thought it was a published article (and therefore a fair use violation).
Do parents have to actively request assignment to a charter school before they are entered into the lottery?
…that is correct.
Parents have to ask to have their child entered into the lottery. That’s what I assumed you were asking, anyway.
So I read the Kane report and the article by Abdulkadiroglu et al that provided a lot of the basis for Kane claiming an apples-to-apples comparison between charter and noncharter schools. In brief, they followed students admitted or not via a lottery to charter schools .
There are two selection bias issues here, one acknowledged by the authors and one not mentioned. The acknowledged bias is that if there wasn’t a lottery for a school, students in that school could not be included in the supposedly “random” comparison. That “may” (in the authors’ words) exclude “less desirable [charter] schools” from the results (p. 32).
The unacknowledged bias is as follows. This is where it’s important to note that parents had to actively apply to the charter schools. (“Whether admitted by lottery or by an application process, parents and students volunteer to attend Charter and Pilot Schools.” p. 8) The “randomized” students who were assigned to charter schools through the lottery then attended a school 100% populated with children of parents who cared enough about their education to take some kind of action to affect it. (Whether it was the right action or not is beside the point.) The “randomized” students who went to a non-charter school were mixed in with other students who may or may not have had similarly motivated parents. Parental involvement and peer group both have a big effect on educational achievement and this didn’t seem to be considered in their study.
I’m surprised that these researchers seemed to have not that much of a “feel” for how people choose schools and what goes on there.