How MCAS Works and Why It Doesn’t Help

The MCAS test is a standardized achievement test. How is it standardized? Every student receives the same test questions. Every student receives the same directions, in fact, all test proctors read from a script. Every student has as much time as she needs to complete the test. Once completed, all the tests are scored the same: open-ended questions on pre-determined rubrics; multiple choice questions on pre-determined answer.

As an achievement test, the purpose of MCAS is to assess students? skills and knowledge, which are reflected in the Massachusetts Curriculum Frameworks. Each question is aligned with at least one standard. Test questions are written by actual teachers, then vetted by psychometricians and the Bias Review Committee addressing issues of bias and diversity.

Statistical Problems

Remember the Sixties line: ?What if they threw a war a nobody came??? To a certain extent, the same is true for a standardized test. No one, let alone the standards-bearers, would administered a standardized test if everyone were to receive an A on it. As paradoxical as it sounds, most standardized tests, the SAT, AP tests, and MCAS tests, are norm-referenced. They are designed to guarantee people receive different scores and that these scores are distributed along a curve. Standardized tests spread scores along a curve. I don?t know the shape of the curve.. On the SAT test, the curve is a normal, or bell-shaped curve. Think about it: the SAT test, which is a requirement for admission to most colleges, is intended to sort students along a pre-determined curve


Although the DOE denies the existence of a curve on the MCAS test, it is a normed test in that it references back to a normative group. According to the DOE, ?The standard scoring system was developed in August of 1998 when 200 panelists were asked to assess the students’ tests and categorize the resultant scores into one of four categories: Failing, Needs Improvement, Proficient, or Advanced.? Every year the MCAS test is referenced back to this original scoring. If the scores on, say, the 2005 test vary from this norm, the cutoffs for failing, needs improvement, proficient, and advanced are adjusted. In spite of new questions every year, the MCAS test remains the same.

Of MCAS and curves, the DOE says,

The panelists [the graders of the original questions], who were mainly educators, were not given quotas of the number of tests to be placed in each category. This allowed for the achievement levels to be determined in the absence of any type of relative scale. The four categories are absolute and therefore allow for each student to score above the failing category. In other words, there is no bell curve, thereby allowing every student who meets the standard to pass the exam.

I believe the DOE site. The normative group does not follow a bell-shaped curve, but I contend that’s because they’ve divided scores into 4 rather than 5 groups. There is no natural middle for a bell-shaped curve. Without the original scoring and statistical documentation, it?s hard to say what the original distribution of scores looked like, but that doesn?t change the fact that there is a curve and MCAS questions are designed to produce a particular result.

Questionable Questions

Aside from statistical considerations, there are certain question qualities that can affect a standardized test. Producing a spread or distribution of scores is a requirement for the MCAS test. For our purposes, test questions can be divided up into easy, medium, and hard difficulty. A high percentage of kids get easy questions right; a small percentage of kids get hard questions right; and about half the kids get medium questions right. If you don’t have a lot of medium difficulty questions, you’ll get a lot of kids at the bottom and a few at the top. In other words, you get a sort of pass-fail test. Test makers have to include a lot of questions of medium difficulty to make sure there are scores across the continuum. I know this is kind of complicated if you’re unfamiliar with statistics, but testing is a technical business.

James Popham, one of the nation’s best know educational researchers and an expert on testsing, says that standardized achievement tests commonly rely on two types of biased questions to create the score distribution they need:

Cultural Bias. The MCAS creators do their best to eliminate overt cultural bias through careful screening. Passages are carefully considered for cultural bias. My understanding, based on what my friend Joe, who worked on the 10th grade English Language Arts, is that test questions are piloted, and if there is a cultural imbalance in right answers, the question in question will be rejected. There may be a cultural bias due to the fact that kids from the white, educated, middle and upper middle class backgrounds have natural cultural advantages in terms of vocabulary and background knowledge, but I’m not prepared to make that argument. I have witnessed my own students who happen to have large vocabularies answering context vocabulary questions correctly because they already know the word in question.

Intelligence Bias. It’s more difficult to eradicate questions biased toward smarter kids. Psychometricians have access to demographic information, but not IQ scores so they can’t filter these questions out. What’s wrong with smarter kids getting more questions right? MCAS is supposed to be measure achievement, learned skills and knowledge, not intelligence. MCAS is supposed to be holding kids accountable for what they’ve learned, not whether or not they’re stupid, but my guess is that some questions that’s exactly what it’s doing.

My English department analyzes our 1oth grade MCAS test scores, and we try to figure out where our department what we can approve on. The problem we have is that neither the questions nor the standards tell us very much. We try to figure out what skill or piece of knowledge a question is requiring. Often we come up with the observation that our kids have failed to make an inference. Such an observation isn’t particularly helpful. Teaching students to make inferences isn’t particularly easy.

There are other issues concerning the efficacy and philosophy of MCAS that I’m not going to touch on. These include the dumbing down of the curriculum to test preparation and drilling in places where schools do poorly, decreasing kids motivation to learn and further widening the achievement gap by allowing more time to be spent on higher-order thinking and creative work, qualities that tend to distinguish Americans from Japan and China. Also worth considering is the fact that American culture has changed in the last 50 years. It’s easy to blame education, and it’s part of our cultural inheritance to think education is the solution to society’s ills. In the cities, there is third world poverty and terrible violence. In suburbs, kids are apt to work and own cars. Their perceived lack of interest in education may be a by-product of a culture with different values, a cultural that is tearing them away from the benefits of a high school education. Finally, there is assumption that what ails our children’s academic achievement is motivation, and that “high standards” can whip them into shape. There is no empirical reason to assume this is the case.

Mark

This post was originally published with Soapblox and contains additional formatting and metadata.
View archived version of this post
.



Discuss

37 Comments . Leave a comment below.
  1. In other words

    When people say "we need higher standards" they're really saying "more kids need to fail".

    Why?  Because "people who graduate high school can't read" whereas in the old days they failed out and were ushered out of society's eyesight and everyone forgot about them.  Now in the days of the qualifications arms race, everyone gets a diploma so their career isn't dead when they're 16.

    The fact that students are doing more thinking and less memorization, the fact that they're using knowledge rather than consuming it, the fact that most 13-year olds are learning algebra these days doesn't matter because the kid in McDonald's is slow taking their order.

    Why? "I went to school thirty years ago and I know what it's like." Oh.

    sabutai   @   Tue 4 Dec 7:00 PM
  2. What are we testing?

    So you're saying that it is impossible for all Massachusetts students to prove that they have met the state's standards because a certain percentage has to fail.

    Regardless of how well our schools do, some kids have to fall into the "failing" group.

    Oy.

    The SAT's are a test intended to separate students. MCAS is supposed to be a simple gauge of whether or not students have reached a certain point.

    Here's a question I don't know the answer to: Does the Bar Exam work on a curve? How about the medical boards? Because those are tests frequently held up as a reason for keeping a high-stakes MCAS. ("Would you want a doctor who didn't pass the boards? A lawyer who didn't pass the bar?")

    Why are the people who are so gung-ho about the need for evaluating students so afraid to take a step back and evaluate the test?

    • Why?

      Why do some students "have" to fail?

      They either know their stuff, or they don't. Mark has excellently laid out the fact that there we should give reasonable consideration to whether or not the MCAS really is testing what students "have" to know when they graduate.

      Honestly, I'm not opposed to a test with a gradution requirement, but I think it should be more on the lines of just "easy" questions - as Mark described - to see if students know the bare basics. It's not as if that's the only measurement of their readiness to graduate. See, we have this little thing called "grades," and if students don't pass their classes, they fail anyway.

      We need to make school more dynamic and interesting - help foster a reason to make kids want to be in school. The MCAS is doing nothing toward that. It's a test that people from the suburbs will pass en masse, but people from cities fail in larger numbers. The MCAS has done one good thing and showed us that we're failing our urban and poor children, but giving up on them and refusing to allow them to graduate isn't a solution to the problem.

      • I think I was misunderstood

        Re-reading my comment, I wasn't clear.

        I'm saying that, as the tests are written and scored, my understanding is that there is an inherent requirement for failure. I don't think this should be the case at all.

  3. How the SAT shaped pretty curves

    During my last year in college (1988-1989) I paid for my ramen noodles and cheap wine by SAT coaching for the Princeton Review.

    We made extensive use of the bell-curve shaping behind it, telling students that the first questions in each section were biased to be answered correctly whether or not kids fully understood them.  So every student would benefit from answering all of the first questions, and no one should spend much time fussing over them.

    Conversely, the last questions were designed to minimize the probability that students would answer them correctly by chance.  Consequently, without training, the percentage of correct answers was lower than it would have been with random choice.  Therefore, the "gut-level" attractive answer would always be wrong.  Mid- to high-scoring students could use this information to cross out the "attractive" answer and choose from the others, whereas lower scoring students benefited from just leaving that part of the test alone so they could spend more time on the first part of the section.

    I don't know to what extent this type of curve-shaping went into the Chemistry GRE test I took.  I was graduating with a B.A., having switched majors rather late, so there was a lot of analytical chemistry I hadn't learned.  It was no matter; I did quite all right.  All I had to do was take a look at the units which were uniform in the answers.  If they were expressed in geeb*flins/stots, the correct answer was available by multiplying the number of geeb by the flins and dividing by total stots.

    Standardized tests can be rather stupid.

    • SAT correlation to future college performance

      I recall, from the book "None of the Above," that SAT scores correlate to a student's performance in the first year of college.

      But High School GPA correlates better.  Which makes the SATs perhaps not so useful.  Or at least not really worth the added expense to students.

      • Esxcept that it helps

        compare a 3.0 in Tough High School A from a 4.0 in High School B, where they hand out full credit just for showing up and appearing to care.

        Which has always been the point of the SAT.

        • Please, where does that high school exist?

          Most people could agree that the MCAS exists for districts like New Bedford. Yet, if New Bedford isn't one of those "tough high schools," it would stand to reason that almost everyone would graduate (you know, because those 4.0s are so easy to get!). Yet, their graduation rate is about 60%. Whoops.

          I went to Swampscott High School, a very good suburban school that's sent more kids to Harvard than any other public school in the country (according to my then-High School Principal). Yet, close to 100% of my class graduated on time.

          Which school is supposed to be the easy one and which one the hard?

          Your logic certainly isn't Vulcan-approved. 

          • Well then let me explain

            Sadly, GPA is just not comparable across different high schools.

            A student from Bronx Science's GPA is going to be lower than the same student's GPA at a marginal high school, at which security is as great an issue as education.

            College admissions folks learn the difference between these schools after awhile, and can perform some sort of mental conversion.  But, for the student from the high school for which the admissions officer has no frame of reference, there is no way to put that student on the spectrum for admissions purposes.  So, Sally and Jim have 3.8s and were in the NHS-- are they better students than the 3.2s from Bronx Science and Hunter?  And how do they compare to the 2 other kids from another unknown high school?

            Enter the SAT, which differentiates them.  As it is supposed to do.  And lets the officer have at least one common point of reference for all applicants.

            • That's what it's supposed to do

              It doesn't necessarily do it.

              I've known very, very, very stupid people who did very, very, very well on the SATs. They couldn't string a coherent sentence together with a map and a compass, but boy did Cornell love their score. Yet, they took a Kaplan or a Princeton review because they had the money and pushy parents... and they got what they paid for.

              I've known extremely good students, who were bright, hard working and highly intelligent, with great GPAs - who either weren't good timed, multiple-choice test takers, didn't think they'd need to know how to beat the SAT "game," or didn't have the funds to take one of those courses... and performed far below their GPAs in a school that wasn't all that easy to have a good GPA in. (Like I said, my high school was one of those 'hard' ones).

              Then, how do you compare and contrast the different tests? Which one is the best? For example, back when the MCAS was actually scored very hard (before it counted for graduation), I was one of 15 people in my class of 180ish to score an advanced on both math and english. We were one of the top 40 schools in the state that year in terms of scores, so it wasn't like my class was filled with dummies.

              Yet, my SAT score was only above average - certainly not in the top 20 of my class. I do far better with open ended questions than multiple choice (I've never met an in-class essay that I couldn't get near full credit on, even if my mastery of the facts was mheh).

              So, who's smarter? Me and my essay-writing prowess, or some Joe Schmoe who can 'beat the system' on the multiple choice? Hell if I know, because I'll admit up front that I know exactly what teachers are looking for in an essay and that's why I'm so damn good at writing them. We each have strengths and one shouldn't be viewed as good or bad. I think, in some ways, the SAT has tried to recognize this (at least, after decades of complaint) - since they changed their test to encompass a writing section. I'd imagine that, if I were to take the SAT today, I'd do far better and score very high on the essay section... and be happy to see some of the sections they removed gone (although, I will say I'd miss analogies, for some reason I was always freakishly good at them).

              I think the bottom line is that we're all intelligent in our own way and we need to create methods to teaching every student to his or her strength. We need programs that encourage their growth in those particular areas. If that means we need to explore oral testing, portfolio testing and other methodology to see who's worthy of going where, then I'm all for it. We should never treat students the same - and have that common reference, as you like to call it - because it's a falsehood to say everyone is the same. What happens is that we pick one area, that's highly prejudiced, to determine intelligence... and it doesn't determine intelligence at all.

              Seriously, if you want to read a good book that's sort of about this, read State Boys Rebellion. It's kind of along the same theme and is an amazingly interesting read and story. I bought my copy of the book for about $2 used on Amazon.

              • I guess we'll have to disagree

                While it is nice to state that everyone is intellegent in their own way, that way lies the road to "I'm OK, you're OK, and we're all special people" perdition.

                There must be a common frame of reference.  Is the SAT perfect?  No.  But I'm unaware of a better tool.  Evidently, neither are college admissions officers.  It is competitive, which means there are winners and losers.  So be it; it beats the system it replaced, which was based on family connections alone.

                I don't even necessarily agree with the criticism of the SAT; what gets me is the concept that metrics are futile because it is all too amorphous to be measured.  If you don't like the test, design a better test, and sell it.

          • The challenge to Rep. Donato

            What's the correlation between graduation rate and grading standards?

            • He said one school was tough, the other hard

              It would stand to say that the 'tough' schools would graduate less students, while the easy ones would graduate more. I corrected him.

              • I get that, but

                what I was asking for was the actual reasoning behind your "it would stand to reason" that schools with higher standards should graduate fewer students. Absent evidence, I would tend to assume the opposite. I would guess that, if you're not holding the socioeconomic situations of the communities constant (e.g., Swampscott v. New Bedford), that the schools which graduate more students also have higher academic standards.

                But I agree that the SAT is not a great solution to the varying standards problem.

        • got that backwards?

          a friend used to teach at private schools in the greater boston area, and eventually chucked it because she was disgusted by the management insisting that mums and daddums not be disappointed by their little muffy & junior getting the grades they deserved (midlin to failure).  so, don't forget the old "buy me a gpa for christmas, pleeeeze daddy!!!!" problem.  it's real.

    • Thankyou, Princeton Review

      From high school SATs, through grad school GREs, I have used nothing but Princeton Review prep books to basically beat ETS's testmakers at the predictable games they create.

      This has had a significant effect my educational progress and, by extension, my career and earnings.

      Your example is typical of the PR approach. It begins by having you throw out the notion that you're being tested on how smart you are, or even what you know. The test is a game and you can maximize your score by memorizing some basics and practicing some test-taking tricks.

      I don't know if teaching to MCAS has gotten this strategic. For the kids' sake, if it's high-stakes, it should. Which should then reveal how limited test-based assessment is.

      • Test taking

        One key difference between MCAS and ETS tests is that the latter deducts points for wrong answers. So, balancing one's time vs. guessing is more strategic for the SAT/GRE, etc..

        Where they are similar is that if you can limit the number of possible answers, you'll increase your chances on those questions where you have to guess. Sure, it's a test taking strategy, but it's a basic decision-making strategy as well. I don't understand why that technique is seen as something special.

        And just because it's a technique that can be used on a standardized test doesn't discount the test. For example, a typical test question might be:

        36 / 2 = x

        a) 17.5 b) 18.5 c) 18.0 d) 72.0

        I know that dividing an even number by 2 will result in a whole number. So, I eliminate a and b. I'm left with either guessing or solving the problem.

        The anti-standardized test folks might want someone to come up with a problem on their own that demonstrate their knowledge of division. I'll say that I want to hang a picture centered on a wall that is 36 feet wide. And then I'll describe that I found the center by dividing 36/2 = 18.

        This assumes that I know 36/2=18. If I didn't, and I didn't know how to solve the problem, rather than having only two possible choices, I have the whole universe to guess from (or unless I'm good at estimating).

        My point is if you know the answer, it really doesn't matter which form the question takes (although the latter is really inefficient). If you don't know the answer, it's easier to fool people with good guesswork, but either way, it's still guesswork.

        • Guesswork counts

          Good guesswork isn't, for practical purposes, "fooling" people. The correct guess counts just as much as having the knowledge, and in the case of MCAS and SAT, has the same real world consequences.

          And smart test-taking involves more than knocking out wrong answers. There's the original example of learning to recognize patterns of difficulty and targeting your time accordingly. And further, you can acquire a feel for how the testmakers think: the kind of wrong answers they like to use, and the particular bits of knowledge they seem obsessed with testing. A wily teacher could probably find out what kinds of written answers too, beyond the official standards, impresses MCAS scorers.

          It's true this kind of strategery tests logical thinking and problem-solving. These tests certainly measure something (including confidence and self-image). But tests, especially the high-stakes ones, rarely admit how big these non-content components are.

          • You're fooling me

            Good guesswork isn't, for practical purposes, "fooling" people.

            In my first example, if you guessed "18" but really couldn't figure out that was the right answer regardless of how much time you had, then yes, you've fooled the person who grades this test into thinking you know how to solve the problem.

            It doesn't matter if this was MCAS or a teacher designed test. The real world consequences that arise from the fact that the student cannot solve a simple arithmetic problem go well beyond a high school diploma. If there are most "test taking techniques" that can be used to get a passing score on MCAS without having the knowledge base to support it, we'd be seeing a rise already. Maybe, just maybe, you actually need to know the material before any of these test taking techniques do squat.

            Let's do this: show me an example of how a teacher would assess that a student had mastered division even though the student got the wrong answer.  Because that's really what this boils down to. People don't like being judged on the basis of one question or one test or one standard.

            these folks seem to think that soft skills and academic knowledge are equal. If a student can show you that they really tried, or worked well with others, is that enough to justify thinking 36/2 = 17 ?!

            • You should be against multiple-choice tests then

              I think you're mistaking my point for an anti-standards, pro self-esteem liberal boogie-man. 

              I'm not against measuring mastery, I'm just pointing out the limits of tests, especially their multiple-choice components, of doing this. They test other things as well as content, and we should be aware of that if we're loading all this importance onto them. An all-open-response test would be a better vehicle for the kind of measurement you're interested in.

              Here's a 10th-grade MCAS question in biology, with my actual thought process for answering it:

              In comparisons of the evolutionary relationships between four species of birds, which of the following would be most useful?

                A. color of feathers  B. gene sequences  C. nesting behaviors  D. patterns of migration

              The first thing I notice is that answer B is not like the others. It's an "inside" thing, rather than an appearance or a behavior. It's probably the right answer then. B sounds right to me on the merits, too, but I have nagging doubts. Is it a trick? Large sections of genes might be indistinguishable, can I assume they mean the small differences that matter? Don't biologists start by observing the "outside" differences of A, C, and D before they ever get to sequencing genes? Does that make them more useful? How are we defining "useful" here? I think my way into momentary paralysis. But I cut through it by zeroing in on the multiple-choice logic that says A, C, and D are indistinguishable, whereas B stands alone. I get it right with a combination of vague content knowledge and more active test-thinking.

  4. Maybe we should get rid of driver's license tests too

    By this logic. And, indeed, any kind of testing whatsoever. The fact is, however, that there are standards in the real world and an educational system that does not provide minimal competency in basic skills -- as high schools established they were prepared to do through, for example, social promotion -- does a huge disservice to students and to the community at large.

    • An MCAS driver's test...

      ...would be calibrated to ensure that a large number of people failed it.  Rather than asking what to do at a red light, it would ask the amount of the fine for not signaling 100 feet before making a turn.

      Is this an important question?  No.  Does knowing it tell you a person's readiness to drive?  No.  Does the driver need to know that?  No.  Is the information hard to find if it comes up?  No.  But that is precisely the type of question that the MCAS asks -- detailed irrlevancy that examine memory over thinking.

      If driver's tests were mishandled as egregiously as the MCAS, people would be furious.  But beacuse it happens in a classroom, few people care.

      sabutai   @   Tue 4 Dec 7:00 PM
      • Using the brand new 7th grade social studies MCAS as a model,

        the driver's test might show the students a graph or table indicating the trends in highway deaths due to speeding, a photo of Henry Ford, an engineering diagram of a turnpike exit, and a quote from Robert Moses.  Using this information (as well as any background they care to bring to their answer) they would then be asked to write an essay describing the history of the interstate highway system. 

        Other driver's license questions based on the Social Studies MCAS might ask students to identify the purpose of a transmission, to identify the components of asphalt, or to describe the gear shift on a bicycle.  Interesting stuff, but do you need it for driving?  Did you study it recently?

        Also, it is my understanding that the 7th grade social studies test covers geography and ancient civilizations.  The 10th grade test covers American history.  So the fall of the Roman Empire to 1800 (as well as the rise of Islam, Asian and African empires, and Mesoamerica) don't really get the MCAS treatment. 

        So a parallel driver's license test would have to skip some substantive things - maybe skills like backing up or parking or keeping to the right.

        I like metaphors!  Thanks for offering this one, Bob. 

    • Bob,

      Do you really think that MCAS and the drivers test are comparable? Seriously?

      Tell you what - let's make MCAS A 20-question multiple choice test that requires 14 correct answers to pass. Does that work for you?

    • Also, Bob

      Do you really think the rise of the MCAS has ended social promotion?  NOT. 

      Like most teachers, I'm not a fan of social promotion, either, and I would like to see serious approaches to ending the practice.  (I admit - I did recommend social promotion recently for one of my 7th graders.  We have hope that he is emerging from the psychological fortress he built around himself when he was homeless, abandoned by his mom, and coping with his dad's drug addiction and subsequent disability.  Because he will be a head taller and physically stronger than the next set of 7th graders coming in, we thought it was better that we continue to work with him as an 8th grader than take the risk of creating a bully and/or adding one more discouraging event to the baggage he is carrying.)

      The underlying causes as well as the symptoms of educational failure - such as social promotion - are not being cured by MCAS or NCLB.

      MCAS may have reduced social graduation (if there is such a thing), but thousands of academically unqualified kids are still getting bumped up a grade each year until they reach 12th grade.  It's the failure to award a diploma that the MCAS accomplishes; I have not seen a connection between testing and better services for failing students.  (I have seen teachers express relief when a difficult student transfers to another school.  "He was a good one to lose.  He counted in three categories," one veteran teacher told me about one student whose MCAS score would be recorded as Latino, Black and low income.)

      If you feel good about kids passing until the ultimate step when they are given a handshake and a boot out the door, I guess I need to stop trying to persuade you.  That approach does bring to mind one of my favorite Lewis Carroll lines, however:

      "It seems a shame," the Walrus said, "to play them such a trick.  After bringing them out so far, and making them trot so quick."

      • Maybe we should have annual tests, indeed

        If you feel good about kids passing until the ultimate step when they are given a handshake and a boot out the door, I guess I need to stop trying to persuade you.

        I never said that, so why do you impute such an argument?

        I think there is a strong case to be made for tests at the end of each year. If the kids pass those, and perhaps also additional requirements, per the various excellent suggestions that a test should not be the only requirement, then they should advance to the next grade. If not, what's the point of promoting them to a grade with harder problems and more advanced instruction. In the case of the child you describe, don't you think that, at least in theory, transferring them to a special school for children with special needs is a better solution than just promoting them to the next grade where they theoretically won't be able to do the work either?

        • Is that a gauntlet I see before me?

          I'm sorry if I mistakenly connected your apparent dogged defense of MCAS in this and other posts to the manifest failure of the standardized testing system to prevent social promotion, and to that system's insistence on denying a diploma at the end of 12th grade to many otherwise qualified students.  I did not mean to wrongfully impute, and I'm glad you don't want kids to reach the end of the line and then suffer an all-or-nothing defeat by standardized testing.

          But you offer another challenge to me, and I accept.  You propose that we have a chutes-and-ladders-style test at the end of each school year.  The kids who pass would move forward, and the kids who don't pass would go to a "special school for children with special needs."

          My reply:

          A) Why test kids only once each year?  The very savvy and MCAS-conscious new principal at our school has been drilling all the faculty on techniques for formative assessment.  Instead of summative assessment (testing at the end of each arbitrary unit of instruction, as you propose), the kids and teacher are in a constant dialogue, with varying degrees of formality, that allow the teacher to check for understanding throughout the lesson or unit.  Test anxiety, teaching to the test, cheating and a host of other problems associated with standardized testing are significantly reduced, and the teacher can adjust instruction along the way to make sure kids are following along.

          B) What if a kid is smart but doesn't like tests?  My difficult student is very bright.  The one time he participated in a formal test this year (I was writing for him because he had a dislocated shoulder, and I think he loved the one-on-one attention) he did very well on the portion he completed, even though he had appeared to be inattentive while we were reading the play, and he quit part way through the exam.  I believe he would shut down just as completely in an special school as he does in a mainstream school - maybe moreso because he would resent being stigmatized, sent away and separated from his friends.  Up until his family crises became acute about four or five years ago, his file says that he was a happy student.  So an end-of-year high-stakes test would not show me anything about his ability or capacity.  It would underscore the fact that this emotionally damaged child does not like to pick up a pencil.  And it would result in removing him from a school environment where he could be making some progress.

          C) Most teachers know that building a relationship with a student is a major part of successful learning.  We don't want assessment reduced to an all-or-nothing test once a year or once every three years.  And we don't want students shipped out to a special school when they fail a test.  We want to build on what we know of that child's needs and work with other teachers to address them as best we can.

          You point out that 8th grade may have harder problems and more advanced instruction than my student encountered in 7th grade.  The state standards tell us to cover all of the standards for previous years, however, and teachers often employ small group techniques and differentiated instruction that allow students to work at their own pace and level.  We are just as thrilled when our IEP kids work as a group to read a whole (easy vocabulary) novel as we are when one of our academic stars wins the city spelling bee or writes an A+ essay.

          D) I'm not an expert, but I believe that state law requires that special ed students be mainstreamed to the extent feasible.  Some teachers would agree that there is a big difference in the ways we should address learning disabilities and emotional disabilities, so you might find some agreement that there should be different programs for different types of IEPs. 

          There is an area at our school where severely disabled kids are given instruction tailored to their unique needs and abilities - some even have one-on-one paraprofessionals who work with them. 

          In all likelihood, however, failing this once-a-year test that you propose would probably not result in removal of a student to another special school.  My student would fail the test and come back to our school the next year, much like the current system.

          • Be still my heart....

            To you sir/madame, 'tis my leg I offer, the cocker spaniel of wisdom.  You are the genuine nads.  I admire a) your patience and b) your panache.  Alas, in my middle age, I lack the vitality to fight this battle into the next millennium.  You have won the heart of the prince/princess.  Go forth; fight the good fight.  I'll buy you two beers at the next study group at the, errrr, library. 

            p.s.  You get an 10 extra points for the "Chutes" reference alone. 

    • Okay, I'll play, too.

      The 10th-grade English MCAS might feature a selection from Steinbeck's Travels With Charley, in which Steinbeck describes driving 10,000 miles across country in the company of his pet poodle, Charley. 

      Students will read the excerpt and answer eight multiple choice questions based on their understanding of the text.  Then students will answer an open response question designed to elicit what metaphoric relevance, if any, they find in Steinbeck's journey.  Another open response question on this excerpt might ask the student to describe how having a poodle, specifically, for company affected Steinbeck's experience.  Yet another open response question might query the student on the effects on Steinbeck's mood of long-distance travel in a van.  Students will be asked, of course, to do all of these tasks using information from the article to support their answers. 

      Scoring a 240 or better demonstrates the student's readiness for driving long distances with a pet as a sole companion.

      • Likely outcomes

        I just left that one blank.  We never studied poodles. --- I thought it said "puddles"!  I wrote a whole page about the risk of hydroplaning at high speeds before I realized that it was talking about a poodle. --- I decided to write about my dog instead.  He can't drive, but he is really good at playing chess. --- Isn't Steinbeck the guy in that Mary Shelley novel? --- What's the difference between Travels with Charley, and Charley on the MTA? --- You never told us we needed to know about metaphors and writing and stuff.  I didn't even know we had the MCAS today.

        • That One Kid, though...

          You will, of course, get That One Kid (aka the accomplished crackpot who one day will make an eccentric contribution to the human experience) who will link the mass movement of people, the transcendant image of poodles in puddles, the iconic nature of the VW minibus, and the kitschy quality of beer steins in one large neo-postmodern pastiche that, somehow, makes sense to him and him alone. 

    • Nobody ever re-calculates the cut score for passing a driver test

      the way the state bureaucracy does with MCAS.  The driver test has a set cut score (14 out of 20, iirc) but the MCAS scores get looked at in the aggregate, after the test is taken, to decide what the cut is for passing.

      I'm basically re-stating Mark Bail's point here.  The MCAS is a high-stakes test, but the test-makers don't firmly set the passing level ahead of time, belying the assumption that test-makers know what knowledge levels are necessary for a passing grade.

      Hence it is not a more objective and reliable measure than an individual teachers assessment of a grade over the course of the class.

      More in depth in Cut Scores: Results May Vary

      • Cut scores

        Setting cut scores requires a degree of subjectivity. But the process isn't done to ensure a certain curve results. It is done so that there is consistency of difficulty and scale.  This allows scores on a particular test to be compared with scores on the next year's test. If you're going to change test questions every year, this sort of quality control is necessary.

        While it is true that an individual teacher's grading system may also employ a degree of subjectivity, such a standard cannot be used to inform state policy as there would be little  basis for comparison of an "A" from Mr. Jones and an "A" from Mrs. Barker. A class grade is not a standard outside of that class.

    • Bob

      What's up with your driving to absurd conclusions lately?

      Who in this thread has been talking about banning testing? Or driver's exams? What the heck does a driver's exam have to do with educational policy anyway? It's a Sean Hannity tactic and I expect better of people who embrace the reality-based community.

      This has been an honest, thought-provoking thread, with lots of great content and an insightful diary by Mark. It deserved a better response by one of the three editors of this website than absurd hyperbole.

  5. Why it doesn't help?

    Interesting post, but I'm confused about your choice of title.

    Often we come up with the observation that our kids have failed to make an inference. Such an observation isn't particularly helpful. Teaching students to make inferences isn't particularly easy.

    So, the standards and MCAS allow you to conclude that your students aren't making correct inferences. In other words, students cannot derive logical conclusions from factual premises. That's a very important skill and is infused within the frameworks from the earliest grades.

    But surely you aren't saying that identifying the problem isn't helpful because it's difficult to teach?

    Also, I have to question how you come up with your observation. If a question asks students to choose the correct answer based on facts presented in a text, and they get the question wrong, there are other possible causes:

    - They were careless, didn't take the test seriously, marked the wrong bubble, etc. - They lack key vocabulary needed for understanding the text - They can't read

    Let's look at an example of test questions. If you can't answer these questions correctly, can it really be said that you can read? Other than this test format, how would you propose that the requisite skills be assessed? Or are these skills not important enough to be required for a high school diploma?

    Has anyone in the "MCAS and" camp offered any example of what an alternative assessment would look like? Let's take inference. How would you assess that?

    I think the MCAS has helped. We've always sort of known that kids graduating from Brookline were better prepared than those of Brockton. The MCAS provides one measure of those differences.

    In this article, Ruth Kaplan says that her SAT scores weren't all that stellar but somehow she got into Wellesley. She, and other anti-MCAS people, forget that without the reputation of the Brookline public schools behind her, she most likely wouldn't have gotten in. Mediocre scores from the valedictorian of a mediocre district would likely not have garnered an acceptance.

    I hope that someday Brockton's MCAS scores rival those of Brookline. I believe we can find evidence in pockets all over the state and country, where individual districts/schools/teachers are beating the odds and meeting the same standards as their elite neighbors.

    • Some more possibilites to be added to your list:

      -They've been tested for days straight and their reading comprehension has gone down the shitter, as it so often happens when you read, read, read, read, read and read some more without a break. (I can't count how many times I've just missed getting a 100% or 95% ect. on a multiple choice exam or essay because my stamina was exceeded toward the end of an exam, time limit or not).

      -The question was unfair, for whatever reason. Mark expressed a number of ways for that to happen, especially cultural parciality (which does still happen, I can assure you, no matter how much we try to weed it out).

      -They're hungry, can't think straight, because their family struggles to pay the grocery bill.

      -They were up all night, because they live in a broken home, and everything they're reading is either fuzzy or - you're right, they just don't care... because their parents are getting a divorce, or whatever.

      -How about this: they were sick that day.

      -Their pencil's eraser sucked and didn't completely erase the bubble.

      There are myriad reasons why tests go bad, not all having to do with whether you know you're stuff. When I took my LSATs, I had been taking practice tests for months that were exactly like the actual test, in test conditions. I was routinely doing fairly well on them. Then, I went in to take the real exam... got enough sleep, carefully examed every question, finished each section with time to spare, felt confident and as though I had done great... and did, by far, the worst I had ever done on it.

      I consider myself a pretty good test taker and it was completely different than what my other scores were generating. If I took it the next day, maybe I would have done the best I've ever done. I really don't know. Standardized tests suck that way.

« Blue Mass Group Front Page

Add Your Comments

You must be logged in to post a comment.

Sat 22 Nov 5:17 AM