Bayes's Theorem and Hiring Your Way To Great Teachers

Bayes' Theorem is the most important statistics theorem that nobody knows.  In essence, it says that when computing the probability of an event occurring, you must take into account the information you already know about what theoretical outcomes are actually possible.  In mathematical terms:  the probability of A given that B has occurred equals the probability that both occur, divided by the probability that B occurs (with or without A).

Bayes's theorem has wide applications in statistical inference, but today we're going to talk about one that's of crucial concern to anyone trying to improve teaching in a particular school:  hiring teachers.  Suppose that your interview protocol allows you to identify satisfactory-or-better teachers with 80% accuracy (if you have such a protocol, tell me!).  And suppose that 80% of the teachers in the applicant pool are satisfactory-or-better.  Then if you interview 100 teachers, here's what happens:

• 80 teachers are satisfactory-or-better.  Of these, your interview protocol says that 80%, or 64, really are satisfactory-or-better, and 20%, or 16, it rates as unsatisfactory.
• 20 teachers are unsatisfactory.  Of these, your interview protocol says that 80%, or 16, are unsatisfactory, and 20%, or 4, are inaccurately rated as satisfactory-or-better.

What's the takeaway?  Well, there are 68 teachers rated as satisfactory-or-better, but of these, 4 are actually unsatisfactory.  Thus if you hire one of the 68 teachers, you have about a 94% chance of getting a satisfactory-or-better teacher.

So far, the odds sound pretty good.  But those odds are highly dependent on the assumptions we made: that 80% of the candidates were satisfactory-or-better, and that your protocol helps you tell good from bad 80% of the time.  In my experience, neither of these things is necessarily true:

• Many of the satisfactory-or-better teachers are pretty happy where they are, and aren't looking for new jobs.  In my experience, the applicant pool for Chicago Public Schools jobs is more like 40% satisfactory-or-better.  (Note: I'm not, not, not saying that only 40% of CPS teachers are satisfactory.  There are lots of satisfactory-or-better CPS teachers--but in my experience, many of them are committed to the schools where they teach.  What I'm saying is that when I was interviewing applicants for jobs at my school, only 40% of the applicants were satisfactory-or-better.)
  
• Teachers can have great credentials and interview well without being great in the classroom.  You can catch that with demo lessons (I've known of candidates who hit home runs in the interview only to totally whiff the demo lesson), but even that won't tell you how well they relate to students and parents over the long-term, how well they collaborate with colleagues, and how committed they really are to improving their practice.  So my guess is that interview protocols are less than 80% reliable, say 70%.

Running that same thought-experiment with our revised assumptions yields very different results:

• Only 40 applicants are satisfactory-or-better, and our protocol identifies 70%, or 28, of them as such.  12 applicants are incorrectly identified as unsatisfactory.
• 60 applicants are unsatisfactory, and our protocol identifies 70%, or 42, of them as such.  18 applicants are incorrectly identified as satisfactory.

Thus 46 applicants are identified as satisfactory-or-better, of whom only 28 really are satisfactory.  So the probability that a given applicant who passes the interview process is actually satisfactory is 28/46, or about 61%.  Under these conditions, then, if you hire five candidates, only three will probably work out.  And you'll be stuck with two whom you sort of wish you didn't hire.  

What are some conclusions we can draw?

• The usefulness of hiring procedures has as much to do with the overall quality of the applicant pool as it does with the theoretical reliability of the procedure itself.  If the pool has lots of unsatisfactory teachers, even a good test will end up with many of the apparently-satisfactory teachers being actually unsatisfactory.
• If you don't have a great applicant pool, firing a teacher who isn't working out will only result in a substantial improvement 60% of the time.  That's better than nothing, but a whole lot less than the "Just fire all the bad teachers" voices usually let on.
• If lots of bad teachers are suddenly fired, the applicant pool will get worse, both because the fired teachers are now in it, and because lots of people are trying to hire the good ones (remember that 3/5 of the teachers we hire, even under the pessimistic assumption, are good teachers).  And then as we've seen, hiring procedures become less effective at securing satisfactory teachers for jobs.  So as a system-wide policy, "fire the bad teachers" is unlikely to produce substantial improvements for a large fraction (probably more than half) of the kids in the system.

So it's unlikely that we can hire--or fire--our way to great teachers.  We need to take the teachers we already have and develop them instead.

Teaching Mathematical "Grit": A Dialogue

This week, the CPAM listserv has been bubbling with discussion about how to teach grit in mathematics; unsurprisingly, both of us have pretty strong opinions on this subject.  So here was our part of the dialogue:

John:  During class, I have them work on one challenging problem at a time. They work. I walk around and listen. They are encouraged to try it themselves first, then discuss their work with their neighbors. I do not give hints or show methods to solve the problem. They know that they have been given a problem, not an exercise. That is, I expect that it will take time to solve it, it is related to what we are working on but is not a copy of other problems they have been asked to work on. I do not ask them to do the problem, but insist that they work on it. Every fifteen seconds, or so, I walk by and observe progress. If you want them to become persistent you must provide situations where persistence is the only way. At some point during the class, we discuss solutions that various students have proposed. If it is a worthy problem, one that requires persistence, there will be several ways to approach it. When the problem has been solved, they have learned the content for tonight's homework. That is the intent of the problem, to teach the new lesson by having them figure out what the next thing is. Then I give them another problem. I never give them a worksheet nor do I give notes. We spend class time working on worthy problems. They learn the content, they learn how to solve problems, they learn "grit" and most of all, they love it.

It takes a while for them to get used to the concept that I will not "show them how to do the problem before I ask them to do it." but once they do, they are actually learning math.

PJ.:  One lesson I learned about this is that it's hard to learn more than one thing at a time.  So a good problem to use to teach grit is one in which the actual skills embedded in the problem aren't that difficult, but maybe they have to think through a lot of different possibilities, go down a blind alley, or something.  It helps a lot if the problem has someplace fairly obvious to start (even if it's ultimately the wrong place) and if there's some obvious way to check that your answer is correct.  Many math contest problems require a lot of grit but fail on these two counts, because there's basically only one thing to do, and the "problem" is seeing what that one thing is.  On the other hand, problems like the camel & bananas problem (one version + solution here), or the farmer with the broken eggs (look here), or the locker problem (look here) can be good places to start.

In terms of classroom practice, I'm not as hands-off as John, but I think it's really important NOT to scaffold the problem, especially in problem-specific ways.  Scaffolding the problem by asking leading questions just leads the students, and teaches them that they needed your help to do the problem, which isn't what you want them to learn.  Following Polya and my friend Doug, I have a series of nudges I tend to give in these situations, and I choose the problem with those nudges in mind.  (That is, I'm thinking about which general strategies I want kids to apply.)  There aren't many of these nudges, and I use the questions a lot, to the point that I can start asking students "What do you think I'm going to tell you to do?"  A not-totally-exhaustive list, in no particular order:

• Try a simpler case.
• Try several simpler cases, and look for a pattern.
• What are the conditions?  Can you state ______ as a mathematical sentence?
• What's the unknown?
• Look for congruent triangles/similar triangles. [I teach a lot of geometry]
• Try chasing angles. [Same]
• Reduce the problem to finding a point. [Same]
• Try dropping one condition and satisfying the others.
• Can you satisfy even one of these conditions?
• What's a related problem?  Can you transform this problem into that one?
• Use algebra

John:  I think it is more what we choose to emphasize when we try to get other people to understand what we are trying to do. I believe teachers tell kids too much so I emphasize that I do not tell them anything. The things you listed are certainly questions I ask my students , as well as things like, can you prove it, can you do it another way, does your answer make sense, does your answer agree with the answers others got, perhaps you should draw a bigger picture.....

[After this, another teacher wrote in about Carol Dweck's work on mindset, and fourth teacher said that she regularly assigns her kids hard problems to work over a week or two; students don't ask her for help unless they're "totally stuck".  I wrote back the following pair of replies:]

P.J.: Two brief extra thoughts:

1. It turns out that kids actually respond when they are taught the science behind cognitive development -- that in itself can be a way to change their mindset -- which is maybe surprising?
2. I actually encourage my students to see me relatively early on (before they're totally stuck), because I worry that their peers will give them too much help.  But I have a small enough roster that I don't have to worry about being overwhelmed.

δ, ε, and Mathematical Thinking

So a few of you have expressed interest (surprise, concern on my behalf...) that I'm doing formal limits with my class.  To be fair, the class is very advanced: only one of the fifteen students has yet to take Calculus, and almost all have spent at least one summer doing math.  But it's still a scary prospect.  So here's a report from day 1.

Why?  Why teach the formal definition?  And how do you motivate it?  We began this unit by doing an empirical investigation of iterates of the function f(x) = rx(1 - x) for 0 ≤ x ≤ 1 in the context of rabbit populations: if x represent this year's population density in a particular warren, f(x) represents next year's population density in that warren.  Students quickly discovered that a variety of long-term behaviors are possible (convergence to a single limit, oscillations between 2 or 4 points, and apparently "random" behavior that we couldn't quite nail down), and depend mostly on r.  So then as we want to refine our ideas and start writing proofs about limits, we realized that we needed a formal definition--otherwise, as I put it in class, "we're not doing mathematics, we're doing what those people across the hall [the science teachers] do."

So that's the class's official motivation.  But why slog through this? A traditional answer is this: if you're going to go on in proof-based mathematics, you need to be able to write proofs using the formal definition of limit.  A less-traditional answer is that we're going to need some formal properties of limits, continuous functions, and sequences in this very class, and this is our first opportunity to start working those muscles.  But the least-traditional, and most important answer is this: I'm using limits to teach mathematical thinking.

What?  A fundamental mathematical activity is defining, and it's very hard.  A good mathematical definition describes a phenomenon precisely, excluding everything else.  In lower-level classes, I tell students that, unlike in English, a mathematical definition is like a poem: every word is crucial in exactly the place that it is.  When possible, we actually analyze those poems: why do we define a trapezoid as "a quadrilateral with at least one pair of parallel sides", or an isosceles trapezoid as "A quadrilateral with at least one pair of parallel sides, such that the two angles formed by one of the parallel sides with the other two sides are congruent"?  We drop conditions, try to draw things that "break" the revised definition, argue about the merits of an inclusive versus exclusive definition (what theorems about parallelograms follow from our definition of trapezoid?).

But it's rare that students get to construct their own definition, and so that's exactly what we did yesterday.  We iterated through four stabs ("It's getting pretty violent in here!" quipped one student).  Each time we started by taking the fuzzy new idea and rephrasing it in mathematical language.  For example, when a student proposed "The terms in the sequence get closer to the limit," we rephrased as "|x_n – L| decreases."   But each time, one student or another would come up with an objection: "Look, we're saying that these terms are getting closer to 1733 1/3 ... but they're also getting closer to 2000, 2100, or anything bigger than that!"  Then we rephrased: "What do you mean by closer?" "As small as you want."  "Okay, then, so how do we say that mathematically?" "Smaller than any number."  "Okay, then we're going to have to name that any number...."

Who?  It helped that a few students had done the formal definition in a class the previous year, but I found an interesting way to handicap them:  I told them they could only give two kinds of contributions, "genuine questions" and "counterexamples" in response to other students' proposals.  That restriction didn't totally quiet them (although it did, somewhat) -- it forced them to think through what they had learned last year, and to apply the underlying ideas to the definition we were working with.  At least two key ideas (and one major counterexample) were found by students who had never studied limits formally.

How?  I've already said that I hamstrung students who had already studied the topic in a way that made them think mathematically without taking the work away from other students.  But I made a few other crucial decisions that really made this half-hour of discussion go well:

• We started with a rich set of concrete examples--a bunch of sequences with a variety of long-term behaviors--on which we could draw as we worked on our definition.  This context put the meat of the activity--asking whether our current stab included the things we wanted to include, and excluded the things we wanted to exclude--within just about every student's grasp.
• We started with sequences.  In the past, I've found that limits of sequences are much easier for students to get going on than the limit of a function at a point.  Sequences are simpler: the values are discrete, they only go in one direction, and the definition only involves one set of absolute values, not two.  Moreover, there are lots of relatively interesting (nonconstant) sequences whose N's can be explicitly calculated from their ε's.  Conceptually, too, there's a natural quality to "we're wondering what happens as time goes on" that "we're wondering what happens when x gets close to, but not exactly equal to, a" seems to lack.

Well?  Did it work?  I'm not sure yet.  The kids were engaged and spent half an hour discussing definitions before we settled on what we called the "working draft" we'd use until further notice.  Most of the students contributed to the discussion at least once, so that's something.  And when we went through a proof, together, the kids seemed to follow.  But I'll know more tomorrow, after kids try one on their own.  I can tell you this, though: it was way more engaging for everyone, way more interesting, than starting class by writing the definition on the board and then slogging through a bunch of examples.

A Mathematical Adventure Underway!

My apologies for the relative silence.  We're back and running, somehow.

Today's entry is relatively brief but pretty exciting.  For the last two summers, I've taught a Chaos "Maxi" course at HCSSiM:  three weeks, 2.5 hours every morning and 3 hours of problems most every evening, with 15 of my best teenage friends.  The course was based on the Chaos half of my "KAM-Geometry" course that I teach at Walter Payton, but I sussed it up with more analysis and other theoretical math, and of course I cared a little less about students "getting" everything (so long as they weren't totally lost).  What a pair of crazy adventures those courses were!

Our annual "Toga Day" (do mathematics while dressed like Ancient Greek mathematicians)

Batman comes to help us prove the completeness of Fractal space under the house-dwarf metric

A board of conjectures

This spring, I'm teaching the high school version of the course -- but I've decided it won't be a high school version.  Instead, we'll do as much of the sussed-up, proof-heavy, analysis stuff as we can: because it's cool, because it makes the course math (instead of "here's another cool observation, wonder why it happens?"), and because I think the kids can do it.

We do the formal definition of limit tomorrow ... I'll let you know how it goes!