girls aren't the problem; the World is the problem!

My whole educational world is sharing this New York Times article, which argues that we need to modify education to make girls less responsible and more confident, to help them succeed. Here is an email I sent to my faculty (NYU Physics) about this article. I'm putting it here to make it an open letter to universities; what I say about NYU at the end is true of many universities, especially urban ones!

This piece (Why Girls Beat Boys at School and Lose to Them at the Office, New York Times, 2019 February 7) captures the point that girls are more responsible workers and better students and more stressed out. All these things are important and show that we should give tons of support to girls here at NYU.

But then the author’s interpretation is that the problem is with girls. That we should change how girls feel and act. That’s a classic statistical mistake of putting unjustifiable causal structure onto data. Isn’t an equally or even far more plausible explanation that the world is strongly biased against girls and women? See, eg, everything about hiring, promotion, and pay differentials? Then the psychological and behavioral reactions of girls reported in this article actually all make perfect sense as a reaction to this structure. And it’s sensible!

So I think the piece way over-steps in arguing that girls should change their reactions to things. I think it’s better and more likely correct to argue that girls are reacting rationally to a strongly biased world and we should support and promote them as much as we can. Similar things could be said (and I will say them!) about other disadvantaged groups, obviously.

But I agree strongly with my colleagues that this is relevant to our work! As I have said many times, NYU is biasing its undergraduate admissions against hard-working female potential science majors in favor of having a hard-to-justify gender ratio. That is making these structural problems even worse for girls!

manipulating betting odds into probabilities and other odds

A colleague and I wanted to enter into a bet on a group-stage game of the World Cup of Football. We wanted our bet to have the following structure: He pays me something if Team A wins, I pay him something if Team B wins, and we push (that is, neither of us pays) if it is a draw between Team A and Team B. We are friends, so we wanted to give reasonable odds on this bet. How to calculate them?

Online, we saw the following odds posted on a reputable sportsbook (yes, such things exist) in European style:

Team A wins	1.625
Team B wins	6.710
tie	3.910

Is it possible to convert these data into what we want to know? Of course the answer is yes, but we need to make assumptions. Before we continue, I should note that these odds would be written in American style as follows:

Team A wins	-160
Team B wins	+571
tie	+291

Because that may be confusing, let's just check in on what these odds mean. Looking at the European odds, what they mean is that if I bet $100 that Team A wins, I will be paid $162.50 if they do indeed win. That is, I will be paid back my $100 plus $62.50 in winnings. If I bet $100 that there will be a draw, then I will be paid $391.00 (my $100 plus $291 in winnings) if they do indeed draw. Looking at the American odds, what they mean is that if I bet $160 that Team A will win, I will be paid $100 in winnings plus my original bet back, or a total of $260, if they do indeed win. And they mean that if I bet $100 that Team B wins, I will be paid $571 plus my original bet back, or a total of $671, if they do indeed win. The +/- sign at the beginning of those odds makes a big difference! So in this sense, to a scientist, usually the European-style odds writing makes more sense.

Strictly, what's written in European-style odds is not the odds, but the odds-plus-one. If the book was not out to make money—If the book was just trying to break even—then these odds would be the book's approximation to the inverse of the probabilities of the outcomes. If you want to see why, you can think about placing a Dutch bet: Putting money on each outcome in proportion to the probability. If you do that, and the odds are just right, then you will get the same payout no matter what the outcome. That's an exercise to the reader! But if we interpret these European odds as inverse probabilities, then the implied probabilities of the various outcomes would be:

Team A wins	0.615
Team B wins	0.149
draw	0.256

Which you can get just by inverting the numbers in the first chart. The observant reader will notice that these three numbers don't add up to one! They add up to a bit more than one. Why? Because the book is making money, and so they pay out odds slightly lower than what's fair, which means that they have an edge over the bettors, on average.

Now if we make the assumption that the book is relatively capable (or that the betting public is pushing the book's odds to something sensible), then we can assume that these are close to the correct probabilities for the three outcomes. That's a great start! But remember that we want odds for a win bet, with the draw as a push. The idea is, we want to know: What's the probability that Team A wins given that there isn't a draw. For that we condition on there not being a draw and re-normalize that probabilities. Probabilities for these outcomes are:

Team A wins (conditioned on no draw)	0.805
Team B wins (conditioned on no draw)	0.195

I got these by re-normalizing the two outcomes my colleague and I care about to get unity. Now we can convert to European-style odds by inverting again, and the odds we get are:

Team A wins (conditioned on no draw)	1.242
Team B wins (conditioned on no draw)	5.128

That is I should bet $100, and my colleague $24.20, and whoever wins gets it all. If there is a draw, we take our money back. Or at larger stakes, I should bet $412.80, my colleague $100, and so on.

where's the information?

It came up frequently in discussions this summer (and last): Where is the information (in, say, a spectrum of a star) about some parameter of interest (say, the potassium abundance of the star, or the radial velocity), and how much information is there? The answer is very simple! But the issues can be subtle, because there is only calculable information within the context of some kind of model. And by “model” here, I mean a probability density function for the data, parameterized by the parameters of interest. That is, a likelihood function.

The fast answer is this: The information about parameter θ is related to the (inverse squared) amount you can move parameter θ and still get reasonable probability for the data. The nice thing is that you can compute this, often, without doing a full inference. It is easiest in linear (or linearized) models with Gaussian noise! That's the question we will answer here.

When you have a linear or linearized model with Gaussian noise, there are derivatives of the expectation Y for the data with respect to the parameter of interest, dY/dθ. Here (for now) Y is an N-vector the size N of your data, and θ is a scalar parameter (let's call it the velocity!). So the derivative dY/dθ is an N-vector. The information about θ in the data is related to the dot product of this vector with respect to itself: The accuracy with which you can measure θ given data with Gaussian noise with N×N covariance matrix C (possibly diagonal if the N data points are independent) is:

σ_θ^-2 = [dY/dθ]^T C^-1 [dY/dθ]

where σ_θ is the uncertainty on θ. That is, the inverse variance on the θ parameter is the inner product of the derivative vectors, where that inner product uses the inverse variance tensor of the noise in the data as its metric! Here we have implicitly assumed that the vectors are column vectors. When the N data points are independent, the C matrix is diagonal, as is its inverse. Note the units too: The inverse variance tensor has inverse Y-squared units, the inner product uses the derivatives to change this to inverse θ-squared units.

(When there are multiple parameters in θ—say K parameters—the inner product generalizes to making a K×K inverse covariance matrix for the parameter vector, and the expected variance on each parameter is obtained by inverting that inverse variance matrix and looking at the diagonals.)

But we started with the question: Where is the information in the data? In this case, it means: Where in the spectrum is the information about the velocity? The answer is simple: It is where the data—or really the inverse variance tensor for the noise the data—makes large contributions to the inverse variance computed above for θ. You can think of splitting the data into fine chunks, and asking this question about every chunk; the chunks or pixels or data subsets that contribute most to the scalar inverse variance are the subsets that contain the most information about θ.