where's the information?

It came up frequently in discussions this summer (and last): Where is the information (in, say, a spectrum of a star) about some parameter of interest (say, the potassium abundance of the star, or the radial velocity), and how much information is there? The answer is very simple! But the issues can be subtle, because there is only calculable information within the context of some kind of model. And by “model” here, I mean a probability density function for the data, parameterized by the parameters of interest. That is, a likelihood function.

The fast answer is this: The information about parameter θ is related to the (inverse squared) amount you can move parameter θ and still get reasonable probability for the data. The nice thing is that you can compute this, often, without doing a full inference. It is easiest in linear (or linearized) models with Gaussian noise! That's the question we will answer here.

When you have a linear or linearized model with Gaussian noise, there are derivatives of the expectation Y for the data with respect to the parameter of interest, dY/dθ. Here (for now) Y is an N-vector the size N of your data, and θ is a scalar parameter (let's call it the velocity!). So the derivative dY/dθ is an N-vector. The information about θ in the data is related to the dot product of this vector with respect to itself: The accuracy with which you can measure θ given data with Gaussian noise with N×N covariance matrix C (possibly diagonal if the N data points are independent) is:

σ_θ^-2 = [dY/dθ]^T C^-1 [dY/dθ]

where σ_θ is the uncertainty on θ. That is, the inverse variance on the θ parameter is the inner product of the derivative vectors, where that inner product uses the inverse variance tensor of the noise in the data as its metric! Here we have implicitly assumed that the vectors are column vectors. When the N data points are independent, the C matrix is diagonal, as is its inverse. Note the units too: The inverse variance tensor has inverse Y-squared units, the inner product uses the derivatives to change this to inverse θ-squared units.

(When there are multiple parameters in θ—say K parameters—the inner product generalizes to making a K×K inverse covariance matrix for the parameter vector, and the expected variance on each parameter is obtained by inverting that inverse variance matrix and looking at the diagonals.)

But we started with the question: Where is the information in the data? In this case, it means: Where in the spectrum is the information about the velocity? The answer is simple: It is where the data—or really the inverse variance tensor for the noise the data—makes large contributions to the inverse variance computed above for θ. You can think of splitting the data into fine chunks, and asking this question about every chunk; the chunks or pixels or data subsets that contribute most to the scalar inverse variance are the subsets that contain the most information about θ.

walk or take the elevator?

I'm just generally excited about getting back into the classroom after a long sabbatical. I'm thinking about problem-set problems for the Physics Majors. Here's what's in my head right now:

NYC has had a hot summer, with most buildings running air conditioning on a thermostat continuously. To save energy, NYU (and other large entities in NYC) asked their employees to conserve energy in various ways, some of which we might take issue with. Here's an uncontroversial one: You should take the stairs, not the elevator.

But is that uncontroversial? What considerations are required to figure out whether this policy would reduce or increase energy consumption? Obviously—if you take the stairs—you use less elevator energy, but then you drop a metabolic load on the building air-conditioning. Which uses more power in the end? Use a combination of web research and simple physical arguments to make cases, and identify weaknesses in your argument as you change assumptions. Things that matter include: Neither humans nor elevators are 100-percent efficient delivery vehicles for potential energy (in fact, can you see a fundamental argument that elevators must spend more than 50 percent of their energy generating heat?). Elevators are heavy but counter-weighted. Some buildings have very busy elevators, so your contribution to the elevator load is only the marginal contribution; in other buildings you are typically the only person in the elevator. Air conditioning systems have efficiencies limited by fundamental ideas in thermodynamics, but are probably much less efficient than the limits. And so on!

Thanks to Andrei Gruzinov (NYU) for starting me thinking about this one.

Syllabus (the book)

I just read Syllabus by Lynda Barry. It is a set of syllabi and daily in-class instructions, along with some reflections, from a few years of teaching cartooning and writing at Wisconsin. It is a combination of hilarious and insightful, both about learning to draw and cartoon, but also about the practice of teaching.

A page from Lynda Barry's Syllabus.

The syllabi and daily exercises include many great practices. For example, when students are listening to something (being read or previously recorded), she has them either draw tight spirals in their notebooks or else color in a line drawing. She has the students sketch in non-photo blue pencil and ink in later; it reduces inhibitions. She has simple forms for students to write daily diary or journal entries so that they make close observations of the world, in words and pictures. She loves most the students who have never drawn (since childhood), and she makes exercises that capitalize on their newness (and defeat those who are more experienced at rendering), like forcing them to do drawings in increasingly short time intervals.

The book is hilarious in part because she shows lots of great student work (it is not clear she has proper permissions here, because she does not individually credit each student drawing), and because the whole book is a collage of drawings, writings, and found objects, in the Lynda Barry style. I doubt there is another book out there on teaching that makes you laugh out loud all the way through.

A great question for me is what of Barry's practices could carry over to a class in Physics?