Rich and Strange Aeons

I'm reading How To Measure Anything and it's had some surprising revelations.

Rule of Five: if you take 5 random samples from a population, there's a 93.75% chance the population median is within the minimum and maximum (range) of the sample. For it not to be, all 5 samples would have to be e.g. below the median. They're random, so the chance of that is 0.5**5 = 3.125%. They could also be above the median, so that's another 3.125%.

The same math gives that a sample set of 3 has a 75% chance of bracketing the population median! A set of 7, 98.4%.


Single Sample Majority: This one's a bit trickier, but say you have a bunch of urns, containing red or green marbles, the urns have a completely uniform distribution. If you draw a single marble from each urn, and bet that the majority of each urn matches the corresponding sample, you'll be right 75% of the time. Bayes:

p(urn|draw) = p(draw|urn) * p(urn)/p(draw)

Uniform, so p(urn) = p(draw)

Uniform, so p(draw|urn) = 0.75. For individual urns it varies: if the urn is 51% red, 51% chance of drawing a red; if 95% red, 95% chance of a red draw, but the urns range over all the percentages.

Honestly this one seems less generally useful than the Rule of Five, but it's still impressive -- if you don't know much, even a single sample can be meaningful.

I'm reading Judea Pearl's _The Book of Why_ (Caltech book club), about causal inference and types of causation. Just got through necessary vs. sufficient causation. It seems to me that captures part of the gun debate: widespread gun ownership is necessary for frequent mass shootings (no guns, no shootings) but not sufficient (we can imagine lots of guns without shootings... like most of the US's own history).

Gun control advocates point out that taking the guns away will stop the shootings; gun rights advocates argue guns aren't really to blame.

Pearl's own illustrative example is a house fire: a match (or some other ignition) is both necessary and sufficient, oxygen in the air is necessary (no oxygen no fire) but not sufficient (just adding oxygen doesn't start a fire). That includes an assumption that oxygen is 'normal', present whether or not the match is (thus enabling the match to be sufficient, because we can assume the oxygen is there).

Are guns like oxygen? For a lot of the US, yes, in being ubiquitous, considered normal, and mostly not killing people. Removing oxygen to prevent fires is usually overkill, and they would say removing guns is too.

Other modern crises:

* greenhouse gases are reasonably necessary and sufficient for global warming. (No gases, no warming; adding greenhouses gases to an 1800 AD background suffices to cause warming.) (As a side note, Pearl points out that climate models allow climate scientists to generate counterfactual 'data' quite easily, by tweaking model parameters.)

* cars and deaths: cars are necessary for 40,000 dead Americans a year (no cars, no such deaths... alternative transport modes aren't nearly so dangerous) and sufficient (adding cars, or rather a car-oriented and -dependent culture[1] is the main reason we have the deaths).

[1] Japan actually has 3/4 as many cars per capita as the US, but 1/3 as many road fatalities per capita; the cars are used less, as well as being smaller and slower in common use. Many fewer deaths per vehicle too, but similar deaths per vehicle-km.

One virtue of my math education was that it was fairly proof heavy; I feel I can demonstrate or prove most of what I know. I may have mentioned this not extending to sin(A+B), which I had to figure out later as an adult. It also didn't include detailed "how do we calculate this shit?" for stuff like trig, log, fractional exponents. Once you hit calculus the answer becomes "Taylor series or something better", but before then?

Feynman talks about it somewhat in Chapter 22 of his Lectures, which is a cool read; he goes from defining multiplication in terms of addition, to numerically suggesting Euler's law, and showing how to build up log tables along the way. It's a heady ride. But he doesn't talk about trig tables.

I've long had my own thoughts about those, and finally wrote some code to try out the calculations.

Basic idea is that given sin(a) = y/r, cos(a) = x/r, and the Pythagorean theorem, there are two right triangles for which we know the values precisely. The 45-45 is trivial, while the 30-60 one can be gotten from sin(60) = cos(30) = sin(30+30) = 2*sin(30)*cos(30). From there, you can apply the half-angle formula as often as you please, to get very small angles, at which point you'll notice that sin(x) ~= (approximately equal to) x for small x. Thus you can approximate an angle like 1 degree which you can't otherwise[1] get to, then use double and addition formulas to build back up to say 10 degrees or anything else.

The question in my mind always was, how good is that? And the code finally answers the question. Sticking to half-angle stuff is *very* accurate, matching the built-in function to 15 decimal places. Using a sin(1 degree) approximation and building up to sin(15 degrees) is accurate to 4 places; starting from 0.125 degrees instead is accurate to 6 places. I don't know how practically good that is; one part in a million sounds pretty good for pre-modern needs -- like, at that point can you make or measure anything that precisely? -- but Feynman says that Briggs in 1620 calculated log tables to 16 decimal places.

[ETA: hmm, I'd been assuming the built-in function, presumably based on power series, was the most accurate version, but maybe I should view it as deviating from the exact roots of the half-angle approach. Or both as having small errors from the 'true' value. Especially as Python's sin(30) returns 0.49999999999999994, not 0.5. Of course, both approaches are using floating point. sin(60) = sqrt(3)/2 to the last digit, but cos(30) is slightly different.]

[1] The two exact triangles basically give you pi/2 and pi/3, and those divided by 2 as much as you want. But you can't get to pi/5 exactly.
a/2 + b/3 = 1/5 #a and b integers
15a + 10b = 6
5(3a+2b) = 6
(note that 2a+3b=5 is totally solvable.)

Though I guess a more valid approach would be
c(1/2)^a + d/3*(1/2)^b = 1/5 #all variables integers
c*5*3*2^b + d*5*2^a = 3*2^(a+b)
5(c*3*2^b + d*2^a) = 3*2^(a+b)
which still isn't soluble.

Likewise for getting to pi/9 (20 degrees):
c(1/2)^a + d/3*(1/2)^b = 1/9
c*9*2^b + d*3*2^a = 2^(a+b)
3(c*3*2^b + d*2^a) = 2^(a+b)

When I was a kid, I learned -- probably in The Gentle Art of Mathematics or some such book -- how to convert numerals from one base to another. Divide n by newbase, record the remainder, replace n by the quotient, repeat until quotient = 0, then write the remainders in reverse order.

This manifestly worked, if you did it, but always felt backwards to me, doubly so -- once for the reverse order thing, and once for the fact that you're only using remainders, not quotients, or so it seemed. Plus, though it gave the right answer, I didn't have much sense of why it worked.

Well, I was thinking about it early this morning in lieu of sleeping again, as I do, and came up with a method[1] that's slow but does both use quotients and yield the digits in MSB (most significant 'bit' first, descending significance). And then I thought about the old remainder method again, and it made more sense[2], and I also realized a key mismatch: the whole cycle of dividing by newbase calls up long division, may even involve long division if newbase is multiple digits in oldbase, and long division gives its answer in MSB. The remainder method chews up the number LSB, and gives the newbase numeral's digits LSB, so it's not backwards at all by its own lights, but it is LSB instead of MSB. Works fine if you write the units first and work to the left.

[1] Trial and error: multiply by increasing powers of newbase until the quotient is zero; backoff to the preceding power, and the resulting quotient is the first digit (MSB) of your answer. Repeat for the remainder. So (base 10 to base 10, for simplicity), n=742: find that 742//1000 = 0, 742//100 = 7, so 7, then repeat for 42. A subtlety is that numeral-based long division may not work, e.g. if you're converting a binary numeral to base ten and dividing by '1010': binary quotients are only 0 or 1. You'd have to go back to basics, division as "how many times can I subtract b from a?"

[2] 10 to 10 again: n=742, divide by 10 and get 2, that's your units; now shift right (or use the quotient) to get 74, divide by 10, get 4, which is your tens digit, and so on. Thinking about the relation between dividing by 2, and shifting the binary representation of a number to the right, made it all clear.

Dunno if I'm making it clear. I think the core insight for me is simply "base conversion felt backwards because you felt it should go in the same order as long division, but it doesn't and is doing something else. Also "put the remainders aside and write in reverse order" is less clear than "write the remainders as you find them, but right to left."

So there are the left and right Riemann sums, and the much better midpoint Riemann sum. Recently I wondered about integrals that took the average of the endpoint of each strip: (f(x)+f(x+step))/2. The thing is that most of the halves combine, so you can just add up f(x) for each whole step, plus f(start)/2 and f(end)/2. How's that compare?

Much better than the left and right sums, but not quite as good as the standard midpoint one. E.g. the integrals of sin(x) over 0 to pi/2 are

left_: 0.9992143962198378
right: 1.0007851925466327
mid__: 1.0000001028083885
mine_: 0.9999997943832352

All the other integrals I tried show a similar pattern: x, x^2, x^3, 1/x, e(x)... the two are close, but midpoint is just a bit closer to the correct answer. Or looked at another way, has close to 1/2 the error... hmm, that factor is consistent too. I should look into that.

Or: if I just recalled my terminology correctly, midpoint Riemann sums have half the error of trapezoidal Riemann sums. Which is not what I would have expected.

This was something covered in pre-calculus, way back in 7th grade. Like most of precalc, it didn't leave a deep conscious impression on me. Or much of any impression, in this case. In my new hobby of revisiting math fundamentals, largely in bed, I tried to figure it out in my head, got annoyed, and looked it up. The formula isn't bad, though not one I was working toward (it uses Ax+By+C=0 lines, I was using y=mx+b), but the proofs I saw were pretty grotty. I came up with a new one I like more.

Simplify! Skipping the *really* simple stages, of horizontal and vertical lines, what's the distance from a line to the origin? Combining the two representations, the line is y=-Ax/B -C/B. A perpendicular line from the origin is y=Bx/A. Via algebra, he point of intersection is x=CA/(A^2+B^2), y=CB/(A^2+B^2). The distance from that to the origin is |C|/sqrt(A^2+B^2). (Adding |absolute value| because distances must be positive, also this C is actually a square root of C^2 from the Euclidean distance formula, so of course pick the positive root.)

Now, given a line and some other point (X,Y), we just have to translate the system so the point coincides with the origin. This turns the line into A(x-(-X))+B(y-(-Y))+C= Ax+AX+By+BY+C=Ax+By+(AX+BY+C)=0 Applying the previous result, the distance is |AX+BY+C|/sqrt(A^2+B^2). Voila!

I need a math icon.

In number theory there is Euler's φ(n) (or totient function), henceforth phi(n) because easier to type, which gives the number of numbers (sorry) less than n which are relatively prime to n. So for n=6, phi(n)=2, because only 1 and 5 are relatively prime to 6, {2 3 4} all sharing a divisor. Confusingly, 1 is both a divisor of 6 and considered relatively prime to it, with a g.c.d(1,6)=1.

The formula for phi is elegant: if n=p^a*q^b*r^c... p, q, r being distinct prime factors, then phi(n) = n * (1 - 1/p) * (1 - 1/q) * (1 - 1/r)...
So for n=6, 6*(1/2)*(2/3)=2
n=12, 12*(1/2)*(2/3)=4, {1 5 7 11}

Davenport (The Higher Arithmetic) gives a proof based on the Chinese remainder theorem, which I think I understood but have forgotten. Andrews (Number Theory) gives a different proof based on Moebius numbers, which has so far resisted my casual and sleep-deprived browsing.

But looked at another way, it seems obvious! That way is the Sieve of Eratosthenes, one of the few episodes I distinctly remember from elementary school math. The formula represents striking out all the multiples of p (including p), then striking from what is left all the multiples of q, and so on. All remaining numbers (including 1, heh) will share no prime factors with n.

So for n=6:
{1 2 3 4 5 6}
{1 3 5} 1/2 numbers remaining
{1 5} 2/3 of those remaining

That doesn't feel like a traditionally rigorous proof, but it also seems pretty straightforward and without obvious holes.

(If you multiply out the terms, you get something you can see as subtracting from the count the multiples of p, q, etc., then adding back in for the multiples of pq, pr, qr, etc. that were subtracted multiple times, and so on, but that seems less obviously correct than the first approach.)

I bought this popular math book at B&N, for my quasi-niece G2, for lack of anything obviously better on the shelf. I read it before turning it over to her, and it was fun; I think I would have liked it at her age (12) though I'm not sure if she will. Of course, it was less eye-openingi for me now: I've taken classes on Bayesian statistics, I've given talks on voting systems, I've read about the file drawer problem or the exponential stockbroker scam. Still, I did learn some things.

A big one was what he calls "don't talk about percentages of things that could be negative." More specifically, say the US adds 18,000 jobs one month, and Wisconsin adds 9,000 jobs. Does that mean Wisconsin was responsible for 50% of US job growth? The governor of WI would like to say so. But say that California added 30,000 jobs -- does that mean it was responsible for 166% of US job growth? Uh... The trick being that the US number is a net sum of positive and negative (say Texas lost 21,000 jobs) numbers; taking percentages of that is meaningless.

Another example: it's said that the top 1% have taken 93% of US income growth. Sounds pretty bad. But the next 9% might (I don't recall the numbers) might have taken 20% of income growth. Uh oh... we can balance this by actual reductions of income in the bottom 90%. So this is better and worse: more people are doing well, but the rest are actually falling behind, not just standing still. But the "middle class" would probably be less outraged by learning that they were also doing well.


Berkson's Fallacy was something I'd vaguely heard about, but forgotten. It's explained well at the link (by him, even), but the short version is that two independent variables can look negatively correlated if you select for either of them. Like, if you notice nice people or hot people, you'll find that of the people you notice, many hot people are jerks. But this needn't mean hot people are actually prone to jerkiness, just that you ignore plain jerks.

You can extend that to any two variables of desirable things. Niceness and richness, niceness and political agreeableness... say I put up with people if they're personally agreeable or politically agreeable; this will lead to my thinking that my political allies are unfortunately prone to being jerks, when it's more that I ignore opponents who are jerks.


A couple of geometrical statistics I hadn't know: correlation being the same as elliptical eccentricity, and Pearson correlation being the cosine of the angle between two vectors. Simple that way, ugly as an algebraic formula of the components.

Also, correlation is not transitive, the way that Dad and Mom are both related to Baby but not each other. Portfolio 1 might be IBM and Apple, correlated with P2 which is Apple and Honda, but not with P3, Honda and GM.


Reminders of the ubiquity of regression to the mean, and of the variance of small populations, are always useful.


Slime molds apparently suffer a voting paradox. They like oats and dislike light, it makes sense that you can balance them between a big pile of oats in the dark and a bigger pile under a UV lamp. It makes less sense that if you add a small pile of oats in the dark, the mold starts going for the big pile in the dark.

'hazard' comes from the Arabic for dice.

He talks a bit about the damage done by the cult of genius, when those magical moments of insight and revelation take a lot of work beforehand to prepare your brain.

Proof advice: try to prove it by day, disprove it by night. (More applicable if you don't know if it's true or false.) Might get insight into why it has to be true, or find out you were wrong all along.

Condorcet wanted the Rights of Man for women, Hilbert refused to endorse the Kaiser in WWI, and defended giving Emmy Noether a position.

In grad school Douglas Hofstadter -- yes, that one -- was my advisor, and I took several of his courses, most of which were only tangentially related to what you'd think of as cognitive science, and just forget about computers. The last one was "Mind and the Atom", which I mostly remember as reading a bunch of early philosophers, then the resurrection of atomic ideas, with Doug sometimes asking "how did they believe this stuff?" partly as incredulity, partly as an actual cognitive question: how did a bright guy like Thales even *think* of "everything is made of water?" I don't remember any answers to that. I remember trying to memorize the periodic table, and reading about how horrible the 14th century was, though that might have been my abusing my eee in class.

Also we had to give skits at one point, from the perspective of various philosophers, with Democritus et al. probably off limits due to their being right and where's the fun in that? Anyway I think I went with Heraclitus.

Or, no, this is why I keep a journal; we had an atomist group and anti-atomist. Everyone I knew was in the atomist group, but it was bigger than the anti one, so I felt I had to join the latter. So maybe someone did have Democritus.

Anyway, we had to write a paper for the end of the class. Probably no assigned topic. I couldn't think of one and procrastinated and procrastinated. I should have gone to him and asked for a topic or something, but I didn't, I don't know why. Probably massive loads of not giving a damn -- not about him, but about grad school in general at that point, and certainly not *grades*, who cares about the grades of some 7th year PhD student, beyond "why are you even taking classes?" Maybe I was depressed again, like senior year at Caltech? I have no idea, now. At any rate I managed to put it off to the night before it was due... and beyond. Had some vague idea of doing a "who was right" paper.

Finally, that morning, I had an idea: write a dialogue! Of, like, the various philosophers arguing it out equipped with modern science knowledge about which of them was most right. I think my main motivation was laziness. See, I've always been really good at writing essays -- short one, anyway, not many long in my life ever -- but I do need a topic, and it does take a bit of effort. Can't write as fast as I can physically type, unless I've really rehearsed the ideas in my head. And even then I'd often thing i had something worked out in my head, then freeze once I sat down at a keyboard.

But I do a lot of thinking as imaginary conversations, me talking to someone, real or imagined, and I've had a fair number of insights in those -- the explanation effect or teaching effect at work, probably -- and more of note here, it just *flows*. Those never block. Repeat, maybe, but not block. So I figured I could set them up as imaginary people, have them talk, transcribe it, and that'd be my paper. Might not be great, but it'd be something to turn in. Easier to write, and more *fun*, and something I'd never done before -- a challenge! Which probably shouldn't be combined with last-minute papers, but eh, it's worked before, as in the papers I wrote for some Caltech humanities classes while affected by the verbose and ornate style affected by Steven Brust through the character, or rather, fictional author, of Paarfi, in The Phoenix Guards, following in the style and structure of The Three Musketeers, or at least of whatever English translations of that book that Brust had read and, obviously, enjoyed.

And of course Doug had done all those Achilles-Tortoise dialogues in Goëdel, Escher Bach, but I just figured that meant he wouldn't reject the paper out of hand based on format, not that I'd get a better grade for it. And of course I had the idea for such dialogues from that book, hanging around in the back of my mind for years to spring.

So I finally start writing around the time it's due, blowing off the last class so I can do the paper, and yeah, it flows out fairly well onto the screen, and a few hours later I'm done and can turn it in. (Ooh, I recorded that; 3600 words in 3.5 hours, including proofreading and distractions. Probably not perfect flow, I can type 30 wpm, or 1800 words per hour. (Also, Firefox spell-checker knows 'wpm', but not 'spellchecker'.)) You can see the result here (10 page PDF); I cleaned up some typos and LaTeX gave it a new date, but otherwise it's the same.

And it *was* fun to write, and probably a lot easier than writing some more normal essay on the same topic would have been, with a topic of some sort, unless I kept the conclusion a surprise, and digesting the various philosophers in my own words, instead of what felt like their own, even though it was all really my own, but there's a cognitive difference in imagining other people, or in imagining yourself as other people -- you can fix some cognitive errors in the lab just by asking them to "think like a trader", and I've found that telling myself "think like a Buddhist" causes a calmness and "think like a happy person" makes me smile.

Also, I just realized tonight that arguably this paper is fanfic or real person fic of an odd sorts, and thus the only example of such I've actually written down, at least that's longer than a paragraph. First fanfic, arguing philosphers, go me. Literally self-insert, too, since I'm in the dialogue.

Anyway.

His opinion: "Some of the best work I've seen from you." and a high grade. Which I'd had before, outside of the ambigrams class I sucked at, so I don't think I cleared a low bar.

My reaction to that: "This... this is not helping." I've thought for a long time that I keep procrastinating because I keep getting away with it; I can't easily think of any "yeah, I got really burned by putting it off" incidents, except maybe all of grad school itself (main discovery: I suck at self-motivation or choosing my own topics), while lots of getting away with it, or even being serendipitously helped by the lateness, where doing the right thing wouldn't have been as good. ("Wow, this book would be the perfect gift for John, and it's only 50 cents! Good thing I didn't give him something lamer yesterday.") Certainly wrote lots of short essays the period before class in high school, to As, and also to a 5/5 on both AP English exams, so it's not just that my teachers were coddling me. But nothing so long, so late. Best work? WTF.

I re-read it, and it still doesn't seem *bad*. Great, I dunno, I don't think I can have that thought about myself, I just have "sucks", "okay", and "other people tell me it's good". Unless I'm specifically re-writing someone else's words or ideas, then I can say "better than *that*". Done a fair bit of that online, at short scales, plus giving an intro quantum computing talk after someone else's intro quantum computing talk because I thought I could do it better. Influenced by Hofstadter, actually, and his emphases on examples, clarity, and examples [sic].

My other fluid writing mode is "stream of consciousness", you just got a full barrel of it.

***

For comparison, here's my first paper for one of his classes (12 page PDF), his Group Theory Visualized course, where i seem to have gone for a high clarity regurgitation of some of what we'd learned, especially first very basic group theory and then what automorphisms and such were, those being a new concept I'd struggled with in the class. On re-reading, I thought I'd done fine -- very clear and even funny -- until I hit section 5. Footnote 5 is opaque, I switched from phi to f for no reason, I think the endomorphism paragraph makes sense but it needs to be much clearer, and the sample mappings table is a great idea but doing something weird at the end. What the hell is *g? I think I figured it out to something sensible, but I shouldn't have had to, that violate the purpose of the paper... Section 7 should show the math, at least if I'm aiming for a naive audience (what audience was I aiming for? No idea, now.) Section 9 uses standard notation for some groups, not the notation I'd built up in the paper. Section 10 says "It may help to recall what factor groups are" when I'd never mentioned them before. Maybe the paper was aimed by my classmates, or just at my teacher? That may make more sense, trying to explain one of our topics. Section 11... proves that.

That was disappointing. I went from "this is great!" to "no, this is flawed but I can fix that" to "what's going on here and why do I care?" Maybe whipping out my best work wasn't such a high bar.

I've known elementary group theory since 7th grade, I suppose it makes sense that I was able to be really clear for the first half.

***

Also surprising: I had barely any tags appropriate for this, just 'philosophy' and 'math'. I guess I rarely LJed about grad school or classes?