tag:blogger.com,1999:blog-17232051.post3212991231324448661..comments2023-02-04T12:49:36.805-05:00Comments on Noahpinion: Why predict percentages?Noah Smithhttp://www.blogger.com/profile/09093917601641588575noreply@blogger.comBlogger25125tag:blogger.com,1999:blog-17232051.post-45433252258665611792012-11-07T07:37:00.261-05:002012-11-07T07:37:00.261-05:00I now have a better idea of what's going on. I...I now have a better idea of what's going on. I hadn't realized that Silver kept updating his odds to the last minute. If he did that and his performance hadn't improved, then it meant that there was something very wrong with his model.<br /><br />But this raises for me another question. If I were using his model for practical purposes, I'd need to know its performance over time. For example, suppose it's Nov. 5 and I need to figure out where to concentrate my resources to maximize their impact on the electoral outcome. One of the variables that I'd consider would be the odds in the various locations that I'd be considering, no? I'm sure Nov. 6 involves different considerations since my resources would become less mobile as it nears Nov. 7, but again, odds would figure into my decisions. And so on.<br /><br />The media focuses on Silver's perfect or near-perfect score because they have a different concern: maintaining the interest of viewers well after the ballots have been closed, when there is nothing, really, that voters and the campaigns can do but wait for the tally and all talk about surges, firewalls and the like are illusory at best.Jun Okumurahttps://www.blogger.com/profile/00291478225274759649noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-44087295427112668652012-11-04T17:05:07.397-05:002012-11-04T17:05:07.397-05:00Probabilities imply fair odds for betting on outco...Probabilities imply fair odds for betting on outcomes. Traders express their views in a similar but better way, in the form of a two way market. In this case for example 69/71, meaning I am willing to "buy" the outcome at odds of 69% and "sell" at 71%. thetanoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-87561086981480118242012-11-03T00:06:48.019-04:002012-11-03T00:06:48.019-04:00Nate Silver's model actually takes into accoun...Nate Silver's model actually takes into account correlations between states. I'm not sure what method he uses to do this, but I'm guessing he relies on some mix of historical correlations among poll results, demographic data, etc.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-32971092980126875232012-11-02T22:45:14.831-04:002012-11-02T22:45:14.831-04:00I'm gonna get picky on you here, Noah.
You me...I'm gonna get picky on you here, Noah.<br /><br />You mean probabilities, not percentages. E.g. "70% _chance_."<br /><br />Percentages are just a different notation for ratios. Ratios can measure lots of different things besides probabilities.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-34674865670247069672012-11-02T18:31:50.067-04:002012-11-02T18:31:50.067-04:00That is definitely true but if you assume uniform ...That is definitely true but if you assume uniform swing which is a pretty decent approximation then the model will predict all or almost all of the states if the polling is good. In any case, the model doesn't have to successfully predict a bunch of independent events it just needs to predict two sets of events [each candidate winning all their 80/20 states].maxhtimmonshttps://www.blogger.com/profile/16725324604158912393noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-8461484377843658042012-11-02T16:34:14.835-04:002012-11-02T16:34:14.835-04:00Do silver or other statisticians in the public opi...Do silver or other statisticians in the public opinion forecasting realm ever use hindcasting as a method of tuning/testing the accuracy of their models? I certainly never hear about it if they do.Ben Johannsonnoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-83103099647975405052012-11-02T16:16:37.702-04:002012-11-02T16:16:37.702-04:00That's right. I didn't mention predictions...That's right. I didn't mention predictions for vote shares, only for win/lose events.Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-17996505528141534152012-11-02T16:13:36.525-04:002012-11-02T16:13:36.525-04:00That'll turn out to be the same thing, I'm...That'll turn out to be the same thing, I'm pretty sure.Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-84792464875765295212012-11-02T16:10:48.522-04:002012-11-02T16:10:48.522-04:00I actually don't understand what you're as...I actually don't understand what you're asking...Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-56236511454651802302012-11-02T12:10:06.616-04:002012-11-02T12:10:06.616-04:00I want a Minimum Squared Error function for Nate S...I want a Minimum Squared Error function for Nate Silver.Davydehttps://www.blogger.com/profile/15161743272166194930noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-3498334196936220862012-11-02T11:56:56.958-04:002012-11-02T11:56:56.958-04:00"Results in each state are strongly correlate..."Results in each state are strongly correlated"<br /><br />I thought about that, but then I got to thinking, 80-20 for candidate A over candidate B can't be strongly correlated with 80-20 for candidate B over candidate A, and that was more or less where I gave up.Jun Okumurahttps://www.blogger.com/profile/00291478225274759649noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-69201381491214529662012-11-02T08:22:37.454-04:002012-11-02T08:22:37.454-04:00The idea here isn't quite right, since Noah is...The idea here isn't quite right, since Noah is operating from a win/lose perspective, while the percentages in Noah's model don't just tell you who will win, but how they will win. As Silver has pointed out, if Obama won California by 20 percent less than Silver has him down for, and Romney won Colorado by 5 percent more than Silver has him down for, he'd be less wrong about Colorado than about California. Roger Gathmannhttps://www.blogger.com/profile/11257400843748041639noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-89388808186282352392012-11-02T08:05:08.674-04:002012-11-02T08:05:08.674-04:00Silver has been pretty transparent about his metho...Silver has been pretty transparent about his methodology and his new book is a pretty good read even if one has a good grasp of probability and finite mathematics. He also has a good data set of past predictions that go beyond just the 2008 presidential race including the Senate races and 2010 midterm elections when he called the Republican sweep of the House. What he does is slightly different from what a bookmaker does. The bookmaker sets preliminary odds of an occurrence happening and then adjusts those odds depending on the wagers coming in so that he/she can assure that a profit will be made. Silver is looking at data streams and is agnostic about the outcome since there is no financial impact on his decision (unless Joe Scarborough takes him up on the charity bet).<br /><br />As we know, his probability percentage is somewhat complicated as it necessarily relies on the combinations/permutations to get to 270 electoral votes. Once all of the certain state outcomes are discarded, one is still left with 7-10 states whose outcomes will determine the result. Today Silver notes that Obama has just over 80% chance of winning. One can judge him on this prediction as well as the individual predictions of each state. He could get the overall election call correct and still miss the results in some states.<br /><br />None of this is rocket science and a lot of us have been doing this same thing in looking at baseball statistics and outcomes where there is a huge data set. It's safe to say that new metric models for baseball performance have been more successful than economic models in recent years.Alan Goldhammernoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-41678013048125309782012-11-02T08:02:09.218-04:002012-11-02T08:02:09.218-04:00As long as you have an Intrade-type market for pre...As long as you have an Intrade-type market for predictions, you can measure the accuracy of any forecaster by using their predictions to make money (long Silver, short Intrade, or vice-versa). If Silver is systematically better than Intrade, over time, Intrade will converge on Silver or it will be possible to make an arbitrarily large amount of money without taking commensurate risk, something markets tend to prevent (magically!?) because otherwise the market will be destroyed (irreparably destabilized).s jaynoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-71096513166245670862012-11-02T07:49:15.654-04:002012-11-02T07:49:15.654-04:00Data from Google's internal prediction market ...Data from Google's internal prediction market and its relevance to probabilities...<br />http://www.cooperationcommons.com/cooperationcommons/blog/howard-rheingold/87-googles-internal-prediction-marketss jaynoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-74000315467290426912012-11-02T07:01:44.828-04:002012-11-02T07:01:44.828-04:00You can only ask that question if you have a good ...You can only ask that question if you have a good answer to the question of why you exist.waltnoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-48498367915851313472012-11-02T05:23:07.077-04:002012-11-02T05:23:07.077-04:00I mean: R(t)=1 if it rains and R(t)=0 if it doesn&...I mean: R(t)=1 if it rains and R(t)=0 if it doesn't rainNick Rowehttps://www.blogger.com/profile/04982579343160429422noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-59197729733644073762012-11-02T05:21:59.585-04:002012-11-02T05:21:59.585-04:00Noah: there must be a better way, that works even ...Noah: there must be a better way, that works even if the forecaster only makes one probabilistic prediction per event. My econometrics is cr*p, but wouldn't something like this work?<br /><br />Suppose two weather forecasters give a probability of rain tomorrow as P(t). Let R(t) be a dummy for rain tomorrow, where R(t)=0 if it rains and R(t)=0 if it doesn't rain. And we have a long sample of R(t) and P(t) for both forecasters.<br /><br />Estimate a regression R(t) = a + b.P(t) for each forecaster<br /><br />The forecaster who gets a bigger b and smaller a is a better forecaster??? Or something like that???<br /><br />Econometricians must have solved this problem (I hope). That's why God invented econometricians, to solve problems like this, that people like me puzzle over but can't figure out. If econometricians can't figure this one out for me, why do they exist?Nick Rowehttps://www.blogger.com/profile/04982579343160429422noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-52579469203514427782012-11-02T03:56:03.409-04:002012-11-02T03:56:03.409-04:00@Jun Okumura
Results in each state are strongly c...@Jun Okumura <br />Results in each state are strongly correlated so if there are a number of states where one candidate is a 80% favorite, then there is a pretty good chance that they will all go for the favored candidate. This also implies that most elections will be predicted be very well but if the model misses it will by a lot. Obviously luck is needed for states that are true toss-ups on election day too. A better metric for judging performance than number of states called correctly would be the product of the probabilities assigned to the outcome of each state.maxhtimmonshttps://www.blogger.com/profile/16725324604158912393noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-81641776147130963442012-11-02T02:55:00.835-04:002012-11-02T02:55:00.835-04:00Mr. Smith:
I remember reading somewhere that Silv...Mr. Smith:<br /><br />I remember reading somewhere that Silver got 49 out of 50 states right in 2008. My admittedly arithmetically-challenged brain tells me that if that were indicative of the long-term accuracy of his model, the model must have made an unbelievable number of in-the-bag calls and made them stick. Here's an illustrative example that I made up:<br /><br />His model calls 45 states 100%-0% and the remaining 5 states 80%-20% for one candidate or other, which brings the overall hypothetical accuracy of the calls to 98%, or what the model's actual record turned out to be.<br /><br />Is the model so powerful, are states that easy to call (yes I realize that many states are deeply blue or red), are the results anomalous, or is my reasoning off by a mile (or, as we call it here, 1.6 km)? <br /> Jun Okumurahttps://www.blogger.com/profile/00291478225274759649noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-8877698150960783552012-11-01T22:29:08.915-04:002012-11-01T22:29:08.915-04:00Noah, I was very sad to see that your previous pos...Noah, I was very sad to see that your previous post on entrepreneurship did not have any Easter Eggs in its photo accompanying the post.<br /><br />Good to see you are back on your game here! Simonnoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-80641706508065809082012-11-01T16:49:22.330-04:002012-11-01T16:49:22.330-04:00I think bookies' odds are actually a fairly di...I think bookies' odds are actually a fairly direct outcome of their hedging activities. ( Here's a write-up , though I seem to recall having seen a rather better one elsewhere: http://e-sportbets.org/how-does-your-bookmaker-calculate-its-odds/ ) That has a lot in common with dynamic hedging strategies used in securities trading. <br /><br />So what Silver is trying to do is not really the same as what a bookie does. He's more concerned with balancing actual voter choices (as told to pollsters) as accurately as possible rather than balancing bettor's *predictions* of voters' choices. Sethhttps://www.blogger.com/profile/16486234948199900568noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-28643546775151608562012-11-01T15:31:23.475-04:002012-11-01T15:31:23.475-04:00Silver is making predictions for individual states...Silver is making predictions for individual states so you can check the quality of his prediction by comparing outcomes with his predictions for the individual states. The electoral college prediction is just an (appropriate) aggregate of the predictions for the individual states.Absalonhttps://www.blogger.com/profile/09131268683451462949noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-38359679327511031592012-11-01T14:08:42.074-04:002012-11-01T14:08:42.074-04:00Starting to look more and more like I'm gonna ...Starting to look more and more like I'm gonna owe you and Chris beers on Nov. 7th. But I prefer to think of our bet as properly hedging against even a small chance of crushing depression on that day.Jolly Greenhttps://www.blogger.com/profile/09631364433925344569noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-58666950851287198492012-11-01T13:03:45.762-04:002012-11-01T13:03:45.762-04:00Look at something like the greatest thing in the w...Look at something like the greatest thing in the world, KenPom. Pomeroy's model, like Silver's, gives probablistic predictions of college basketball games. Now, there are a lot more basketball games in any year than there are elections, but KenPom checks by seeing if his 50-55% favorites are winning 50-55% of their games, that the 70% favorites win 70% of the time, and so on. If we're patient, the way to test Silver is to see if he's hitting his percentages.Anonymoushttps://www.blogger.com/profile/11611167410303250685noreply@blogger.com