tag:blogger.com,1999:blog-953859622641559363.post4571503309646577872..comments2017-09-18T08:59:09.824-07:00Comments on A Numbers Game: Judge Side BiasA Numbers Gamehttp://www.blogger.com/profile/18412505663480533746noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-953859622641559363.post-70619096565221344842009-05-04T01:19:00.000-07:002009-05-04T01:19:00.000-07:00Thanks for asking around, Rob. I do plan on adjust...Thanks for asking around, Rob. I do plan on adjusting for team strength, but I'll probably first look at resolution bias.A Numbers Gamehttps://www.blogger.com/profile/18412505663480533746noreply@blogger.comtag:blogger.com,1999:blog-953859622641559363.post-86229302910086447152009-05-04T01:17:00.000-07:002009-05-04T01:17:00.000-07:00Gonzo, the formula has no controls for division. Y...Gonzo, the formula has no controls for division. You make a good point.<br /><br />I suspect (though I have not checked) that there is a slight negative bias in JV, as some teams may be made up of one debater who should be in the open division and one who should be in the novice division. The better debater, as a 2N, can often win the debate without much help from his/her partner.<br /><br />The confidence interval takes into account the number of ballots a judge has produced, and creates larger confidence intervals for judges with fewer ballots.<br /><br />I think the main problem with the numbers is related to the total number of judges. So many judges were tested, finding a few with skewed ballot counts may simply be identifying the tails of a distribution with no greater variance than if each round had been decided by a coin flip.A Numbers Gamehttps://www.blogger.com/profile/18412505663480533746noreply@blogger.comtag:blogger.com,1999:blog-953859622641559363.post-8350609245149075802009-05-04T01:08:00.000-07:002009-05-04T01:08:00.000-07:00Bias from the PRNG is something I hadn't considere...Bias from the PRNG is something I hadn't considered yet, Anonymous. That's an intriguing idea.<br /><br />I do plan on adjusting the numbers based on team strength, but your idea is much more interesting than that. I think a source code review of the common software would be revealing.A Numbers Gamehttps://www.blogger.com/profile/18412505663480533746noreply@blogger.comtag:blogger.com,1999:blog-953859622641559363.post-76106440040816475772009-04-30T10:16:00.000-07:002009-04-30T10:16:00.000-07:00To follow up on my previous comment, I asked a few...To follow up on my previous comment, I asked a few statistics people for suggestions on dealing with weighting. Here is the response: <br /><br /><I>The main problem is that you have a tiny sample size. With 50-100 debates per year, you could potentially just eliminate all cases where a team that ended up higher in the rankings beat a team below them, retroactively scaled.<br /><br />So once you have the final rankings, if #1 beat #4 in the first round, just eliminate that from consideration. Presumably they're a better debating team if they won. Do this for all 50-100 matches, see what you have left.<br /><br />Potentially, only eliminate ones where the final rankings differ by more than 2-3 or some arbitrary margin. Or repeat the analysis with various sets of criteria.<br /><br />Then take whatever is left, and test them. You could try something like a sign test, where + is affirmative, and - is the opposite, and just see if there's any sort of proof of bias at a 90% level. It's not going to be hard and fast, but it might be interesting.</I>I would suggest something like exclusion of rounds where one team was ahead by 2 rounds at the end of the debate (i.e. exclude a 6-2 4-4 round).Rob Millsnoreply@blogger.comtag:blogger.com,1999:blog-953859622641559363.post-18490354652054825902009-04-30T09:06:00.000-07:002009-04-30T09:06:00.000-07:00Does your formula control for any covariance, such...Does your formula control for any covariance, such as the overall win percentages of aff versus neg in a given division? For example, there's a possibility that novice debates are more likely to skew negative than open debates, etc. Seems like there's a lot of potential confounding variables that go into the decisions. Also, as anonymous commented, how do you correct for strength of teams? It seems that a judge might just see several presets that skew the sample - given the low power of a given judge's sample size (at most 70 or 80, much lower for many of those judges whom you've indicated are biased outside of a given confidence interval) even a random few highly imbalanced presets can wreck havoc on your findings...Gonzohttps://www.blogger.com/profile/15406999667395007539noreply@blogger.comtag:blogger.com,1999:blog-953859622641559363.post-21960477469795822752009-04-30T07:30:00.000-07:002009-04-30T07:30:00.000-07:00You need to develop a system to weight wins and lo...You need to develop a system to weight wins and losses. It could be that 'stronger' teams tend to be negative in front of certain judges due to the random number generator.Anonymousnoreply@blogger.com