Sunday, November 8, 2009

RKS scale not dramatically increasing predictive accuracy

If moving to the Ross K. Smith 100-point scale increases the accuracy of speaker points, and if speaker points accurately reflect a team's abilities, we might expect that the scale change would help speaker points predict future winners. To my surprise, there is not a dramatic increase in the predictive accuracy of speaker points in the three large tournaments that moved to the new scale while maintaining the same number of teams and preliminary rounds.

I used the following model: Predictions are made for rounds five and later, using at least the first four preliminary rounds. The model makes no prediction if the two teams have different win-loss records. If they have the same win-loss record and the same total points, the model also does not make a prediction. Otherwise, it predicts that whichever team had the higher points for the predicting rounds will win the debate.

If two teams have the same record for rounds one through four, but they don't meet until round eight, the model still predicts the winner based on their points in rounds one through four, since the teams might well have debated in round five. It also makes predictions based on their points in rounds one through five, one through six, and one through seven, assuming their win-loss records are the same after rounds five, six, and seven. All predictions are based on rounds one through four, five, six, or seven; no predictions ever discard any early prelims.

Wake Forest, 8 prelims

Year Teams Tied points, no prediction Correct prediction Incorrect prediction Accuracy
2006-2007 138 11 89 50 63.0%
2007-2008 134 1 101 79 56.1%

UNLV open division, 7 prelims

Year Teams Tied points, no prediction Correct prediction Incorrect prediction Accuracy
2008-2009 56 0 32 23 58.2%
2009-2010 54 0 20 11 64.5%

Harvard, 8 prelims

Year Teams Tied points, no prediction Correct prediction Incorrect prediction Accuracy
2008-2009 80 9 57 27 66.1%
2009-2010 87 4 68 33 66.7%

Wednesday, November 4, 2009

RKS translation at Harvard

The Harvard tournament this year had a very explicit and very simple translation from the 30-point scale to the 100-point (or Ross K. Smith) scale. Using the results packet, we can see how the judging pool interpreted the RKS scale:

If we apply the Harvard translation to historical tournaments, the switch to RKS this year was accompanied by more point inflation than usual. The older data (drawn from debateresults.com) shows that a median speaker used to earn between 27.8 and 28.0 points (78 to 80 points, under RKS). This year, he or she earned 28.25 points (82.5, under RKS). A 28.0 (80, under RKS) used to mean 55th to 65th percentile. This year, it meant about 35th percentile.

It will be interesting to see how many judges circled the indicator on the front of their ballots indicating that they were conforming to the suggested scale translation.

It should also be noted that Harvard's points this year were substantially lower than Kentucky's points this year: