Fairness

Once a valid application has been submitted, a minimum of five fellow applicants and five Wise Head Panel members will be assigned to score each valid submission. Wise Heads will offer both scores and comments for each of five distinct traits. Each of the five traits will be scored on a 0-5 point scale, in increments of 0.1. Those scores will combine to produce a total score. Examples of possible scores for a trait are: 1.4, 3.7, etc.

The most straightforward way to ensure that everyone is treated by the same set of standards would be to have the same Wise Heads score every application; unfortunately, due to the number of applications, that is not possible.
‍
Since the same Wise Head will not score every application, the question of fairness needs to be carefully explained. One Wise Head scoring an application may take a more critical view, giving every assigned submission a range of scores only between 1.0 and 2.0, as an example; meanwhile, another Wise Head may be more generous and score every submission between 4.0 and 5.0.
‍
For illustrative purposes, let’s look at the scores from two hypothetical Wise Heads:

The first Wise Head is far more generous in scoring than the second Wise Head, who gives much lower scores. If your application was rated by the first Wise Head, it would earn a much higher total score than if it was assigned to the second Wise Head.
‍
We have a way to address this issue. We ensure that no matter which Wise Heads are assigned to you, each application will be treated fairly. To do this, we utilize a mathematical technique relying on two measures of distribution, the mean and the standard deviation. The mean takes all the scores assigned by a Wise Head, adds them up, and divides them by the number of scores assigned, giving an average score.
‍
Formally, we denote the mean like this:

The standard deviation measures the “spread” of a Wise Head's scores. As an example, imagine that two Wise Heads both give the same mean (average) score, but one gives many zeros and fives, while the other gives more ones and fours. It wouldn't be fair if we didn’t consider this difference.
‍
Formally, we denote the standard deviation like this:

To ensure that the evaluation process is fair, we rescale all the scores to match the Wise Head population. In order to do this, we measure the mean and the standard deviation of all scores across all Wise Heads. Then, we change the mean score and the standard deviation of each Wise Head to match.
‍
We rescale the standard deviation like this:

Then, we rescale mean like this:

Basically, we are finding the difference between both distributions for a single Wise Head and those for all of the Wise Heads combined, then adjusting each score so that no one is treated unfairly according to which Wise Heads they are assigned.
‍
If we apply this rescaling process to the same two Wise Heads in the example above, we can see the outcome of the final resolved scores. They appear more similar, because they are now aligned with typical distributions across the total judging population.

We are pleased to answer any questions you have about the scoring process by email.