We calculated inter-evaluator agreement while the a straightforward proportion amongst the instances when all of the the latest evaluators provided a similar benefit (every proper or every wrong) separated because of the total number out of instances evaluated by the same evaluators.

Error analysis

We analyzed 100 program problems chose randomly of predications considered due to the fact untrue experts from the about three judges.