You shouldn’t be purely concerned with the brier score/calibration/accuracy of your beliefs or predictions if the upside and downside risks are very different, this is Scott Alexander’s (or Yudkowsky’s?) point in that fable about the volcano predictors.
Apparently the pro publica scandal led to 3 papers with impossibility theorems about different conceptions of fairness. So different conceptions of fairness can’t all be satisfied. kleinman was one such author. This is a great example of the increased need to make it explicit with ethics going forward. I think the issue was, given different rates of offence across groups, you can’t have equal false positive/false negative ratios and equal calibration.
C. Thi. Nguyen that fairness and signalling relative improvement and regression to students across time is impossible to perfectly achieve both if you make mistakes in marking (is this the same as the fairness impossibility systems in ML?)