Harmony in Diagnosis: Exploring Consensus and Variability in Clinical Judgement

This research explores the variability in decision-making among clinical psychologists by applying machine learning techniques to reverse-engineer their judgments. Using a dataset of 861 patients evaluated by 29 judges (both experienced psychologists and trainees), we trained decision tree and linear regression models for each judge to capture their decision patterns. We employed methods such as Jaccard similarity and pairwise Mean Squared Error (MSE) to quantify the distance between judges’ models, providing a clear metric of variability. Additionally, we applied SMOTE for data augmentation to enhance model training and improve robustness in comparisons. To gain deeper insights into the decision-making process, we used Large Language Models (LLMs) to explain and compare individual and pairwise models, highlighting key differences in clinical judgment. Our findings reveal significant variability in how judges weigh psychological traits, offering valuable insights into the potential for improving consistency and understanding judgment discrepancies in clinical practice. You can also find the related code in the following GitHub Repository.