Claudia Beleites
Chemometric Consulting
Södeler Weg 19
61200 Wölfersheim/Germany
e-mail: claudia dot beleites at chemometrix dot eu

Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to Grading of Astrocytoma Tissues

Paper deriving and explaining the theory behind softclassval

C. Beleites, R. Salzer and V. Sergo,
Chemom. Intell. Lab. Syst., 122 (2013), 12 - 22.

Keywords: soft classification, partial class membership, classifier validation, borderline cases, ambiguous reference, biomedical spectroscopy

Download links:

DOI: 10.1016/j.chemolab.2012.12.003
official version
arXiv: 1301.0264
full text of accepted manuscript including supplementary material


We use partial class memberships in soft classification to model uncertain labelling and mixtures of classes. Partial class memberships are not restricted to predictions, but may also occur in reference labels (ground truth, gold standard diagnosis) for training and validation data.

Classifier performance is usually expressed as fractions of the confusion matrix, like sensitivity, specificity, negative and positive predictive values. We extend this concept to soft classification and discuss the bias and variance properties of the extended performance measures. Ambiguity in reference labels translates to differences between best-case, expected and worst-case performance.

We show a second set of measures comparing expected and ideal performance which is closely related to regression performance, namely the root mean squared error RMSE and the mean absolute error MAE. All calculations apply to classical crisp as well as to soft classification (partial class memberships as well as one-class classifiers).

The proposed performance measures allow to test classifiers with actual borderline cases. In addition, hardening of e.g. posterior probabilities into class labels is not necessary, avoiding the corresponding information loss and increase in variance.

We implemented the proposed performance measures in R package “softclassval” which is available from CRAN and at

Our reasoning as well as the importance of partial memberships for chemometric classification is illustrated by a real-word application: astrocytoma brain tumor tissue grading (80 patients, 37 000 spectra) for finding surgical excision borders. As borderline cases are the actual target of the analytical technique, samples which are diagnosed to be borderline cases must be included in the validation.

literature reference article
Published: 2013-01-03