Keywords

machine learning algorithms, cross-validation, confidence, fitness

Abstract

Neural network and machine learning algorithms often have parameters that must be tuned for good performance on a particular task. Leave-one-out cross-validation (LCV) accuracy is often used to measure the fitness of a set of parameter values. However, small changes in parameters often have no effect on LCV accuracy. Many learning algorithms can measure the confidence of a classification decision, but often confidence alone is an inappropriate measure of fitness. This paper proposes a combined measure of Cross- Validation and Confidence (CVC) for obtaining a continuous measure of fitness for sets of parameters in learning algorithms. This paper also proposes the Refined Instance-Based (RIB) learning algorithm which illustrates the use of CVC in automated parameter tuning. Using CVC provides significant improvement in generalization accuracy on a collection of 31 classification tasks when compared to using LCV.

Original Publication Citation

Wilson, D. R. and Martinez, T. R, "Combining Cross-Validation and Confidence to Measure Fitness", Proceedings of the IEEE International Joint Conference on Neural Networks IJCNN'99, CD paper #163, 1999.