Keywords
machine learning, learning algorithm, permutation test, p-value
Abstract
The paired-difference t-test is commonly used in the machine learning community to determine whether one learning algorithm is better than another on a given learning task. This paper suggests the use of the permutation test instead hecause it calculates the exact p-value instead of an estimate. The permutation test is also distribution free and the time complexity is trivial for the commonly used 10-fold cross-validation paired-difference test. Results of experiments on real-world problems suggest it is not uncommon to see the t-test estimate deviate up to 30-50% from the exact p-value.
Original Publication Citation
Menke, J., and Martinez, T. R., "Using Permutations Instead of Student's t Distribution for pvalues in Paired-Difference Algorithm Comparisons", Proceedings of the IEEE International Joint Conference on Neural Networks IJCNN'4, pp. 1331-1336, 24.
BYU ScholarsArchive Citation
Martinez, Tony R. and Menke, Joshua, "Using Permutations Instead of Student’s t Distribution for p-values in Paired-Difference Algorithm Comparisons" (2004). Faculty Publications. 1032.
https://scholarsarchive.byu.edu/facpub/1032
Document Type
Peer-Reviewed Article
Publication Date
2004-07-29
Permanent URL
http://hdl.lib.byu.edu/1877/2449
Publisher
IEEE
Language
English
College
Physical and Mathematical Sciences
Department
Computer Science
Copyright Status
© 2004 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Copyright Use Information
http://lib.byu.edu/about/copyright/