Abstract

Modern deep neural networks achieve strong performance on large-scale datasets, but often require substantial training time. Online batch selection methods seek to reduce this cost by updating models on informative subsets of each batch rather than on all available examples. Recently introduced methods leverage teacher models and report substantial speedups, particularly in noisy-label settings. However, comparisons are often based on the number of epochs required to reach a target test accuracy, a coarse metric that is sensitive to implementation details and may obscure important differences in learning dynamics. In this thesis, we implement several online batch selection methods in a unified codebase and compare their behavior using progress, a recently introduced metric that measures how predictive distributions move through the probability simplex. We find that methods prioritizing difficult examples, especially those using teacher models, induce a two-phase learning process that we call delayed prediction dynamics. In the initial latent phase, predictive probabilities remain near the high-entropy center of the simplex, even as accuracy may increase. This is followed by an activation phase, in which same-class predictive probabilities move synchronously toward the correct simplex corners. Linear probing and neural tangent kernel diagnostics provide no conclusive evidence that the latent phase reflects enhanced feature learning, suggesting instead that these methods may alter prediction-space dynamics without necessarily inducing richer representations.

Degree

College and Department

Computational, Mathematical, and Physical Sciences; Mathematics

Rights

https://lib.byu.edu/about/copyright/