Reducing Generalization Error in Multiclass Classification through Factorized Cross Entropy Loss

Author

Oleksandr Horban, Claremont McKenna CollegeFollow

Researcher ORCID Identifier

https://orcid.org/0009-0003-7604-2020

Graduation Year

2024

Date of Submission

4-2024

Document Type

Open Access Senior Thesis

Award

Best Senior Thesis in Mathematics

Degree Name

Bachelor of Arts

Department

Mathematical Sciences

Reader 1

Mike Izbicki

Terms of Use & License Information

Terms of Use for work posted in Scholarship@Claremont.

Rights Information

© 2024 Oleksandr Horban

Abstract

This paper introduces Factorized Cross Entropy Loss, a novel approach to multiclass classification which modifies the standard cross entropy loss by decomposing its weight matrix W into two smaller matrices, U and V, where UV is a low rank approximation of W. Factorized Cross Entropy Loss reduces generalization error from the conventional O( sqrt(k / n) ) to O( sqrt(r / n) ), where k is the number of classes, n is the sample size, and r is the reduced inner dimension of U and V.

Recommended Citation

Horban, Oleksandr, "Reducing Generalization Error in Multiclass Classification through Factorized Cross Entropy Loss" (2024). CMC Senior Theses. 3630.
https://scholarship.claremont.edu/cmc_theses/3630