At CREST, Zong Shang introduced an innovative method for analyzing benign overfitting in statistics.
The Essentials: At CREST, Zong Shang introduced an innovative method for analyzing benign overfitting in statistics.
Zong Shang, researcher at CREST (Ecole Nationale de la Statistique et de l'Analyse de l'Information [Bruz]).
Thesis defended in 2026 at the doctoral school Hadamard Doctoral School of Mathematics.
Referenced in the ABES/STAR network, this thesis meets the rigor standards of French higher education.
Context and Research Question
The phenomenon of overfitting is a major issue in statistical learning theory and mathematical statistics. The uniform convergence argument, traditionally used to analyze statistical estimators, has shown its limits when confronted with benign overfitting. This phenomenon, which occurs when estimators perfectly interpolate the training data while remaining performant on new data, is not well explained by classical methods. This finding underscores the need for a more suitable mathematical framework for studying estimator behavior in overfitting contexts.
Zong Shang's thesis addresses this problem by proposing a feature space decomposition method. This approach aims to refine the uniform convergence argument and offer a new perspective on overfitting, particularly within the framework of linear regression and linear classification. Consequently, this research focuses on analyzing the minimal-lq-norm interpolating estimator and the minimal-l2-norm interpolating estimator, in order to better understand the mechanisms underlying overfitting.
Methodology
The feature space decomposition method developed by Shang relies on a systematic analysis of estimators in linear regression and classification. This method makes it possible to explore the interactions between data features and estimator performance, particularly with regard to benign overfitting. Applications of this methodology include ridge regression and other spectral methods, such as gradient descent and gradient flow.
The approach consists of characterizing the population excess risk of estimators using non-asymptotic descriptions of their estimation error. By integrating geometric aspects of functional analysis, the feature space decomposition method makes it possible to obtain precise results on estimator performance across various regression scenarios. This systematic approach helps establish a solid theoretical framework for the analysis of statistical methods, focusing on the geometric and analytical aspects of regression problems.
Key Findings
The results of this research highlight several significant contributions. First, the feature space decomposition method makes it possible to analyze the phenomenon of benign overfitting in interpolating estimators. This result is crucial for theoretical statisticians, as it offers a new perspective on population excess risk.
Second, the thesis establishes a partial order on the set of spectral methods for a given linear regression problem. This advance makes it possible to organize the various statistical approaches according to their effectiveness and their ability to handle overfitting.
Third, Shang succeeds in generalizing the Dvoretzky-Milman theorem for the lq norm under a general probability measure. This development, which follows naturally from the feature space decomposition method, enriches the existing theoretical framework by connecting the geometric aspects of functional analysis to statistical methods.
Finally, the thesis introduces three novel concepts: a partial order on spectral methods, a generalized saturation effect, and a mathematical definition of the feature-learning property. These concepts are essential for characterizing the excess risk associated with estimators and provide analytical tools for statistics researchers.
Discussion and Outlook
The implications of Zong Shang's research extend beyond theoretical considerations. The feature space decomposition method provides practical tools for statisticians and researchers in statistical learning, facilitating a more nuanced understanding of overfitting and estimation methods. By integrating geometric elements into statistical analysis, this thesis opens new avenues of research for exploring estimator behavior in varied contexts.
Future research could turn toward applying the feature space decomposition method to other data analysis domains, notably complex machine learning models and neural networks. Furthermore, exploring the practical implications of the generalization of the Dvoretzky-Milman theorem could enrich the understanding of statistical methods in real-world contexts.
In sum, Zong Shang's thesis represents a significant advance in the field of statistics, proposing an innovative theoretical framework for addressing overfitting and enriching the tools available to practitioners. The contributions of this research strengthen our understanding of the mechanisms underlying overfitting and lay the groundwork for future work in the field.
Accéder à l'étude complète
Sources and Access
Zong Shang. Feature space decomposition. Statistics [math.ST]. Institut Polytechnique de Paris, 2026. English. ⟨NNT : 2026IPPAG004⟩. ⟨tel-05604065⟩