Adaptation in Estimation and Annealing

We study general penalized log likelihood procedures and propose Information Theoretic conditions required of the penalty to obtain adaptive risk bounds. We demonstrate our conditions are natural and are satisfied in some canonical problems in statistics. We then investigate whether penalties are always required to obtain adaptation in the rates of estimation for M estimators. We show that the plain least squares estimators, without any penalization, in certain canonical shape-constrained regression problems indeed adapt to certain parametric complexities in the parameter space. We attempt to give a geometric characterization of this adaptation behaviour. We then move on to the important issue of computation. Motivated by a classical function estimation problem in non-parametric statistics, we study the possibility of a randomized algorithm, inspired by Simulated Annealing, being able to optimize multimodal functions in high dimensions. We explore the performance of this algorithm in low dimensions and explain the challenges faced in high dimensions. We provide myriads of possible routes for solving the statistically relevant optimization problem with the hope of encouraging further research in this direction.