Three Essays on Feature and Model Selection for Classification and Regression Problem

This thesis comprises of three essays on feature and model selection for classification and regression problems. The first essay focuses on selection of features for classification problems based on the notions of redundancy and complementarity. Redundancy and complementarity are the non-additive effects that result from feature interaction. While redundancy leads to a decrease in the predictive power of a subset of features, complementarity leads to information gain and improves prediction. This essay examines how combining complementarity with relevance and redundancy can lead to superior prediction for classification problems. A filter-based feature selection heuristic is proposed that combines these three criteria using an adaptive multi-objective optimization framework. The heuristic is adaptive in the sense that it updates the relative trade-off between these criteria adaptively based on the redundancy-complementarity ratio of the candidate subset. The proposed heuristic differs from many existing methods in that it distinguishes redundancy from complementarity explicitly, and does not penalize all dependencies. Using empirical study, we show that this approach can yield superior classification performance compared to many existing feature selection methods.;The second essay extends this notion of non-additivity to selection of interaction terms for linear regression problems. In a regression problem, an interaction effect is said to exist if the effect of one variable on the outcome depends on the value of the other variable, called the moderator variable. Existing literature on interaction mostly use sequential regression with regularization or penalty parameters to select relevant interaction effects. In this work, we examine the redundancy and complementarity that results from the correlation between the predictors. Although such synergy or redundancy does not statistically imply an interaction, we hypothesize that it is a potential indicator of the existence of an interaction effect. Based on this hypothesis, two methods of finding interaction terms are proposed. The proposed methods select an interaction term based on the principle of non-additivity. Using empirical study, we show that the proposed methods can select relevant interaction effects relatively quickly and produce comparable prediction accuracy with smaller number of features.;The last essay deals with a model selection problem in the context of securities class-action cases in United States. Insurance companies that provide Directors and Officers (D&O) insurance coverage to public limited companies are highly sensitive to class-action litigations due to high cost of settlement and legal fees. Predicting the probability of dismissal of a case early in the trial may give significant competitive advantage to the insurers in deciding the appropriate course of action- whether to fight or settle. This essay looks into this problem and proposes a hybrid probability model that combines the best of two well-established methods used for prediction. Using past data of securities class-action cases filed in US federal courts between 2002-2010, we show this probability model can predict the probability of dismissal of a case based on five important predictors. This model is useful for insurance companies that underwrite D&O policy.