Mid Sweden University

We present a method for evaluating the discriminative power of compact feature combinations (blocks) using the distance-based scoring measure, yielding an algorithm for selecting feature blocks that significantly contribute to the outcome variation. To estimate classification performance with subset selection in a high dimensional framework we jointly evaluate both stages of the process: selection of significantly relevant blocks and classification. Classification power and performance properties of the classifier with the proposed subset selection technique has been studied on several simulation models and confirms the benefit of this approach.

We focus on Bayesian network (BN) classifiers and formalize the feature selection from a perspective of improving classification accuracy. To exploring the effect of high-dimensionality we apply the growing dimension asymptotics. We modify the weighted BN by introducing inclusion-exclusion factors which eliminate the features whose separation score do not exceed a given threshold. We establish the asymptotic optimal threshold and demonstrate that the proposed selection technique carries improvements over classification accuracy.

Incorporating subset selection into a classification method often carries a number of advantages, especially when operating in the domain of high-dimensional features. In this paper, we focus on Bayesian network (BN) classifiers and formalize the feature selection from a perspective of improving classification accuracy. To exploring the effect of high-dimensionality we apply the

growing dimension asymptotics, meaning that the number of training examples is relatively small compared to the number of feature nodes. In order to ascertain which set of features is indeed relevant for a classification task, we introduce a distance-based scoring measure

reflecting how well the set separates different classes. This score is then employed to feature selection, using the weighted form of BN classifier. The idea is to view weights as inclusion-exclusion factors which eliminates the sets of features whose separation score do not exceed a given threshold. We establish the asymptotic optimal threshold and demonstrate that the proposed selection technique carries improvements over classification accuracy for different a priori assumptions concerning the separation strength.

A concept of feature informativeness was introduced as a way of measuring the discriminating power of a set of features. A question of interest is how this property of features affects the discrimination performance. The effect is assessed by means of a weighted discriminant function, which distributes weights among features according to their informativeness. The asymptotic normality of the weighted discriminant function is proven and the limiting expressions for the errors are obtained in the growing dimension asymptotic framework, i.e., when the number of features is proportional to the sample size. This makes it possible to establish the optimal in a sense of minimum error probability type of weighting.

Discrimination performance, measured by the limiting error probability, is considered from the point of view of feature discriminating power. For assessing the latter, a concept of feature informativeness is introduced. A threshold feature selection technique is considered. Selection is incorporated into the discriminant function by means of an inclusion-exclusion factor which eliminates the sets of features whose informativeness do not exceed a given threshold. An issue is how this selection procedure affects the error rate when sample based estimates are used in the discriminant function. This effect is evaluated in a growing dimension asymptotic framework. In particular, the increase of the moments of the discriminant function induced by the curse-of-dimensionality is shown together with the effect of the threshold-based feature selection. The asymptotic normality of the discriminant function, which makes it possible to express the overall error probability in a closed form and view it as a function of a given threshold of selection.

We present a growing dimension asymptotic formalism. The perspective in this paper is classification theory and we show that it can accommodate probabilistic networks classifiers, including naive Bayes model and its augmented version. When represented as a Bayesian network these classifiers have an important advantage: The corresponding discriminant function turns out to be a specialized case of a generalized additive model, which makes it possible to get closed form expressions for the asymptotic misclassification probabilities used here as a measure of classification accuracy. Moreover, in this paper we propose a new quantity for assessing the discriminative power of a set of features which is then used to elaborate the augmented naive Bayes classifier. The result is a weighted form of the augmented naive Bayes that distributes weights among the sets of features according to their discriminative power. We derive the asymptotic distribution of the sample based discriminative power and show that it is seriously overestimated in a high dimensional case. We then apply this result to find the optimal, in a sense of minimum misclassification probability, type of weighting.

Discrimination problems in a high-dimensional setting are considered. New results are concerned with the role of dimensionality in the performance of the discrimination procedure. Assuming that data consist of a block structurc two different asymptotic approaches are presented. These approaches are characterized by different types of relations between the dimensionality and the size of the training samples.par Asymptotic expressions for the error probabilities are obtained and a consistent approximation of the discriminant function is proposed. Throughout the paper the importance of the dimensionality in the asymptotic analysis is stressed.