We present a method for evaluating the discriminative power of compact feature combinations (blocks) using the distance-based scoring measure, yielding an algorithm for selecting feature blocks that significantly contribute to the outcome variation. To estimate classification performance with subset selection in a high dimensional framework we jointly evaluate both stages of the process: selection of significantly relevant blocks and classification. Classification power and performance properties of the classifier with the proposed subset selection technique has been studied on several simulation models and confirms the benefit of this approach.