Analysis of correlation based dimension reduction methods

Shin, Yong; Park, Cheong

doi:10.2478/v10006-011-0043-9

Abstract

Dimension reduction is an important topic in data mining and machine learning. Especially dimension reduction combined with feature fusion is an effective preprocessing step when the data are described by multiple feature sets. Canonical Correlation Analysis (CCA) and Discriminative Canonical Correlation Analysis (DCCA) are feature fusion methods based on correlation. However, they are different in that DCCA is a supervised method utilizing class label information, while CCA is an unsupervised method. It has been shown that the classification performance of DCCA is superior to that of CCA due to the discriminative power using class label information. On the other hand, Linear Discriminant Analysis (LDA) is a supervised dimension reduction method and it is known as a special case of CCA. In this paper, we analyze the relationship between DCCA and LDA, showing that the projective directions by DCCA are equal to the ones obtained from LDA with respect to an orthogonal transformation. Using the relation with LDA, we propose a new method that can enhance the performance of DCCA. The experimental results show that the proposed method exhibits better classification performance than the original DCCA.

References

Anton, H. and Busby, R. (2003). Contemporary Linear Algebra, John Wiley and Sons, Denver, CO.
Search in Google Scholar
Baudat, G. and Anouar, F. (2000). Generalized discriminant analysis using a kernel approach, Neural Computation 12(10): 2385-2404.10.1162/089976600300014980
Search in Google Scholar
Billings, S. and Lee, K. (2002). Nonlinear fisher discriminant analysis using a minimum squared error cost function and the orthogonal least squares algorithm, Neural Networks 15(2): 263-270.10.1016/S0893-6080(01)00142-3
Search in Google Scholar
Chen, L., Liao, H., M. Ko, Lin, J. and Yu, G. (2000). A new LDA-based face recognition system which can solve the small sample size problem, Pattern Recognition 33(10): 1713-1726.10.1016/S0031-3203(99)00139-9
Search in Google Scholar
Duda, R., Hart, P. and Stork, D. (2001). Pattern Classification, Wiley Interscience, New York, NY.
Search in Google Scholar
Fukunaga, K. (1990). Introduction to Statistical Pattern Recognition, 2nd Edn., Academic Press, San Diego, CA.
Search in Google Scholar
Fukunaga, K. and Mantock, J. (1983). Nonparametric discriminant analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 5(6): 671-678.10.1109/TPAMI.1983.476746121869158
Search in Google Scholar
Garthwaite, P. (1994). An interpretation of partial least squares, Journal of the American Statistical Society 89(425): 122-127.10.1080/01621459.1994.10476452
Search in Google Scholar
He, X. and Niyogi, P. (2003). Locality preserving projections, Proceedings of the Advances in Neural Information Processing Systems Conference, Vancouver, Canada, pp. 153-160.
Search in Google Scholar
Hotelling, H. (1936). Relations between two sets of variates, Biometrika 28(3): 321-377.10.1093/biomet/28.3-4.321
Search in Google Scholar
Hou, C., Nie, F., Zhang, C. and Wu, Y. (2009). Learning an orthogonal and smooth subspace for image classification, IEEE Signal Processing Letters 16(4): 303-306.10.1109/LSP.2009.2014283
Search in Google Scholar
Howland, P. and Park, H. (2004). Generalizing discriminant analysis using the generalized singular value decomposition, IEEE Transactions on Pattern Analysis and Machine Intelligence 26(8): 995-1006.10.1109/TPAMI.2004.4615641730
Search in Google Scholar
Jolliffe, I. (1986). Principal Component Analysis, Springer, New York, NY.10.1007/978-1-4757-1904-8
Search in Google Scholar
Nie, F., Xiang, S. and Zhang, C. (2007). Neighborhood minmax projections, Proceedings of the International Joint Conference on Artificial Intelligence, Hyderabad, India, pp. 993-998.
Search in Google Scholar
Pardalos, P. and Hansen, P. (2008). Data Mining and Mathematical Programming, CRM Proceedings & Lecture Notes, Vol. 45, American Mathematical Society, Montreal.
Search in Google Scholar
Park, C. and Park, H. (2008). A comparison of generalized linear discriminant analysis algorithms, Pattern Recognition 41(3): 1083-1097.10.1016/j.patcog.2007.07.022
Search in Google Scholar
Roweis, S.T. and Saul, L.K. (2000). Nonlinear dimensionality reduction by locally linear embedding, Science 290(5500): 2323-2326.10.1126/science.290.5500.2323
Search in Google Scholar
Sugiyama, M. (2006). Local fisher discriminant analysis for supervised dimensionality reduction, Proceedings of the IEEE International Conference on Machine Learning, Pittsburgh, PA, USA, pp. 905-912.
Search in Google Scholar
Sun, Q., Zeng, S., Liu, Y., Heng, P. and Xia, D. (2005). A new method of feature fusion and its application in image recognition, Pattern Recognition 38(12): 2437-2448.10.1016/j.patcog.2004.12.013
Search in Google Scholar
Sun, T. and Chen, S. (2007). Class label versus sample label-based CCA, Applied Mathematics and Computation 185(1): 272-283.10.1016/j.amc.2006.06.103
Search in Google Scholar
Sun, T., Chen, S., Yang, J. and Shi, P. (2008). A supervised combined feature extraction method for recognition, Procedings of the IEEE International Conference on Data Mining, Pisa, Italy, pp. 1043-1048.
Search in Google Scholar
Tenenbaum, J.B., de Silva, V. and Langford, J.C. (2000). A global geometric framework for nonlinear dimensionality reduction, Science 290(5500): 2319-2323.10.1126/science.290.5500.2319
Search in Google Scholar
Wegelin, J. (2000). A survey of partial least squares (PLS) methods, with emphasis on the two block case, Technical report, Department of Statistics, University of Washington, Seattle, WA.
Search in Google Scholar
Yan, S., Xu, D., Zhang, B., Zhang, H.-J., Yang, Q. and Lin, S. (2007). Graph embedding and extensions: A general framework for dimensionlity reduction, IEEE Transactions on Pattern Analysis and Machine Intelligence 29(1): 40-51.10.1109/TPAMI.2007.250598
Search in Google Scholar
Yang, J. and Yang, J.-Y. (2003). Why can LDA be performed in PCA transformed space?, Pattern Recognition 36(2): 563-566.10.1016/S0031-3203(02)00048-1
Search in Google Scholar
Yang, J., Yang, J., Zhang, D. and Lu, J. (2003). Feature fusion: Parallel strategy vs. serial strategy, Pattern Recognition 36(6): 1369-1381.
Search in Google Scholar
Ye, J. (2005). Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems, Journal of Machine Learning Research 6(4): 483-502.
Search in Google Scholar
Yu, H. and Yang, J. (2001). A direct LDA algorithm for highdimensional data-with application to face recognition, Pattern Recognition 34(10): 2067-2070.10.1016/S0031-3203(00)00162-X
Search in Google Scholar

Analysis of correlation based dimension reduction methods

Abstract

Paradigm

My account