Bayesian estimation and regularization techniques in categorical data analysis

Kalina, J.

doi:10.2478/jamsi-2025-0011

Abstract

This paper explores Bayesian estimation for categorical data, focusing on simple yet effective models that provide a foundation for applying more advanced methods accurately and reliably in real-world applications. We begin by revisiting Bayesian estimators for the binomial distribution and investigating their properties. Next, we develop hypothesis tests for categorical data (sign test, homogeneity test, symmetry test) based on regularized maximum likelihood estimates of the probabilities. Finally, we formulate regularized versions of common association measures for contingency tables and study the regularized version of mutual information, particular for the situation where the regularized version can effectively handle zero counts.

References

Agresti, A. 2002. Categorical Data Analysis, 2nd ed. Wiley, Hoboken.
Search in Google Scholar Back to article
Baek, J. and Park, J. 2023. Mixture of networks for clustering categorical data: A penalized composite likelihood approach. American Statistician 77, 259–273.
Search in Google Scholar Back to article
Bayes, T. 1763. An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London 53, 370–418.
Search in Google Scholar Back to article
Braun, F., Caelen, O., Smirnov, E., Kelk, S., and Lebichot, B. 2017. Improving card fraud detection through suspicious pattern discovery. In Lecture Notes in Computer Science. Vol. 10351. Springer, Cham, 181–190.
Search in Google Scholar Back to article
de Toledo, P., Núñez, F., and Usabiaga, C. 2020. Matching in segmented labor markets: An analytical proposal based on high-dimensional contingency tables. Economic Modelling 93, 175–186.
Search in Google Scholar Back to article
Deng, J. and Deng, Y. 2022. Maximum entropy of random permutation set. Soft Computing 26, 11265–11275.
Search in Google Scholar Back to article
Gelman, A., Carlin, J., Stern, H., Dunson, D., Vehtari, A., and Rubin, D. 2013. Bayesian Data Analysis, 3rd ed. Chapman and Hall/CRC, Boca Raton.
Search in Google Scholar Back to article
Golden, R. 2020. Statistical Machine Learning: A Unified Framework. Chapman & Hall/CRC Press, Boca Raton.
Search in Google Scholar Back to article
Gupta, R., Gupta, S., Singh, J., and Kais, S. 2025. Entropy-assisted quality pattern identification in finance. Entropy 27, 4, 430.
Search in Google Scholar Back to article
Hausser, J. and Strimmer, K. 2013. Entropy inference and the james-stein estimator with application to nonlinear gene association networks. Journal of Machine Learning Research 10, 1469–1484.
Search in Google Scholar Back to article
Issouani, E., Bertail, P., and Gautherat, E. 2024. Exponential bounds for regularized hotelling’s t2 statistic in high dimension. Journal of Multivariate Analysis 203, 105342.
Search in Google Scholar Back to article
Jaynes, E. 2003. Probability Theory: The Logic of Science. Cambridge University Press, Cambridge.
Search in Google Scholar Back to article
Johnson, A., Ott, M., and Dogucu, M. 2022. Bayes Rules! An Introduction to Applied Bayesian Modeling. CRC Press, Boca Raton.
Search in Google Scholar Back to article
Kalina, J. and Matonoha, C. 2020. A sparse pair-preserving centroid-based supervised learning method for high-dimensional biomedical data or images. Biocybernetics and Biomedical Engineering 40, 2, 774–786.
Search in Google Scholar Back to article
Kalina, J. and Rensová, D. 2015. How to reduce dimensionality of data: Robustness point of view. Serbian Journal of Management 10, 131–140.
Search in Google Scholar Back to article
Kalina, J. and Schlenker, A. 2015. A robust supervised variable selection for noisy high-dimensional data. BioMed Research International 2015, 320385.
Search in Google Scholar Back to article
Kalina, J. and Tichavský, J. 2022. The minimum weighted covariance determinant estimator for high-dimensional data. Advances in Data Analysis and Classification 16, 977–999.
Search in Google Scholar Back to article
Ledoit, O. and Wolf, M. 2022. The power of (non-)linear shrinking: A review and guide to covariance matrix estimation. Journal of Financial Econometrics 20, 187–218.
Search in Google Scholar Back to article
Lindskou, M., Eriksen, P., and Tvedebrink, T. 2020. Outlier detection in contingency tables using decomposable graphical models. Scandinavian Journal of Statistics 47, 347–360.
Search in Google Scholar Back to article
Loftus, S. 2024. An Introductory Handbook of Bayesian Thinking. Elsevier, London. Ming, H. and Yang, H. 2024. l₀ regularized logistic regression for large-scale data. Pattern Recognition 146, 110024.
Search in Google Scholar Back to article
Pose, F., Bautista, L., Gianmuso, F., and Redelico, F. 2021. On the permutation entropy bayesian estimation. Communications in Nonlinear Science and Numerical Simulation 99, 105779.
Search in Google Scholar Back to article
Rao, C. 2002. Linear Statistical Inference and Its Applications. Wiley, New York.
Search in Google Scholar Back to article
Smith, M. and Ruxton, G. 2020. Effective use of the mcnemar test. Behavioral Ecology and Sociobiology 74, 133.
Search in Google Scholar Back to article
Sohaee, N. 2023. Error and optimism bias regularization. Journal of Big Data 10, 8.
Search in Google Scholar Back to article
Subramanian, I., Verma, S., Kumar, S., Jere, A., and Anamika, K. 2020. Multi-omics data integration, interpretation, and its application. Bioinformatics and Biology Insights 2020, 14.
Search in Google Scholar Back to article
Tuyl, F., Gerlach, R., and Mengersen, K. 2008. A comparison of bayes-laplace, jeffreys, and other priors: The case of zero events. American Statistician 62, 40–44.
Search in Google Scholar Back to article
Wang, A., Henao, R., and Carin, L. 2024. Transformer in-context learning for categorical data. arXiv:2405.17248.
Search in Google Scholar Back to article
Wang, K., Li, J., and Tsung, F. 2023. Efficient and interpretable monitoring of high-dimensional categorical processes. IISE Transactions 55, 886–900.
Search in Google Scholar Back to article
Wang, R. and Li, J. 2023. Block-regularized 5×2 cross-validated mcnemar’s test for comparing two classification algorithms. Submitted.
Search in Google Scholar Back to article
Zhang, Y., Zaidi, N., Zhou, J., Wang, T., and Li, G. 2024. Effective interpretable learning for large-scale categorical data. Data Mining and Knowledge Discovery 38, 2223–2251.
Search in Google Scholar Back to article
Zhou, X., Heng, Q., Chi, E. C., and Zhou, H. 2024. Proximal mcmc for bayesian inference of constrained and regularized estimation. The American Statistician 78, 4, 379–390.
Search in Google Scholar Back to article

Bayesian estimation and regularization techniques in categorical data analysis

Abstract

Paradigm

My account