f-divergence Analysis of Generative Adversarial Network

Hasan, Mahmud; Sang, Hailin

doi:10.2478/fcds-2025-0018

Abstract

We aim to establish estimation bounds for various divergences, including total variation, Kullback-Leibler (KL) divergence, Hellinger divergence, and Pearson χ² divergence, within the GAN estimator. We derive an inequality based on empirical and population objective functions of the GAN model, achieving almost surely convergence rates. Subsequently, this inequality was employed to derive estimation bounds for total variation, Kullback-Leibler (KL) divergence, Hellinger divergence, and Pearson χ² divergence, leading to almost surely convergence rates and differences between the expected outputs of the discriminator on real data and generated data. Our study demonstrates better results compared to some existing ones, which are a specific case of the general objective function.

References

Agresti, A. (2002), Categorical Data Analysis, Wiley Series in Probability and Statistics, New York.
Search in Google Scholar Back to article
Arjovsky, M., Chintala, S., Bottou, L. (2017), Wasserstein Generative Adversarial Networks, in Proceedings of the 34th International Conference on Machine Learning (ICML), 70, pp. 214-223.
Search in Google Scholar Back to article
Armanious K., Jiang C., Fischer M., Küstner T., Hepp T., Nikolaou K., Gatidis S., Yang B., (2020), MedGAN: Medical Image Translation using GANs for MRI Harmonization and Beyond, Neural Networks.
Search in Google Scholar Back to article
Arora, S., Ge, R., Liang, Y., Ma, T., Zhang, Y. (2017), Generalization and Equilibrium in Generative Adversarial Nets (GANs), in Proceedings of the 34th International Conference on Machine Learning (ICML), 70, pp. 224-232.
Search in Google Scholar Back to article
Beran, R. (1977), Minimum Hellinger Distance Estimates for Parametric Models, The Annals of Statistics, pp. 445-463.
Search in Google Scholar Back to article
Chung, J. K., Kannappan, P., Ng, C. T., Sahoo, P. K. (1989), Measures of the Distance Between Probability Distributions, Journal of Mathematical Analysis and Applications, pp. 280-292.
Search in Google Scholar Back to article
Csiszar, I. (1967), Information-Type Measures of Difference of Probability Distributions and Indirect Observations, Studia Scientiarum Mathematicarum Hungarica, 2, pp. 299-318.
Search in Google Scholar Back to article
Dziugaite, G. K., Roy, D. M., Ghahramani, Z. (2015), Training Generative Neural Networks via Maximum Mean Discrepancy Optimization, in Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence, pp. 258-267.
Search in Google Scholar Back to article
Goldberg, P. W., Jerrum, M. R. (1995), Bounding the Vapnik-Chervonenkis Dimension of Concept Classes Parameterized by Real Numbers, in Proceedings of the 6th Annual Conference on Computational Learning Theory, pp. 361-369.
Search in Google Scholar Back to article
Goodfellow, I., Abadie, J. P., Mirza, M., Xu, B., Farley, D. W., Ozair, S., Courville, A., Bengio, Y. (2014), Generative Adversarial Nets, Advances in Neural Information Processing Systems (NIPS), 27, pp. 2672-2680.
Search in Google Scholar Back to article
Hasan, M., Sang, H. (2023), Error Analysis of Generative Adversarial Network, arXiv preprint arXiv:2310.15387.
Search in Google Scholar Back to article
Huang, J., Jiao, Y., Li, Z., Liu, S., Wang, Y., Yang, Y. (2022), An Error Analysis of Generative Adversarial Networks for Learning Distributions, Journal of Machine Learning Research (JMLR), pp. 5047-5089.
Search in Google Scholar Back to article
Isola, P., Zhu, J. Y., Zhou, T., Efros, A. A. (2017), Image-to-Image Translation with Conditional Adversarial Networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Search in Google Scholar Back to article
Ji, K., Zhou, Y., Liang, Y. (2021), Understanding Estimation and Generalization Error of Generative Adversarial Networks, IEEE Transactions on Information Theory, 67, pp. 3114-3129.
Search in Google Scholar Back to article
Karras, T., Aila, T., Laine, S., Lehtinen, J. (2018), Progressive Growing of GANs for Improved Quality, Stability, and Variation, Proceedings of the International Conference on Learning Representations (ICLR).
Search in Google Scholar Back to article
Karras, T., Laine, S., Aila, T. (2019), A Style-Based Generator Architecture for Generative Adversarial Networks, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Search in Google Scholar Back to article
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T. (2020), Analyzing and Improving the Image Quality of StyleGAN, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Search in Google Scholar Back to article
Karras, T., Aittala, M., Laine, S., Herva, E., Aila, T. (2021), Alias-Free Generative Adversarial Networks (StyleGAN3), Advances in Neural Information Processing Systems (NeurIPS).
Search in Google Scholar Back to article
Li, Y., Swersky, K., Zemel, R. (2015), Generative Moment Matching Networks, in Proceedings of the 32nd International Conference on Machine Learning (ICML), 37, pp. 1718-1727.
Search in Google Scholar Back to article
Liang, T. (2017), How Well Can Generative Adversarial Networks (GAN) Learn Densities: A Nonparametric View, arXiv preprint arXiv:1712.08244.
Search in Google Scholar Back to article
Liang, T. (2021), How Well Generative Adversarial Networks Learn Distributions, Journal of Machine Learning Research (JMLR), 22, 1-41.
Search in Google Scholar Back to article
MacKay, D. J. (2003), Information Theory, Inference, and Learning Algorithms, Cambridge University Press.
Search in Google Scholar Back to article
Mirza, M., Osindero, S. (2014), Conditional Generative Adversarial Nets, arXiv preprint arXiv:1411.1784.
Search in Google Scholar Back to article
Nowozin, S., Cseke, B., Tomioka, R. (2016), f-GAN: Training Generative Neural Samplers Using Variational Divergence Minimization, Advances in Neural Information Processing Systems (NIPS), 29, 271-279.
Search in Google Scholar Back to article
Oberman, A. M., Calder, J. (2018), Lipschitz Regularized Deep Neural Networks Converge and Generalize, arXiv preprint arXiv:1808.09540.
Search in Google Scholar Back to article
Radford, A., Metz, L., Chintala, S. (2016), Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Proceedings of the International Conference on Learning Representations (ICLR).
Search in Google Scholar Back to article
Renyi, A. (1961), On Measures of Entropy and Information, Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability.
Search in Google Scholar Back to article
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H. (2016), Generative Adversarial Text-to-Image Synthesis, Proceedings of the 33rd International Conference on Machine Learning (ICML), 48, pp. 1060-1069.
Search in Google Scholar Back to article
Trefethen, L. (2019), Approximation Theory and Approximation Practice, SIAM.
Search in Google Scholar Back to article
Vershynin, R. (2018), High-Dimensional Probability: An Introduction with Applications in Data Science, Cambridge University Press.
Search in Google Scholar Back to article
Vidyasagar, M. (2003), Vapnik-Chervonenkis Pseudo and Fat-Shattering Dimensions, Communications and Control Engineering, Springer.
Search in Google Scholar Back to article
Xue, Y., Xu, T., Zhang, H., Long, L. (2018), SegAN: Adversarial Network with Multi-scale L1 Loss for Medical Image Segmentation, Neuroinformatics.
Search in Google Scholar Back to article
Yi, X., Walia, E., Babyn, P. S. (2019), Generative Adversarial Network in Medical Imaging: A Review, Medical Image Analysis, 58, 101552.
Search in Google Scholar Back to article
Zhang, P., Liu, Q., Zhou, D., Xu, T., He, X. (2018), On the Discrimination-Generalization Trade-Off in GANs, Proceedings of the 6th International Conference on Learning Representations (ICLR).
Search in Google Scholar Back to article
Zhu, J. Y., Park, T., Isola, P., Efros, A. A. (2017), Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2242-2251.
Search in Google Scholar Back to article

f-divergence Analysis of Generative Adversarial Network

Abstract

Paradigm

My account