Image Super-Resolution Reconstruction Method Based on Improved Generative Adversarial Network

This study adopts Real-ESRGAN as the baseline architecture, with technical improvements primarily focused on network structure optimization. Compared to traditional generative adversarial architectures, the model introduces Residual-in-Residual Dense Blocks (RRDB) as a structural innovation. These blocks integrate multi-level residual connections and dense feature reuse mechanisms, significantly enhancing the network's feature extraction capability and gradient propagation efficiency. Specifically, the RRDB employs a cascaded residual learning framework, which effectively extracts multi-scale image features while optimizing information flow efficiency. Moreover, the network eliminates Batch Normalization (BN) layers, further improving its overall performance.

The residual block demonstrates the effect of removing the Batch Normalization (BN) layer. As a technique proposed in 2015 to accelerate the convergence of neural networks, BN can effectively address the vanishing gradient problem, speed up training, and prevent overfitting. However, in generative adversarial networks for super-resolution reconstruction, the BN layer performs mean and variance normalization on all data. With the increase in network depth, it may introduce artifacts into the generated images, especially when generating high-resolution images. To avoid the instability caused by the BN layer, Real-ESRGAN chooses to remove it, thereby improving the quality of image reconstruction, reducing computational complexity, and effectively enhancing computational efficiency when measured by Peak Signal-to-Noise Ratio (PSNR).

As shown in Figure 1, Real-ESRGAN employs the Residual-in-Residual Dense Block (RRDB) as the core building unit (the detailed structure of the module is shown in Figure 2), replacing the basic Residual Block (Resblock) in traditional generative networks. With the increase in the number of network layers, the effective receptive field of the model is gradually expanded through the cumulative effect of successive convolutional operations. This characteristic enables it to integrate a broader range of contextual correlation information, thereby enhancing the geometric consistency of super-resolution reconstruction.

Figure 1.

Replace the original Resblock with the RRDB structure

Figure 2.

RRDB structure

In the adversarial training framework, this study adopts an improved relativistic discriminator architecture, whose objective function is defined as follows:

The mathematical expression of the discriminator loss is shown in equation (1): 1LDRa=−IExz[ log(DRa(xr,xf)) ]−IExf[ log(1−DRa(xf,xr)) ]

The mathematical expression of the generator's adversarial loss is shown in equation (2): 2LDRa=−IExr[ log(DRa(xr,xf)) ]−IExf[ log(1−DRa(xf,xr)) ]

Here, x_f = G (x_i) represents the generated high-resolution image, x_r represents the real high-resolution image, and xi is the low-resolution input image. This optimization approach has significant advantages in terms of edge details and image quality of the generated images.

In addition, regarding the perceptual loss calculation strategy, Real-ESRGAN innovatively sets the feature extraction point before the activation layer of the VGG network, with the comparison effect shown in Figure 3. Compared to SRGAN, which calculates the loss after the activation layer, this improvement effectively avoids the feature sparsification problem caused by the ReLU function.

Figure 3.

Representative feature maps before and after activation

Despite the excellent performance of Real-ESRGAN based on Convolutional Neural Networks (CNN) in local feature extraction and detail recovery, its Residual-in-Residual Dense Block (RRDB) module is limited by the local receptive field characteristic of convolutions. Although stacking convolutional layers can gradually expand the receptive field, CNNs still struggle to effectively model global context information in complex scenes, such as high-frequency texture-dense urban architecture or long-range dependent remote sensing images.

As shown in Figure 4, the Vision Transformer (ViT) overcomes the locality limitations of CNNs through its self-attention mechanism, enabling the direct modeling of long-range dependencies between arbitrary pixels. Specifically, ViT divides the input image into non-overlapping patches and utilizes Multi-Head Self-Attention (MSA) to dynamically compute the correlation weights between different regions, thereby capturing global context. Moreover, the positional encoding design of ViT preserves the spatial information of the image, avoiding the issue of spatial information loss caused by unordered inputs in traditional Transformers.

Figure 4.

The architecture diagram of ViT

As shown in Figure 5, this paper intuitively compares the essential differences in the feature modeling mechanisms of ViT and CNN. The red box indicates the region of interest for the current operation, and the darker the heatmap color, the higher the attention weight. CNNs employ convolutional operations with local receptive fields, gradually expanding the receptive field through hierarchical stacking. In contrast, ViT directly establishes global dependencies through its self-attention mechanism. Experiments have shown that ViT can dynamically capture the correlations between distant pixels (e.g., the high-frequency structure formed by the four corner points of a building's window frame). Compared to the local processing approach of CNNs, ViT demonstrates a significant advantage in detail recovery.

Figure 5.

Comparison of CNN's local receptive field (left) and ViT's global attention weight distribution (right).

The generator network of Real-ESRGAN consists of 23 Residual-in-Residual Dense Blocks (RRDBs), which are primarily used to gradually extract low-resolution image features and enhance image details. The RRDB module captures local features and a certain range of global information through multi-level residual and dense connections. However, when dealing with high-frequency information, such as small targets or long-range dependencies, it may fail to adequately model complex global features, resulting in insufficient enhancement of local features.

To address this issue, this paper proposes to enhance the global feature modeling capability by introducing the Vision Transformer (ViT) module. ViT has strong global modeling capabilities, especially in capturing long-range pixel dependencies and overall image structure. By incorporating the ViT module, the shortcomings of the RRDB module in modeling global features can be effectively compensated for, particularly in improving the global consistency of generated images and the detail restoration in complex scenes.

To ensure a balance between local and global feature modeling while controlling computational overhead, this paper replaces the convolutional and RRDB modules in layers 9 to 11 of the Real-ESRGAN generator with the ViT-Base module.

This modification further enhances the network's ability to model global features. The improvement not only boosts the overall performance of the network but also shows significant advantages, especially in handling small-object superre-solution tasks. The improved network architecture is shown in Figure 6.

Figure 6.

The improved generator network structure

In the process of introducing the ViT module, this paper selected the ViT-Base model as the foundational module to replace the RRDB module. The multi-head self-attention mechanism of ViT can effectively capture global features and improve the global consistency of images through long-range dependency modeling. However, ViT has a large number of parameters, and due to its global attention mechanism, it may introduce significant computational overhead when used within the GAN generator framework. To avoid these potential issues, this method only replaces layers 9 to 11 in the generator, retaining the convolutional and RRDB modules in the shallow and deep layers to ensure a balance between local detail recovery and global feature modeling capabilities.

As shown in Table 1, the table presents the specific parameter settings of the ViT-Base module. These parameters are based on the basic structure of ViT and have been adjusted to meet the requirements of the generator network.

The experimental environment of this paper is based on the Ubuntu 18.04 operating system, with an RTX 2080Ti GPU and 16 GB of memory. The experiments were essentially conducted using the officially recommended parameter settings and were implemented using the Python 3 and PyTorch frameworks. The training set consists of the DIV2K dataset (800 images) and the Flickr2K dataset (2650 images), totaling 3450 high-resolution images covering a wide range of scenes, including natural landscapes, urban areas, and architecture. The test sets include the Set5, Set14, BSD100, and Urban100 benchmark datasets to evaluate the model's performance across different scenarios.

In this experiment, we used the DIV2K dataset (800 images) and the Flickr2K dataset (2650 images) as the training datasets, totaling 3450 high-resolution images, covering a variety of natural and man-made scenes. To simulate the local degradation features in real-world scenarios, the images were first randomly cropped into 256×256 pixel blocks, and then various data augmentation operations were performed to enhance the model's robustness to complex degradation conditions. The augmentation operations included: applying horizontal or vertical flipping with a 50% probability and randomly rotating the images within the range of –90° to 90° to improve the model's adaptability to directional changes; for color perturbation, the brightness was uniformly sampled and adjusted by ±15%, and the contrast and saturation were uniformly sampled and adjusted by ±10% to simulate different lighting and imaging environments; for noise injection, Gaussian noise with a standard deviation ranging from 0 to 0.03 was added, and JPEG compression noise with a quality factor ranging from 20 to 95 was introduced to simulate the degradation of images during acquisition, transmission, or compression processes.

The training process consists of two stages: pretraining and formal training, with the overall procedure designed to ensure the stability of training and the progressive improvement of model performance. During the pre-training stage, the generator is initialized and optimized using only the L1 loss function for 20k iterations, with a fixed learning rate of 2×10⁻⁴. Performance validation is conducted every 17,280 iterations. This stage effectively mitigates the mode collapse issue caused by an overly powerful discriminator in the early stages of adversarial training, with the training loss stabilizing around 0.03 after 20k iterations.

Subsequently, the formal training stage commences, lasting for 500k iterations. The generator and discriminator are jointly optimized using a combination of L1 loss, perceptual loss, and adversarial loss. The learning rate is dynamically adjusted to further enhance training stability. During the training process, the loss curve is recorded every 216 iterations, and the PSNR and SSIM metrics are output every 1,728 iterations to evaluate the model's performance in terms of reconstruction quality. As shown in Figure 7, the training loss drops below 0.025 after the 300kth iteration, indicating that the model has essentially converged, with stable and reliable training results.

Figure 7.

The improved generator network structure

To verify the performance of the improved model, comparative experiments were conducted with several mainstream super-resolution methods, including VDSR, EDSR, RCAN, and SwinIR. Quantitative metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) were used for evaluation on the Set5, Set14, BSD100, and Urban100 datasets, supplemented by visual comparisons. The results are shown in Table 2.

As shown in Table 2, the proposed method in this paper demonstrates excellent performance in the image super-resolution task, especially under multi-scale magnification and complex scenarios (such as Urban100). Both quantitative and qualitative experiments indicate that our method outperforms existing approaches in terms of PSNR and SSIM, including traditional networks like SRCNN, EDSR, and RCAN, as well as advanced models like SwinIR.

Specifically, at the ×2 magnification scale, our method achieves a PSNR of 33.95 dB on the Urban100 dataset (an improvement of 0.38 dB over SwinIR) and an SSIM of 0.9465 (an increase of 0.027). Our method also performs outstandingly on benchmark datasets such as Set5 and Set14, demonstrating its generalizability. For the more challenging ×4 task, our method achieves a PSNR of 33.00 dB on Set5 (an improvement of 0.12 dB) and a PSNR of 27.73 dB with an SSIM of 0.8200 on Urban100, showing significant performance advantages.

As shown in Figure 8, the proposed method in this paper demonstrates significant advantages in texture recovery in complex scenes, particularly in terms of clarity and naturalness in high-frequency areas such as building edges and texture patterns. Comparative experiments indicate that the reconstruction results of SRCNN are blurry and contain artifacts; although EDSR shows some improvement, the details are still not clear; RCAN and HAN enhance clarity but struggle to accurately recover complex textures; NLSA suffers from structural blurring and color distortion; and SwinIR exhibits texture reconstruction errors in images such as Barbara and img062.

Figure 8.

Comparison of 4x upsampling experimental results for img062.png in the Urban100 dataset.

In contrast, the proposed method in this paper demonstrates more accurate and clear texture edge recovery in the tests of Barbara and img062. In terms of visual perception, the SR results of Barbara are very close to the HR image, especially in the detail restoration of book textures. In the img062 image, the shape and structure of the window grid are closest to the HR image, fully reflecting the superiority of the proposed method in handling high-frequency information in complex scenes.