I. Introduction
The rapid advancement of generative models such as generative adversarial networks (GANs), autoencoders, and, more recently, diffusion-based models has significantly improved the realism of synthetic facial images and videos, commonly referred to as deepfakes. These technologies enable the creation of highly convincing forged media in which individuals appear to say or do things that never occurred. While such methods have legitimate applications in entertainment and virtual content creation, their misuse poses serious threats to digital trust, biometric authentication systems, political stability, journalism, and cybersecurity. As deepfakes continue to improve in terms of visual realism and cost, the need for a reliable, efficient, and explainable detection solution has become one of the prominent research challenges in multimedia forensics and computer vision [1].
The initial deepfake detection techniques were based on manual forensic clues like irregularities of eye-blinking, lack of head poses, edges of the face, or physiological signal anomalies [2]. Even though these methods are effective in the manipulations of early generations, they are fragile and can be readily bypassed as generative models grow. As deep learning has grown, it has been shown that the convolutional neural network (CNN)-based detectors, including XceptionNet, EfficientNet, and ResNet variants, perform well on the benchmark datasets such as FaceForensics++ (FF++) [3]. Nonetheless, these approaches mostly work on raw RGB images, and are plagued by three significant weaknesses, namely, (i) not generalizing well to unobserved datasets and types of manipulations, (ii) being highly sensitive to compression, artifacts, and adversarial perturbations, and (iii) encountering low interpretability, which complicates forensic analysis and trust in allowed deployment settings [4]. To overcome these drawbacks, autoencoder-based reconstruction-based detection models and variational autoencoders (VAEs) have become more and more popular. The underlying intuition of these methods is that a model that has been trained on genuine facial images learns a powerful prior of natural facial structure and texture, resulting in systematic reconstruction errors when shown in manipulated content. One way you can see these failures is in residual maps, which show the difference between the input image and the image reconstruction. They can frequently reveal subtle manipulation artifacts that would not be noticeable in the original frame. Residual-based representations are more interpretable, and have demonstrated the ability to cross-dataset. The majority of the available VAE-based deepfake detectors are, however, confined to spatial analysis, and they lack temporal information, which is essential in detecting video-based forgeries [5].
Temporal consistency is a key distinguishing factor between real and manipulated videos. Deepfake generation methods typically operate on frames independently or with limited temporal constraints, leading to subtle inter-frame inconsistencies, such as flickering artifacts, unstable facial geometry, or incoherent texture evolution over time. While several works have explored temporal modeling using recurrent neural networks, optical flow, or transformer-based video architectures, these methods generally operate on raw video frames and inherit the same robustness and generalization issues as spatial CNN-based detectors. Moreover, the main feature of forensic systems, such as adversarial robustness, is still understudied in both spatial and temporal deepfake detectors [6].
This paper presents a proposal of a robust spatio-temporal residual learning Framework of deepfake detection, which explicitly overcomes the above challenges. Our method also does not use raw pixel inputs but uses adversarially balanced residual representations that are produced by a VAE purely trained on real facial images. The proposed model generates both noise, compression, and adversarial perturbation resistant residual maps through the addition of adversarial training with fast gradient sign method (FGSM) and projected gradient descent (PGD) to VAE learning, and is also sensitive to manipulation. These residuals are easily decipherable and informative in the downstream classification.
To perform image-level detection, the residual maps are classified with the help of a fine-tuned efficientNetV2 model, which provides a decent trade-off between accuracy and computational efficiency. To apply the framework to videos, we propose a tmporal residual consistency module (TCM) which can be applied to video sequences of residual maps and their temporal differences. The TCM models inter-frame differences with either gated recurrent unit (GRU)-based sequential modeling or 1D convolutional temporal filters that allow the identification of subtle temporal artifacts that are not available to frame-wise detectors. The fusion of spatial and temporal scores gives the final predictions, which are a single frame-work used by both images and videos.
In comparison to the current methods of deepfake detection, the suggested framework has several key advantages:
Adversarially robust residual learning: A VAE that is adversarially perturbed learns stable and manipulation-sensitive residual maps that are resistant to noise and adversarial attacks.
Explicit temporal inconsistency modeling: The proposed TCM uses inter-frame residual dynamics for better detection of high-quality video deepfakes.
Better interpretability: The residual visualizations give forensic insight into the areas that have been manipulated, which leads to higher levels of transparency and trust.
Strong cross-dataset generalization: The residual-based design reduces overfitting to dataset-specific artifacts and improves robustness under domain shift.
This is a unified design that enhances robustness, interpretability, and generalization across datasets.
II. Related Work
With the recent fast development of generative models, for example, GANs, autoencoders, and diffusion models, deepfake detection has become an essential field of research. The initial methods of detection were hand-drawn characteristics such as abnormalities in the eye-blinking and pattern of head movement, or mismatch due to face-swapping and reenactment. As deep learning emerged, CNN-based models like XceptionNet, EfficientNet, and ResNet were now more prevalent, and had better spatial feature extraction, but with worse generalization to unseen data [7].
The spread of deepfake technology has prompted the creation of effective detection tools to ensure the integrity of digital media and to fight fake news. The literature review will analyze the current strategies in detecting deepfakes with special emphasis on CNN-based, reconstruction-based, and temporal modeling, and adversarial robustness, which are directly applicable to the presented structure.
The deep convolutional neural networks have emerged as the most popular way of detecting deepfakes because of their strong spatial feature extraction capacity. The FF++ was the first comprehensive benchmark dataset created by Rossler et al. [8] with several manipulation methods such as DeepFakes, Face2Face, FaceSwap, and NeuralTextures. Their analysis showed that XceptionNet structures have a high performance with more than 95% accuracy on controlled data. Nevertheless, these models showed pronounced deterioration in performance on the scenarios that were not seen or manipulations not performed, which points to major generalization drawbacks. Xception architecture developed by Chollet [9], using depthwise separable convolutions, was subsequently used widely in the detection of deepfakes, owing to its ability to represent spatial hierarchies. It is based on this that Tan and Le [10] proposed EfficientNet, a system of models that fundamentally scales network depth, width, and resolution through compound coefficients. EfficientNetV2 additionally increased the rate of training and the efficiency of the parameters as compared to the competitive accuracy. With such architectural achievements, CNN-based detectors that use raw RGB inputs are weak to compression artifacts, adversarial perturbations, and changes of domain between training and deployment environments. Nguyen et al. [11] carried out comprehensive cross-dataset evaluation experiments that indicated that models which have almost perfect accuracy on FF++ dropped to 60%–70% accuracy in WildDeepFake and Celebrity DeepFake (Celeb-DF) datasets. This failure of its performance highlights the overfitting behavior of raw-input CNNs to dataset non-locality and not core manipulation signatures. Li et al. [12] also revealed that CNNs tend to use low-level compression artifacts and color discrepancies instead of finding semantically meaningful forgery patterns to learn, which leads to low generalization to high-quality deepfakes.
Autoencoders and VAEs have also been proposed as alternative effective deepfake detectors based on the reconstruction error as a signal of anomaly. The underlying theory is that any model of facial images trained only on real ones will have problems generating accurate replicas of the manipulated areas, and this will reveal forgery artifacts as residual images. The theoretical basis of latent representation learning using variational inference was derived by Kingma and Welling [13]. The VAEs minimize a loss function that is a combination of reconstruction fidelity and Kullback-Leibler (KL) divergence regularization, which allows one to control generation and detect anomalies. Durall et al. [14] used this concept in the detection of deepfakes and trained autoencoders using real faces, and compared reconstruction errors to detect synthetic images. Their residual-based methodology offered interpretable heatmaps of manipulated facial parts, which have forensic explainability, which cannot be seen with black-box CNN classifiers. Zhao et al. [15] suggested multi-attentional deepfake detection, which is a hybrid of the autoencoder reconstruction with the attention mechanisms to highlight the locations of manipulation artifacts. Their approach showed better interpretability but was susceptible to Joint Photographic Experts Group (JPEG) compression and failed to model the time to analyze video. The DeepVision, proposed by Jung et al. [16], uses the reconstruction loss in autoencoders as auxiliary features in CNN classifiers to obtain minor gains over the baseline methods. Nonetheless, these methods were not systematically oriented to the issues of adversarial and cross-dataset robustness. Nataraj et al. [17] researched how reconstruction mistakes of convolutional autoencoders used with Support Vector Machine (SVM) classifiers could be employed to identify GAN-generated images. Although successful with synthetic faces of particular GAN approximations, their approach had poor extrapolation to face-swapping and reenactment methods common in contemporary deepfaking. The major shortcoming of current reconstruction-based approaches is the absence of adversarial training to stabilize residual generation to perturbations, noise, and compression, which is a necessary condition to deploy in the real world.
Video deepfakes add artifacts of temporal dimension due to the discrepancies between frame-by-frame manipulations. Several studies have aimed at using these temporal cues to achieve better detection results. Sabir et al. [18] introduced Recurrent Convolutional Networks, which were a combination of CNNs to extract spatial features followed by Long Short-Term Memory (LSTM) layers to model time. Their method produced great improvements on video datasets and could process raw RGB frames directly, which was compromised by small temporal irregularities. Gueira and Delp [19] proposed pipelines that are time-conscious by the application of ResNet50 extractors and LSTM networks to detect the abnormalities between frames. On the one hand, in showing the usefulness of temporal modeling, their approach was too computationally expensive, and was not able to cope with high-quality deepfakes with high temporal coherence. Amerini et al. [20] investigated the application of optical flow-based detection of suspicious motion patterns in deepfake videos. Optical flow computation is computationally expensive with low sensitivity to camera motion, which reduces its practical use. Transformer-based architecture has more recently been used in video deepfake detection. We introduced video transformer networks by Bonettini et al. [21], which can jointly process the spatiotemporal features with self-attention mechanisms, and have competitive results on the benchmark datasets. Transformers, however, occupy a lot of computational power and extensive training data and therefore, are difficult to deploy. Importantly, the current temporal approaches do not primarily examine raw pixel sequences but manipulation-sensitive residual features, which is likely the reason why, potentially, critically, but understudied, in deepfake forensics, critically, but underexplored, is the aspect of potentially adversarial robustness. Adversarial examples such as perturbation of inputs to imperceptible levels to confuse classifiers, are dangerous to deployed detection systems. Goodfellow et al. [22] proposed FGSM that produces adversarial perturbations through the linearization of the loss function on input samples. Madry et al. [23] suggested PGD, which upgrades FGSM by the use of an iterative refinement procedure that generates more powerful adversarial instances by optimizing over a sequence of steps with projections onto perturbation budgets. Carlini and Wagner [24] constructed advanced optimization-based attacks that proved that even a highly accurate classifier might be systematically deceived with a small amount of distortion. When considering the deepfake detector, Gandhi and Jain [25] tested the adversarial robustness of popular CNN architectures, which demonstrated a considerable amount of vulnerability: models with a clean accuracy of 95% collapsed to under 60% with PGD attacks with small perturbation budgets. This observation shows that there is a severe breach in the performance in labs and the real world. Adversarial training, which was proposed by Madry et al. [23], is a principled defense mechanism in which adversarial examples are included during training. Robust decision boundaries that are not easily adversarially manipulated are learned by models trained using FGSM and PGD perturbations. Nevertheless, little has been done on adversarial training of deepfakes detection, as the majority of the literature has concentrated on cleaner accuracy measures. Adversarial training for face recognition systems was used by Zhang et al. [26], which showed greater robustness and reduced clean accuracy trade-offs, which can be directly applied to deepfake forensics. It is interesting to note that current VAE-based deepfake detectors have not embraced the concepts of adversarial robustness. Normal VAE training on clean data generates encoders and decoders that are very sensitive to perturbations on the input, leading to unstable residual maps in the presence of noise, compression, or adversarial attacks. This sensitivity compromises the residual-based detectors in the real-world situation where the manipulated material can be handled by post-processing, compression, or deliberate adversarial obfuscation.
Recent studies have investigated hybrid architectures that entail the integration of more than two modalities of detection. A similar case was suggested by Zhao et al. [27], who suggested integrating CNN predictions with the analysis of biological signals (pulse detection using facial videos) so that the methods become more resilient. Nevertheless, these approaches need a particular signal presence and are not always applicable to all forms of manipulation. Haliassos et al. [28] proposed LIPS, which emphasizes the aspect of audio-visual synchronization discrepancies in face-reenactment videos, and this method has proven to be valuable in cross-modal research, but it needs audio information which is not always available in the case of image-only scenarios. Shiohara and Yamasaki [29] integrated frequency-domain analysis and spatial CNN features, and capitalized on the fact that the images generated by GANs contain artifacts in the frequency space. Although promising, frequency-based approaches are susceptible to adversarial attacks that are made to specifically exploit frequency components. Computational complexity and overhead spatial, temporal, and frequency-domain feature integration is an interesting area of research, but presents architectural complexity. Nevertheless, despite significant advances in deepfake detection studies, there are still a few significant gaps:
Limited Adversarial Robustness: Current detectors, especially those based on reconstruction, are not trained adversarially so that they are susceptible to perturbations and artifacts of compression that can occur in practical applications [9, 16, 22, 23, 26].
Temporal Modeling in Residual Space: Although temporal methods can be used, most of them work with raw frames instead of manipulation-sensitive residual features. The temporal inconsistency of reconstruction error space in modeling is not yet studied [14, 15, 18,19,20].
Unified Image-Video Framework: The majority of methods are either image or video detectors that are not based on unified architectures that can use both spatial and temporal features to adapt to input modality [10, 27–29].
Cross-Dataset Generalization: Cross-dataset generalization is still a major issue that can be overfit to data artifacts. More refined generalization could be provided by residual-based methods, which concentrate on reconstruction abnormalities as opposed to pixel-level statistics [8, 11, 21].
Interpretability: Black-box CNN classifiers do not have much forensic explainability. Reconstruction-based techniques provide interpretable residual images but have not been used with effective time modeling and adversarial training [12, 17].
The framework is able to fill these gaps with a proposed adversarially learned VAE that triggers the process of stable residual generation, a TCM that learns to recognize inter-frame discrepancies, and a shared spatial-temporal fusion framework that can be used across both videos and images. The purpose of this integrated approach is to have a higher level of accuracy, robustness, generalization, and inter-pretability than the existing methods. The detailed summary of related work presented in Table 1 gives an insight into the main contribution of the existing approaches along with their limitations, compared to the offered framework.
Table 1:
Comparative analysis of deepfake detection methods: key contributions, limitations, and relationship to proposed framework
| Reference | Category | Method | Key contribution | Research gaps |
|---|---|---|---|---|
| Rossler et al. [8] | Spatial (Dataset + CNN) | FF++ dataset & CNN baselines | Comprehensive benchmark with multiple manipulation types | Poor cross-dataset generalization |
| Chollet [9] | Spatial (CNN) | Xception architecture | Depthwise separable convolutions for efficient feature extraction | Limited robustness to adversarial attacks |
| Tan and Le [10] | Spatial (CNN) | EfficientNetV2 | Improved training efficiency and accuracy through compound scaling | Operates on raw inputs, vulnerable to compression |
| Nguyen et al. [11] | Spatial (analysis) | Cross-dataset evaluation study | Revealed severe performance drop across datasets | Identified problem but no robust solution |
| Li et al. [12] | Spatial (analysis) | CNN artifact analysis | CNNs exploit compression artifacts instead of semantic forgery | Needs alternative representations |
| Kingma and Welling [13] | Reconstruction | VAE | Foundation for latent representation and reconstruction | Not specific to deepfake detection |
| Durall et al. [14] | Reconstruction | Autoencoder-based detection | Reconstruction error reveals manipulation artifacts | No adversarial robustness or temporal modeling |
| Zhao et al. [15] | Reconstruction + attention | MADD | Combines reconstruction with attention for localization | Sensitive to compression, no temporal modeling |
| Jung et al. [16] | Reconstruction + CNN | DeepVision | Uses reconstruction loss as auxiliary feature | Limited robustness evaluation |
| Nataraj et al. [17] | Reconstruction + ML | Autoencoder + SVM | Uses reconstruction errors with classical ML | Limited to specific GAN types |
| Sabir et al. [18] | Temporal (RNN) | Recurrent CNN + LSTM | Captures temporal inconsistencies | Computationally expensive, uses raw frames |
| Güera and Delp [19] | Temporal (RNN) | ResNet + LSTM | Models inter-frame dependencies | Struggles with high-quality deepfakes |
| Amerini et al. [20] | Temporal (motion) | Optical flow-based detection | Detects unnatural motion patterns | Sensitive to camera motion, expensive |
| Bonettini et al. [21] | Temporal (transformer) | Video transformer networks | Self-attention for spatiotemporal learning | High computational cost |
| Zhao et al. [27] | Hybrid (multi-modal) | Ensemble multi-modal detection | Combines CNN with biological signals | Requires specific signals |
| Haliassos et al. [28] | Hybrid (audio-visual) | LIPS | Uses lip-sync inconsistencies | Requires audio data |
| Shiohara and Yamasaki [29] | Hybrid (frequency) | Frequency-domain analysis | Exploits GAN frequency artifacts | Vulnerable to adversarial attacks |
| Goodfellow et al. [22] | Robustness | FGSM | Fast adversarial attack method | Limited attack strength |
| Madry et al. [23] | Robustness | PGD | Strong iterative adversarial training | Computationally intensive |
| Carlini and Wagner [24] | Robustness | C&W attack | Demonstrates model vulnerabilities | Not specific to deepfake detection |
| Gandhi and Jain [25] | Robustness (evaluation) | Adversarial evaluation | Shows CNN vulnerability under attacks | No robust solution proposed |
| Zhang et al. [26] | Robustness (defense) | Adversarial training | Improves robustness in recognition | Not applied to deepfake detection |
| Proposed work | Hybrid (spatial + temporal + robustness) | Adversarial VAE + TCM + EfficientNetV2 | Unified residual-based spatial-temporal framework with adversarial robustness | Addresses robustness, temporal modeling, generalization, interpretability |
[i] CNN, convolutional neural network; C&W, Carlini-Wagner; FGSM, fast gradient sign method; GAN, generative adversarial networks; LIPS, logical inferences per second; MADD, multi-attentional deepfake detection; ML, machine learning PGD, projected gradient descent; RNN, Recurrent Neural Network; TCM, Temporal consistency module; VAE, variational autoencoder.
This literature review provides a theoretical and empirical basis for the proposed Robust spatio-temporal residual learning framework, which shows the systematic coverage of weaknesses of existing solutions through adversarial VAE training, temporal residual consistency modeling, and unified spatial-temporal fusion. Unlike Durall et al. [14], which rely on reconstruction error without robustness considerations, our approach introduces adversarial training to stabilize residual representations. Compared to Zhao et al. [15], which uses attention mechanisms on spatial features, our framework operates in residual space and extends to temporal modeling. Furthermore, existing methods focus either on images or videos, whereas our approach provides a unified framework for both modalities.
III. Proposed Methodology
The proposed robust spatio-temporal residual learning framework is directly structured to identify deepfakes, which is achieved through combined analysis of spatial features and temporal abnormalities that are created when faces are manipulated. As opposed to traditional CNN-based detectors that take RGB inputs directly, our framework exploits the reconstruction property of an adversarially trained VAE in order to capture manipulation-sensitive residual features. These types of residuals reveal the concealed artifacts concealed by the raw-frame models, and a temporal consistency module (TCM) detects the frame-to-frame anomalies occurring in forged videos. The architecture includes a series of four significant steps,-such as adversarial training and residual extraction to VAE, spatial classification with EfficientNetV2, temporal modeling based on residual generation to videos, and spatial-temporal integration to complete the final prediction. Figure 1 depicts the system in general.
Figure 1:
Architecture of the proposed system. CNN, convolutional neural network; FGSM, fast gradient sign method; GRU, gated recurrent unit; PGD, projected gradient descent; VAEs, variational autoencoders.
a. Adversarially robust VAE training and residual extraction
The first component of the framework focuses on learning a robust representation of natural facial structures. Let x ∈ RH × W × 3 denote an input image or video frame. The VAE consists of an encoder E(·) and decoder D(·) that generate a latent embedding and subsequent reconstruction:
The model is trained exclusively on real facial images to ensure reconstruction fidelity for authentic inputs and systematic reconstruction failures for manipulated ones. The objective function includes an ℓ1 reconstruction term and a KL divergence term:
a.i. Adversarial training for robustness
The VAE is adversarially trained against FGSM and PGD so as to improve stability to perturbations as:
where π denotes projection onto an ∈. This ensures that the residual maps are still informative even in the case of adversarial noise. Training produces both clean and adversarial samples which are reconstructed leading to a VAE that generates stable outputs when noise, compression, or attacks are used. This property plays a major role in enhancing the reliability of residual based detection.a.ii. Residual map extraction
After the VAE has been trained, residual maps are created by calculating the absolute difference between the positive and negative input frame and its reconstruction:
These maps point out the manipulation artifacts, including blending inconsistency, curved edges, and small texture irregularities. The adversarial strength of the VAE is that such residual features are informative even in case of perturbation, allowing them to serve effectively as downstream classification features. The image-level and video-level detection use residual maps as the main input feature in the proposed framework.
b. Spatial deepfake classification using efficientNetV2
In image-level, the residual maps R (optionally grouped with the original RGB frame) are inputted to an EfficientNetV2-S classifier that has been finetuned.
Let fsp(u) represent the spatial classification network which produces a deepfake probability:
Where u ∈ {R, x,[x,R]} depending on the chosen feature configuration.
The model is trained using a binary cross-entropy loss:
Training with residual values of the additive auto-decoder is faster to converge and is better at localized face artifacts than models trained on raw-frames.
c. Temporal consistency module
Deepfake videos tend to have temporal inconsistency due to the fact that frame-level manipulations fail to provide coherent facial dynamics. To use this cue, we generate residual sequences as:
And their corresponding temporal difference maps are:
These features are processed by the TCM, implemented using either:
□ a 1D-CNN architecture for local temporal pattern filtering, or
□ a GRU-based sequential model for long-term temporal modeling.
The temporal model outputs a video-level deepfake score:
This component effectively captures flickering artifacts, inconsistent facial geometry, and temporal texture shifts that spatial models alone fail to detect. GRU is used for modeling long-term temporal dependencies across frames, while 1D-CNN captures short-term local temporal patterns efficiently with lower computational cost.
d. Spatial–temporal fusion for video classification
For videos, both spatial and temporal predictions provide complementary forensic evidence. Hence, the final deepfake score is computed via weighted late fusion:
where α ∈ [0,1] is chosen empirically.This fusion strategy enables:
□ image-only detection (using Ssp) when temporal information is unavailable.
□ enhanced video-level detection by combining spatial and temporal cues.
The integrated architecture consistently outperforms spatial-only baselines, especially in high-quality and temporally coherent manipulations.
IV. Experimental Setup
The experiments were designed in a systematic manner to test the efficiency of the suggested robust spatio-temporal residual learning framework in image-level and video-level deepfake detection. The FF++ and WildDeepFake datasets were used to conduct all experiments, and a variety of manipulation techniques and real-world variations are included. The entire pipeline is made up of three key components, namely: (i) an adversarially trained VAE to generate manipulation-sensitive residual maps, (ii) EfficientNetV2 as the spatial classifier, and (iii) a temporal consistency module (TCM) to capture inter-frame inconsistency in videos. The experimental settings are provided in Table 2 as given below:
Table 2:
Experimental setup
| Component | Summary |
|---|---|
| Datasets | FF++ (C40), 4 manipulation types; WildDeepFake for cross-domain evaluation |
| Preprocessing | 30 FPS frame extraction; (256 × 256) resizing; ImageNet normalization; basic augmentations |
| Models | VAE (latent dim 256); EfficientNetV2-S classifier; GRU/1D-CNN temporal module |
| Training settings | Adam optimizer; LR (1 × 10−4); VAE: 40 epochs; Classifier: 25–30 epochs |
| Evaluation metrics | Accuracy, Precision, Recall, F1, AUC |
| Robustness tests | FGSM and PGD applied during evaluation |
| Implementation | PyTorch 2.0, NVIDIA Tesla V100 GPU, mixed-precision (FP16) |
a. Dataset preparation
In the case of FF++, we made use of the C40 compression-level frames that have been obtained out of all four forms of manipulation: DeepFakes, FaceSwap, Face2Face, and NeuralTextures. To avoid cross-video leakage, the dataset was split by the video ID into training, validation and test subsets as per the protocols. Individual videos are broken down into sequences of frames at a frequency of about 30 frames per second (FPS) and a fixed number of samples of 30–40 frames per video were randomly drawn during training and testing. The WildDeepFake dataset was only evaluated in cross-domain to determine the generalization in the face of domain shift. It is also appropriate because of its varied sources of manipulation to test the robustness of stress-testing.
b. Preprocessing and input formation
All frames were resized to 256 × 256 and normalized using ImageNet statistics. For VAE training, only real images were used to ensure the residual map captured unnatural reconstruction error patterns for fakes. Residual maps were computed as the absolute pixel-wise difference between the input image x and its VAE reconstruction For video experiments, temporal residual sequences {Rt} were constructed, and temporal difference maps ΔRt = |Rt−Rt−1| were included to emphasize temporal inconsistencies.
c. Model architecture and training details
The VAE utilized a convolution encoder–decoder structure with a 256-dimensional latent space. Adversarial perturbations were applied during VAE training using FGSM (ɛ = 2/255) and PGD (ɛ = 4/255, step size = 1/255, 7 iterations) to enforce robustness against localized and imperceptible noise. EfficientNetV2-S was used as the spatial classifier due to its favorable trade-off between performance and computational efficiency. Residual maps (or combined 6-channel input) were used as classifier input.
For video-level fusion, a Temporal Consistency Module (TCM) was designed using either a 1D CNN or a GRU operating over residual and difference sequences. Spatial and temporal scores were fused via a weighted linear combination. All models were trained using the Adam optimizer with an initial learning rate of 1 × 10−4. The VAE was trained for 40 epochs, and the classifier for 25–30 epochs with early stopping based on validation loss. Batch size was set to 32 for images and 4 for videos due to memory constraints. Data augmentation included random horizontal flip, Gaussian noise, color jitter, and random affine transformations.
d. Evaluation protocol
Image-level predictions were computed per frame, while video-level predictions were obtained by aggregating spatial and temporal scores across frames. Video-level prediction is computed using mean aggregation:
Then, Final prediction:
Metrics included accuracy, precision, recall, F1-score, and area-under-the-curve (AUC). For cross-dataset evaluation, the model was trained exclusively on FF++ and tested on the full WildDeepFake test set without fine-tuning. We also performed adversarial robustness evaluation using FGSM and PGD attacks at varying perturbation strengths to quantify stability.
e. Implementation details
All experiments were conducted on a single NVIDIA Tesla V100 GPU with 16 GB memory using PyTorch 2.0. The complete pipeline was implemented in Google Colab Pro. Mixed-precision training (FP16) was enabled to reduce training time. For reproducibility, we fixed all random seeds and logged training metrics using TensorBoard.
V. Results and Discussion
The proposed robust spatio-temporal residual learning framework was evaluated on the FF++ and WildDeepFake datasets for both image-level and video-level deepfake detection. The system integrates an adversarially trained VAE to generate manipulation-sensitive residuals and uses EfficientNetV2 as the spatial classifier. For videos, the temporal consistency module (TCM) processes inter-frame residual dynamics to capture temporal inconsistency artifacts.
On FF++ (test split), the proposed residual-guided EfficientNetV2 achieved 93.8% accuracy, 92.3% F1-score, and an AUC of 0.96, outperforming raw-RGB baselines. Residual-only training provided improved sensitivity to fine-grained artifacts, while combining raw inputs with residuals further enhanced performance as shown in Table 3. Cross-dataset generalization was evaluated by training on FF++ and testing on WildDeepFake. The proposed method achieved 80.1% accuracy, 78.6% F1-score, and 0.84 AUC, significantly outperforming raw-frame CNNs, demonstrating improved robustness to domain shift as defined in Table 4.
Table 3:
Performance on FF++ test set
| Method | Accuracy (%) | Precision (%) | Recall (%) | F1-score (%) | AUC |
|---|---|---|---|---|---|
| Raw EfficientNetV2 | 90.30 | 89.10 | 88.40 | 88.70 | 0.92 |
| Residual-only (VAE reconstruction error) | 92.10 | 91.80 | 91.00 | 91.40 | 0.94 |
| Proposed: VAE + EfficientNetV2 | 93.80 | 93.10 | 91.90 | 92.30 | 0.96 |
| Proposed (Adversarial-robust VAE) | 93.50 | 92.80 | 92.00 | 92.40 | 0.96 |
Table 4:
Cross-dataset generalization (Train: FF++ → Test: WildDeepFake)
| Method | Accuracy (%) | F1-score (%) | AUC |
|---|---|---|---|
| Raw EfficientNetV2 | 70.40 | 68.20 | 0.74 |
| Residual-only | 76.90 | 75.10 | 0.8 |
| Proposed: VAE + EfficientNetV2 | 80.10 | 78.60 | 0.84 |
Adversarial robustness experiments using FGSM and PGD as analyzed in Table 5 show that the base-line raw EfficientNetV2 experiences a substantial drop in AUC under perturbation (0.92 → 0.80), whereas the proposed adversarially trained VAE retains much higher resilience (0.96 → 0.90), reducing degradation to only ~4%–6% (Figure 2). For video-level evaluation, the TCM adds an additional 5.1% improvement in accuracy over the spatial-only pipeline, illustrating the importance of temporal residual modeling. The combination of adversarially stabilized residuals, EfficientNetV2 spatial features, and temporal consistency analysis enables the proposed approach to achieve strong performance, interpretability, and robustness across diverse datasets and attack scenarios (Table 6).
Table 5:
Robustness under adversarial attacks
| Method | Clean AUC | Under FGSM | Under PGD |
|---|---|---|---|
| Raw EfficientNetV2 | 0.92 | 0.86 | 0.8 |
| Proposed VAE + EfficientNetV2 | 0.96 | 0.93 | 0.9 |
| Improvement over baseline | 0.04 | 0.07 | 0.1 |
Table 6:
Ablation study on FF++
| Configuration | Accuracy (%) | F1-score (%) |
|---|---|---|
| EfficientNetV2 (raw frames) | 90.30 | 88.70 |
| Residual-only | 92.10 | 91.40 |
| Residuals + Input (6-channel) | 93.80 | 92.30 |
| Proposed (Residual + TCM video fusion) | 95.4 (videos) | 94.20 |
Figure 2:
ROC curves comparing raw EfficientNetV2 and proposed residual-based methods. AUC, area-under-the-curve.
Adversarial training significantly improves robustness under perturbations while maintaining comparable clean performance as defined in Table 7. The training curves for the VAE and classifier demonstrate stable convergence behavior, with both training and validation losses decreasing smoothly across epochs (Figure 3). The EfficientNetV2 classifier trained on residual maps shows faster convergence and lower validation loss compared to raw-frame training. Accuracy plots similarly indicate that the residual-based classifier achieves higher validation accuracy throughout training. Receiver operating characteristic (ROC) curve analysis further demonstrates that the proposed approach significantly outperforms baseline models, achieving an AUC of 0.96 compared to 0.92 for raw EfficientNetV2. The adversarially trained VAE variant maintains high performance even under FGSM and PGD attacks, with only minor degradation in AUC (Figure 4). The confusion matrix indicates a balanced detection capability across real and fake classes, with fewer misclassifications compared to baseline CNNs. For video-level analysis, scatter plots of spatial vs temporal scores reveal clear separability between real and fake samples, validating the complementary nature of residual spatial features and temporal consistency modeling (Figure 5).
Figure 3:
Classifier loss curves.
Figure 4:
Confusion matrix.
Figure 5:
Spatial vs temporal score scatter plot.
Table 7:
Effect of adversarial VAE training
| VAE type | Clean AUC | FGSM AUC | PGD AUC |
|---|---|---|---|
| Clean VAE | 0.95 | 0.87 | 0.82 |
| Adversarial VAE | 0.96 | 0.93 | 0.9 |
The proposed framework outperforms baselines on FF++ and WildDeepFake (Table 8). Residual-based classification improves sensitivity to subtle artifacts, while temporal modeling enhances robustness against high-quality video deepfakes. Cross-dataset results confirm strong generalization. Residual visualizations highlight manipulated facial regions, providing interpretable forensic evidence.
VI. Conclusion
In this paper, the authors proposed a single and strong image and video deepfake detector through the use of adversarial and stable residual learning and temporal inconsistency modeling. A VAE that is solely trained with adversarial perturbation augmentation with real facial data learns robust residual representations revealing subtle manipulation artifacts. These spatial residuals are categorized with EfficientNetV2 to make image-level predictions, and an inter-frame anomaly (temporal consistency module) is used to make video-level predictions. The experimental findings based on various datasets prove that the proposed method is much better in terms of generalization, robustness, and interpretability over traditional raw-input CNN-based detectors. The combination of space and time cues allows the model to identify high-quality, time-coherent deepfakes that present significant challenges to the existing systems. The proposed framework achieves 93.8% accuracy and 0.96 AUC on FF++, improves cross-dataset performance to 80.1%, and enhances video detection by 5.1%. Future research can build on this framework with transformer-based temporal models, multi-frequency residual analysis or real-time deployment policies of large-scale forensic applications.