Benchmarking Tabular Data Synthesis: Evaluating Tools, Metrics, and Datasets on Prosumer Hardware for End-Users

2.1 Scope of tools

The tool selection process began with a query of scientific databases, including IEEE Xplore, ScienceDirect, Google Scholar, SpringerLink, NeurIPS proceedings, and MDPI journals. We included tools designed for TDS, excluding those for image or text generation. Selection criteria required peer-reviewed publications in English, with full-text and code availability, and alignment with TDS models included in Figure 1. In this taxonomy, hybrid models are those that combine different TDS models, for example, a VAE with diffusion. Imputation and discriminative models were excluded due to the absence of state-of-the-art tools.

To ensure a manageable comparison, we included only vanilla versions or significant advancements of tools representing each model type, excluding minor variations. The initial process identified 37 tools. The final verification in April 2024 excluded tools lacking publicly available code, highly customized tools like KAMINO (Ge et al., 2021) and STaSy (Kim, Lee, and Park, 2023), as they required specialized configurations incompatible with our standardized benchmark, and we focused on PyTorch-based tools, leaving TensorFlow-based implementations for possible future extensions. For all tools except SMOTE, we used the author-provided code. For SMOTE, we used Python packages imbalanced-learn (Lemaître, Nogueira, and Aridas, 2017) and smogn (Cantalupo, 2021).

Obsolete versions were replaced by advanced implementations, such as CTAB-GAN+ over TableGAN and GANBLR++ over GANBLR. Time-series tools were also excluded, as they are covered in custom benchmarks (Ang et al., 2023). The final tools are presented in Table 1.

2.2 Scope of datasets

The initial 37 tools used 114 datasets in their evaluations. We first removed duplicates and modified versions of the same dataset, then prioritized datasets with diverse sizes, domains, column types, and characteristics like distributions and skewness. Publicly available datasets from the UCI repository (Dua and Graff, 2017) and Kaggle (LLC, 2010) were favored. Excessively large datasets were also removed to ensure compatibility with the hardware setup (Section 3). The final selection of 15 datasets is listed in Table 2, where mixed columns follow the definition in (Zhao et al., 2021) as columns containing both categorical and numerical values.

2.3 Scope of metrics

The initial compilation of evaluation metrics included 71 metrics sourced from the publications of the 37 tools, as well as from TabSynDex (Chundawat et al., 2024) and Goncalves et al. (Goncalves et al., 2020). Also, we identified key purposes for synthesizing data, along with functional and non-functional requirements users can impose on a TDS tool (Davila et al., 2025). Based on these, we designed specific tests to evaluate the tools, which include 23 of the 71 metrics from the tool publications, across six evaluation dimensions. These tests are shown in Table 3.

This study does not include evaluations for differential privacy, temporal dependencies, text columns, inter-table correlations, or integrity constraints. Including these aspects would significantly broaden the range of use cases, blurring the focus of this benchmark. Instead, these topics are reserved for future work.

4.1 Correlations

Relationships between columns are essential for structural integrity and accurate downstream tasks like predictive modeling. We used the Correlation Difference metric, which we calculate as in Correlations, where we first choose randomly column pairs in the dataset (always the same pairs for all experiments), identify from the configuration what type of columns they are, and calculate their correlation. For continuous columns, we use Pearson’s correlation; for binary categorical, we use the Point-Biserial; and for all other columns, we use Spearman and Kendall. All of them use the functions from the SciPy library. For all experiments which converged, we calculate the difference between the correlations of the original dataset’s column-pairs and the correlation of the synthetic datasets.

Figure 3 shows correlation preservation, with scores normalized from zero (perfect preservation) to one (worst performance). Empty cells are cases where no dataset was generated, mainly due to: (i) PrivBayes not supporting continuous target columns (making it unsuitable for regression use cases) (Zhang et al., 2017), or (ii) tools running out of resources, reflecting typical end-user hardware constraints.

Figure 3

Only AutoDiff, REaLTabFormer, SMOTE, and TVAE converged for all use cases, with AutoDiff and REaLTabFormer consistently preserving correlations. GAN-based tools (CTAB-GAN+, CTGAN, GANBLR++) struggled with datasets having many columns, often failing to converge or preserve correlations. GOGGLE (Graph Neural Networks) did not converge with any of the datasets using prosumer hardware, further discussed in subsequent evaluations. Notably, SMOTE and TVAE show low correlation errors, despite their relative simplicity.

To ensure fair comparisons, PrivBayes and GOGGLE were excluded from further evaluations. The remaining seven datasets, where all tools converged (abalone, diabetes, insurance, churn2, wilt, adult, cardio), were used for analysis. PrivBayes was separately assessed for classification tasks without continuous target columns to evaluate Bayesian Networks-based tools.

4.2 Column distributions

Preserving column-wise distributions is critical in TDS and closely related to the Privacy vs. Utility trade-off (Park et al., 2018), which describes the challenge of generating useful synthetic data for its original task while protecting sensitive information. Accurate column-wise distributions ensure that the synthetic dataset retains the statistical properties of individual columns, maintaining its utility.

We compared synthetic and real column distributions using visual inspection and quantitative metrics to assess how the tool captures the original statistical characteristics. Section 4.6 then analyzes privacy, the second part of the trade-off. Figure 4 shows one of the distribution plots used for visual inspection, showing how different tools replicate column-wise distributions across datasets. Visual inspection offers an intuitive initial check of whether synthetic data captures key characteristics. This assessment was performed for all datasets and tools, with plots available in Github Repository.

Figure 4

Figure 4 presents plots for continuous and categorical columns. For continuous columns, real distributions are shown in blue and synthetic in red, showing whether tools capture column characteristics without assuming simple distributions like the Gaussian. For categorical columns, bar graphs display class counts, with real classes in blue and synthetic in red, making it easy to compare the number of classes and class proportions. For example, one can easily identify how GOGGLE collapses and reaches extremely high density values.

While visual inspection is a helpful starting point, it is inherently subjective and may overlook subtle discrepancies. To ensure consistency and precision, we used quantitative metrics such as the Wasserstein Distance for continuous data and Jensen–Shannon Divergence for categorical data. These metrics provide objective, numerical evaluations, capturing subtle differences in distributions that visual methods might miss, such as slight shifts in central tendency or variability. Quantitative metrics are essential for scalable and reproducible benchmarking, with results detailed in the following sections.

4.3 Dataset imbalance

Dataset imbalance is a key challenge in synthetic data generation (Davila et al., 2025), and it occurs when certain classes significantly outnumber others, potentially biasing predictive models toward dominant classes. To evaluate how well tools address this challenge, we used the metrics Wasserstein Distance (WD), Kolmogorov–Smirnov (KS) Statistic, Jensen–Shannon (JS) Divergence, and Kullback–Leibler (KL) Divergence, as implemented in the Python package (SciPy, 2024a, b, c, d).

For continuous columns, WD measures the cost of transforming one distribution into another, indicating how well tools maintain class proportions without distorting statistical properties. KS statistic captures the maximum difference between cumulative distribution functions (CDFs), reflecting alignment between real and synthetic class distributions.

For categorical data, JS Divergence measures the similarity between real and synthetic class probability distributions, with smaller values indicating better alignment. KL Divergence evaluates information loss when approximating distributions, penalizing deviations from target class balance. Percentage Class Count Difference provides an intuitive measure of relative class count differences between real and synthetic datasets.

Table 5 presents the evaluation results, with zero as the ideal value for all metrics. Top results are bolded and underlined. Each metric was calculated across all datasets for each tool, repeated five times for statistical significance, averaged, and normalized using Min–Max scaling for comparison.

The top-performing tool was SMOTE, followed by CTAB-GAN+ and TabDDPM. As expected, SMOTE performed well because it directly oversamples minority classes by interpolating between similar points, making it ideal for dataset imbalance. However, this can risk creating synthetic samples that are too similar or reinforce linear patterns, as presented in (Brandt and Lanzén, n.d.) and evidenced in the further dimensions of this benchmark. In contrast, CTGAN and GANBLR++ produce relatively lower metrics, possibly because GAN models can have difficulty capturing minority classes due to mode collapse, where the generator fails to learn the minor classes’ full distribution.

4.4 Dataset augmentation

Dataset augmentation is also a primary use case for synthetic data (Davila et al., 2025), where users expand datasets to improve model robustness and generalization by generating new data points that preserve the original dataset’s statistical properties while enhancing diversity.

To evaluate how well a tool augments the data, we used again WD, KS Statistic, and correlation differences for continuous columns, but additionally Quantile Comparison, and JS Divergence with Percentage Number of Classes Difference for categorical columns. Quantile Comparison ensures synthetic data reproduces the original data’s spread across quantiles, crucial for maintaining variability and improving model robustness. Percentage Number of Classes Difference measures whether the synthetic data retains the original number of classes, with lower values indicating better representation.

Table 6 summarizes the results, where lower values represent better performance. Top results are bold and underlined. Metrics were calculated across all datasets for each tool, repeated five times for statistical significance, averaged, and normalized using Min–Max scaling for comparison.

The top-performing tools were TabSyn, SMOTE, TabDDPM, and REalTabFormer. Unlike class imbalance handling, effective augmentation requires generating a diverse set of new samples that not only maintain statistical fidelity but also introduce meaningful variation. This is particularly captured by the Quantile Comparison metric, which evaluates how well tools reproduce the distributional spread across quantiles, a key aspect for improving model generalization. TabSyn showed the top performance on this metric, suggesting it effectively balances variability and structure. SMOTE and REalTabFormer followed closely, indicating that both sampling-based and Transformer-based methods can successfully generate data that enhances diversity while preserving core statistical properties. In the classification-only assessment, PrivBayes achieved the highest Percentage Number of Classes Difference. Simpler tools like CTGAN and TVAE perform well on basic metrics but struggle with complex distributions, supporting what is seen in Figure 4.

4.5 Missing values

Handling missing values is a key challenge in synthetic data generation. Missing values can distort analyses and degrade model performance by introducing gaps in the data. We evaluated tool performance by introducing 5%, 10%, and 20% missing values into the datasets and assessing their ability to maintain the integrity of continuous and categorical variables. For continuous columns, we used WD, KS statistic, correlation differences, and Quantile Comparison. For categorical columns, we used JS Divergence, Percentage Class Count Difference, and Percentage Number of Classes Difference, which ensures class diversity is not artificially altered when missing values affect certain classes.

Only 10 tools were evaluated, excluding GOGGLE and GANBLR++ due to poor initial performance, and TabDDPM due to errors with missing data in the configuration files. A visual inspection, shown in Figure 5, highlights the effect of missing values on the distributions of the diabetes dataset (Kahn, n.d.). Continuous plots display real distributions in blue, with red, green, and orange for synthetic datasets generated with 5%, 10%, and 20% missing values. Categorical bar plots show real classes in blue, with synthetic classes in red, green, and orange. The first column, SkinThickness, follows a multimodal distribution.

Figure 5

Tools, such as AutoDiff, REaLTabFormer, SMOTE, TabSyn, and TabuLa, replicated distributions with minimal deviations, regardless of missing values. In contrast, CTGAN struggled, with deviations increasing as missing values rose, and it failed to preserve class proportions. CTAB-GAN+ captured simpler distributions but missed the multimodal nature of SkinThickness.

Metrics were calculated across all datasets for each tool at 5%, 10%, and 20% missing values, with evaluations repeated five times for statistical significance. Results were averaged and normalized using Min–Max scaling for fair comparison. Table 7a, 7b, and 7c show the detailed results. SMOTE demonstrated the highest robustness, maintaining stable performance across all metrics and missing value levels. AutoDiff, REaLTabFormer, TabSyn, and TabuLa also maintained acceptable performance at 20% missing values, while other tools struggled with higher levels of missing data.

4.6 Privacy

Another key aspect of the Privacy vs. Utility trade-off is assessing how closely synthetic data resembles real data while ensuring privacy. This is crucial when real datasets contain sensitive or personally identifiable information. Synthetic data offers an alternative to traditional anonymization methods like masking, which often cause significant information loss. The challenge is to generate data that is useful for downstream tasks without exposing individual records.

We evaluated privacy using Distance to Closest Record (DCR) (Hernandez et al., 2022; Park et al., 2018) and Nearest Neighbor Distance Ratio (NNDR) (Zhao et al., 2024). DCR measures the Euclidean distance between each synthetic record and its nearest real counterpart. Lower DCR values enhance utility but increase privacy risk, while higher values suggest better privacy at the cost of utility. NNDR compares the distance between a synthetic record and its closest real record to its second-closest synthetic record, with higher values indicating better privacy protection through greater dispersion.

Table 8 summarizes the results, where higher values indicate stronger privacy protection. Top-performing tools are bolded and underlined. Each metric was calculated across all datasets and tools, repeated five times for statistical reliability, averaged, and normalized using Min-Max scaling for comparison.

TabDDPM, CTAB-GAN+, and REalTabFormer performed best in privacy. Despite its strong performance in previous evaluations, SMOTE underperformed in privacy, generating data points too similar to the original, demonstrating a poor Privacy vs. Utility trade-off. In contrast, CTGAN achieved high privacy scores but struggled with column distributions and correlations, also demonstrating the trade-off.

Tools like PrivBayes allow configurable parameters such as the privacy budget (𝜖), which controls the trade-off between utility and privacy under Differential Privacy (DP). DP adds controlled noise to ensure that including or excluding any single record does not significantly affect outputs, offering strong privacy guarantees. However, evaluating DP mechanisms is beyond the scope of this benchmark.

4.7 Machine learning utility

Machine learning (ML) Utility evaluates how well synthetic datasets replace real ones in training ML models for classification and regression tasks. To assess ML Utility, we used nine classifiers and four regressors, as shown in Table 9. Models were tested on real data to measure predictive accuracy and robustness. The metrics included Accuracy, AUC, and F1 Score (micro, macro, weighted) for classification, and EVS, MAE, and R² for regression (Pedregosa et al., 2011). Accuracy measures overall correctness, AUC assesses class separation, and F1 Score balances precision and recall. EVS captures target variability, MAE measures average error, and R² indicates the proportion of variance explained by the model.

Table 10 presents the results as differences between model performance on real and synthetic data. Zero means equal performance, positive values favor real data, and negative values favor synthetic data. For MAE, lower values are better, so its inverse is shown for consistency. While accuracy and F1 Micro appear similar, accuracy measures overall correctness, while F1 Micro accounts for class imbalance. Similarly, EVS focuses on variability, while R² also penalizes systematic prediction errors.

Table 10 shows that AutoDiff, TabuLa, REalTabFormer, and TabSyn preserve ML utility better. In general, tools perform better in classification than in regression, likely because the discrete nature of classification targets is easier to learn. SMOTE presented a strong classification performance; however, it is closely similar to the real data, which suggests limited record variability.

4.8 Computational performance

Computational performance measures the resource usage and execution speed during synthetic data generation, assessing computational efficiency and practical usability across tools. Advanced tools, such as deep learning-based models, often rely on GPUs, while simpler methods like PrivBayes and SMOTE prioritize lightweight execution without specialized hardware. This comparison highlights whether complex tools justify their resource demands with superior results in other dimensions (e.g., dataset balancing, or ML utility).

Table 11 shows the results for CPU, GPU, and memory usage, as well as runtime, measured by the mean, max, and standard deviation across five runs for all tools and datasets.

PrivBayes is the most resource-efficient tool (106s runtime, 17% CPU, no GPU), while SMOTE (248s, 25% CPU, no GPU) is the best for mixed data. REalTabFormer has the highest runtime (7682s) with high memory (21% mean, 39% max) and GPU usage (28% mean, 77% max), reflecting its complex architecture.

GOGGLE and GANBLR++ show high resource demands due to graphical modeling, with GOGGLE using 45% CPU and GANBLR++ peaking at 88% CPU.

GAN models like CTAB-GAN+ balance efficiency and runtime (863s, 18% CPU, 7% memory), while Transformer models (REalTabFormer, TabuLa) are resource-intensive, with TabuLa reaching 90% GPU usage. Diffusion-based TabDDPM offers balanced performance (11% CPU, 19% memory, 666s runtime). Hybrid models TabSyn (317s runtime, 17% CPU, 19% memory) and AutoDiff (14% CPU, 15% memory) provide efficient resource usage with minimal GPU reliance, maintaining strong performance across evaluations.

4.9 Aggregated benchmark results and discussion

We aggregated the results per model type across the six evaluation dimensions. Metrics were normalized, averaged per tool, and re-normalized to a 1–10 scale, where 10 means best performance, and 1 represents poor performance or failed use cases.

We assigned equal weight to all metrics within each of the six dimensions, aiming to provide general, comparable conclusions across models. We acknowledge that alternative weighting could be applied, depending on the priorities of specific use cases. Figure 6 compares the different TDS model types, using these normalized scores to provide a consolidated view of their overall strengths and limitations.

Figure 6

Sampling models like SMOTE excel in handling imbalance, augmentation, and missing values, with top resource efficiency, making them ideal for fast, lightweight data synthesis. However, their low variability affects ML utility and privacy, as they often generate data too close to the original. While this limits their use in privacy-sensitive or high-variability tasks, their simplicity and low computational demands make them accessible for users with limited hardware, especially for balancing and augmentation needs in large datasets.

Graph Neural Networks (GNNs) like GOGGLE and GANBLR++ perform poorly with our prosumer setup, often failing to converge without high-performance clusters. Their reliance on complex graphical structures introduces computational overhead, making them unsuitable for prosumer hardware. They were still shown in Figure 6 for completeness, but the dimensions where they were excluded were left blank, since the results are not comparable. This is also the case for PrivBayes, which shows that Bayesian Networks are highly efficient, with strong privacy and performance scores, but their limited ability to model complex relationships affects their results for dataset augmentation and balancing, making them better suited for privacy-prioritized applications.

GAN-based models such as CTGAN and CTAB-GAN+ deliver moderate performance, excelling in augmentation and imbalance handling but struggling with missing values and privacy. Despite these challenges, GANs generate realistic synthetic data, provided the data is complete and pre-processed. Diffusion models achieve consistently high scores across all dimensions, balancing privacy, utility, and performance, but require significant tuning and resources for optimal use. Finally, Hybrid and Transformer models stand out with high scores in augmentation, missing values, and ML utility. Transformers (e.g., REaLTabFormer, TabuLa) deliver excellent results but demand substantial computational resources, reflecting their complexity. Hybrid models (e.g., AutoDiff, TabSyn) combine strengths from multiple models, offering balanced, efficient performance suitable for diverse use cases, making them versatile tools for most TDS applications.

4.10 Discussion on hyperparameter tuning

The experiments above were carried out, as mentioned in Section 3, using the parameters tuned by the original authors, without performing additional hyperparameter tuning. We chose this approach to avoid researcher bias and to benchmark the tools in their out-of-the-box configuration.

However, complex TDS models benefit greatly when their hyperparameters are tuned for each dataset. Simple TDS tools, such as SMOTE and PrivBayes, require no hyperparameter tuning because they rely on fixed algorithmic procedures rather than parameterized learning processes. In contrast, GANs are well known to be unstable during training, meaning that key hyperparameters, such as learning rates, optimizer choice, gradient penalty, and batch size, can cause training to collapse or oscillate. (Arjovsky, Chintala, and Bottou, 2017; Mescheder, Geiger, and Nowozin, 2018) For this reason, GAN-based tools typically incorporate stabilization techniques, such as Wasserstein loss or spectral normalization, which should not be, or only very carefully, altered.

Diffusion models, on the other hand, are much more stable to train because their training is explicit likelihood-based, as shown in Figure 1. Yet, their effectiveness is sensitive to the chosen hyperparameters (Kotelnikov et al., 2023), such as the number of diffusion steps, learning rate, or noise schedule. As a result, hyperparameter tuning can significantly improve the performance of diffusion-based models. The situation is similar for Transformer-based tools, which depend heavily on their hyperparameters (Casola, Lauriola, and Lavelli, 2022), such as number of layers, hidden dimension, number of attention heads, or learning rate schedules.

This study acknowledges that a further benchmark including hyperparameter tuning would increase fairness toward certain models, such as Diffusion or Transformers, which are capable of producing superior results when properly tuned. Nevertheless, Diffusion, Transformer-based, and Hybrid models already achieved better performance in our benchmark with respect to correlations and distributions, which are the basis of synthetic data quality. In the aggregated results, they obtained the highest scores across all dimensions, except computational performance.

We expect that including hyperparameter tuning in future experiments will further increase the quality of the generated synthetic data, but at the expense of higher computational cost. Exploring this trade-off is left for future work.