Implementation of Sand cat Swarm Optimization for Uniform T-Way Test Suite Generation

Software systems are deeply embedded in various domains, including finance, health care, aviation, and critical infrastructure, making their reliability a fundamental concern. Demand for software applications has raised the bar for developed software quality assurance [1]. Software products must be dependable and high quality to fulfill every customer’s expectations and requirements [2]. A single software failure can lead to catastrophic consequences, ranging from financial losses and security breaches to safety risks and operational disruptions. Software testing serves as a quality assurance mechanism, systematically identifying defects and vulnerabilities that could compromise system integrity. With an estimated 40% of total development expenses, software testing is an especially expensive portion of the software development process [3]. However, as modern software applications become increasingly complex, traditional method or testing approaches face significant challenges in achieving comprehensive validation within practical time and resource constraints [4].

Ensuring software reliability requires efficient test design strategies that balance coverage and efficiency Interaction-related faults are among the crucial errors that must be disclosed because any breakdown in these interactions would impact main functionalities of the system [5]. Exhaustive testing is impractical due to the exponential growth of test cases, making combinatorial testing a preferred approach for systematically covering parameter interactions because selecting at least one test case must be covered for each t-way (t is the interaction strength) combination of input parameters of a configuration system that cover all t-way interactions among input parameters [6, 7]. The process of combinatorial testing involves defining the input space (sampling the input configuration), constructing optimized test cases that ensure interaction coverage, and executing the test suite to detect faults [8].

Generating optimal combinatorial test suites remains computationally challenging, making difficult combinatorial testing a challenge and a significant research limitation [9]. Metaheuristic algorithms, such as genetic algorithms (GAs), particle swarm optimization (PSO), ant colony optimization (ACO), and whale optimization algorithm (WOAs) have been applied to optimize test suite size and address the minimum covering array generation (MCAG) problem [10]. However, the nondeterministic polynomial (NP) hard nature of test suite optimization, where no single strategy can guarantee that it always produces the best test suite size for all configurations, presents challenges for existing algorithms, creating opportunities for new metaheuristic approaches to improve efficiency and coverage [11]. Current methods struggle with selective pressure, premature convergence, inefficient exploration, and high computational costs, limiting their effectiveness in large-scale configurations [12].

Nonetheless, according to the no free lunch (NFL) theorem, no single optimization algorithm can solve all optimization domains [13]. This suggests that an algorithm’s performance is highly dependent on the nature of the problem, making the integration of metaheuristic algorithms into specific applications, such as t-way combinatorial test generation, an ongoing research challenge. Motivated by this, the present study explores the application of SCSO in t-way testing.

SCSO is a recently introduced metaheuristic inspired by the adaptive hunting behavior of sand cats, which attack or search for prey according to the sound frequency [14]. Unlike traditional swarmbased optimization techniques, SCSO incorporates a stochastic movement strategy that dynamically adjusts its search behavior, effectively balancing exploration and exploitation. It has demonstrated strong performance in finding optimal solutions with fewer parameters and reduced computational overhead [15]. This adaptability makes it particularly suitable for optimization problems that require efficient navigation of large search spaces. While SCSO has been successfully applied to various optimization domains, its potential for combinatorial test suite generation remains unexplored. Given its unique search mechanism, SCSO could serve as an alternative approach for minimizing test suite size while ensuring comprehensive parameter interaction coverage.

Motivated by its reported advantages, this study aims to evaluate the performance of SCSO in t-way combinatorial test suite generation and compare it against existing metaheuristic-based approaches. The remainder of this paper is structured as follows. Section 2 presents a comprehensive review of combinatorial testing strategies and metaheuristic optimization techniques. Section 3 details the methodology, including the adaptation of SCSO for test suite generation. Section 4 discusses the experimental setup, data sets, and evaluation criteria. Section 5 presents the results and analysis, followed by Section 6, which concludes the study and outlines potential future research directions.

Software testing plays a crucial role in ensuring the quality and reliability of modern software systems [16]. As applications become increasingly complex, the number of configurable parameters expands, leading to an exponential growth in possible input combinations. Exhaustive testing (a method that tests all combinations), while theoretically ensuring complete accuracy, becomes impractical due to excessive computational costs and time constraints, as the combinatorial explosion problem makes it unfeasible for a large number of input parameters [12]. To address this challenge, combinatorial testing has emerged as an effective approach, focusing on covering t-way interactions between input parameters, rather than testing every possible combination. This method provides a structured way that effectively reduces the number of test cases needed compared to exhaustive testing while maintaining adequate coverage of parameter interactions [17].

The core principle behind t-way testing is based on empirical observations that show most software failures are caused by the interaction of a small subset of input parameters rather than the entire input space that can reach an incorrect result [18]. Studies have shown that pairwise (2-way) testing can detect a large percentage of defects while increasing the interaction strength (t) enhances fault detection. By systematically covering all possible t-way interactions, combinatorial testing ensures that critical parameter interactions are tested, improving software quality without requiring an infeasible number of test cases. The effectiveness of this approach depends on selecting an appropriate t, where higher values provide better fault detection at the cost of increased test suite size.

Combinatorial test suite generation relies on the construction of covering arrays (CAs), which are mathematical structures that ensure that all t-way combinations appear in at least one test case. A covering array is represented as CA(N; t, k, v), where N denotes the number of test cases, t is the interaction strength, k is the number of parameters, and v represents the possible values each parameter can take [10]. The goal of test suite generation is to minimize N while maintaining full t-way coverage, as a smaller test suite reduces execution time and testing costs. Constructing an optimal CA array is computationally challenging, requiring efficient strategies to generate minimal test suites while maintaining complete interaction coverage.

Exhaustive testing involves evaluating every possible combination of input parameters, which quickly becomes infeasible due to the combinatorial explosion. T-way testing can cover all of the important interaction components at least once, and the size of the test suite is reduced in proportion to the interaction strength, “t” [19]. It can improve the effectiveness of software testing from different configuration systems while simultaneously lowering the anticipated cost and time [20]. It also makes combinatorial testing highly efficient, especially for large-scale software systems where time and cost are major concerns. It allows teams to detect potential issues early, optimize resources, and accelerate product delivery without getting stuck in a never-ending cycle of exhaustive testing [21].

Combinatorial testing can be categorized into two variations based on interaction strength: uniform interaction strength and variable interaction strength [10]. Uniform strength interaction means that all input parameters share the same level of interaction where the CA maintains a consistent interaction strength across all parameters, ensuring uniform test coverage [22]. Variable strength alone guarantees and has more than one interaction strength for generating test cases [22]. It assigns different t-way levels to parameters based on system criticality. This approach optimizes test coverage without unnecessarily increasing test suite size [10].

Metaheuristic algorithms have gained significant attention in combinatorial testing due to their ability to explore large solution spaces and optimize test suite size while ensuring full coverage. It is because it has capable to escape from local optima and perform a robust search of a search space [23]. Algorithms such as GA, ACO, and WOA leverage heuristic-based search techniques are able to find near-optimal test suites under several conditions which are balancing exploration (global search) and exploitation (local refinement) to minimize the number of test cases. These methods have proven particularly useful in large-scale and highly configurable software systems, whereas traditional techniques struggle to provide efficient solutions.

As combinatorial testing continues to evolve, its integration with optimization techniques remains a key area of research. The challenge of generating optimal test suites efficiently requires a balance between computational feasibility and maximizing interaction coverage. Metaheuristic approaches provide promising solutions to this problem by leveraging intelligent search mechanisms to minimize test suite size while maintaining high defect detection efficiency. It can exploit stochastic behavior to search for the best test cases through only several iterations [17]. With the growing complexity of modern software systems, combinatorial testing is becoming an indispensable methodology in ensuring robust and efficient software validation.

2.1.

Problem Definition Model

Figure 1 illustrates the concept of t-way testing using an “online learning system” application. This system filters learning content based on five parameters: user type, device used, level course difficulty, content format, and interaction mode. The user type is categorized into two groups: student and teacher. The device used is classified as mobile phone or laptop, while the course difficulty falls into three levels: beginner, intermediate, and advanced. The content format is video and text, and the interaction mode is categorized as self-paced, live, and hybrid. To simplify the explanation, each parameter is assigned a distinct notation (e.g., U1 and U2 for user type, D1 and D2 for device used, L1, L2, and L3 for level course difficulty). Table 1 provides a simplified representation of the parameters and their possible values for this system.

Table 1.

Input parameter	U	L	D	M	C
Possible Value	U1	L1	D1	M1	C1
	U2	L2	D2	M2	C2
		L3		M3

Figure 1.

Filter for online learning system

To test such a system, exhaustive testing, which involves generating all possible combinations of parameter interactions to achieve complete coverage, could be applied. For the given system, full-strength interaction testing (where t = 5) results in 72 test cases (2 × 3 × 2 × 3 × 2 = 72). While exhaustive testing theoretically ensures complete accuracy, it becomes impractical when dealing with a large number of input parameters due to the combinatorial explosion problem [12]. Figure 2 shows test cases for exhaustive testing based on the online learning system application.

Figure 2.

List of generated test cases for exhaustive testing

To address this issue, the number of test cases can be significantly reduced by applying t-way testing with a lower interaction strength (t). For instance, pairwise testing (t = 2) ensures that all pairs of parameter interactions are tested, thereby reducing the number of test cases while maintaining sufficient coverage. T-way testing focuses on testing parameter interactions up to a defined strength (t). For example, 2-way testing ensures that all pairwise interactions are covered, significantly reducing the number of test cases compared to exhaustive testing. Techniques such as uniform strength (pairwise) testing provide flexibility in prioritizing critical interactions.

In pairwise testing (t = 2), only specific pairs of parameters are tested for interactions, while the remaining noninteracting parameters are assigned random valid values (denoted as DX) to form complete test cases. For the given system, the interacting pairs include UL, UD, UM, UC, LD, LM, LC, DM, DC, and MC. Figure 2 demonstrates the construction of test cases using t-way testing with t= 2. Any duplicate test cases are removed, and only unique combinations are retained as final test cases.

As shown in Figure 3, t-way testing significantly reduces the number of test cases from 72 (in exhaustive testing) to 18, achieving a reduction of over 75%. Despite this reduction, the test cases maintain adequate coverage of parameter interactions, ensuring effective fault detection. This approach not only reduces the number of test cases but also minimizes resource consumption, such as time and cost, making the testing process more efficient.

Figure 3.

Test case generation for t = 2 and list of test cases for t-way

Research in t-way combinatorial testing has significantly evolved, transitioning from traditional computational approaches to modern metaheuristic methods. Computational approaches, rooted in algebraic techniques, have been fundamental in t-way test suite generation. These methods employ heuristic rules and greedy strategies to cover uncovered test combinations [24]. However, while computational methods offer flexibility and support for complex configurations, they struggle with efficiency, as longer computational times are required to handle extensive test combinations [25]. This limitation highlights the need for more scalable solutions. To address these challenges, metaheuristic algorithms have emerged as an effective alternative for t-way test suite generation. Metaheuristics typically begin with a random solution and iteratively apply search techniques to improve fitness. Though not perfectly accurate, these methods efficiently produce near-optimal solutions in less execution time than computational methods. Metaheuristic algorithms enable a more robust and adaptive learning process, ultimately enhancing the network’s ability to solve various challenging problems involving t-way interactions [26].

Within this context, t-way testing strategies are broadly categorized into two fundamental approaches: one-test-at-a-time (OTAT) and one-parameter-at-a-time (OPAT). OTAT starts with an empty test suite, where test cases are added one by one until all interactions are covered. Whenever a test case is chosen, it is included in the final suite (vertical extension). OTAT strategies have been extensively explored in research due to their ease of implementation and ability to generate effective test suites efficiently [19]. On the other hand, OPAT begins with an initial test suite and progressively adds parameters one at a time until all parameters have been incorporated. This method is referred to as horizontal extension, where test cases evolve gradually with additional parameters. After completing the horizontal extension, further test cases may need to be added through vertical extension to ensure full interaction coverage. OPAT provides an alternative approach to test suite generation but requires more complex handling of parameter dependencies, making it less commonly adopted in research [19].

Due to its structured nature and incremental test case construction, OTAT is often considered a more convenient and practical approach for t-way testing. Its systematic process simplifies test generation and reduces computational complexity, making it easier to implement compared to OPAT. As a result, OTAT strategies have gained more attention as one of the most promising research areas in t-way combinatorial testing [10].

Among the early computational methods developed for t-way interaction testing was the Test Configuration Generator (TConfig), introduced by A. W. Williams in the late 1990s [27] for web-based interaction testing. It is classified as a computational approach, since it relies on systematic algorithms. TConfig follows the OPAT approach, constructing test configurations in a structured, step-by-step manner rather than generating all possible test combinations at once. The tool primarily employs combinatorial testing techniques, specifically using CAs to ensure that every pair-wise (or higher-order) combination of parameter values is represented efficiently in the test set. It leverages the recursive block algorithm (Williams) and the in-parameter-order (IPO) greedy algorithm (Lei and Tai) to construct these test configurations, focusing on minimizing test cases while maintaining high interaction coverage.

In 1998, Lei and Tai introduced the IPO strategy [28], which was a groundbreaking computational method for pairwise interactions. It also applies the OPAT approach. Enhancements such as in-parameterorder-general (IPOG) and IPOG-D extended its capabilities to support variable-strength testing up to t = 6, optimizing test suite generation through horizontal and vertical extensions [29, 30]. Subsequent variants like IPOG-F2 and SCIPOG integrated lightweight heuristics and advanced constraint-handling techniques, ensuring efficient and precise test case generation for modern systems [20,31].

At the beginning of the new decade, in 2010, the test vector generator (TVG), developed by Arshem, was introduced as a public domain tool for generating GUI tests using computational t-way interaction testing. It supports three interaction strength types: input-output relationship (IOR), variable strength, and uniform strength. Although it is claimed that TVG can support up to six strengths, practical execution has only achieved up to five [32]. TVG employs a greedy method and OTAT for test case generation and utilizes three algorithms: t-reduced, plus-one, and random set. Among these, t-reduced produces the most optimized test suites, though limited details are available on the workings of each algorithm.

In contrast to these computational approaches, metaheuristic-based t-way interaction testing strategies have gained significant attention and have been widely explored in recent years.

Among the earliest breakthroughs, the particle swarm test generator (PSTG), introduced by Ahmed and Zamli [33] in 2010, utilized PSO as a metaheuristic method for generating t-way combinatorial test suites. Initially designed for uniform interaction strengths, PSTG was improved by the same authors in 2011 [34] to support variable-strength interaction as well. PSTG follows an OTAT approach, incrementally generating test cases to optimally cover interactions through particle swarm operations. PSTG effectively balances global search (exploration) and local search (exploitation) to produce compact test suites. Advantages include simpler implementation, fewer parameters to tune compared to other metaheuristics, and strong performance in generating smaller test sets. However, PSTG may experience higher computational overhead from repeated particle evaluations and typically requires careful parameter adjustments to consistently perform well across different test scenarios.

A year later, in 2011, the harmony search strategy (HSS) [35] was introduced by Alsewari and Zamli as a metaheuristic approach specifically designed for generating t-way test suites. It employs an OTAT method, incrementally building individual test cases to optimally cover interactions. Initially supporting only uniform interaction strengths, HSS was enhanced in 2012 to include support for variable-strength interactions. Its search mechanism balances exploration and exploitation effectively using harmony memory operations. The advantages of HSS include producing comparatively compact test suites, simple implementation, and minimal parameter tuning. Nevertheless, it may incur higher computational costs due to frequent harmony evaluations and can require careful parameter calibration to maintain optimal results across various testing configurations.

Subsequently, in 2015, the cuckoo search (CS) strategy for combinatorial test suite generation was introduced by Ahmed et al. [36]. It is a metaheuristic approach employing the OTAT method, where each iteration produces one complete test case optimized through a stochastic global search. CS leverages Levy flights to efficiently balance exploration (global search) and exploitation (local search), allowing it to effectively generate combinatorial test suites. The strategy supports both uniform and mixed interaction strengths, making it suitable for variable-strength combinatorial testing. Key advantages include minimal parameter tuning, robustness in escaping local optima, and competitive performance with simpler tuning requirements compared to other metaheuristics like GA and PSO. Nevertheless, CS can incur higher computational overhead due to the iterative Levy flight operations, particularly for complex or large-scale test generation scenarios, and it may still experience issues with convergence speed in some contexts.

In the same year, the swarm intelligent test generator (SITG) was introduced by Rabbi et al. [37]. The strategy applies PSO to generate t-way test suites by treating each particle as a complete test case rather than a numerical solution. It follows the OTAT approach, where test cases are added incrementally to maximize interaction coverage. The process begins with a randomly initialized swarm, where each particle represents a candidate test case. Instead of using traditional PSO velocity updates, SITG evaluates each test case based on its ability to cover previously uncovered t-way interactions, adjusting positions accordingly. The best test cases are iteratively selected and added to the final test suite (FTS), ensuring that all required interactions are covered. SITG supports both uniform and variable-strength interactions, with testing conducted up to t = 6. The approach efficiently balances exploration and exploitation, leading to compact test suites that effectively cover interactions, especially for higher-strength scenarios (t ≥ 4). However, it introduces computational overhead due to frequent velocity evaluations and requires careful parameter tuning to prevent premature convergence to suboptimal solutions. Despite these challenges, SITG remains an effective and adaptive strategy for combinatorial test case generation.

In 2016, the high-level hyper-heuristic (HHH) strategy using tabu search was introduced by Zamli et al. [38] for t-way combinational test suite generation. It employs the OTAT approach and explicitly supports both uniform and variable interaction strengths, with reported experimental validations up to t = 6. In its implementation, HHH iteratively generates each test case by dynamically selecting among four low-level metaheuristics: teaching learning-based optimization (TLBO), the global neighborhood algorithm (GNA), PSO, and CS. Differing from conventional single-method strategies, HHH utilizes tabu search to intelligently switch among these metaheuristics using adaptive operators for improvement, diversification, and intensification, effectively balancing exploration and exploitation. Advantages of HHH include flexibility in adapting its search strategy according to problem dynamics, reduced likelihood of becoming trapped in local optima, and improved test suite compactness. However, employing multiple metaheuristics through tabu search significantly increases algorithm complexity, computational overhead, and sensitivity to parameter tuning, potentially limiting its practical applicability without careful calibration.

Following this, in 2017, the adaptive teachinglearning-based optimization (ATLBO) method was introduced by Din and Zamli [39] to enhance the conventional TLBO by integrating fuzzy logic for adaptive combinational t-way test suite generation. ATLBO applies an OTAT approach, incrementally constructing each test case through iterative optimization. In this implementation, candidate solutions represent individual test cases that evolve based on the TLBO mechanism, comprising a global search (teacher phase) and local search (learner phase). Unlike traditional TLBO, ATLBO dynamically selects between these two search phases using a Mamdani fuzzy inference system, which continuously evaluates solution quality, intensification, and diversification measures to decide the optimal search direction at each iteration. ATLBO explicitly supports uniform and variable interaction strengths, with experimental validation reported up to t = 6. Its main advantages over conventional TLBO include increased adaptability, improved convergence speed, and better robustness against premature convergence. Nevertheless, the fuzzy logic integration introduces additional complexity, computational overhead, and the need for carefully designed fuzzy inference rules, potentially complicating practical applications.

Building upon reinforcement learning principles, in 2018, the Q-learning sine cosine algorithm (QLSCA) was introduced by Zamli et al. [40] as a metaheuristic approach specifically tailored for generating combinational t-way test suites using an OTAT methodology. QLSCA constructs test suites by iteratively generating each individual test case through a dynamic search guided by reinforcement learning (Q-learning). In each iteration, candidate solutions are updated by adaptively selecting among four distinct search operations: sine search, cosine search, Levy flight, and elitism. Unlike traditional SCA, where the search operations are determined through fixed parameters, QLSCA dynamically chooses these operations based on a learned Q-value, promoting balanced exploration and exploitation. The method explicitly supports uniform interaction strength and has been experimentally validated up to t = 4. Advantages of QLSCA include improved adaptability, reduced reliance on manually tuned parameters, and enhanced capability to avoid premature convergence compared to traditional SCA. However, incorporating the reinforcement learning mechanism introduces additional complexity, computational overhead, and increased sensitivity related to the learning parameters, which must be tuned carefully to maintain consistent performance.

In 2019, the artificial bee colony for variable strength (ABCVS) strategy was introduced by Alazzawi et al. as a metaheuristic approach derived from the artificial bee colony (ABC) algorithm for combinatorial t-way test suite generation [41]. ABCVS employs an OTAT approach and explicitly supports both uniform and variable interaction strengths, validated up to t = 6. Unlike conventional ABC, ABCVS enhances the balance between exploration and exploitation by dynamically adjusting the number of employed, onlooker, and scout bees based on test case coverage and diversity. This adaptive mechanism ensures optimal interaction coverage while minimizing test suite size. The advantages of ABCVS include improved optimization efficiency, high scalability, and better adaptability in handling complex t-way interactions. Additionally, its swarm intelligence approach helps prevent premature convergence while maintaining test diversity. However, ABCVS requires careful tuning of parameters such as colony size and search limits, and its iterative nature introduces computational overhead, which may impact execution time in large-scale test configurations.

Continuing this trend, in 2020, the ant colony optimization algorithm using fuzzy logic (ACOF) strategy was introduced by Ahmad et al. [42] as an advanced metaheuristic derived from Dorigo’s original ACO. ACOF employs an OTAT approach and supports both uniform and variable interaction strengths, explicitly tested up to t = 6. Unlike traditional ACO, ACOF integrates a Mamdani fuzzy inference system to dynamically adjust two critical parameters: the pseudo-random proportional selection rule and the number of ants utilized per iteration. Specifically, fuzzy logic determines the optimal selection probability and dynamically allocates ant resources, enhancing exploration and exploitation balance. Advantages include generating compact test suites and significantly improving execution time compared to conventional ACO variants, due to its adaptive mechanisms. However, integrating fuzzy logic increases algorithm complexity, requires carefully designed fuzzy rules, and may add computational overhead during the inference process.

In the same year, WOA for t-way test suite generation was introduced by Hassan et al. [43]. This metaheuristic strategy employs the OTAT approach and explicitly supports both uniform and variable-strength interactions, with successful tests conducted for interaction strengths up to t = 6. The WOA method initializes multiple candidate solutions (“whales”) within the combinational search space and iteratively updates their positions using whale-inspired mechanisms such as encircling prey (exploitation) and bubble-net attacking (exploration). This adaptive process helps achieve a balanced explorationexploitation trade-off, enhancing diversity and efficiency in generating optimized test suites. However, the approach faces potential computational overhead due to iterative candidate updates and can experience excessive exploration, which may slow convergence.

Advancing further, in 2021, the gravitational search test generator (GSTG) strategy was introduced by Htay et al. [11], based on the gravitational search algorithm (GSA), which is a populationbased metaheuristic inspired by Newtonian gravity. GSTG uses the OTAT approach, iteratively selecting the best test case by mimicking gravitational interactions between candidate solutions, with each candidate represented as a test case. The algorithm uniquely employs gravitational forces to guide less optimal solutions (lighter masses) toward optimal ones (heavier masses), thus effectively balancing exploration and exploitation. GSTG supports both uniform and variable interaction strengths, explicitly tested up to t=10. Its advantages include strong exploration capabilities, effective avoidance of local optima due to its gravity-inspired search mechanism, and competitive performance in generating optimal or near-optimal test suites. However, GSTG has disadvantages, such as higher computational complexity, especially with increasing interaction strength or parameters, and sensitivity to parameters like gravitational constants, requiring careful tuning for optimal results. These issues are consistent with typical challenges noted in the literature regarding GSAs.

More recently, in 2022, the improved particle swarm optimization (improved PSO) strategy for t-way test suite generation was introduced by Prasad et al. [44]. It is a metaheuristic method extending traditional PSO, specifically adapted for generating t-way combinatorial test cases. Improved PSO follows the OTAT approach, uniquely simplifying the particleupdate mechanism by directly adjusting particle positions based on coverage of uncovered interactions. Unlike conventional PSO, it eliminates reliance on typical parameters such as inertia weight, acceleration coefficients, and complex velocity calculations, significantly reducing the need for extensive parameter tuning. The strategy supports both uniform and variable interaction strengths, explicitly tested up to t = 6. Its advantages include easier implementation, fewer parameters to tune, and notably improved performance and efficiency compared to conventional PSO. However, disadvantages include computational overhead due to iterative particle evaluations and sensitivity to parameter adjustments, potentially causing premature convergence and trapping in local optima, consistent with common challenges highlighted in PSO-related literature.

Most recently, in 2024, the wingsuit flying search (WFS) optimization algorithm for t-way test suite generation was introduced by Rose et al. [45] as a metaheuristic, parameter-free approach inspired by wingsuit flying. WFS employs a unique three-phase implementation: generating initial points via Halton sequences, adaptively adjusting neighborhood sizes, and progressively narrowing the search space through decreasing discretization steps. It follows the OTAT approach, iteratively building each test case. WFS supports both uniform and variable interaction strengths, explicitly tested up to interaction strength t = 10. Its main advantages include ease of use due to its parameter-free nature, efficient performance, and the capability of producing compact test suites without extensive parameter tuning. However, it tends to heavily emphasize exploitation in later stages of optimization, increasing the risk of convergence to local optima, especially in complex test configurations [46].

This continuous evolution from computational methods to modern metaheuristic-based strategies highlights the ongoing advancements in t-way interaction testing techniques, significantly improving efficiency and adaptability over time.

Figure 4 shows the recommended framework for SCSO’s approach. To achieve the SCSO method, a number of components are created and used, including the test case generator (TCG) and tuple generator (TG) where TG is derived from [24, 47]. The framework consists of three main sections. The first section, input parameter setting, defines the test parameters, their values, interaction parameters, and interaction strength, which serve as the foundation for generating test cases. The SCSO framework support only uniform strength, which are implemented using the OTAT test case generation approach. The second section, SCSO strategy, involves the TG, producing a tuple list (TL) representing possible parameter combinations. These tuples are then processed by the TCG, which incorporates the SCSO algorithm to generate optimized test cases. The final section, output, delivers the final test suite, ensuring that the generated test cases comprehensively cover the defined interactions.

Figure 4.

SCSO’s framework

SCSO is a bio-inspired optimization algorithm modeled after the hunting and survival strategies of sand cats, called “Felis margarita.” The SCSO concept was first introduced by Seyyedabbasi and Kiani to solve optimization problems [15]. This novel approach combines unique behavioral patterns of sand cats to solve optimization problems efficiently Exploration and exploitation phases based on the hunting behavior of sand cats is shown in Figure 5.

Figure 5.

Exploration and exploitation of SCSO

The process begins by initializing the population, determining the population size required for the SCSO to achieve the best outcome. Based on the requirements, a population size of 10 seems sufficient for the iterations, as it can yield the best possible results efficiently. Furthermore, it will take a long time if the set is more than 10. The associated structure is a vector, and SCSO is thus a population-based technique. Each possible solution is determined by evaluating a predetermined fitness function. The SCSO will determine the optimal values of the parameters based on this function’s definition of the problem’s parameters.

4.1.

Global Sensitivity Factor, rG→ \overrightarrow {{{\rm{r}}_{\rm{G}}}}

1 rG→=SM−(SM×iterciterMax) \overrightarrow {{r_G}} = SM - \left( {{{SM \times ite{r_c}} \over {ite{r_{Max}}}}} \right)

The SCSO algorithm introduces a dynamic global sensitivity factor from Eq. (1) to balance exploration and exploitation effectively. The sensitivity is calculated, where SM = 2 inspired by sand cats’ 2 kHz hearing ability. High initial sensitivity ensures broad exploration during early iterations, covering diverse regions of the search space. As iterations progress, the global sensitivity factor decreases gradually, shifting the algorithm’s focus to exploitation by refining solutions. Toward the final iterations, it approaches zero, minimizing exploration and emphasizing intensification. This adaptive behavior mirrors sand cats’ transitioning from scanning for prey to capturing targets, ensuring efficient navigation through the search space.

4.2.

Sensitivity for Each Sand Cat

2 r→=rG→×rand(0,1) \vec r = \overrightarrow {{r_G}} \times r\,and(0,1)

Each sand cat’s individual sensitivity r is dynamically adjusted using Eq. (2), where r scales the agent’s responsiveness and rand(0,1) introduces randomness. This stochastic variability allows agents to explore diverse regions of the search space effectively while maintaining adaptability. The sensitivity mechanism ensures a balanced search dynamic, enabling agents to avoid local optima while still refining promising solutions. By mimicking sand cats’ natural adaptability to environmental cues during hunting, this mechanism enhances the robustness and diversity of the optimization process.

4.3.

Decision Parameter (R)

3 R=2×rG→×rand(0,1)−rG→ R = 2 \times \overrightarrow {{r_G}} \times r\,and\,(0,1) - \overrightarrow {{r_G}}

The decision parameter R from Eq. (3) determines whether agents explore new areas (R > 1) or exploit known solutions (R ≤ 1) during each iteration. It is calculated, where r G → \overrightarrow {{r_G}} is the global sensitivity factor.

If R > 1, the agent enters the exploration phase, where it moves to less-visited regions of the search space to promote diversity and avoid premature convergence. In this phase, the position of the agent is updated using the formula below.

4 r→·(Posbc→(t)−rand(0,1)·Posc→(t)) \vec r\,\cdot\,\left( {\overrightarrow {Po{s_{bc}}} (t) - r\,and\,(0,1)\cdot\overrightarrow {Po{s_c}} (t)} \right)

Here, r→ \vec r represents the sensitivity factor that scales the agent’s movement, ensuring its adjustments align with the global search dynamics. Posbc→(t) \overrightarrow {Po{s_{bc}}} (t) (t) refers to the position of the best candidate solution at iteration t, serving as a reference for guiding exploration. rand(0,1) introduces randomness to the movement, adding stochastic variability, while Posc→(t) \overrightarrow {Po{s_c}} (t) (t) represents the current position of the agent. The subtraction Posbc→(t)−rand(0,1)·Posc→(t) \overrightarrow {Po{s_{bc}}} (t) - r\,and(0,1)\cdot\overrightarrow {Po{s_c}} (t) creates a directional vector that leads the agent toward unexplored areas of the search space, with the random term ensuring unpredictable movement patterns. This calculated randomness enables the agent to explore a broader region effectively, increasing the probability of discovering better solutions while avoiding local optima. This formula encapsulates the essence of balancing guidance from the best solutions and randomness, making exploration robust and efficient in SCSO.

On the other hand, if R ≤ 1, the agent enters the exploitation phase, which focuses on refining its current position by moving toward promising solutions. The agent’s position is updated using Eq. (5).

5 Posb→(t)−Posrnd→cos(θ)·r→\overrightarrow {Po{s_b}} (t) - \overrightarrow {Po{s_{rnd}}} cos\,(\theta )\,\cdot\,\vec r

The position update mechanism leverages both deterministic and stochastic components to guide the agent’s movement during the search process. Here, Posb→(t) \overrightarrow {Po{s_b}} (t) represents the best-known position at iteration t, serving as a key anchor point for the agent to refine its search toward optimal solutions. Posrnd→ \overrightarrow {{\rm{}}Pos{{\rm{}}_{rnd}}} introduces an element of randomness by selecting a random position within the search space, ensuring variability in movement. The cosine function, cos(θ), incorporates the angle θ, a randomly chosen value between 0° and 360°, to determine the direction of the agent’s movement. This angle is further multiplied by r→ \vec r, the sensitivity factor, which scales the movement magnitude. Together, these terms ensure that the agent’s motion is both guided by the best solutions and randomized to maintain diversity and avoid convergence on suboptimal regions.

The random angle θ is selected using the roulette wheel selection algorithm, a probabilistic approach that enables dynamic and unbiased selection of movement directions. By randomly selecting θ, the agent’s path is less predictable, reducing the likelihood of being trapped in local optima. The cosine function applied to θ introduces controlled oscillations in the movement trajectory, allowing the agent to explore more refined areas of the search space without losing diversity. R dynamically regulates the ratio of exploration to exploitation, making sure that the search agent both investigates novel possibilities and converges on the best answers. Achieving an ideal search procedure in optimization issues requires striking this balance.

Figure 6 describes the SCSO algorithm for selecting an optimal test suite based on test coverage. The process begins by initializing a population of search agents and calculating their fitness using a test coverage-based function. Parameters r and R are also initialized. The algorithm iterates while the TL is not empty. Within each iteration, every search agent selects a random angle using a roulette wheel selection mechanism. Based on the absolute value of R, the search agent updates its position using either the exploration equation [Eq. (5)] if | R| ≤ 1 or the exploitation equation [Eq. (4)] otherwise. After updating positions, the algorithm evaluates the fitness of all search agents and selects the best test case candidate. If the selected candidate improves coverage, it is added to the FTS, and the covered interactions are removed from TL. The population is then updated based on the best solution. This process continues until all interactions in TL are covered, and the algorithm terminates, returning the optimized FTS.

Figure 6.

Pseudocode for Sand Cat Swarm Optimization Algorithm

The SCSO algorithm is developed and compiled using the Java programming language within the Eclipse 2024 (4.34.0) software. The running environment for the project is a desktop PC operating on Windows 11, equipped with a 2.19 GHz Intel® Core™i7-12700F CPU and 32 GB of RAM. The three groups of experiments are as follows:

Group 1: CA(t,v⁷), t varied from 2 to 6 and v varied from 2 to 5 based on [38,43,45,47].

Group 2: CA(t, 2¹⁰) based on [38,44,45,47].

Group 3: CA(t, 3⁷) based on [41,44].

This project focuses solely on uniform strength configurations, as SCSO is a newly introduced approach to t-way testing. To evaluate its performance in generating test suite sizes for uniform strength, three groups of experiments were conducted, based on configurations derived from two separate journal papers. These experiments benchmarked the SCSO against results from various existing strategies discussed in Section 2. The experimental configurations were adapted from prior works by [38, 41, 43–45, 47], where the SCSO has been executed and benchmarked with those various experiments results. The results of these experiments are documented and summarized in Table 2, Table 3, and Table 4. Detailed experimental configurations for each group are presented accordingly.

The results of all experiments are presented in the provided tables, with the best test suite sizes highlighted in bold and cells are darkened for clarity. Cells marked as “X” indicate that results are unavailable or not reported in the respective articles. Based on the data in Table 2, SCSO delivers good competitive performance across all configurations of t and v.

The metaheuristic-based techniques generally outperform computation-based strategies in Table 2. Among these techniques, the HHH strategy demonstrated the highest performance, achieving the best results in 40% (8 out of 20) of the test cases. Following closely, ACOF secured 30% (6 out of 20), while QLSCA performed well with 25% (5 out of 20). Strategies such as GSTG, WOA, and HSS each accounted for 20% (4 out of 20) of the top results, whereas ATLBO followed with 15% (3 out of 20). Meanwhile, WFS, PSTG, CS, TConfig, and IPOG each contributed to 10%.

(2 out of 20) of the best-performing cases. However, SCSO, t-way test suite generation strategy based on ant colony algorithm (TTSGA), and TVG recorded the lowest performance, failing to achieve the best results in any of the test cases (0%).

The analysis of Table 3 reaffirms the dominance of ACOF and GSTG, both achieving the highest performance with 60% (3 out of 5) of the test cases, setting a strong benchmark. Meanwhile, HHH and HSS demonstrated balanced performance, each securing 40% (2 out of 5) of the best test cases. In contrast, WFS, CS, improved PSO, SITG, and TTSGA trailed behind, contributing to only 20% (1 out of 5) of the test cases. The lowest-performing strategies in this group were SCSO, PSTG, IPOG, TConfig, andTVG, all of which failed to secure any top-ranking test cases (0%).

Table 4 highlights the dominance of improved PSO, which achieves the highest performance with 80% (4 out of 5) of the test cases, outperforming all other strategies. Meanwhile, SITG, ABCVS, and TConfig each secure 20% (1 out of 5) of the test cases, demonstrating a more limited impact. SCSO, despite not leading the group, manages to contribute 40% (2 out of 5) of the best test cases, showcasing moderate performance and competitive potential. In contrast, IPOG fails to secure any top-ranking test cases (0%), making it the least effective strategy in this configuration. This pattern of results reinforces improved PSO’s dominance, the balanced yet limited success of SITG and TConfig, and the moderate competitiveness of SCSO, while IPOG remains at the bottom, struggling to deliver optimal results.

SCSO demonstrates consistent competitiveness across multiple groups, showing its potential in diverse test scenarios. While it performs moderately and has yet to surpass the top-ranked metaheuristic strategies, its stability across configurations highlights its strength. This suggests that with further refinements or hybridization with other metaheuristic techniques, SCSO could achieve even greater optimization and emerge as a strong contender among the top-performing strategies.

The performance of SCSO has been evaluated using statistical analyses to assess its effectiveness in generating optimal test suite sizes for uniform strength interaction test. Two nonparametric tests, the Wilcoxon Rank test and the Friedman test, were employed due to the small sample size and nonnormal distribution of results. The Wilcoxon Rank test analyzed paired strategies with SCSO to determine statistically significant differences at a 95% confidence level (α = 0.05). A null hypothesis was used, where a p-value ≤ α indicated significant differences, leading to the null hypothesis being rejected. Conversely, a p-value > α retained the null hypothesis, suggesting no significant difference. The Friedman test ranked strategies based on mean rank, with smaller values indicating better performance in generating test suite sizes when the null hypothesis was rejected.

A total of 30 results from three groups of experiments were analyzed across Table 5 to Table 6, although some configurations were excluded due to unavailable data for certain strategies. These analyses highlight SCSO’s competitive performance compared to other strategies, particularly in scenarios where it demonstrated statistically significant advantages or achieved better mean rankings in test suite generation.

Table 5 presents the results of testing conducted using the Wilcoxon and Friedman tests, involving data from Groups 1 and 2. A total of 25 test types were analyzed, with the results offering insights into the comparative performance of different strategies. Table 5 also provides insights into whether the null hypothesis was retained or rejected for each paired strategy involving SCSO. For strategies like ACOF, TTSGA, HHH, HSS, and GSTG, the null hypothesis is rejected, meaning these strategies significantly outperform SCSO. For instance, the pairing with ACOF shows a p = 0.002, confirming a clear advantage for ACOF. Similarly, the null hypothesis is rejected when SCSO is compared to TVG, IPOG, WOA, QLSCA, and ATLBO, but in these cases, it is SCSO that demonstrates superior performance, as indicated by its lower mean rank.

In comparisons with PSTG, CS, TConfig, and WFS, the null hypothesis is retained, signifying no statistically significant differences. This implies that SCSO performs similarly to these strategies. For example, against PSTG (p = 0.055) and CS (p = 0.083), SCSO neither significantly outperforms nor underperforms, reflecting comparable performance levels.

When the null hypothesis is rejected, it means one strategy is statistically better than the other based on the test results. For SCSO, this occurs when it shows clear advantages over strategies like TVG, IPOG, and others, or when it is outperformed by stronger strategies like ACOF and TTSGA. On the other hand, retaining the null hypothesis, as seen in its comparisons with PSTG, CS, and WFS, suggests that SCSO is on par with these strategies, offering neither significantly better nor worse performance.

The majority of the paired strategies that outperformed SCSO, such as ACOF, TTSGA, HSS, improved PSO, and GSTG achieved better rankings primarily because they integrate additional techniques, such as hybridization with other optimization algorithms, rule-based enhancements, or ensemble approaches. These combinations allow them to improve search efficiency, balance exploration and exploitation, and adapt to problem-specific constraints more effectively. For example, ACOF benefits from the selforganizing behavior of ACO while incorporating fuzzy logic to handle uncertainty and improve decisionmaking. The fuzzy rules help refine the solution space more effectively, reducing randomness and increasing convergence speed, making ACOF superior to SCSO.

Another reason these strategies appear superior is that HHH inherently carries four different metaheuristics, which are TLBO, GNA, PSO, and CS, while SCSO operates as a standalone approach. HHH’s adaptive mechanism selects the most suitable metaheuristic at different stages of optimization, ensuring a well-balanced search process. This multi-metaheuristic structure naturally provides an advantage in performance, as it allows HHH to adapt to different problem complexities dynamically. However, this also makes comparisons with SCSO unfair, as HHH benefits from the combined strengths of multiple algorithms, whereas SCSO is evaluated based on a single optimization framework.

ATLBO holds a significant advantage due to its adaptive teacher and learner phases, which dynamically adjust using a fuzzy inference system. Unlike SCSO, which follows a fixed optimization structure, ATLBO intelligently adapts its search strategy based on real-time conditions, allowing it to balance exploration and exploitation more effectively. The teacher phase drives global improvements by guiding learners toward better solutions, while the learner phase enhances local refinement through peer-to-peer learning. This adaptability not only improves solution quality but also reduces the risk of premature convergence. However, this advantage also makes direct comparisons with SCSO less fair, as ATLBO’s superior flexibility comes from its additional fuzzy rule-based decision-making system, which SCSO does not incorporate. Essentially, ATLBO benefits from an embedded adaptive learning mechanism, making it more versatile yet computationally demanding compared to SCSO.

Table 6 presents the results of testing conducted using the Wilcoxon and Friedman tests, involving data from Groups 2 and 3 with total of 10 types of testing. The table presents a statistical analysis comparing the performance of SCSO against four other strategies: improved PSO, TConfig, SITG, and IPOG. The comparison is based on the Wilcoxon Rank test and the Friedman Mean Rank test. The results indicate that SCSO demonstrates comparable performance to improved PSO, TConfig, ABCVS, and SITG, as the p- values for these comparisons (0.944,0.086,0.715, and 0.513, respectively) are greater than the significance threshold of 0.05, leading to the retention of the null hypothesis. While SCSO shows slight advantages in certain samples, the Friedman Mean Rank test confirms that the differences are not statistically significant. However, when compared to IPOG, SCSO exhibits a clear statistical advantage, with a p-value of 0.005 leading to the rejection of the null hypothesis. SCSO outperforms IPOG in all samples and achieves a much better Friedman Mean Rank, solidifying its superiority in this comparison.

The analyses highlight that SCSO demonstrates a balanced performance, holding its ground against some strategies while excelling over weaker ones. SCSO shows comparable performance to Improved PSO, TConfig, SITG, PSTG, ABCVS, and CS, as indicated by retained null hypotheses and similar mean ranks. However, it decisively outperforms IPOG and TVG, with statistically significant results and superior Friedman Mean Ranks. Conversely, SCSO struggles against stronger strategies like ACOF, TTSGA, HHH, and HSS, which achieve better rankings in direct comparisons. Overall, SCSO proves to be a robust and competitive strategy, particularly effective against less dominant methods while maintaining reliability in varied contexts.

This paper presents the first implementation of SCSO in t-way testing for test suite generation. The results indicate that SCSO performs competitively, outperforming 15.79% of competing strategies, matching 42.11%, but being outperformed in 42.11% of cases. While SCSO shows strengths in handling uniform interaction strengths, it struggles against more advanced techniques such as ACOF, TTSGA, and WOA, highlighting its limitations in maintaining effective exploration throughout the search process.

SCSO mimics sand cat hunting behavior, where movement intensifies as the search progresses toward an optimal solution. However, as it nears the global optimum, its search radius contracts, leading to reduced exploration and increased reliance on exploitation. This transition increases the risk of premature convergence, causing SCSO to become trapped in local optima, particularly in complex search spaces. The benchmark results support this, as SCSO underperforms in nearly half of the cases, suggesting that an improved balance between exploration and exploitation is needed.

To enhance SCSO’s effectiveness, future work should focus on improving its exploration capability. Hybridizing it with global optimization techniques such as simulated annealing, genetic algorithm, or Levy flight could help mitigate premature convergence and improve search diversity. Additionally, extending its application to variable-strength and input-output relationship testing could enhance its adaptability. Exploring integration with other metaheuristic strategies may further refine its efficiency, making SCSO a more competitive approach for t-way test suite generation.