Investigating Auditory–Visual Perception Using Multi-Modal Neural Networks with the SoundActions Dataset

2.1 Auditory–Visual theory

Musicological theories of sound perception have evolved over the last decades, supported by the development of sound technologies. Some of these theories have been combined with film theories and related technologies. In this subsection, we briefly introduce Pierre Schaeffer’s concept of reduced listening, sound objects, and Michel Chion’s synchresis principle and the phenomenon of added value.

2.1.2 Schaeffer’s typology of reduced listening

One of the challenges of reduced listening is to find a terminology to describe the sound itself—that is, without mentioning any causal or semantic aspects such as events, objects, actions, materials, and environments, we need to look for new descriptions of the properties of the sound. Starting from the foundation of sound and temporality, Pierre Schaeffer introduced a universal typology for describing the temporal property of sound: impulsive, sustained, and iterative. Figure 1 illustrates the energy curve of these three types of sounds and their corresponding sound‑producing actions: an ‘impulsive’ event is a short but abrupt action/sound (e.g., clapping), a ‘sustained’ event is a long sound with stable action/sound energy (e.g., a legato note of a violin), and an ‘iterative’ event is a long sound but with fluctuating sound energy, while the action energy can be either fluctuating or stable (e.g., a vibrato note or an engine sound).

Figure 1

Despite being a valuable theoretical framework for electroacoustic music analysis, musicologists have also questioned Schaeffer’s typology. One of the major criticisms is that Schaeffer’s typology focuses on separable, ‘monophonic’ objects and thus cannot ‘handle the “polyphony” of superimposed sound objects nor more complex music productions featuring continuously evolving elements’ (Lartillot, 2024, p. 275). Musicologists have proposed various types of new analysis frameworks as improvements of Schaeffer’s typology, including R. Murray Schaefer’s ‘soundscape’ analysis (Schafer, 1993), Denis Smalley’s Spectro‑Morphology (Smalley, 1986), Stephane Roy’s Hierarchical and Functional Analysis (Roy, 2004), Lasse Thoresen’s Graphical Formalisation (Thoresen and Hedman, 2007, 2015), Ecrin’s Audio‑Content description (Geslin et al., 2002), Structural and Functional Analysis (Delalande et al., 1996; Thoresen and Hedman, 2007), and Pierre Couprie’s Morphology (Couprie, 2003). These theories addressed and improved the drawbacks of Schaeffer’s typology. Still, some act more like personal compositional frameworks instead of generalized theories, while some are piece‑by‑piece analysis tools that are not categorizable or computable.

2.2 Audio–Video datasets and labeling

Nowadays, recording audio–video data is easy thanks to the widespread availability of high‑quality personal recording devices and access to online streaming platforms for sharing. However, collecting an audio–video dataset still requires much effort to standardize the content, define classes, and organize metadata. Table 1 shows the popular audio–video datasets that provide labels. This section will briefly introduce their characteristics.

Gemmeke et al. (2017) introduced AudioSet, a large‑scale dataset of sound events with manual labels. The dataset contains over 2 million audio–video samples with a summed duration of more than 5,800 h, collected via YouTube. Apart from the data, they also introduced an ontology of hierarchical sound events that greatly influence sound event detection. As the top layer of the AudioSet ontology, they classified sound events into seven main categories: (1) human sounds, (2) animal sounds, (3) natural sounds, (4) music, (5) sounds of things, (6) source‑ambiguous sounds, and (7) channel, environment, and background sounds. As a continuation of some classical sound event detection works (Gaver, 1993; Nakatani and Okuno, 1998; Sager et al., 2016; Salamon et al., 2014), the AudioSet ontology focuses on causal listening (Chion and Murch, 2019a, pp. 22–34), which means the labels describe the sound‑producing action or the material of the sound‑producing objects. The AudioSet ontology also does not include visual aspects of the events, such as positional information and surroundings.

Tian et al. (2018) introduced the AVE dataset, a subset of AudioSet data with frame‑level event labels. The AVE dataset contains 4,143 video clips covering 28 event categories, which is also a subset of the AudioSet ontology. While the size and number of categories are smaller than AudioSet, the frame‑level labels enable a new type of task, auditory–visual event localization (AVEL), which recognizes the type of event and the duration and timing of the event. Another sub‑dataset is the LLP (Tian et al., 2020), containing 11,849 video clips of 25 categories. The LLP dataset has specified separate time information on the audio and video streams since the sounds and related actions might happen simultaneously. For example, a video clip might have the sound of a dog first, and the figure of the dog appears after one second. In this case, the ‘dog’ label for audio and video would have different start times. Tian et al. (2020) introduce a new task called ‘audio‑visual video parsing’ (AVVP), in which the models give separate output on sound and visual events. Although the timelines are separated, the ontology of audio and video labels in the LLP dataset remains the same subset of the AudioSet ontology.

The AudioSet data and ontology have widely influenced many other datasets. Meanwhile, datasets with different ontologies are also created to meet the needs for various tasks. The Kinetics‑400 dataset (Kay et al., 2017) focuses on human actions, containing 306,245 clips in 400 action classes. The actions ontology is first gathered from previous action and motion capture datasets and then added or modified during the crowd‑sourcing labeling process by the Mechanical Turk workers. The VGGSound dataset (Chen et al., 2020) contains over 200,000 clips in 300 classes, with most class labels containing two parts of information: the action and the object, for example, ‘playing violin’ or ‘llama humming.’ For inseparable events, for example, ‘thunder,’ there is only one descriptor in the label. Unlike AudioSet, the VGGSound dataset uses a flat hierarchy in which each sample only contains one label layer.

The datasets mentioned above were collected from YouTube, which is a good source for building large datasets; however, there are many challenges regarding privacy, copyright, quality control, and reproducibility. Huh et al. (2023) and Damen et al. (2022) collected a large dataset of original, manually labeled, egocentric audio–video recordings of kitchen activities. The dataset contains 117,000 frame‑level segments of events totaling more than 100 h in duration. The EPIC‑KITCHENS and EPIC‑SOUNDS datasets introduce a more detailed ontology. First, they separated visual events and auditory events in both temporal and semantic dimensions. For example, the action of ‘cutting a tomato’ is only recognizable as a visual event. In contrast, the auditory event is ‘cutting objects,’ and the visual and auditory events can have different durations. Reviewers assigned auditory labels only heard the sound of the samples during labeling, and reviewers assigned video labels only saw the video of the samples. Second, they classified events into two types, actions and collisions of different materials, which clarifies whether the auditory–visual event is caused by an action or by some objects.

Finally, the MediaEval community proposed several tasks that are relevant to audio–video information retrieval, including E‑Sports game storytelling (Lux et al., 2019), Multimedia for Recommender System (Deldjoo et al., 2019), visual and audio analysis of movies video for emotion detection Batziou et al. (2018), and Multi‑modal Person Discovery in Broadcast TV (Poignant et al., 2015). While not designed explicitly for data‑rich ML approaches, these tasks and datasets show the potential for audio–video analysis in more professionally focused applications.

The ontologies used by existing datasets primarily fall into Schaeffer’s category of causal listening or causal perception. To our knowledge, CMU‑MOSEI (Bagher Zadeh et al., 2018) is the only audio–video dataset with non‑causal labels focusing on human facial emotions. This could be seen as falling into Schaeffer’s definition of semantic perception. In the existing audio–video datasets, there are very few descriptions of the auditory or visual characteristics of the audio–video data. For example, one clip in the AudioSet class ‘music’ could be a quiet, middle‑frequency‑ranged chamber music piece without much visual action. Another sample with the same label could be a loud, all‑frequency‑band experiment noise music piece with abrupt visuals. To our knowledge, no existing audio–video dataset provides reduced‑type labels.

2.3 Audio–Video machine learning models

Before the latest revival of contrastive multi‑modal learning led by CLIP (Radford et al., 2021), the first attempts in multi‑modal deep learning (Ngiam et al., 2011; Yuhas et al., 1989) were actually on audio and video data. The pioneer works (Alwassel et al., 2020; Arandjelovic and Zisserman, 2017; Aytar et al., 2016; Korbar et al., 2018; Ma et al., 2021; Owens et al., 2016; Tian et al., 2021) experimented with two main issues: different network structures of information fusion (early‑, mid‑, and late‑fusion), and the understanding of cross‑modal temporality by adding recursive or sequential networks. In the age of transformers, latent Audio‑Visual Hybrid Adapters (LAVisH) (Duan et al., 2023; Lin et al., 2023) have become the most powerful technique in audio–video models since they solve two problems at the same time: parameter‑efficient fine‑tuning (PEFT) and crossmodal information fusion. With the help of LAVisH, it is now possible to use existing large vision and audio transformer models pre‑trained on large datasets and perform fine‑tuning on smaller datasets in various downstream tasks, including AVEL, AVVP, Audio‑Visual Segmentation, and Audio‑Visual Question Answering.

Despite the rapid development in model structures and training efficiency, the limitation of existing datasets—all being causal or semantic—makes it hard to test the models’ ability to recognize reduced listening components in audio–video samples. Furthermore, even though modality comparisons are often included in the ablation studies, existing works usually only compare three types of situations, using (1) audio, (2) video, and (3) audio and video. It is hard to validate whether the models have learned synchresis or the added‑value phenomenon based on these three cases alone. We will further discuss the gap between audio–video learning and auditory–visual theory in Section 2.5.

2.4 Deep ensemble learning and ensemble of PEFT

Deep ensemble learning is a technique that combines the parameters or outputs of multiple deep learning models to achieve superior results compared to the individual models (Ganaie et al., 2022). As its continuation after the PEFT methods (Xu et al., 2023) becomes widely adopted, some recent works attempt to ensemble the PEFT modules instead of the entire model. Kim et al. (2024) investigated the ensemble of sequential (Houlsby et al., 2019) and parallel (He et al., 2022) adapters, showing superior performance in multiple language or vision tasks compared to single adapters. Kwok et al. (2024) and Lin et al. (2024) extend the investigation into a broader range of PEFT methods, including LoRA (Hu et al., 2022) and PiggyBack (Mallya et al., 2018). Kwok et al. (2024) conclude that the ensemble of multiple different PEFT methods, that is, LoRA+parallel adapter, outperforms the ensemble of the same PEFT method with different random seeds. Since the development of the ensemble of PEFT and audio‑visual adapters happened around the same time, no attempts have been made to apply the ensemble method to audio‑visual adapters yet.

2.5 The gap between audio–video learning and auditory–visual theory

Musicologists and ML scientists share an interest in auditory–visual phenomena, but they study them using widely different theoretical and methodological approaches. In this section, we discuss three major issues worth considering when working between the disciplines: the difference in the definitions of ‘multi‑modality’ and ‘audio‑visuality,’ the lack of reduced listening audio–video datasets, and comparing the concept of modality imbalance and added value.

3.1 The soundactions dataset

The SoundActions dataset¹ is a collection of 365 selected real‑world, action–sound pairs with various causal and reduced listening labels. Each recording in the dataset focuses on a single action or a continuous series of actions. Most recordings are short, around 5–10 s, although some are considerably longer. Figure 2 shows a summary of all SoundActions samples, and Figure 3 shows the durations of samples in the SoundActions dataset.

Figure 2

Figure 3

3.1.1 Data collection, naming, and cleaning

The last author made one recording for the SoundActions dataset daily throughout 2022, hence the 365 recordings. All recordings were made with a Samsung S21 Ultra mobile phone and a Røde VideoMic Me‑C microphone. In most recordings, the phone stood on a tripod, with the microphone facing the sound‑producing object. All recordings were made in real‑world settings, but care was taken to reduce the background noise. Figure 4 shows the recording process of one sample in the SoundActions dataset.

Figure 4

Since the recordings were made over the span of a year, seasonal actions include walking in the snow in the winter and trimming bushes in the summer. The actions in the dataset are pre‑selected and made as diversified as possible under the idea of the environment–action–object typology (Jensenius, 2022), described as follows:

• Environment: The action’s surroundings. We included nine classes of environments: ‘outdoor,’ ‘office,’ ‘room,’ ‘corridor,’ ‘hall, ‘bathroom,’ ‘kitchen,’ ‘car,’ and ‘other indoors.’

• Action: The (inter)action that leads to sound production with the object(s). Actions are described as free texts, for example, ‘stepping,’ ‘cutting,’ ‘pressing.’

• Object: The sound‑producing object(s) involved in the interaction. Sometimes, it is one object; other times, it is two or more equal objects (e.g., two stones hitting each other); and other times, it is a tool interacting with an object (e.g., a drumstick). Objects are described as free texts, for example, ‘snow,’ ‘button,’ ‘hands,’ ‘key.’

In addition to the listed typologies, we also included 15 classes of materials: ‘metal,’ ‘wood,’ ‘plastic,’ ‘organic,’ ‘skin,’ ‘stone,’ ‘fabric,’ ‘rubber,’ ‘liquid,’ ‘snow,’ ‘ceramic,’ ‘paper,’ ‘leather,’ ‘electronic,’ and ‘others.’

After recording, the files were manually trimmed (using FFmpeg to avoid recompression) to contain only one sound‑producing action or a series of actions forming one coherent sound object, following Schaeffer’s theory. The file names contain information about the sound‑producing action and objects, such as ‘cutting paper’ or ‘pouring rice.’

3.2 Audio–Video fine‑tuning on reduced‑type labels

The first experiment was to use an audio–video deep learning model to recognize the reduced‑type labels in the SoundActions dataset. We used DG‑SCT (Duan et al., 2023), the current state‑of‑the‑art model for auditory–visual event recognition, to investigate the ability to recognize reduced‑type labels under different conditions². As demonstrated in Figure 5, three main factors are considered in the experiments:“

1. Fine‑tuning range: Two kinds of fine‑tuning have been tested: fine‑tuning only the final layer after all transformer and adapter layers, labeled as ‘cls,’ or fine‑tuning all adapters, labeled as ‘all.’ The ‘cls’ groups aim to test the model’s ability after training on large‑scale, causally labeled datasets, and the ‘all’ groups tend to investigate the best ability of the model over reduced‑type labels. The ‘cls’ groups have 531K trainable parameters out of the total 460M parameters, while the ‘all’ groups have 71.1M out of 460M trainable parameters.

2. Modality combination: To investigate the model’s mechanism on extracting information from different modalities, we tested three different combinations of input modalities: the combined audio–video input (‘av’), audio‑only (‘a’), or visual‑only (‘v’). When only one modality is used, we create an all‑zero matrix as a dummy input for the other modality. Similar to the fine‑tuning modality, the validation data is provided with ‘av,’ ‘a,’ or ‘v.’ If the model is fine‑tuned with ‘a’ or ‘v,’ we will not provide the unseen modality in validation. However, validation modalities are withdrawn for models fine‑tuned with ‘av’ to test their reliability on different modalities.

3. Label type: We use the two reduced‑type labels provided by the SoundActions dataset: ‘PerceptionType,’ which includes Impulsive, Sustained, Iterative, and Unknown, and ‘Enjoyability,’ which includes Yes, Neutral, No, and Unknown.

Figure 5

Due to the limited size of the SoundActions dataset, we used five‑fold validation for each group of the experiments and calculated the mean and standard deviation as the final metrics. The training and validation data might use different modalities according to the experiment setups. The official checkpoint of DG‑SCT on the AVE task was used as the pre‑trained model for the experiment. The audio backbone HTS‑AT (Chen et al., 2022) was pre‑trained on AudioSet (Gemmeke et al., 2017), the video backbone Swin‑T (Liu et al., 2022) was pre‑trained on ImageNet 22 K (Deng et al., 2009), and the DG‑SCT adapters were pre‑trained on the AVE dataset Tian et al. (2018). All backbones and adapters are fixed for ‘cls’ groups, and only the final CMBS layer (Xia and Zhao, 2022) with 531K trainable parameters is activated. For ‘all’ groups, backbones are fixed, while adapters and the CMBS layer are activated, resulting in 189M trainable parameters.

The fine‑tuning experiments were conducted with the learning rate of $5\times 10^{-4}$ after a grid search between $1\times 10^{-3}$ and $4\times 10^{-2}$. The learning rate is reduced by half if the training loss stops decreasing for more than five epochs, and the training is terminated when the validation loss stops decreasing for more than 10 epochs. The ‘cls’ groups were trained with a batch size of 16, and ‘all’ groups used batch size 4 but accumulated four batches, due to heavier computational needs. Both groups used nine types of video data–augmentation transforms, while no audio augmentation was used. An NVIDIA A100 GPU was used to conduct the experiments. PyTorch (Paszke et al., 2019), PyTorch‑Lightning (Falcon and The PyTorch Lightning team, 2019), and Weights and Biases (Biewald, 2020) were used for the training scripts and experiment logs.

3.3 EoPMA: Ensemble of perception mode adapters

The second experiment aims to improve the audio–video model using Schaeffer’s theory of three listening modes (Schaeffer et al., 1967) and its extension to auditory– visual perception modes (Chion and Murch, 2019a). Instead of being triggered separately or exclusively in human perception, the three listening modes ‘overlap and combine in the complex and varied context of (the film) sound track’ (Chion and Murch, 2019c, p. 33). Inspired by this, we propose the EoPMA method, which combines multiple adapters trained with different perception mode labels. Figure 6 shows the structure of a classic audio–video adapter and the EoPMA model. Since no speech data are involved in the AVE or SoundActions datasets, we include two modes of adapters: the causal perception adapters that are trained on causal labels in the AVE dataset, and the reduced perception adapters that are trained on reduced labels in the SoundActions dataset. After each adapter layer, the audio and/or video embeddings from the two adapters are combined and fed into the next transformer layer, depending on the modality setup.

Figure 6

There are two benefits of using the adapter structure for mixing the perception modes: (1) compared to fine‑tuning the entire audio–video backbone, the parameter efficient nature of the adapter model makes the training more efficient and also reduces the overfitting on small datasets such as SoundActions, and, (2) compared to only adjusting the final embeddings that loses the adjustment of low‑level features in the shallow layers, the insertion of adapters throughout all backbone layers makes the model able to focus on reduced‑type features at all abstraction levels.

Similar to Duan et al. (2023), we define the audio input of layer $l$ as $a^{(\mathit{l})}\in {\mathbb{R}}^{T\times (L^{(l)}\cdot F^{(l)})\times C_a^{(l)}}$ and the video input of such as $v^{(\mathit{l})}\in {\mathbb{R}}^{T\times (H^{(l)}\cdot W^{(l)})\times C_v^{(l)}}$, where $T,C$ denote the timestamp and channel, $L,F$ denote the length of each time frame and the frequencies of the audio spectrogram, and $H,W$ denote the height and width of the video frame. Then, we denote the causal adapter as ${\mathrm{\Omega }}_c^{v2a}(v^{(l)},a^{(l)}),{\mathrm{\Omega }}_c^{a2v}(a^{(l)},v^{(l)})$, where ${\mathrm{\Omega }}_c^{v2a}$ is the video‑to‑audio attention block and ${\mathrm{\Omega }}_c^{a2v}$ is the audio‑to‑video attention block. Analogously, we define the reduced adapter as ${\mathrm{\Omega }}_r^{v2a}(v^{(l)},a^{(l)}),{\mathrm{\Omega }}_r^{a2v}(a^{(l)},v^{(l)})$.

The operations of causal adapters in each layer can be written as:

where $a_{fc}^{(l)}$ denotes the causal, audio attention of the first attention block, while $v_{fc}^{(l)}$ denotes the video attention. Analogously, we obtain the reduced attention $a_{fr}^{(l)}$, $v_{fr}^{(l)}$ by using the reduced adapters. Then, the attentions from the two perception‑type adapters are added by the hyperparameter $\beta \in [0,1]$ that controls the ratio between the causal and reduced embeddings:

where $a_y^{(l)}$ is the interim audio embedding after the first attention block, while $v_y^{(l)}$ is the interim video embedding. Then, within the same layer $(l)$, the processed is repeated with the second group of causal attention blocks ${\mathrm{\Omega }}_{yc}^{v2a}$, ${\mathrm{\Omega }}_{yc}^{a2v}$ and reduced attention blocks ${\mathrm{\Omega }}_{yr}^{v2a}$, ${\mathrm{\Omega }}_{yr}^{a2v}$.

where $a_{yfc}$, $v_{yfc}$ are the causal audio and caudal video attentions. The reduced attentions $a_{yfr}$, $v_{yfr}$ are obtained analogously. Finally, we obtain the output audio and video embedding of the layer $a^{(l+1)},v^{(l+1)}$:

where MHA and MLP refer to ‘multi‑head attention’ and ‘multi‑layer perception,’ respectively.

In the inference tests for EoPMA, we used the pre‑trained adapters from the original DG‑SCT checkpoints on the AVE dataset as the causal perception adapters and the SoundActions fine‑tuned adapters (groups C and D in Table 3) as reduced perception adapters. The SoundActions fine‑tuned adapters with the highest validation accuracy within each group are used for the ensemble. No further training is required for the EoPMA inference test. Instead, we tested three variables: the hyperparameter $\beta$, the SoundActions adapters from different fine‑tuning setups as in Table 3, and the modality combinations in the ensemble process. We tested the EoPMA on the validation set of the AVE dataset and compared the accuracy with the classic audio–video adapters, including LAVisH (Lin et al., 2023) and DG‑SCT (Duan et al., 2023).

Four comparison groups are added to the experiment: (1) the original DG‑SCT adapters trained on the AVE dataset; (2) using the reduced‑tuned model only to validate on the AVE dataset; (3) ensemble the final embeddings of reduced‑tuned model and the original AVE‑trained model; and (4) ensemble of two original AVE‑trained models that are trained with different random seeds, same with the method by Kwok et al. (2024). The hypothesis is that group 2 would have less accuracy than group 1 because group 2 is trained on a different task; groups 3 and 4 should perform better than group 1; and, finally, the EoPMA method would perform better than all the comparison groups.

There are two main differences between EoPMA and the previous works on PEFT ensemble (Kim et al., 2021; Kwok et al., 2024; Lin et al., 2024): First, even though the two perception mode adapters in EoPMA have the same structure, they are not trained on the same dataset, only with different random seeds as in Kwok et al. (2024). Instead, the perception mode adapters are trained with different perception mode labels. Second, the ensemble method used by Kwok et al. (2024) is a post‑processing system that only combines the last few layers of the models, whereas, in EoPMA, the embeddings are merged at every layer.

4.1 Reduced‑Type fine‑tuning results

Table 3 lists the results of all experiment groups. As discussed in Section 2.5, we analyze the results with auditory–visual theory from three aspects: the ability of the models to recognize reduced‑type labels, the level of synchresis in the models, and the added value and modality imbalance of the models. Results of ‘cls’ fine‑tuning demonstrate the model’s ability to recognize reduced‑type labels after being pre‑trained on causal labels from AudioSet and AVE. The accuracy of groups A1 and B1 shows that the model has learned the general properties of audio–video data in the original dataset.

Comparing the performance of different fine‑tuning types (A1 vs. C1, B1 vs. D1), we see that fine‑tuning all adapters (‘all’) gives better performance than only fine‑tuning the classifier (‘cls’), since fine‑tuning all adapters makes the model able to adjust the low‑level information in the shallow layers. While the validation accuracies shown in Table 3 do not differ significantly, we notice that the training accuracy of the ‘cls’ groups is much lower than the ‘all’ groups. This means that while both group types are suffering from overfitting due to the small dataset, the ‘all’ groups have more generalizable performance.

The results within each group (e.g., A1 to A5) show the modality imbalance while using the same fine‑tuning type and labels. We can see that both groups using ‘PerceptionType’ labels have a significant performance drop when modalities are withdrawn in either fine‑tuning or validation. When fine‑tuning with only one modality (e.g., A2 and A3), the result demonstrates the best performance with that single modality, while withdrawing one modality during validation (e.g., A4 and A5) reveals the model’s reliance on one modality when both modalities are provided. We can see that the model performs worse in the second case, which suggests that, when fine‑tuning using both modalities, the model has learned to fuse the information from both modalities, which resembles the synchresis phenomena in human perception, as mentioned in Section 2.1.3.

Finally, comparing the performance on the two different labels (Group A&C vs. B&D), we observed two phenomena: 1) When using ‘cls’ fine‑tuning (A1 vs. B1), the performance on ‘PerceptionType’ is lower than ‘Enjoyability’; however, when using ‘all’ fine‑tuning (C1 vs. D1), ‘PerceptionType’ performs slightly better. 2) The performance of ‘PerceptionType’ groups (A, C) has more drastic changes when using different modality combinations. Observation 1 suggests that the ‘Enjoyability’ labels are similar to the original labels used for pre‑training, indicating that they contain more causal‑type information. In contrast, ‘PerceptionType’ labels are less similar and contain more reduced‑type information that the model did not learn during pre‑training. Observation 2 suggests that the ‘PerceptionType’ labels require a higher level of synchresis, and it relies more on audio information rather than on video (C2 vs. C3, C4 vs. C5).

Figure 7 shows a visualization of dimension‑reduced embedding spaces of different modalities, calculated using principal component analysis (Pearson, 1901). When comparing PT4–PT5 and EJ4–EJ5, we see that, when only one modality is provided during fine‑tuning, the video groups (PT5, EJ5) perform significantly better than the audio groups (PT4, RJ4). This shows that the labels correlate more with video information, which means the participants relied more on the video than on the audio when labeling. However, when both audio and video are provided during fine‑tuning (PT1–3, EJ1–3), instead of using the easier video embeddings (PT3, EJ3), the audio embeddings contribute more to the separation (PT2, EJ2). This interesting result of the ML model is similar to the added value concept in Chion’s theory (Chion and Murch, 2019a).

Figure 7

4.2 EoPMA results

Table 4 shows the validation accuracy of applying different ensemble methods to the AVE dataset. As a comparison, the SoundActions fine‑tuned models with the original AVE classifier do not perform well on the AVE evaluation since the label sets of the two datasets focus on different perception modes. When ensembling the output embeddings before going into the classifier, the performance is slightly improved compared to the original model trained on AVE. However, the ensemble of two AVE‑trained models with different random seeds is not better, matching findings by Kwok et al. (2024). Meanwhile, the EoPMA method’s accuracy is higher than that when using only different random seeds, which holds for both reduced‑type labels. This suggests that the reduced perception adapters extracted additional information from the same audio–video input that helped the causal perception adapters, even when the AVE task is causal. The PerceptionType labels are more helpful than Enjoyability labels, which might be due to the better agreement level and label quality (Table 2).

Figure 8 plots the accuracy change along different $\beta$ values. The range of $\beta$ values that improve the accuracy on the AVE validation set is between $0.001$ and $0.027$, which is relatively small since the task is still causal. Nonetheless, the improvement is consistent mainly within a specific range of $\beta$ values and among all setups of reduced labels and modality combinations. Resonating with the observations in Figure 7, we also find that, while $PT\text{\_}a\text{\_}a$ (trained on audio only and assembles audio embeddings only, similar to PT4) performs worse than $PT\text{\_}v\text{\_}v$ (trained on video only and assembles video embeddings only, similar to PT5), the audio–video model’s audio embedding $PT\text{\_}av\text{\_}a$ (similar to PT2) has a better performance than $PT\text{\_}av\text{\_}v$ (similar to PT3). It again shows the similarity of modality imbalance in the DG‑SCT model with the concept of added value. The improvement in accuracy might seem insignificant, but it is achieved with only 365 samples in the SoundActions dataset compared to the original AVE dataset with 4,143 samples. This shows the potential of reduced‑type labels to make further improvements if the dataset can be extended in the future.

Figure 8