Perceptual and automated estimates of infringement in 40 music copyright cases

2.1 Perceptual Experiments

In one previous experimental study, Lund (2011) used two past court cases (Swirsky v. Carey and Gaste v. Kaiserman) and manipulated MIDI representations of the works to change aspects such as tempo, rhythm, and instrumentation. Lund found that such manipulations reduced the accuracy of participants’ judgments of copyright infringements even though it was assumed that such non-melodic features should not play a role in decisions. Lund argued that this demonstrated that the “lay listener test” was flawed because it relies on subjective listening to audio recordings that may differ in non-melodic aspects. However, this study only tested two cases, and did not compare full audio recordings with these MIDI representations, so it remains unknown whether listeners are in fact more accurate when listening to MIDI representations than when listening to full audio recordings.

2.2 Automatic Analysis

Automatic analysis of musical similarity is a long-standing challenge in music information retrieval (Casey et al., 2008), and a number of studies have applied this to the domain of copyright infringement (Müllensiefen & Pendzich, 2009; Robine et al., 2007; Savage et al., 2018; Selfridge-Field, 2018; Yuan et al., 2020; see Cason & Müllensiefen, 2012 and Malandrino et al., 2022 for detailed review). Often, this application is to a small number of famous cases: for example, Robine et al. (2007) applied a string-matching algorithm to five famous cases (Bright Tunes v. Harrisongs; Fantasy v. Fogerty; Heim v. Universal; Repp v. Webber; Selle v. Gibb). At larger scale, Müllensiefen and Pendzich (2009) developed an algorithm for judging melodic similarity that compares the profile of successive pitch intervals in two disputed songs against each other, while weighting them against a database of comparable profiles from 14,063 pop songs using a formula for estimating perceptual salience. When they applied this algorithm to a database of 20 past music copyright decisions focused on melodic similarity from the Music Copyright Infringement Resource (MCIR) (Cronin, 2018), they found the best-performing version of their algorithm was able to accurately identify 90% (18/20) of past cases.

Savage et al. (2018) developed a Percent Melodic Identity (PMI) method for quantifying melodic evolution based on automatic sequence alignment algorithms used in molecular genetics to measure melodic similarity (Savage & Atkinson, 2015). When they applied this method to the same set of cases as Müllensiefen & Pendzich, it accurately predicted 80% (16/20) of cases, despite being a simpler method that didn’t require calibration to an existing database of popular songs.

Recently, Malandrino et al. (2022) applied a range of automated algorithms to a larger sample of 164 cases from the MCIR and tested these against 50 famous plagiarism cases involving melodic similarity. They found some algorithms were able to predict up to 88–90% of decisions. They also found that their automated method helped to assist participants with perceptual judgments of a subset of six cases randomly chosen from their set of 50 cases.

While the related task of cover song detection has a long history of study in music information retrieval (Serrà et al., 2010; Yesiler et al., 2019), to our knowledge no audio similarity algorithms had been tested for their ability to evaluate copyright infringement until our preliminary study (Yuan et al., 2020). However, many general audio similarity algorithms have been evaluated through the Music Information Retrieval Evaluation Exchange (MIREX) competition. We thus chose the audio similarity algorithm implemented in Musly, an open-source library of audio music similarity algorithms, because it has consistently performed at or near the top of audio similarity algorithms as evaluated in MIREX (Flexer & Grill, 2016).

2.3 Our Preliminary Study

We conducted a preliminary study with a small dataset of 17 music copyright court cases (14 from the USA and 3 from Japan; Yuan et al. 2020). Two automated algorithms focused on melody (Percent Melodic Identity [PMI]) and rhythm/timbre (Musly), proved to be effective in determining music similarity, matching past legal decisions with identical accuracy of 12/17 cases (~70%). Experiments were also conducted to collect perceptual data from 20 human participants, but we found no significant differences between the three conditions where the disputed sections were edited to contain either full audio, melody only, or lyrics only, with participants matching past decisions in between 50–60% of cases in all three conditions.

Since our preliminary study had limited size and scope with a dataset of only 17 court decisions and perceptual ratings from only 20 participants, we wanted to expand the testing data by including more usable music copyright cases. In particular, we wanted to change the sampling method from focusing only on cases which have court decisions focused on melodic similarity, to also include disputes involving similarities in lyrics, rhythm, timbre, or other non-melodic features, in order to fairly compare the roles of melodic vs. non-melodic features in copyright infringement judgments. To increase diversity and cross-cultural generalizability, we also aimed to expand our sample to include more languages and countries.

3.1 Musical Stimulus Preparation

The disputed segments of the musical works (mean length: 25s; range: 3–54s) were presented in one of three different versions: full-audio (the recorded versions including all instrumental and/or vocal parts), melody-only (MIDI rendition of the pitches and rhythms of the main melody), and lyrics-only (lyrics shown as visual text, without any accompanying audio).⁶

If several consecutive sections were disputed, we chose the longest continuous disputed section. For instance, in Francescatti v. Lady Gaga, both the repeated words “Juda” and “Judas” in the choruses and melodies in the breakdown sections were argued to share similarities, and consequently we selected sections spanning from the chorus through breakdowns of both songs for stimulus preparation. If several different and non-consecutive sections were disputed, we combined them for the study. For example, in Peters v. West, the plaintiff complained about the lyrics and rhyme scheme used in the defendant’s works, so we chose the sections of potential lyrical similarity and combined them into a single audio file for each song. If a disputed section repeated multiple times with undisputed sections in between (e.g., chorus was disputed but not verse), we selected only one disputed section and ignored the repeated ones. If longer sections were disputed in vague terms (e.g., groove, genre, style) but shorter sections were disputed with specific issues (e.g., melodic/lyrical similarities), we focused only on the specific sections.

For the melody-only condition, we transcribed the melodies from the original audio recordings. Usually, we selected the vocal soloist line as the melody. If the melody included vocal harmony, we chose the top notes, but ignored background vocal harmony if there was a soloist. If there was no vocal melody and/or if an instrumental melody was disputed, we used whichever instrument was highest and/or most melodic. If the vocal melody of the verse and/or the chorus and an instrumental melody from another part were both disputed, we chose the lead vocal melody and predominant instrumental melody. In order to control for all non-melodic factors including instrumentation, key, and tempo, the transcribed melodies from the original audio recordings were edited as necessary to exactly correspond to the audio recordings: these transcribed melodies were transposed to have a tonic of C, and were then recorded using the MIDI piano in MuseScore, played back at a tempo that was the average of the tempi from the plaintiff and defendant recordings. A metronome sound was added to all transcribed melodies to support listeners to perceive the rhythms/meters of songs and avoid any confusion in the absence of the instrumental background that would normally provide the metric context.

For the lyrics-only condition, we found the lyrics from online resources or manually transcribed the lyrics when the lyrics were not available online. There were six court cases involving instrumental works without lyrics (cf. Table 1) – these were not included in the lyrics-only study.

4.1 Experiment Design

We conducted an online perceptual experiment where participants were each asked to judge substantial similarity for the 40 cases.⁷ To avoid possible order effects caused by participants evaluating the same melody/lyrics multiple times, we split the experiment in two: one with only full-audio versions of musical works, and one with melody-only and lyrics-only versions presented to the participants. Each participant only completes one experiment (this means each participant is never exposed to the same melody or lyrics twice). In our preliminary study we also included randomly selected pairs of melodies whose relationships were not in dispute (i.e., comparing a defendant’s work against a randomly selected plaintiff’s work) as well as disputed pairs of melodies to ensure that perceptual judgments were more accurate than would be expected by random chance. However, because these demonstrated very clear non-random judgment (near 100% accuracy for random pairs, with much lower accuracy of ~50–60% for disputed pairs; Yuan et al., 2020) and because these random pairs doubled the length of the experiment, we removed the random pairing for the expanded experiment; that is, in the new experiments, the participants compared only the originally disputed pair of musical works presented as either full-audio, melody-only, or lyrics-only versions.

This gave a total of 40 different pairs of musical works to evaluate in the experiment with only full-audio, and a total of 74 (40 cases shown as melody-only + 34 cases shown as lyrics-only) different pairs of musical works to evaluate in the experiment with melody-only and lyrics-only versions. Note that the 6 cases involving instrumental works without lyrics were not included in the experiment with lyrics-only versions. The pairs of musical works were presented in fully random order in each experiment (without separate blocks for different conditions; e.g., any given sample might be melody-only or lyrics-only in the experiment with melody-only and lyrics-only versions). Each experiment took approximately 1–2 hours for one participant to complete evaluations for all pairs.

For each pair, the participant is given a pair of music excerpts, “A” and “B”. “A” is always a plaintiff’s work while “B” is always a defendant’s work. After listening to the full-audio or MIDI or reading the lyrics of the two music works and reading the jury instructions for the “lay listener test” used by Lund (2011; see introduction), the participant needs to answer two questions: 1) How similar do you think A and B are? (5-point Likert scale: “not at all similar”, “a little similar”, “somewhat similar”, “very similar”, and “extremely similar”), and 2) Do you think B infringes on the copyright of A? (yes/no answer). When lyrics-only versions are shown, the participants are allowed to answer N/A for both questions if they are not able to make the judgment, specifying the reasons (e.g., cannot understand the foreign languages shown).

4.2 Results

We collected perceptual data from 51 participants recruited by the first author (YY), 28 of whom did the experiment with only full-audio and 23 of whom did the version with melody-only and lyrics-only. All experiments were conducted online in English (during the COVID-19 pandemic) and all participants provided informed prior consent to participate in the study.

For the full-audio experiment, 14 participants were male and 14 were female. Ages ranged from 23 to 57, and the mean age was 36. The native languages of the participants were Chinese (26 participants), English (1), and Japanese (1). The 26 Chinese native speakers also reported language skills of English (26 participants), Cantonese (2), French (1), Japanese (1), Spanish (1), and Thai (1). The English native speaker reported Japanese language skills. The Japanese native speaker reported English language skills. 7 reported substantial music experience of more than 2 years while 21 had less than 2 years of musical experience.

In the experiment with melody-only and lyrics-only versions of musical works, 8 were male and 15 were female. Ages ranged from 22 to 58, and the average age was 39. The native languages of the participants were Chinese (22 participants), and Japanese (1). The 22 Chinese native speakers also reported language skills of English (22 participants), Cantonese (2), and Japanese (2). The Japanese native speaker reported English language skills. 4 reported substantial music experience of more than 2 years while 19 had less than 2 years of musical experience.

Figure 1 shows how accurately the 51 participants’ judgment of infringement matched the official court decisions when they were given full-audio, melody-only, or lyrics-only versions of music pieces from the 40 court cases. Note that “accuracy” was defined as the proportion of participants agreeing with the court’s decision, whether that decision was of infringement or no infringement.⁸ In Figure 1A individual data points represent mean accuracy for individual cases (n = 40) averaged across the 28 and 23 participants, while in Figure 1B, individual data points represent mean accuracy for individual participants (n = 28 for full-audio condition and n = 23 for melody-only and lyrics-only conditions) across the 40 cases.

Figure 1

The results in Figure 1 showing mean participant accuracy of 57.9%, 57.5%, and 51.5% for full-audio, melody-only, and lyrics-only groups respectively are similar to the ones we found in our preliminary study with a smaller dataset and fewer participants (58%, 54%, and 49%, respectively). Variability is much greater in Figure 1A than Figure 1B, suggesting that the main source of variation comes from individual copyright cases, not the way different individuals perceive musical similarity. This point is also suggested by a comparison of musicians and non-musicians, who performed similarly (Figure 1B). No clear differences are notable for the small subsets of cases with Japanese or Chinese languages.

Notably, cases in all conditions show skewing toward higher accuracy for non-infringing cases (Figure 1A). This suggests that participants hold a high threshold for judging copyright infringement and tend to err on the side of no infringement. This raises the possibility that participants may not have been able to accurately predict infringement, with their performance instead due to chance and the slight baseline skew toward no infringement (22/40 cases).

To investigate this possibility we performed signal detection analysis using Receiver Operating Characteristic (ROC) curves for both mean perceived similarity and proportion of perceived infringement for all three conditions. The results confirmed that participants performed significantly above chance for full audio and melody-only conditions (AUC > 0.7, p < .05), but were at chance levels for the lyrics-only condition (AUC < .5, p > .7; Figure 2). These patterns were consistent whether based on perceived similarity or perceived infringement (Figure 2). Because lyrics-only data was thus effectively random, we exclude it from further analysis in the rest of the manuscript.

Figure 2

5.1 Melodic Similarity (Percent Melodic Identity [PMI])

We chose the PMI (Percent Melodic Identity) method to calculate melodic similarity because it has been validated in previous research using a similar sample of copyright cases (Savage et al., 2018). Like Judge Learned Hand’s “comparative method” (Fishman, 2018) to test musical similarity, the PMI method begins by transposing two melodies transcribed in staff notation into a same key, eliminating rhythmic information by assigning all notes equal time values, and then aligning and counting the confluence of notes.⁹ Following the procedure, we prepared note sequences of disputed melodies all transposed to a C tonic for consistency (just as was done when preparing MIDI files). The PMI algorithm then automatically aligns each sequence pair using the Needleman-Wunsch global pairwise alignment algorithm (Needleman & Wunsch, 1970), and counts the number of identical notes (ID). The percentage of identical notes shared between melodies, named percent melodic identity (PMI) (Savage et al., 2018), is calculated by dividing ID by the average length of the melody pair (L₁ and L₂), as follows:

5.2 Audio Similarity (Musly)

Musly currently implements two music similarity algorithms. One implements the Mandel-Ellis audio similarity algorithm (Mandel & Ellis, 2005). The other one, which is the default one, improves the Mandel-Ellis algorithm to compute audio similarity for best results. Specifically, it computes a representation of each song’s audio signal based on 25 Mel-Frequency Cepstral Coefficients (MFCCs) to estimate a Gaussian model and eventually a single timbre model to be compared, computes similarity between each pair of timbre models using Jensen-Shannon approximation, and normalizes the similarities with Mutual Proximity (Schnitzer, 2014; Schnitzer et al., 2011). We used the default algorithm because it has been found to have higher accuracy (Schnitzer, 2014).

We prepared the full-audio version of the music excerpts from the dataset of court cases and fed them to the default algorithm of Musly to compute similarity. The output of the Musly algorithm is a distance matrix where distances, i.e. differences, between every two songs are listed. Because the Musly default algorithm normalizes the results, all the distances range between 0 and 1. Consequently, we calculated the audio music similarity by subtracting distance values from 1 and multiplying by 100 to convert the results into percentage terms for consistency with our other methods.

6.1 PMI vs. Perceptual Data

Mean perceptual similarity of each court case was calculated by averaging participants’ individual ratings of similarity. Figure 4A shows the relationship between PMI values and perceptual similarity under the three different conditions. Correlation analyses show that the PMI melodic similarity is significantly correlated with perceptual similarity for both full-audio and melody-only conditions (full: r = 0.41, p = 0.009; melody: r = 0.77, p = 6.6 × 10^–9), but not for the lyrics-only condition (r = 0.20, p = 0.26).

Figure 4

6.2 Musly vs. Perceptual Data

We also compared the Musly-calculated audio music similarity with the perceptual data collected. Figure 4B shows the correlation between Musly similarity and perceptual similarity of the 40 tested court cases under three different conditions for perceptual judgment. The Musly audio similarity measure has only weak, non-significant correlations with perceptual similarity for all three condition groups of “full-audio”, “melody-only”, and “lyrics-only” (full: r = 0.065, p = 0.69; melody: r = 0.039, p = 0.81; lyrics: r = -0.21, p = 0.23). Musly thus shows both weaker predictive power when compared to the actual court decisions and weaker correlations with participants’ perceptual ratings than PMI.

6.3 Multiple logistic regression analysis including automated and perceptual variables

To determine which feature(s) best predicted copyright judgments, we ran a multiple logistic regression using variables that showed significant predictive power for court decisions in the signal detection analyses in Figures 2, 3 (i.e., excluding lyrics-only data where participant judgements were at chance level and no algorithm for textual similarity was employed). Because mean perceived similarity and portion perceived infringement were highly correlated (r = 0.93), which creates problems with multi-collinearity, we only used the similarity ratings in the regression model, as they are more closely related to court decisions than the infringement judgements (cf. Figure 2). We thus included four variables in our regression:

1. “Perceived audio similarity” (mean averaged across the 28 audio-only participants for each of the 40 cases)

2. “Perceived melodic similarity” (mean averaged across the 23 melody-only participants for each of the 40 cases)

3. “Algorithmic audio similarity” (as calculated for each of the 40 cases using the Musly algorithm, as described in Section 5.2)

4. “Algorithmic melodic similarity” (as calculated for each of the 40 cases using the percent melodic identity [PMI] algorithm, as described in Section 5.1)

For completeness, we ran both frequentist and Bayesian models once using the full sample of n = 40 cases and once replicating the sub-sample of n = 17 cases used in the Yuan et al. (2020) preliminary study, with and without variable selection. The main results are summarized in Table 2 (cf. Supplementary Materials Tables S1–5 for alternative analyses). While the sub-sample of n = 17 cases was too small to draw strong conclusions from, all analyses provided qualitatively consistent results confirming the results of the ROC analyses suggesting that human perceptual ratings of similarity listening to full-audio excerpts best predicted the outcome of court cases. This variable demonstrated the highest standardized regression coefficients after controlling for the other features through multiple logistic regression (z = 2.0, p = .04; Table 2). In contrast, perceived melodic similarity consistently predicted decisions less well than perceived audio similarity, and was excluded by the stepwise AIC (Akaike Information Criterion) variable selection procedure (Table 2).

7.1 Obvious melodic infringement settled out of court (Harumaki Gohan v. Mori)

This case showing the highest similarity ratings by both humans and the PMI algorithm is instructive as a window into a major challenge of this research: selection bias. This case is part of the MCIR database, but unlike almost all other MCIR cases, this case was not accompanied by an official legal decision, but was instead settled out of court because the similarities were so striking. Indeed, the PMI algorithm confirms the obvious: the two melodies are 100% identical (though the lyrics are different; Figure 6).

Figure 6

Such settlements are actually common, but this case is unusual because the publishing company formally admitted infringement and fired Mori in a statement entitled “当社元提携作家による不正行為についてのお詫び [“Apology for Infringement by Our Former Songwriter”]. In most similar cases, defendants employ “no fault” settlements where they pay the plaintiffs to resolve the case without any legal admission of infringement (e.g., HaloSongs v. Sheeran). The implications of this trend on the present and future research are discussed below in the section entitled “the problem of reliable ground-truth legal decisions”.

7.2 Obvious non-infringing similarities in titles (Francescatti v. Lady Gaga)

This case showed the lowest similarities for both humans and the PMI algorithm, and despite the similarities in the title (“Juda” vs. “Judas”), the rest of the lyrics were also rated as not very similar (mean 1.8 on a 1–5 scale; Figure 7). This represents the converse of the Harumaki v. Gohan case, where similar meritless infringement claims are very common, yet are usually dropped or settled out of court prior to a legal decision due to the expensive nature of a lawsuit and low probability of winning. Cases such as this that do make it all the way to a formal legal decision tend to have similar titles, perhaps because similarities in titles carry more weight or are easier to “sell” to a jury or media without requiring extensive musical analysis. Indeed, 10 of the 40 cases analyzed have similar or identical titles, though only 3 of these (Three Boys v. Bolton; Grand Upright v. Warner; and Boosey v. Empire Music) were judged as infringing.

Figure 7

7.3 Subconscious melodic plagiarism with minimal perceived similarity (Bright Tunes Music v. Harrisongs)

This case showed the most striking discrepancy between human and algorithmic ratings. Human participants listening to the full audio versions rate them as not very similar (<2 on a 1–5 scale), while the PMI algorithm identifies >50% identical notes in the melodic sequences (Figure 8). In this case, the melodic similarities were striking enough that a judge concluded that George Harrison’s “subconscious knew it already had worked in a song his conscious mind did not remember”, and found him liable for infringement to the tune of several million dollars.

Figure 8

7.4 Similarity of non-original musical expression (Vargas v. Pfizer)

This case represents the final extreme of acoustic similarity in the absence of original melodic material. Instead, we find nearly identical rhythmic patterns played using nearly identical drum sounds (kick drum, snare, hi-hat) that nevertheless share no melodic similarities (as there is no melody, only rhythm; Figure 9). In this case, the rhythmic patterns were common enough that the judge granted summary judgment (dismissed the case without a full trial) on the grounds that the plaintiffs had failed “to preclude the possibility of independent creation”.

Figure 9

7.5 Unreliable ground truth legal decision (Williams v. Gaye)

Our final example comes not from the extremes of Figure 5 but from the middle to highlight the issue of unreliable ground truth decisions. This case ultimately found Pharrell Williams and Robin Thicke liable for infringing on Marvin Gaye’s “Got To Give It Up” in their Number 1 2013 hit “Blurred Lines”, awarding over $5 million in damages. However, while the appeal process finally concluded with the 9^th Circuit District Court affirming the decision, the correctness of the decision remains highly contested. The 9^th Circuit District Court ruling included a dissenting opinion from one of the three judges, arguing that “the Gayes’ expert, musicologist Judith Finell, cherry-picked brief snippets to opine that a “constellation” of individually unprotectable elements in both pieces of music made them substantially similar.” Indeed, the “signature phrase” cited by Finnell as being the smoking gun for similarity is short enough and shows minimal enough similarities (45%) as to not be significantly more similar than would be expected by chance from two random melodies composed from the same scales (p = .2; Figure 10).

Figure 10

While the majority appeal decision concluded that the ruling was legally valid and “not reviewable after a full trial”, this does not necessarily imply that the jury’s decision was correct, but simply that it could not be legally over-ruled after the fact. Since this decision, other judicial opinions have come to opposite decisions, reversing jury decisions involving similar “constellations” of non-protectable elements (e.g., Skidmore v. Zeppelin; Gray v. Perry). Instead, the standard for such “constellations” appear to be correcting to the standard requiring “virtual identity” between the two works to establish infringement, as has been held in numerous earlier cases involving other copyrighted authorship like sculptural works (Satava v. Lowry, 2003) graphical user interfaces (Apple Computer v. Microsoft, 1994) and personal organizers (Harper House v. Thomas Nelson, 1989).

Our findings suggest that adding perceptual experiments or automated algorithms to this complicated process will not necessarily help. The PMI method does quantify a relatively low level of melodic similarity, while the full audio experiments suggest a somewhat higher but still not striking level of similarity. But many other cases with both higher and lower levels of melodic or acoustic similarity have been decided in both directions in the past, and the measures we use appear to be too coarse to offer conclusive evidence in such a polarizing case that has seen hundreds of musicians, musicologists, lawyers, and other interested parties weigh in on opposing sides.