Modular Bibliographical Profiling of Historic Book Reviews

2.1 Book reviews and natural language processing

Within NLP, identifying details of the book or books focused on in a book review can be framed as a citation extraction or NER task, but both frames have important shortcomings or limitations. Lessons learned from both domains, meanwhile, are relevant to bibliographical profiling. Citation extraction, which focuses on automated recognition and classification of in-text citations and references, is arguably a less unified area of NLP research than NER, as there are stakeholders in computer science and various areas of application (Iqbal et al., 2021). However, the task of profiling a reference using interdependent information—title, author name, source periodical—is similar to profiling a book review. NER focuses on identifying entities such as institutions, people, places, dates, dollar amounts, events, works of creative expression, etc. The body of methods literature on NER is large, and applications to digital humanities (DH) domains, especially by digital historians, are widespread (Ehrmann et al., 2024). Recent methods work in NER has focused on leveraging deep learning (Al-Moslmi et al., 2020; Li et al., 2020; Yadav & Bethard, 2019; Sevgili et al., 2022). Recent applications in digital history have focused on evaluating the suitability of novel machine learning (ML) architectures (e.g., BiLSTMs, transformers) to historical materials and using models with character and sub-word information to address “spelling variations and OCR errors” (Ehrmann et al., 2024, p. 31). Insights drawn from tackling these problems are likely to be relevant to profiling historical book reviews.

2.2 Book review analysis in digital humanities

In DH contexts, applications of NER and citation extraction techniques appear to be widespread, and profiling book reviews appears to be a growing area of interest. There have been several DH articles in the past five to seven years that appear to depend on extracting information from book reviews or book reviews indices like the Book Review Index (Hegel, 2018; Sinykin et al., 2019; Underwood & Sellers, 2016). For the most part, in DH scholarship, NLP methods are described sparingly or not at all. Code and raw data are shared regularly by some scholars and inconsistently or not at all by others. In some cases, copyright or licensing restrictions make it difficult to share these materials. In other cases, procedural details are omitted or glossed over because they are framed as self-evident or unimportant. This is especially the case when DH scholarship is published with a “traditional” humanities audience in mind (e.g., in scholarly journals and scholarly monographs not intended specifically for DH or CA audiences). Some of these tasks might be computationally uncomplicated, but lack of detail in the literature generally makes it difficult to assess the complexity of both the tasks performed and solutions applied.

2.3 Analogous work in digital humanities

How the “mechanical details” of computational inquiry relate to the theoretical or critical work of DH is a subject of much discussion (Drucker, 2017). Whereas many NLP and ML papers are foremost concerned with methodological novelty and performance benchmarks, DH scholarship must consider well-established and novel approaches, with a focus on adapting them to humanities contexts and interrogating the values they convey. Computational methods prompt practitioners to specify epistemic assumptions, often in the form of data curation and writing code. In various areas, DH scholarship has engaged closely with computational methods and considered consequences beyond those typically prioritized in computer science. Such scholarship has striven to understand the technical and theoretical components of methods well enough to identify cases where seemingly trivial decisions can have crucial downstream implications. As stated above, perhaps the closest analogous area is the application of NER in the context of digital history, where Ehrmann et al. advocate that “the next generation of historical NER systems” prioritize transferability “across historical settings”; “more systematic and focused” NER methods; using gold standards and of shared tasks for greater comparability; developing finer-grained historical NER; and more resource sharing (Ehrmann et al., 2024, p. 31). MBP is conceived as a scholarly intervention situated between theory and practice, and hopefully contributing to both.

4.1 Exploratory data analysis

The methods explored in this paper differentiate between (1) approaches based on any Boolean (True/False) string matching criteria (literal, fuzzy, or pattern matching) and (2) approaches where a feature extraction phase is paired with a metric-based similarity scoring phase (MBP). As Warren et al. have argued, confidence estimates are often preferable when dealing with records of unequal informational value and are “better suited to the grey areas of humanistic research” (Warren et al., 2016). Using a string-matching approach, one might loop through a list of authors, titles, or publishers and classify any in-text match of a string as a mention of the corresponding entity. The dataset used in this article generates a list of 1,155 unique author surnames for 1,093 reviews. Of these surnames, approximately 90% can be found in their corresponding reviews, but that leaves about 10% of surnames unmentioned in the review text. With book titles, one can tokenize the text, control for capitalization, and limit each title to a four-gram with stopwords included or excluded and find approximately 70% of book titles in their corresponding reviews (out of 1,087 distinct title strings). With publisher names, repeats are much more common even in a small sample, so the dataset yields 258 distinct strings or n-grams (after controlling for non-standard tail words like “corp” and “inc”). NLP-based matching can discover more than half of these publisher names in their corresponding review text. In some cases, authors, titles, or publishers are simply not mentioned in book reviews. In other cases, OCR errors lead to missed matches or false negatives. Table 4 summarizes match rates as well as false positive counts.

Among author, publisher, and title, author surnames are the most likely to be matched in reviews, at a rate of approximately 90%.⁴ If we look closer at recall, precision, and F1 scores, there are several surprises. First, the number of false positives drives down all three performance statistics. False positives occur most often when the entity being matched overlaps with commonly used words or phrases, as with author surnames like Long, Day, Church, and London; book titles like Poems, Missouri, and Bliss; or title n-grams like “history of the [noun]” or “the story of a [noun]”. They can also occur in the case of entity overlap or ambiguity, such as when a book titled Balsac (a biography of the author) boosts the likelihood of Balsac being the author under review or a reference to Appleton’s Encyclopedia boosting the likelihood of Appleton being the book’s publisher (Kemp, 1909).

Hopefully it is clear that simplistic NLP-based matching approaches make certain kinds of errors more likely than they should be. Perhaps most importantly, the ceiling for false positives is raised. With traditional classification or matching tasks, false positives cannot be greater than the number of observations (N). With simplistic string matching, each review can produce one correct title or publisher, one or more correct authors, and as many false positives as the number of candidates one uses for matching. Allowing more fuzziness to regular expressions or N-gram comparisons may seem like an appealing way to reduce false negatives, but one should expect false positives to increase at a greater rate than true positives. As fuzziness is widened, true positives tend to follow a pattern of diminishing returns, while false positives will either stay steady or increase, with an upward bound of the number of tokens or n-grams in a document.

4.2 Modular bibliographical profiling

This study is designed to control for known sources of uncertainty and make gains in book review profiling related to feature extraction and matching book review text to author, title, publisher triads. The most common errors associated with reconciling book review features to bibliographical records or identifiers seem to be best isolated by eliminating any errors related to incorrect page segmentation and misidentifying non-review content as book reviews. Focusing on reviews with only one correct bibliographical target simplifies performance evaluation while maintaining opportunities for generalization. The tasks of page segmentation, review identification, and review type classification are crucial to analysis of historical book reviews, but they warrant full length studies of their own.

In this study, the following procedure is employed:

1. All single-focus book reviews in the dataset (N = 1,093) are used for each task

2. Each review’s full text is pre-processed (e.g., tokenization, punctuation preserved or removed, capitalization preserved or ignored)

3. A feature extraction or selection strategy is employed to isolate text features

4. A list of bibliographical entity candidates is derived by taking the correct labels for all authors, titles, and publishers from the metadata and reducing each triad to a single entry of raw text

5. Reviews and triads alike are converted to term-frequency tables

6. A similarity measure is employed to compare each review to each triad

7. For each review, a ranked list of most similar triads is returned

8. If the top-ranked triad is the correct entry, the match is considered correct

9. If the top-ranked triad is not the correct entry, a false positive recorded for the matching record and a false negative is recorded for the true match

10. If the match is incorrect, the rank of the correct match is used for “in the ballpark” statistics (below)

No partition of training and test set is used here because every bibliographical record in the dataset is unique. Instead, all true labels are in the candidate set, and each matching task has a number of potential false positives equal to the total number of candidate labels, minus one (1,092). When conducting this task using random guessing, we would expect an accuracy rate of about 0.001%. Table 5 details the feature selection approaches that are considered.

The feature extraction phase is designed to be modular, so that any feature extraction can be swapped for another without changing any other aspects of the code.

Strategy 1 (naive BOW) involves tokenizing all book reviews and all author-title-publisher triads, and then transforming them into a single BOW model such that the term-frequency matrix includes every unique token from all reviews and all triads.

Strategy 2 (rule-based entity extraction) uses a complex set of rules to find likely author surnames, and then infer forenames based upon those surnames. All possible titles and publishers were reduced to n-grams of non-stopword tokens with a maximum length of four (e.g., “Gone with the Wind” produces the tokens “gone” and “wind” and “From the Mixed-Up Files of Mrs. Basil E. Frankweiler”, produces the tokens “mixed-up”, “files”, “mrs”, “basil”).⁵

Strategy 3 (pre-trained NER) began by running NER on all reviews using a pre-trained model (Spacy’s “en_core_web_trf”). Entities belonging to the categories of Person, Geopolitical Entity, Nationality, Organization, Facility, Event, Location, Product, Work of Art, and Law were retained and all other recognized entities were dropped. As with Strategies 1 and 2, a BOW model was generated with a vector space that contained all tokens from the list of qualifying recognized entities.

For the matching portion of the experiments, the same process was used for all three strategies. From within the derived vector space, every book review vector was compared to every author-title-publisher triad, and a ranked list of review-to-triad cosine similarities was generated. If the correct record ranked first in the list of similarity scores, the match was scored as correct.

6.1 Dataset limitations and reuse potential

The dataset used for this paper is put forth as a well-balanced (albeit relatively small) sample of single-focus reviews from Proquest’s APS database originally published in prominent weekly and monthly publications in the United States between 1880 and 1925. The constraints of this sample imply several limits to the broader representativeness of the data, especially temporal, cultural, and linguistic representativeness. All reviews are written in English, and it can be expected that at least some of the performance statistics reported here would differ if similar methods were applied to reviews in other languages.

This dataset should prove to be useful in a range of classroom contexts (review analysis, testing NLP methods on shorter texts, etc.) and can be used to benchmark the performance of additional approaches to entity profiling, with the caveat that, with each reuse, the risk of overfitting increases, which would potentially prevent refinements from generalizing to other reviews. Finally, this dataset may aid in future efforts to locate reviews in digitized periodicals and extract bibliographical entities from them, which could result in more representative datasets over time.

6.2 Modularity as a template

The dataset used for this study is structured in order to function as one component of a larger pipeline. This workflow is intended to act as a template for future analysis of this sort, as well as work on this dataset and the creation of comparable datasets. A modular approach such as the one laid out above should be given serious consideration in the context of large-scale bibliographical profiling of book reviews, and beyond. This kind of approach has both procedural advantages and a strong potential to improve performance (measured in terms of recall and precision). These benefits include:

1. Modularity: The approach is designed to create discrete partitions between the feature extraction phase and the similarity scoring phase so that any feature extraction strategy (including strategies not explored in this paper) could be used interchangeably, as could additional similarity measures.

2. Simplicity: Increasingly intricate rules are eschewed in favor of more general NLP strategies. The dataset eliminates several well-known confounds for book review analysis and provides a basis to compare the performance of the feature extraction and entity linking steps of the process.

3. Extensibility: Items 1 and 2 should make these profiling approaches easier to adapt and extend. Additional refinements are expected and encouraged, with the caveats described in section 3.1.

4. Non-Consumptive Formats: All of the above methods can be used on corpora of book reviews where only non-consumptive format is available. Periodicals will still need to be segmented and OCRed at the article level, but it is hoped that this work will motivate entities like ProQuest, JSTOR, and HTRC to segment more periodicals and/or release more non-consumptive formats representing the article level of periodicals.

5. Books as Bibliographical Entities: The method established above cannot create bibliographical records that do not exist by “mixing and matching” author, title, and publisher combinations. Adding this control creates the opportunity to establish a performance baseline that can be manipulated in future analysis.

6. Novelty Detection: Building on item 4, a subsequent study could arbitrarily remove an incrementing percent of true labels from the candidate set (or introduce an incrementing percent of “nuisance candidates”) to assess how predictably performance decreases under new uncertainty. It could also help researchers identify obscure or as-yet-unknown authors, publishers, or books. This potential is especially important when one considers the fact that matching bibliographical records from predetermined could be a source of bias amplification, specifically in the form of confirmation bias.

7. Applications Beyond Book Reviews: It is hoped that this work will help motivate and enable research occupying a middle space between work traditionally categorized as theory/methods or practice/application. Opportunities to extend the “praxis work” of DH are abundant.