A Comparison of Topic Modeling Approaches Using Networked Discussion Forum Posts From the City-data.com Corpus

(1.1) Limitations and Caveats

City-Data.com forum rules prohibit flaming, hate speech, and trolling; however, posts do feature frank depictions of violence and potentially controversial statements that some may find offensive.

While the discussion board format of City-Data.com provides users with a direct means to quote previous posts when replying, this is not a strict requirement. Some replies attend to the content of previous posts with explicit attribution. Thus, some replies are not accessible by the presence of a “Quoted” field. Moreover, the quote_body and quote content may not always present the full content of the original post. Users can be selective about their citations. Consequently, quote_body and quote content may not always recover original posts. Lastly, because forum users can post across forum, quoted post content may have not accessed the originating page, thus, leaving gaps in quote_body and quote content.

Object name

City-Data.com Corpus

Format names and versions

CSV

Creation dates

07/14/2022 – 08/14/2022

Dataset creators

Ryan M. Omizo (Temple University)

Language

English

License

Creative Commons Attribution 4.0 International

Repository name

Zenodo 10.5281/zenodo.10086354

Publication date

2023-11-09

(3.1) Experimental Trials

I conduct 16 topic modeling trials designed to leverage the networked structure of the City-Data.com Corpus. I evaluate topic quality of models trained on post-level segments, thread-level segments, and topics guided by prior information (see Li et al. 2018; El-Assady et al. 2019; Popa and Rebedea 2021; Gourru et al. 2018). Guided topic modeling uses prior information about the data to center model priorities. Post and thread-level topic modeling test how the unitization of City-Data.com’s networked content influences topics modeling and is primarily a data preparation step. Guided topic modeling intervenes in the topic modeling process. To guide the kmeans clustering process (the basis of the sentence embedding-based topic model), I manually designate initial cluster centers. This initial positioning will guide subsequent re-centerings as the model converges to reduce the distance between intra-cluster datapoints (Arthur & Vassilvitskii 2007). To guide the LDA topic modeling, I adapt Li et al.’s (2018) method of injecting seed words extracted from the data into the topic modeling process.ⁱ In one experiment conducted on the 20 Newsgroups dataset, Li et al. (2018) used forum label text (e.g., talk.politics.guns) as seeds with the assumption that forum labels provide distinguishing categorical information about potential topics.

I test three types of topic seeds:

1. Topic titles – Following Li et al. (2018), I use thread topic titles as seeds (see Table 1).

2. Initial forum posts – Initiating posts declare the horizons of participating in forum discussions. Empirical work by Sobkowicz and Sobkowicz (2010), See Jagarlamundi et al. 2012 for example, demonstrates that social medial discussions evidence strong first mover advantage similar to scientific papers (Newman 2009). Papers that appear early in the rise of a discipline will outpace the citation rate of newer papers. Here, the hypothesis is that posts that appear first gain more engagement and this increased engagement will condition the content of a sizable portion of the thread. (see Table 7 in Appendix A).²

3. Posts with the highest degree – Leveraging the network properties of the City-Data.com corpus, I create a graph of each forum and extract posts with the most incoming and outgoing links or node degree (Gerlach et al. 2018; Duan et al. 2021; Yang et al. 2016). Posts that receive numerous quoted replies and/or are replying to other posts serve as proxies for engagement. Like the intuition behind the use of initial forum posts, posts that are bound up in more extensive conversations, condition more content because respondents must stay on topic to sustain discourse (see Table 8 in Appendix A).

I calculate topical coherence and diversity scores to measure model quality. Topical coherence includes several measures that indicate how well topical term lists reflect the underlying data. In this study, I use Minmo et al.’s (2011; see also Hinneburg et al. 2014) UMASS method. UMASS coherence measures the probability that co-occurring words in the topical term list occur in documents divided by the total number of documents. Unlike other coherence measures such as UCI-coherence, which compare topical terms to a large reference corpus like Wikipedia dumps, UMASS coherence is derived from the original dataset. I balance coherence scores against topical diversity scores (Mimno et al. 2011). I employ Gensim’s coherence measures (Řehůřek, R., & Sojka 2011) to calculate UMASS coherence. Topical diversity refers to a family of metrics that indicate the variability of topical terms, thus, the range of data explainable by topical term lists. Quality topics are both coherently related to the underlying data, but also distinctive enough to offer thorough faceting (Dieng et al. 2020). For this study, I employ four diversity measures (Terragni 2023):

1. Proportion of unique words (PUW) – PUW determines the ratio of unique topical terms for all topics (Dieng et al. 2020). Scores closer to 1 indicate diverse topics; scores closer to 0 indicate repetitive topical terms.

2. Jaccard Distance between topical term lists (JD) – Proposed by Tran et al. (2013), this diversity measure evaluates the Jaccard distance between topical term lists. Greater distances between topical terms indicate more topical diversity (Terragini 2023).

3. Word embedding centroid distance (WE-CD) (Bianchi et al. 2020b) – WE-CD calculates the distance between collocations in the topical term list to a reference corpus of word embedding. This metric determines how diverse topical term lists are in comparison to generalized usage in embedding models of large volumes of texts like Wikipedia or the Common Crawl Corpus of internet sites. For this paper, I use the FastText Common Crawl word embedding model with 300 dimensions and 2 million subword vectors (Bojanowski et al. 2016; Joulin et al. 2016a; Joulin et al. 2016b).³

In all, topical diversity measures used in this study determine intra-topical term list diversity, inter-topical term list diversity, and generalized topical term list diversity when compared to a reference corpus. To calculate each measure, I use the top-25 terms per topic.

Because the Metro forum also discusses criminal activity, I set the topic number at 4 to correspond to the following thematic categories for topic modeling: coronavirus, crime, plan, and retail.⁴

Tables 3 and 4 illustrate the SE-Topics and LDA modeling coherence and diversity scores across different data segmentations.

In general, LDA topic modeling produced better topical coherence and diversity scores compared to sentence embedding approaches with an average UMASS coherence of –2.31 and PUW and JD diversity scores near 1.0. Mean WE-CD for LDA topic models is also 0.12 points greater, signifying that LDA topic models are more semantically distant from the reference corpus than SE-Topics with high coherence. Box and whisker plots of topical coherence and diversity scores (see Figures 1 and 2) also indicate that LDA topic modeling results are more consistent across segment types. SE-Topical coherence and diversity scores indicate a wider interquartile range between the best performing segments (high degree posts) and the lowest performing segments with a difference of 2.93. On the other hand, there is only a 0.53 difference in coherence scores between the best performing LDA topics (high degree posts and threads) and the worst performing topic model (topic titles).

Figure 1

Figure 2

Put another way, the best scoring SE-Topics—guided initial posts and guided high degree —perform as well as the two worst scoring LDA topic models.

The discrepancies in coherence and diversity scores, however, are complicated by qualitative assessments.

Comparing SE-Topics and LDA topics guided by high degree posts, we can discern close similarities among term collocations within individual topics. Both SE-Topics and LDA Topics represent the Retail thread and the development rhetoric of the Plan thread; however, SE-Topics have agglomerated crime and coronavirus discourse (see Topic 0 in Table 5). The LDA topic model seeded by high degree posts (see Table 6), however, has captured discussions about government restriction of public services that ensured in the first months of the 2020 COVID-19 pandemic in Philadelphia with terms such as money, state, local, care, and help.

Thread segmentation modestly improves LDA topical coherence (mean –2.2 compared to mean –2.4). Coherence scores for SE-Topics, on the other hand, worsen with thread segmentation (mean –4.41 compared to mean –3.49).

SE-Topics models benefited the most from the injection of prior information. The results of the SE-Topics informed by the embeddings bundled with the most quoted replies are particularly interesting when compared with the generally poor performance of thread-based segments. Both permutations depend on the networked structure of the City-Data.com Corpus to structure topics. Threads utilize the chained structure of posts and quoted replies to crystalize texts that emerge through interaction; high degree node priors utilize the linkages of posts with other posts. Although more information is encoded when threads are joined into single texts, the centrality of posts within conversations lends better guidance to the formation of more cohesive topics. This makes intuitive sense in that posts with numerous linkages will anchor common content. Threads, on the other hand, are more intrinsically diverse segments, striated by conversational turns and/or dissensus among interlocutors.

Guided topic modeling returned better topical coherence and diversity scores than unguided models (with the sentence embedding model guided by initial posts the exception). Posts with the most incoming and outgoing linkages (degree) produced the best scoring topic models. This finding suggests that network structure can influence the development of topical content in extended asynchronous conversations. Messages enmeshed in replies are more likely conserved as interlocutors attend to given information as they add commentary.