Supervised probabilistic latent semantic analysis with applications to controversy analysis of legislative bills

2.1Probabilistic topic models

In 1999, three papers [1, 2, 13] introduced the model of Probabilistic Latent Semantic Analysis. One variant of the model appeared in 1998 [14] and all these models were originally discussed in an earlier technical report [15]. PLSA was a probabilistic implementation of latent semantic analysis (LSA) introduced by Deerwester et al. [6]. LSA was extended from the vector space model. It aimed to represent documents in a low dimensional vector space consisting of common semantic factors. Differing from LSA in projecting document or word vectors into the latent semantic space, PLSA extracted the aspects related to documents. This aspect model was interpreted as a mixture model containing latent semantic mixtures. Parameters of mixture probabilities were estimated by the maximum-likelihood (ML) principle. PLSA did not provide a straightforward way to make inferences about new documents not seen in the training data and the parameterization of the model was susceptible to overfitting. Latent Dirichlet Allocation (LDA) addressed these limitations by proposing a Bayesian probabilistic topic model.

PLSA and LDA established the field of probabilistic topic models. Many extensions of the two basic models have been proposed. In Zhai et al. [16], PLSA was extended to include a background component to explain the non-informative background words and a cross-collection mixture model was proposed to support comparative text mining. Mei and Zhai [17] propose a general contextual text mining model which is an extension of PLSA to incorporate context information. They further regularize PLSA with a harmonic regularizer based on a graph structure in the data [18]. One active area of topic modeling research is how to relax and extend the assumptions of PLSA and LDA to uncover more sophisticated structure in the texts. For example, the work by Rosen-Zvi et al. [19] extends LDA to include authorship information. Recently, probabilistic topic models are proposed for unsupervised many-to-many object matching [20] and cross-lingual tasks [21]. There are many other topic models proposed. Blei [22] gives an overview of the field of probabilistic topic models.

The original PLSA and LDA and most of their variants are unsupervised models. Blei and McAuliffe [12] proposed supervised LDA (sLDA) to capture real-valued document rating as a regression response. The generative process of sLDA is similar to LDA, but with an additional step: draw a reponse variable. The sLDA model is trained by maximizing the joint likelihood of the contents and the responses of documents. They tested sLDA on two real-world datasets: movie reviews with ratings and web pages with popularity, and the experimental results demonstrated the advantages of sLDA versus regularized regression, and versus an unsupervised LDA analysis followed by a separate regression. Other extensions include multi-class sLDA [23], which directly captures discrete labels of documents as a classification response; and discriminative LDA (DiscLDA) [24], which also performs classification, but with a mechanism different from that of sLDA; and MedLDA [25], which leverages the maximum margin principle for estimation of latent topical representations. Recently, Jameel et al. [26] integrate class label information and word order structure into a supervised topic model for document classification. More variants of supervised topic models can be found in a number of applied domains, such as Labeled LDA [27], automatic summarization of changes in dynamic text collections [28], modeling of numerical time series [29], inferring topic hierarchies [30], and query expansion [31]. In computer vision, several supervised topic models have been designed for understanding complex scene images [32, 33]. Mimno and McCallum [34] also proposed a topic model for considering document-level meta-data; for example, publication date and venue of a paper.

Most of the above supervised topic models are based on LDA. There exist very few work on extending PLSA to the supervised setting. One such work was to use the spoken content of a multimedia document as a query for retrieving similar or relevant documents [35]. The query was used to train the model in a supervised fashion with respect to a query-document similarity objective function. Fergus et al. [36] extend PLSA to include spatial information in a translation and scale invariant manner, and utilized this modified PLSA model to learn an object category. Another work added a category-topic distribution in PLSA for human action recognition [37]. However, these models do not associate the topic distribution of the document with the response variable. Consequently, the discovered topics may not be indicative of the response. Aliyanto et al. [38] proposed a version of supervised PLSA to estimating technology readiness level, but they assumed the topics of each word in a document are observed which are actually not available in many real-world applications. In this paper, we follow the way LDA was extended to sLDA by directly associating the documents’ topic distributions with the response. The response is at the document level instead of the word level and it is more readily accessible. The learned topics depend on both the document’s content and response. To the best of our knowledge, no prior work has extended PLSA in a similar manner.

Recently, with the rise of deep learning, novel topic models based on neural networks have been proposed. Salakhutdinov and Hinton [39] proposed a two layer restricted Boltzmann machine (RBM) called the replicated-softmax to extract low level latent topics from a large collection of unstructured documents. Larochelle and Lauly [40] proposed a neural auto-regressive topic model inspired from the replicated softmax model but replacing the RBM model with a neural auto-regressive distribution estimator (NADE). Kingma and Welling [41] proposed variational autoencoders by combining topic modeling and neural networks. Cao et al. [42] proposed neural topic model (NTM), and it is supervised extension (sNTM) where words and documents embedding are combined. Moody [43] proposed the lda2vec, a model combining LDA and word embeddings. Dieng et al. [44] integrated to a recurrent neural network based language model global word semantic information extracted using a probabilistic topic model. Gupta et al. [45] integrated to an LSTM recurrent neural network, a neural auto-regressive topic model. Murakami and Chakraborty [46] investigated the use of word embedding with NTM for interpretable topics from short texts. Grootendorst [47] proposed BERTopic to generate document embedding with pre-trained transformer-based language models and then produce topic representations with the class-based TF-IDF procedure. Two recent surveys [48, 49] provided comprehensive reviews of neural topic models, with nearly a hundred models developed and a wide range of applications in neural language understanding such as text generation, summarization and language models. Despite the popularity of deep learning, our work has focused on traditional probabilistic methods because they are often easier to implement and more efficient to train, which may be more suitable in resource constrained environments where only limited computation and storage are accessible. Nevertheless we will explore to combine the proposed model with neural networks in a future work.

2.2Controversy analysis of legislative bills

Legislative voting is a major area of research. Most of the research is focused on the ideal point estimation of the ideological positions of legislators. This is primarily for the purpose of predicting their voting patterns. An early work in this area presented a spatial model of legislative voting [50]. Londregan [51] estimated the preferred positions of legislators by modeling the legislative agenda. Cox and Poole [52] used a spatial model to assess the role of partisanship in influencing the votes of legislators. Variational methods were applied to predict votes [53]. Thomas et al. [54] modeled voting behavior from congressional debate transcripts. Gerrish and Blei [55] demonstrated roll call predictive models which link legislative text with legislative sentiment. They [56] further derived approximate posterior inference algorithms based on variational methods to predict the positions of legislators. Fang et al. [57] analyzed public statements from legislators to build a contrastive opinion model of the legislators. Gu et al. [58] conducted ideal point estimations of legislators on the latent topics of voted documents.

Some of the work cited above utilized topic models. For example, Gerrish and Blei [55] extended LDA to build a generative model of votes and bills called the ideal point topic model. The model infers two bill related latent variables. One of the latent variables explains bills that all legislators will vote for or against while the other variable explains bills that do not have unanimous approval or disapproval. In addition, the model infers a latent variable for the legislators’ ideal points. Another example, Fang et al. [57] present the cross-perspective topic model which unifies two identically extended LDA models to contrast the opinion words of a bipolar legislative body. The opinion words reflect the subjective positions of the polar entities on various topics. The model discriminates between opinion words and topics words by treating them as two separate observed variables.

On the broader field of controversy analysis, much work has been done detecting contradictions in textual data. One of the early works studied the dynamics of conflicting opinions in texts by visually inspecting graphs [59]. Tsytsarau et al. [60] further investigated two types of contradictions, namely, “overlapping contradicting opinions” and “change of sentiment”. Many supervised learning approaches have been proposed for classifying texts into one of the two opposing opinions using annotated controversial corpora including sentences [61], documents [62] and document collections [61]. Some recent work addresses the task of identifying controversial contents on Wikipedia [63, 64, 65] and on social media [66, 67, 68].

Figure 1.

Graphical model representation of (a) PLSA and (b) sPLSA.

3.1Notations

Assume the corpus 𝒟 contains M documents with K topics. Nd is the number of words in document d. Each document d has two set of observed variables: Wd⁢n, which is the nt⁢h word of d; and cd, which is the response of d, such as the rating of a review. Table 1 includes the main notations in the paper.

3.2Generative process

Similar to many other topic models, sPLSA assumes that a document consists of multiple topics. Therefore, there is a distribution θ𝐝 over a fixed number of K topics for each document d. Like PLSA, this distribution is a multinomial distribution where each element θd⁢k in the vector represents the probability that topic k appears in document d, i.e., θd⁢k=P(k|d). In addition, we assume each topic represents a distribution over words w in a fixed vocabulary of size V, denoted by β𝐤. This distribution is also a multinomial distribution where each element βk⁢w represents the probability that term w is chosen by topic k, i.e., βk⁢w=P(w|k).

The essential difference between PLSA and sPLSA lies in the modeling of the response variable cd connected to document d. Under the sPLSA model, each document and response arises from the following generative process:

Figure 1 illustrates the graphical model representation of PLSA and sPLSA, respectively.

It is worth noting that our approach for modeling cd is different from that of sLDA. sLDA approximates a response variable, which in our case is cd, as a linear combination of the mean Zd⁢n values. sLDA represents each Zd⁢n=k as an indicator vector of length K where the kt⁢h position is set to 1 and the others are set to 0. sLDA evaluates the mean Zd⁢n by taking the mean value of the vectors, which is expressed as 𝐳𝐝¯=∑n=1NdZd⁢n. In Section 4, we empirically show that using a linear combination of θd⁢k instead of zd⁢k¯ yields vk values that better factorize the response of the latent topics.

3.3Likelihood function

The likelihood function in supervised PLSA consists of two parts. The first part is the likelihood for observing all the words in the corpus, 𝐖, given the topic distributions for the documents, 𝚯, and the word distributions for the topics, β. Mathematically, it is as follows:

where n⁢(d,w) is the number of times word w appears in document d. Therefore, the log likelihood of the observed words is

The second part of the likelihood function comes from the likelihood of the response variable. As shown in the generative process, we assume a linear model with Gaussian noise for modeling the response cd. Specifically, we express cd as follows:

where vk is the coefficient for θd⁢k. The expression indicates that cd is a random variable drawn from a Gaussian distribution with a mean ∑k=1Kθd⁢k⁢vk and a variance σ2. The likelihood of observing all the responses is as follows:

where 𝐜 is a vector of all cd in the corpus, and 𝐯 is a vector of all vk. vk can be viewed as the contribution of topic k to the overall response. That is the higher a vk value is the more its latent topic contributes to the response variable.

We assume a Gaussian prior on the coefficients vk, i.e., vk∼𝒩⁢(0,η2), which is equivalent to L2 norm regularization. By ignoring some constants which do not impact the outcome of the likelihood maximization, the log likelihood of observing all the responses can be specified as:

Equations (2) and (5) share 𝚯 as a parameter to estimate. This means we will need to unify both likelihoods into a single unified likelihood equation in order to estimate 𝚯. We accomplish this by normalizing the two likelihoods, and then linearly combining Eqs (2) and (5) as follows:

where λ is a weighing constant and is a real number λ∈[0,1]. Its value affects the perplexity of the latent topics, β, inferred by the unified likelihood. We discuss this in Section 4.

3.4Parameter estimation

Now that we have established the unified likelihood, we can use it to derive formulas for iteratively updating the parameters 𝒗, 𝜷, and 𝚯 in order to converge the likelihood to its maximum value. At a high-level, we iteratively update the parameters one at a time until the likelihood converges. We illustrate the process in Fig. 2.

Figure 2.

The iterative updates of the parameter estimation process.

3.5Inference

After the parameter estimation is completed, we do the following to infer the latent topics and their factorized response values:

4.1Dataset

We tested sPLSA using bills which were placed for a vote in the United States Congress. The objective of our test is to generate the latent topics of the bills, and then rank them by controversy. We do this by first assigning a controversy score to each bill followed by inferring the factorized controversy score of each topic using sPLSA. We assign a controversy score to each bill by using the spread of the number of yes and no votes. The formula we use is as follows:

where cd is the controversy score of bill d, ad is the number of yes votes for the bill, and bd is the number of no votes for the bill. A value of 0 indicates no controversy and occurs when the votes are either all yes or all no. A value of 1 indicates maximum controversy and occurs when the number of yes and no votes are evenly split. sPLSA uses the cd value of the bills as the response variable and generates the latent topics of the bills. We use the vk values generated by the model to rank the latent topics by controversy.

The reason why we selected congressional bills and their controversy scores as our dataset is to demonstrate applying sPLSA to a real world problem. Specifically, we want to identify contentious issues in the United States Congress by generating their latent topics. By inferring their relative controversy using sPLSA, we can rank the topics by controversy, and identify the contentious issues by selecting the most controversial topics.

We collected bills starting from the 100th Congress and ending with the 114th Congress. This is for the years 1987 to early 2016. We used the Vote API of GovTrack 22 to obtain information about the votes. Next, we discarded the votes that are not associated with a bill. For votes that are associated with a bill, we kept the final votes for the bill. Finally, we obtained the digital content of the bills from the website of the U.S. Government Publishing Office.33 We only obtained the content of bills that had a plain text version.

We were able to collect the votes and content of 6,403 bills. 5,531 bills were from the House of Representatives and 872 bills were from the Senate. 6,160 bills had more yes votes than no votes, and 243 bills had more no votes than yes votes. Figure 3 shows the distribution of the bills’ controversy score.

Figure 3.

Histogram of the distribution of the response variable calculated using Eq. (27).

We did the following preprocessing of the bills to create our dataset:

We then created the dataset as a bag-of-words representation of each bill.

4.2Setup

We randomly partition our dataset as follows: 80% for training, 10% for validation, and 10% for testing. We initialize μ by sampling from a Gaussian distribution of mean 0 and variance 1. We initialize 𝚯 from the initial values of μ using Eq. (23). We initialize β by sampling from a uniform distribution and then normalizing each β𝐤, the word distribution of topic k, so that it becomes a valid probability distribution. We initialize 𝐯 by setting all vk=0. We initialize values for λ, K, and η depending on the experiment we are running. Our implementation of sPLSA iteratively updates in lockstep 𝐯, β, and μ until the unified likelihood converges. At the beginning of each iteration, we update 𝚯 using Eq. (23).

4.3Evaluation metrics

We test a trained model by folding-in the test dataset similar to the way specified in [1]. This is essentially the same as training the model with the test dataset except β and 𝐯 are not updated, and their values are obtained from the trained model. The only parameter we estimate in the folding-in process is 𝚯. We evaluate the performance of the model with test data as follows:

The reason why we can correlate each vk with sk is because the θ𝐝 are sparse. An example of the sparsity is illustrated in Fig. 4 where we aggregate the average probability of the kt⁢h order statistic of the topics in all θ𝐝 for K=20 and λ=0.5. We can clearly see from the plot that the 20th order statistic is by far the most dominant topic.

Figure 4.

The sparsity of θ𝐝 by plotting the average probability of the kt⁢h order statistic of the topics in all θ𝐝 for K=20 and λ=0.5.

4.4Results

Table 2 shows the top 5 words for the 1st, 2nd, median, 2nd last, and last controversial topics for four experiments. Each experiment selected a unique K∈{10,20,30,40}, and all the experiments set λ=0.5 and η=1. As we will see later, our choice of λ and η are optimal for our dataset. In addition to the top 5 words, the table shows the vk coefficient of each topic. As we mentioned earlier, the vk values estimate the controversy of the topics and we use them to select the topics shown in the table. For K=10, we find that some of the topics overlap. For example, the words “fundraising”, “expense” and “fisa” (Foreign Intelligence Surveillance) appear in multiple topics. This is because the number of topics is insufficient. On the other hand, K=40 has more granular topics that overlap less. For K=40, we can infer from the words that the 1st topic is about higher education, the 2nd is about funding, the 3rd is about child education, the 4th is either about religion or sequester related budget cuts, and the 5th topic is about housing. We can therefore conclude that K has to be large enough in order to avoid topic overlap. We also observe that as the value of K increases so does the variance of the vk values. This means that the most controversial topics of larger K values are more controversial than the most controversial topics of smaller K values. This makes intuitive sense since the overlap between the most controversial topics and other topics gets smaller as K increases.

4.4.3Impact of η

We trained the model with K=20 and λ=0.5 for each η∈{10-3,10-2,10-1,1,10,102,103}. We then tested each model with the validation dataset, and obtained the results shown in Table 4. Overall, we can see from the table η=1 yields the best results.

Table 4

The perplexity and Pearson correlation values on the validation dataset for different values of η

η	0.001	0.01	0.1	1	10	100	1000
Perplexity	2072.6	2307.3	2062.4	2102.3	2056.7	2151.2	2053.5
Correlation	0.335	-0.302	0.982	0.987	0.951	0.966	0.976

Figure 5.

The prediction RMSE values of sPLSA and sLDA at various values of K.

Figure 6.

Comparison of the training time of sPLSA and sLDA for various values of K.

4.4.4Impact of λ

We trained the model for each combination of K∈{10,20,30,40} and λ from 0 to 1 in increments of 0.1. We then tested the model using the test dataset. Figure 7 shows the perplexity values for the various combinations of K and λ.

Figure 7.

The perplexity for various combinations of K and λ.

From the figure, we can generally see that as λ increases the perplexity increases as well. For smaller K values this increase is noisy, but for larger K values it gets smoother. The increase in perplexity accelerates as λ approaches 1. We also notice that as K gets larger the overall perplexity gets lower.

Figure 8 shows the values of the Pearson correlation between vk and sk for the various combinations of K and λ.

Figure 8.

The Pearson correlation between vk and sk for various combinations of K and λ.

From the figure, we can generally see that the correlation increases steeply from λ=0 to approximately λ=0.3. It then decelerates rapidly and levels off to within a noisy range. We can conclude from Figs 7 and 8 that for a fixed K improving the perplexity by increasing λ generally deteriorates the correlation and vice-versa. However, there is a range of λ values between 0.4 and 0.5 where the perplexity is not that far from the lowest perplexity and the correlation is not much different from the maximum correlation. Our ideal λ is therefore in the range of [0.4, 0.5] for our dataset.

4.5Case study

For each topic listed in Table 2 where K=40, we sampled the bill which has the highest probability for the topic. We summarize the bills and analyze their connectedness to their corresponding topics in Tables 8, 9, 10, 11, and 12. As we can see from the tables, the controversy score of the bills closely aligns with the controversy level of the topics. In addition, the themes of the topics we specified in the beginning of Section 4.4 partially or fully match the theme of the bills with the exception of the bill for the second least controversial topic. This is primarily because the theme of the second least controversial topic is hard to determine based on the top words of the topic.