The Performance of Probabilistic Latent Semantic Analysis

Introduction

In this blog post, I want to investigate the performance of probabilistic latent semantic analysis: a subject I have been teaching (but also studying) for a course. Probabilistic latent semantic analysis proceeds from a document-term matrix, a standard data matrix in the field of text mining. It should look something like this, where the rows of the matrix represent $n$ documents and the columns $p$ terms (words). Usually, $p > n$. a

$$A = \begin{pmatrix} doc_1,term_1 & doc_1, term_2 & \dots & doc_1, term_p \\ \vdots & \dots & \ddots & \vdots \\ doc_n, term_1 & \dots & \dots & doc_n, term_p \end{pmatrix}$$ The standard maximum likelihood estimator for $Pr(d_i, t_j)$ is $x_{ij} / m$ where $m$ is the total word count in all documents. This has a simple interpretation: count of word $j$ in document $i$ / total word count in all documents.

PLSA

Probabilistic Latent Semantic Analysis (PLSA) is an attempt to decompose this matrix using something similar to a singular value decomposition. In particular, given a probability matrix:

$$P = \begin{pmatrix} p(d_1, t_1) & \dots & p(d_1, t_p) \\ \vdots & \ddots & \vdots \\ p(d_n, t_1)& \dots & p(d_n, t_p) \end{pmatrix}$$

We can have construct an approximation $U\Sigma V^T$ with $U=N \times r$ ($r$ classes):

$$U = \begin{pmatrix} p(d_1 | c_1) & \dots & p(d_1 | c_r) \\ \vdots & & \vdots \\ p(d_n | c_1) & \dots & p(d_n | c_r) \end{pmatrix}$$

$\Sigma = \text{diag}(P(c_r))$, and $V^T$ ($r \times p$) has elements $V^T_{ij} = P(t_j | c_i)$. Naturally, the object of interest is usually $U$: this represents the probabilities of the document belonging to class $1$ to $r$.

In R, the package svs can be used to carry out probabilistic latent semantic analysis:

library(svs)

In what follows, I’ll demonstrate the capacity of PLSA to distinguish two types of documents on its own: I’ll scrape and convert into a document text matrix several pages about football, and several pages about tennis, set $k$ (the number of classes) equal to 2, and investigate the output.

Example

Here, I first web-scrape the text of several wikipedia pages:

Now, I use the tidytext package to put these into a document-term matrix:

library(tidytext)
# Compute the texts into a data.frame
text_df <- tibble(Text = texts) |> 
  rowwise() |> 
  mutate(Text = paste(Text, collapse="")) |> 
  ungroup()

# Put all words together grouped by document
text_data <- text_df |>
  group_by(row_number()) |> 
  unnest_tokens(word, Text) |> 
  rename('document' = 'row_number()')

# Convert to a document-term matrix
# Filter out stop_words and numbers
stop_words <- bind_rows(stop_words, data.frame(word = as.character(0:10000), lexicon="Custom"))

dtm <- text_data |>
  count(document, word) |>
  filter(!is.element(word, stop_words)) |> #!str_detect(word, paste(as.character(0:10000), collapse="|"))) |> 
  cast_dtm(document, word, n)

The document-term matrix (dtm) looks like this:

as.data.frame(as.matrix(dtm)) |> dim()

## [1]   14 6197

Now, let’s compute the frequencies and apply LPSA:

library(svs)
X <- as.matrix(dtm)
out <- fast_plsa(X, k=2, symmetric=T)

Now, I want to find out which class each of the documents have been assigned to:

apply(
  out$prob1, 1, which.max
)

##  1  2  3  4  5  6  7  8  9 10 11 12 13 14 
##  1  2  1  1  1  2  2  2  2  2  2  2  2  2

.. which means that the majority of the documents is classified in the correct corresponding cluster.

Comparison

We can compare the results with a so-called latent semantic analysis, which is just a singular value decomposition.

out_lsa <- fast_lsa(X)

out_lsa$pos1[, 'Dim1']

##            1            2            3            4            5            6 
## -0.803081398 -0.338888792 -0.191561592 -0.298909337 -0.271174528 -0.133052909 
##            7            8            9           10           11           12 
## -0.146478395 -0.026522482 -0.011199147 -0.008402141 -0.019986480 -0.013202093 
##           13           14 
## -0.001734649 -0.001084412

In this case, we can see that the median of the first dimension already separates the documents perfectly in two classes. The first 7 observations having very low values and the second 7 values having very high values. So we can take this to be an indicator for which class the documents belong to:

data.frame(doc_no = 1:14) |> 
  mutate(class = if_else(out_lsa$pos1[,'Dim1'][doc_no] > median(out_lsa$pos1[,'Dim1']), 1, 2))

##    doc_no class
## 1       1     2
## 2       2     2
## 3       3     2
## 4       4     2
## 5       5     2
## 6       6     2
## 7       7     2
## 8       8     1
## 9       9     1
## 10     10     1
## 11     11     1
## 12     12     1
## 13     13     1
## 14     14     1

Conclusion

In this setting, I have demonstrated a simple example of latent probabilistic semantic analysis, and latent semantic analysis, and I would prefer a simpler method to a potentially more complicated method. Thank you for reading!