org.apache.spark.mllib.clustering
Return the topics described by weighted terms.
Return the topics described by weighted terms.
This limits the number of terms per topic. This is approximate; it may not return exactly the top-weighted terms for each topic. To get a more precise set of top terms, increase maxTermsPerTopic.
Maximum number of terms to collect for each topic.
Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.
Return the topics described by weighted terms.
Return the topics described by weighted terms.
WARNING: If vocabSize and k are large, this can return a large object!
Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.
Number of topics
Number of topics
Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, alpha, eta)
Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, alpha, eta)
Note:
Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)
Convert model to a local model.
Convert model to a local model. The local model stores the inferred topics but not the topic distributions for training documents.
For each document in the training set, return the distribution over topics for that document ("theta_doc").
For each document in the training set, return the distribution over topics for that document ("theta_doc").
RDD of (document ID, topic distribution) pairs
Inferred topics, where each topic is represented by a distribution over terms.
Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.
WARNING: This matrix is collected from an RDD. Beware memory usage when vocabSize, k are large.
Vocabulary size (number of terms or terms in the vocabulary)
Vocabulary size (number of terms or terms in the vocabulary)
:: Experimental ::
Distributed LDA model. This model stores the inferred topics, the full training dataset, and the topic distributions. When computing topics for new documents, it may give more accurate answers than the LocalLDAModel.