public class ALS extends Estimator<ALSModel> implements ALSParams
ALS attempts to estimate the ratings matrix R as the product of two lower-rank matrices,
X and Y, i.e. X * Yt = R. Typically these approximations are called 'factor' matrices.
The general approach is iterative. During each iteration, one of the factor matrices is held
constant, while the other is solved for using least squares. The newly-solved factor matrix is
then held constant while solving for the other factor matrix.
This is a blocked implementation of the ALS factorization algorithm that groups the two sets of factors (referred to as "users" and "products") into blocks and reduces communication by only sending one copy of each user vector to each product block on each iteration, and only for the product blocks that need that user's feature vector. This is achieved by pre-computing some information about the ratings matrix to determine the "out-links" of each user (which blocks of products it will contribute to) and "in-link" information for each product (which of the feature vectors it receives from each user block it will depend on). This allows us to send only an array of feature vectors between each user block and product block, and have the product block find the users' ratings and update the products based on these messages.
For implicit preference data, the algorithm used is based on
"Collaborative Filtering for Implicit Feedback Datasets", available at
http://dx.doi.org/10.1109/ICDM.2008.22, adapted for the blocked approach used here.
Essentially instead of finding the low-rank approximations to the rating matrix R,
this finds the approximations for a preference matrix P where the elements of P are 1 if
r > 0 and 0 if r <= 0. The ratings then act as 'confidence' values related to strength of
indicated user
preferences rather than explicit ratings given to items.
| Modifier and Type | Class and Description |
|---|---|
static class |
ALS.CholeskySolver
Cholesky solver for least square problems.
|
static class |
ALS.InBlock<ID>
In-link block for computing src (user/item) factors.
|
static class |
ALS.InBlock$ |
static interface |
ALS.LeastSquaresNESolver
Trait for least squares solvers applied to the normal equation.
|
static class |
ALS.LocalIndexEncoder
Encoder for storing (blockId, localIndex) into a single integer.
|
static class |
ALS.NNLSSolver
NNLS solver.
|
static class |
ALS.NormalEquation
Representing a normal equation (ALS' subproblem).
|
static class |
ALS.Rating<ID>
Rating class for better code readability.
|
static class |
ALS.Rating$ |
static class |
ALS.RatingBlock<ID>
A rating block that contains src IDs, dst IDs, and ratings, stored in primitive arrays.
|
static class |
ALS.RatingBlock$ |
static class |
ALS.RatingBlockBuilder<ID>
Builder for
ALS.RatingBlock. |
static class |
ALS.UncompressedInBlock<ID>
A block of (srcId, dstEncodedIndex, rating) tuples stored in primitive arrays.
|
static class |
ALS.UncompressedInBlockBuilder<ID>
Builder for uncompressed in-blocks of (srcId, dstEncodedIndex, rating) tuples.
|
| Constructor and Description |
|---|
ALS() |
| Modifier and Type | Method and Description |
|---|---|
ALSModel |
fit(DataFrame dataset,
ParamMap paramMap)
Fits a single model to the input data with provided parameter map.
|
ALS |
setAlpha(double value) |
ALS |
setImplicitPrefs(boolean value) |
ALS |
setItemCol(String value) |
ALS |
setMaxIter(int value) |
ALS |
setNonnegative(boolean value) |
ALS |
setNumBlocks(int value)
Sets both numUserBlocks and numItemBlocks to the specific value.
|
ALS |
setNumItemBlocks(int value) |
ALS |
setNumUserBlocks(int value) |
ALS |
setPredictionCol(String value) |
ALS |
setRank(int value) |
ALS |
setRatingCol(String value) |
ALS |
setRegParam(double value) |
ALS |
setUserCol(String value) |
static <ID> scala.Tuple2<RDD<scala.Tuple2<ID,float[]>>,RDD<scala.Tuple2<ID,float[]>>> |
train(RDD<ALS.Rating<ID>> ratings,
int rank,
int numUserBlocks,
int numItemBlocks,
int maxIter,
double regParam,
boolean implicitPrefs,
double alpha,
boolean nonnegative,
StorageLevel intermediateRDDStorageLevel,
StorageLevel finalRDDStorageLevel,
long seed,
scala.reflect.ClassTag<ID> evidence$1,
scala.math.Ordering<ID> ord)
Implementation of the ALS algorithm.
|
org.apache.spark.sql.types.StructType |
transformSchema(org.apache.spark.sql.types.StructType schema,
ParamMap paramMap)
:: DeveloperAPI ::
|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitalpha, getAlpha, getImplicitPrefs, getItemCol, getNonnegative, getNumItemBlocks, getNumUserBlocks, getRank, getRatingCol, getUserCol, implicitPrefs, itemCol, nonnegative, numItemBlocks, numUserBlocks, rank, ratingCol, userCol, validateAndTransformSchemagetMaxIter, maxItergetRegParam, regParamgetPredictionCol, predictionColaddOutputColumn, checkInputColumn, explainParams, get, getParam, isSet, paramMap, params, set, set, validate, validateuidinitializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarningpublic static <ID> scala.Tuple2<RDD<scala.Tuple2<ID,float[]>>,RDD<scala.Tuple2<ID,float[]>>> train(RDD<ALS.Rating<ID>> ratings, int rank, int numUserBlocks, int numItemBlocks, int maxIter, double regParam, boolean implicitPrefs, double alpha, boolean nonnegative, StorageLevel intermediateRDDStorageLevel, StorageLevel finalRDDStorageLevel, long seed, scala.reflect.ClassTag<ID> evidence$1, scala.math.Ordering<ID> ord)
public ALS setRank(int value)
public ALS setNumUserBlocks(int value)
public ALS setNumItemBlocks(int value)
public ALS setImplicitPrefs(boolean value)
public ALS setAlpha(double value)
public ALS setUserCol(String value)
public ALS setItemCol(String value)
public ALS setRatingCol(String value)
public ALS setPredictionCol(String value)
public ALS setMaxIter(int value)
public ALS setRegParam(double value)
public ALS setNonnegative(boolean value)
public ALS setNumBlocks(int value)
public ALSModel fit(DataFrame dataset, ParamMap paramMap)
Estimatorpublic org.apache.spark.sql.types.StructType transformSchema(org.apache.spark.sql.types.StructType schema,
ParamMap paramMap)
PipelineStageDerives the output schema from the input schema and parameters. The schema describes the columns and types of the data.
transformSchema in class PipelineStageschema - Input schema to this stageparamMap - Parameters passed to this stage