org.apache.spark.ml.recommendation
Class ALS

Object
  org.apache.spark.ml.PipelineStage
      org.apache.spark.ml.Estimator<ALSModel>
          org.apache.spark.ml.recommendation.ALS

All Implemented Interfaces:

java.io.Serializable, Logging, Params

public class ALS
extends Estimator<ALSModel>

:: Experimental :: Alternating Least Squares (ALS) matrix factorization.

ALS attempts to estimate the ratings matrix R as the product of two lower-rank matrices, X and Y, i.e. X * Yt = R. Typically these approximations are called 'factor' matrices. The general approach is iterative. During each iteration, one of the factor matrices is held constant, while the other is solved for using least squares. The newly-solved factor matrix is then held constant while solving for the other factor matrix.

This is a blocked implementation of the ALS factorization algorithm that groups the two sets of factors (referred to as "users" and "products") into blocks and reduces communication by only sending one copy of each user vector to each product block on each iteration, and only for the product blocks that need that user's feature vector. This is achieved by pre-computing some information about the ratings matrix to determine the "out-links" of each user (which blocks of products it will contribute to) and "in-link" information for each product (which of the feature vectors it receives from each user block it will depend on). This allows us to send only an array of feature vectors between each user block and product block, and have the product block find the users' ratings and update the products based on these messages.

For implicit preference data, the algorithm used is based on "Collaborative Filtering for Implicit Feedback Datasets", available at http://dx.doi.org/10.1109/ICDM.2008.22, adapted for the blocked approach used here.

Essentially instead of finding the low-rank approximations to the rating matrix R, this finds the approximations for a preference matrix P where the elements of P are 1 if r > 0 and 0 if r <= 0. The ratings then act as 'confidence' values related to strength of indicated user preferences rather than explicit ratings given to items.

See Also:

Serialized Form

Nested Class Summary

static class	ALS.Rating<ID> :: DeveloperApi :: Rating class for better code readability.
static class	ALS.Rating$

Constructor Summary

ALS()

ALS(String uid)

Method Summary

DoubleParam	alpha() Param for the alpha parameter in the implicit preference formulation (>= 0).
ALS	copy(ParamMap extra) Creates a copy of this instance with the same UID and some extra params.
ALSModel	fit(DataFrame dataset) Fits a model to the input data.
double	getAlpha()
boolean	getImplicitPrefs()
String	getItemCol()
boolean	getNonnegative()
int	getNumItemBlocks()
int	getNumUserBlocks()
int	getRank()
String	getRatingCol()
String	getUserCol()
BooleanParam	implicitPrefs() Param to decide whether to use implicit preference.
Param<String>	itemCol() Param for the column name for item ids.
BooleanParam	nonnegative() Param for whether to apply nonnegativity constraints.
IntParam	numItemBlocks() Param for number of item blocks (>= 1).
IntParam	numUserBlocks() Param for number of user blocks (>= 1).
IntParam	rank() Param for rank of the matrix factorization (>= 1).
Param<String>	ratingCol() Param for the column name for ratings.
ALS	setAlpha(double value)
ALS	setCheckpointInterval(int value)
ALS	setImplicitPrefs(boolean value)
ALS	setItemCol(String value)
ALS	setMaxIter(int value)
ALS	setNonnegative(boolean value)
ALS	setNumBlocks(int value) Sets both numUserBlocks and numItemBlocks to the specific value.
ALS	setNumItemBlocks(int value)
ALS	setNumUserBlocks(int value)
ALS	setPredictionCol(String value)
ALS	setRank(int value)
ALS	setRatingCol(String value)
ALS	setRegParam(double value)
ALS	setSeed(long value)
ALS	setUserCol(String value)
static <ID> scala.Tuple2<RDD<scala.Tuple2<ID,float[]>>,RDD<scala.Tuple2<ID,float[]>>>	train(RDD<ALS.Rating<ID>> ratings, int rank, int numUserBlocks, int numItemBlocks, int maxIter, double regParam, boolean implicitPrefs, double alpha, boolean nonnegative, StorageLevel intermediateRDDStorageLevel, StorageLevel finalRDDStorageLevel, int checkpointInterval, long seed, scala.reflect.ClassTag<ID> evidence$1, scala.math.Ordering<ID> ord) :: DeveloperApi :: Implementation of the ALS algorithm.
StructType	transformSchema(StructType schema) :: DeveloperApi ::
String	uid()
Param<String>	userCol() Param for the column name for user ids.
StructType	validateAndTransformSchema(StructType schema) Validates and transforms the input schema.

Methods inherited from class org.apache.spark.ml.Estimator

fit, fit, fit, fit

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.Params

clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, setDefault, shouldOwn, validateParams

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

ALS

public ALS(String uid)

ALS

public ALS()

Method Detail

train

public static <ID> scala.Tuple2<RDD<scala.Tuple2<ID,float[]>>,RDD<scala.Tuple2<ID,float[]>>> train(RDD<ALS.Rating<ID>> ratings,
                                                                                                   int rank,
                                                                                                   int numUserBlocks,
                                                                                                   int numItemBlocks,
                                                                                                   int maxIter,
                                                                                                   double regParam,
                                                                                                   boolean implicitPrefs,
                                                                                                   double alpha,
                                                                                                   boolean nonnegative,
                                                                                                   StorageLevel intermediateRDDStorageLevel,
                                                                                                   StorageLevel finalRDDStorageLevel,
                                                                                                   int checkpointInterval,
                                                                                                   long seed,
                                                                                                   scala.reflect.ClassTag<ID> evidence$1,
                                                                                                   scala.math.Ordering<ID> ord)

:: DeveloperApi :: Implementation of the ALS algorithm.

Parameters:

ratings - (undocumented)

rank - (undocumented)

numUserBlocks - (undocumented)

numItemBlocks - (undocumented)

maxIter - (undocumented)

regParam - (undocumented)

implicitPrefs - (undocumented)

alpha - (undocumented)

nonnegative - (undocumented)

intermediateRDDStorageLevel - (undocumented)

finalRDDStorageLevel - (undocumented)

checkpointInterval - (undocumented)

seed - (undocumented)

evidence$1 - (undocumented)

ord - (undocumented)

Returns:

(undocumented)

uid

public String uid()

setRank

public ALS setRank(int value)

setNumUserBlocks

public ALS setNumUserBlocks(int value)

setNumItemBlocks

public ALS setNumItemBlocks(int value)

setImplicitPrefs

public ALS setImplicitPrefs(boolean value)

setAlpha

public ALS setAlpha(double value)

setUserCol

public ALS setUserCol(String value)

setItemCol

public ALS setItemCol(String value)

setRatingCol

public ALS setRatingCol(String value)

setPredictionCol

public ALS setPredictionCol(String value)

setMaxIter

public ALS setMaxIter(int value)

setRegParam

public ALS setRegParam(double value)

setNonnegative

public ALS setNonnegative(boolean value)

setCheckpointInterval

public ALS setCheckpointInterval(int value)

setSeed

public ALS setSeed(long value)

setNumBlocks

public ALS setNumBlocks(int value)

Sets both numUserBlocks and numItemBlocks to the specific value.

Parameters:

value - (undocumented)

Returns:

(undocumented)

fit

public ALSModel fit(DataFrame dataset)

Description copied from class: Estimator

Fits a model to the input data.

Specified by:

fit in class Estimator<ALSModel>

Parameters:

dataset - (undocumented)

Returns:

(undocumented)

transformSchema

public StructType transformSchema(StructType schema)

Description copied from class: PipelineStage

:: DeveloperApi ::

Derives the output schema from the input schema.

Specified by:

transformSchema in class PipelineStage

Parameters:

schema - (undocumented)

Returns:

(undocumented)

copy

public ALS copy(ParamMap extra)

Description copied from interface: Params

Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly.

Specified by:

copy in interface Params

Specified by:

copy in class Estimator<ALSModel>

Parameters:

extra - (undocumented)

Returns:

(undocumented)

See Also:

defaultCopy()

rank

public IntParam rank()

Param for rank of the matrix factorization (>= 1). Default: 10

Returns:

(undocumented)

getRank

public int getRank()

numUserBlocks

public IntParam numUserBlocks()

Param for number of user blocks (>= 1). Default: 10

Returns:

(undocumented)

getNumUserBlocks

public int getNumUserBlocks()

numItemBlocks

public IntParam numItemBlocks()

Param for number of item blocks (>= 1). Default: 10

Returns:

(undocumented)

getNumItemBlocks

public int getNumItemBlocks()

implicitPrefs

public BooleanParam implicitPrefs()

Param to decide whether to use implicit preference. Default: false

Returns:

(undocumented)

getImplicitPrefs

public boolean getImplicitPrefs()

alpha

public DoubleParam alpha()

Param for the alpha parameter in the implicit preference formulation (>= 0). Default: 1.0

Returns:

(undocumented)

getAlpha

public double getAlpha()

ratingCol

public Param<String> ratingCol()

Param for the column name for ratings. Default: "rating"

Returns:

(undocumented)

getRatingCol

public String getRatingCol()

nonnegative

public BooleanParam nonnegative()

Param for whether to apply nonnegativity constraints. Default: false

Returns:

(undocumented)

getNonnegative

public boolean getNonnegative()

validateAndTransformSchema

public StructType validateAndTransformSchema(StructType schema)

Validates and transforms the input schema.

Parameters:

schema - input schema

Returns:

output schema

userCol

public Param<String> userCol()

Param for the column name for user ids. Default: "user"

Returns:

(undocumented)

getUserCol

public String getUserCol()

itemCol

public Param<String> itemCol()

Param for the column name for item ids. Default: "item"

Returns:

(undocumented)

getItemCol

public String getItemCol()