org.apache.spark.mllib.regression

Class GeneralizedLinearAlgorithm<M extends GeneralizedLinearModel>

java.lang.Object

org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm<M>

All Implemented Interfaces:

java.io.Serializable, Logging

Direct Known Subclasses:

LassoWithSGD, LinearRegressionWithSGD, LogisticRegressionWithLBFGS, LogisticRegressionWithSGD, RidgeRegressionWithSGD, SVMWithSGD

public abstract class GeneralizedLinearAlgorithm<M extends GeneralizedLinearModel>
extends java.lang.Object
implements Logging, scala.Serializable

:: DeveloperApi :: GeneralizedLinearAlgorithm implements methods to train a Generalized Linear Model (GLM). This class should be extended with an Optimizer to create a new GLM.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
GeneralizedLinearAlgorithm()

Method Summary

Methods

Modifier and Type	Method and Description
protected boolean	addIntercept() Whether to add intercept (default: false).
protected abstract M	createModel(Vector weights, double intercept) Create a model given the weights and intercept
int	getNumFeatures() The dimension of training features.
boolean	isAddIntercept() Get if the algorithm uses addIntercept
protected int	numFeatures() The dimension of training features.
protected int	numOfLinearPredictor() In GeneralizedLinearModel, only single linear predictor is allowed for both weights and intercept.
abstract Optimizer	optimizer() The optimizer to solve the problem.
M	run(RDD<LabeledPoint> input) Run the algorithm with the configured parameters on an input RDD of LabeledPoint entries.
M	run(RDD<LabeledPoint> input, Vector initialWeights) Run the algorithm with the configured parameters on an input RDD of LabeledPoint entries starting from the initial weights provided.
GeneralizedLinearAlgorithm<M>	setIntercept(boolean addIntercept) Set if the algorithm should add an intercept.
GeneralizedLinearAlgorithm<M>	setValidateData(boolean validateData) Set if the algorithm should validate data before training.
protected boolean	validateData()
protected scala.collection.Seq<scala.Function1<RDD<LabeledPoint>,java.lang.Object>>	validators()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

GeneralizedLinearAlgorithm

public GeneralizedLinearAlgorithm()

Method Detail

validators

protected scala.collection.Seq<scala.Function1<RDD<LabeledPoint>,java.lang.Object>> validators()

optimizer

public abstract Optimizer optimizer()

The optimizer to solve the problem.

Returns:

(undocumented)

addIntercept

protected boolean addIntercept()

Whether to add intercept (default: false).

validateData

protected boolean validateData()

numOfLinearPredictor

protected int numOfLinearPredictor()

In GeneralizedLinearModel, only single linear predictor is allowed for both weights and intercept. However, for multinomial logistic regression, with K possible outcomes, we are training K-1 independent binary logistic regression models which requires K-1 sets of linear predictor.

As a result, the workaround here is if more than two sets of linear predictors are needed, we construct bigger weights vector which can hold both weights and intercepts. If the intercepts are added, the dimension of weights will be (numOfLinearPredictor) * (numFeatures + 1) . If the intercepts are not added, the dimension of weights will be (numOfLinearPredictor) * numFeatures.

Thus, the intercepts will be encapsulated into weights, and we leave the value of intercept in GeneralizedLinearModel as zero.

Returns:

(undocumented)

getNumFeatures

public int getNumFeatures()

The dimension of training features.

Returns:

(undocumented)

numFeatures

protected int numFeatures()

The dimension of training features.

Returns:

(undocumented)

createModel

protected abstract M createModel(Vector weights,
            double intercept)

Create a model given the weights and intercept

Parameters:

weights - (undocumented)

intercept - (undocumented)

Returns:

(undocumented)

isAddIntercept

public boolean isAddIntercept()

Get if the algorithm uses addIntercept

Returns:

(undocumented)

setIntercept

public GeneralizedLinearAlgorithm<M> setIntercept(boolean addIntercept)

Set if the algorithm should add an intercept. Default false. We set the default to false because adding the intercept will cause memory allocation.

Parameters:

addIntercept - (undocumented)

Returns:

(undocumented)

setValidateData

public GeneralizedLinearAlgorithm<M> setValidateData(boolean validateData)

Set if the algorithm should validate data before training. Default true.

Parameters:

validateData - (undocumented)

Returns:

(undocumented)

run

public M run(RDD<LabeledPoint> input)

Run the algorithm with the configured parameters on an input RDD of LabeledPoint entries.

Parameters:

input - (undocumented)

Returns:

(undocumented)

run

public M run(RDD<LabeledPoint> input,
    Vector initialWeights)

Run the algorithm with the configured parameters on an input RDD of LabeledPoint entries starting from the initial weights provided.

Parameters:

input - (undocumented)

initialWeights - (undocumented)

Returns:

(undocumented)