public final class QuantileDiscretizer extends Estimator<Bucketizer> implements QuantileDiscretizerBase, DefaultParamsWritable, HasInputCols, HasOutputCols
QuantileDiscretizer takes a column with continuous features and outputs a column with binned
categorical features. The number of bins can be set using the numBuckets parameter. It is
possible that the number of buckets used will be smaller than this value, for example, if there
are too few distinct values of the input to create enough distinct quantiles.
Since 2.3.0, QuantileDiscretizer can map multiple columns at once by setting the inputCols
parameter. If both of the inputCol and inputCols parameters are set, an Exception will be
thrown. To specify the number of buckets for each column, the numBucketsArray parameter can
be set, or if the number of buckets should be the same across columns, numBuckets can be
set as a convenience.
NaN handling:
null and NaN values will be ignored from the column during QuantileDiscretizer fitting. This
will produce a Bucketizer model for making predictions. During the transformation,
Bucketizer will raise an error when it finds NaN values in the dataset, but the user can
also choose to either keep or remove NaN values within the dataset by setting handleInvalid.
If the user chooses to keep NaN values, they will be handled specially and placed into their own
bucket, for example, if 4 buckets are used, then non-NaN data will be put into buckets[0-3],
but NaNs will be counted in a special bucket[4].
Algorithm: The bin ranges are chosen using an approximate algorithm (see the documentation for
org.apache.spark.sql.DataFrameStatFunctions.approxQuantile
for a detailed description). The precision of the approximation can be controlled with the
relativeError parameter. The lower and upper bin bounds will be -Infinity and +Infinity,
covering all real values.
| Constructor and Description |
|---|
QuantileDiscretizer() |
QuantileDiscretizer(String uid) |
| Modifier and Type | Method and Description |
|---|---|
QuantileDiscretizer |
copy(ParamMap extra)
Creates a copy of this instance with the same UID and some extra params.
|
Bucketizer |
fit(Dataset<?> dataset)
Fits a model to the input data.
|
static QuantileDiscretizer |
load(String path) |
static MLReader<T> |
read() |
QuantileDiscretizer |
setHandleInvalid(String value) |
QuantileDiscretizer |
setInputCol(String value) |
QuantileDiscretizer |
setInputCols(String[] value) |
QuantileDiscretizer |
setNumBuckets(int value) |
QuantileDiscretizer |
setNumBucketsArray(int[] value) |
QuantileDiscretizer |
setOutputCol(String value) |
QuantileDiscretizer |
setOutputCols(String[] value) |
QuantileDiscretizer |
setRelativeError(double value) |
StructType |
transformSchema(StructType schema)
:: DeveloperApi ::
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitgetNumBuckets, getNumBucketsArray, getRelativeError, handleInvalid, numBuckets, numBucketsArray, relativeErrorgetHandleInvalidgetInputCol, inputColgetOutputCol, outputColclear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwntoStringwritesavegetInputCols, inputColsgetOutputCols, outputColsinitializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarningpublic QuantileDiscretizer(String uid)
public QuantileDiscretizer()
public static QuantileDiscretizer load(String path)
public static MLReader<T> read()
public String uid()
Identifiableuid in interface Identifiablepublic QuantileDiscretizer setRelativeError(double value)
public QuantileDiscretizer setNumBuckets(int value)
public QuantileDiscretizer setInputCol(String value)
public QuantileDiscretizer setOutputCol(String value)
public QuantileDiscretizer setHandleInvalid(String value)
public QuantileDiscretizer setNumBucketsArray(int[] value)
public QuantileDiscretizer setInputCols(String[] value)
public QuantileDiscretizer setOutputCols(String[] value)
public StructType transformSchema(StructType schema)
PipelineStageCheck transform validity and derive the output schema from the input schema.
We check validity for interactions between parameters during transformSchema and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate().
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema in class PipelineStageschema - (undocumented)public Bucketizer fit(Dataset<?> dataset)
Estimatorfit in class Estimator<Bucketizer>dataset - (undocumented)public QuantileDiscretizer copy(ParamMap extra)
ParamsdefaultCopy().copy in interface Paramscopy in class Estimator<Bucketizer>extra - (undocumented)