3.2. Tuning the hyper-parameters of an estimator

Hyper-parameters are parameters that are not directly learnt within estimators.In scikit-learn they are passed as arguments to the constructor of theestimator classes. Typical examples include C, kernel and gammafor Support Vector Classifier, alpha for Lasso, etc.

It is possible and recommended to search the hyper-parameter space for thebest cross validation score.

Any parameter provided when constructing an estimator may be optimized in thismanner. Specifically, to find the names and current values for all parametersfor a given estimator, use:

estimator.get_params()

A search consists of:

• an estimator (regressor or classifier such as sklearn.svm.SVC());

• a parameter space;

• a method for searching or sampling candidates;

• a cross-validation scheme; and

• a score function.

Some models allow for specialized, efficient parameter search strategies,outlined below.Two generic approaches to sampling search candidates are provided inscikit-learn: for given values, GridSearchCV exhaustively considersall parameter combinations, while RandomizedSearchCV can sample agiven number of candidates from a parameter space with a specifieddistribution. After describing these tools we detailbest practice applicable to both approaches.

Note that it is common that a small subset of those parameters can have a largeimpact on the predictive or computation performance of the model while otherscan be left to their default values. It is recommended to read the docstring ofthe estimator class to get a finer understanding of their expected behavior,possibly by reading the enclosed reference to the literature.

3.2.1. Exhaustive Grid Search

The grid search provided by GridSearchCV exhaustively generatescandidates from a grid of parameter values specified with the param_gridparameter. For instance, the following param_grid:

param_grid = [
  {'C': [1, 10, 100, 1000], 'kernel': ['linear']},
  {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},
 ]

specifies that two grids should be explored: one with a linear kernel andC values in [1, 10, 100, 1000], and the second one with an RBF kernel,and the cross-product of C values ranging in [1, 10, 100, 1000] and gammavalues in [0.001, 0.0001].

The GridSearchCV instance implements the usual estimator API: when“fitting” it on a dataset all the possible combinations of parameter values areevaluated and the best combination is retained.

Examples:

• See Parameter estimation using grid search with cross-validation for an example ofGrid Search computation on the digits dataset.

• See Sample pipeline for text feature extraction and evaluation for an exampleof Grid Search coupling parameters from a text documents featureextractor (n-gram count vectorizer and TF-IDF transformer) with aclassifier (here a linear SVM trained with SGD with either elasticnet or L2 penalty) using a pipeline.Pipeline instance.

• See Nested versus non-nested cross-validationfor an example of Grid Search within a cross validation loop on the irisdataset. This is the best practice for evaluating the performance of amodel with grid search.

• See Demonstration of multi-metric evaluation on cross_val_score and GridSearchCVfor an example of GridSearchCV being used to evaluate multiplemetrics simultaneously.

• See Balance model complexity and cross-validated scorefor an example of using refit=callable interface inGridSearchCV. The example shows how this interface adds certainamount of flexibility in identifying the “best” estimator. This interfacecan also be used in multiple metrics evaluation.

3.2.2. Randomized Parameter Optimization

While using a grid of parameter settings is currently the most widely usedmethod for parameter optimization, other search methods have morefavourable properties.RandomizedSearchCV implements a randomized search over parameters,where each setting is sampled from a distribution over possible parameter values.This has two main benefits over an exhaustive search:

• A budget can be chosen independent of the number of parameters and possible values.

• Adding parameters that do not influence the performance does not decrease efficiency.

Specifying how parameters should be sampled is done using a dictionary, verysimilar to specifying parameters for GridSearchCV. Additionally,a computation budget, being the number of sampled candidates or samplingiterations, is specified using the n_iter parameter.For each parameter, either a distribution over possible values or a list ofdiscrete choices (which will be sampled uniformly) can be specified:

{'C': scipy.stats.expon(scale=100), 'gamma': scipy.stats.expon(scale=.1),
  'kernel': ['rbf'], 'class_weight':['balanced', None]}

This example uses the scipy.stats module, which contains many usefuldistributions for sampling parameters, such as expon, gamma,uniform or randint.

In principle, any function can be passed that provides a rvs (randomvariate sample) method to sample a value. A call to the rvs function shouldprovide independent random samples from possible parameter values onconsecutive calls.

Warning

The distributions in scipy.stats prior to version scipy 0.16do not allow specifying a random state. Instead, they use the globalnumpy random state, that can be seeded via np.random.seed or setusing np.random.set_state. However, beginning scikit-learn 0.18,the sklearn.model_selection module sets the random state providedby the user if scipy >= 0.16 is also available.

For continuous parameters, such as C above, it is important to specifya continuous distribution to take full advantage of the randomization. This way,increasing n_iter will always lead to a finer search.

A continuous log-uniform random variable is available throughloguniform. This is a continuous version oflog-spaced parameters. For example to specify C above, loguniform(1,100) can be used instead of [1, 10, 100] or np.logspace(0, 2,num=1000). This is an alias to SciPy’s stats.reciprocal.

Mirroring the example above in grid search, we can specify a continuous randomvariable that is log-uniformly distributed between 1e0 and 1e3:

from sklearn.utils.fixes import loguniform
{'C': loguniform(1e0, 1e3),
 'gamma': loguniform(1e-4, 1e-3),
 'kernel': ['rbf'],
 'class_weight':['balanced', None]}

Examples:

• Comparing randomized search and grid search for hyperparameter estimation compares the usage and efficiencyof randomized search and grid search.

References:

• Bergstra, J. and Bengio, Y.,Random search for hyper-parameter optimization,The Journal of Machine Learning Research (2012)

3.2.3. Tips for parameter search

3.2.3.1. Specifying an objective metric

By default, parameter search uses the score function of the estimatorto evaluate a parameter setting. These are thesklearn.metrics.accuracy_score for classification andsklearn.metrics.r2_score for regression. For some applications,other scoring functions are better suited (for example in unbalancedclassification, the accuracy score is often uninformative). An alternativescoring function can be specified via the scoring parameter toGridSearchCV, RandomizedSearchCV and many of thespecialized cross-validation tools described below.See The scoring parameter: defining model evaluation rules for more details.

3.2.3.2. Specifying multiple metrics for evaluation

GridSearchCV and RandomizedSearchCV allow specifying multiple metricsfor the scoring parameter.

Multimetric scoring can either be specified as a list of strings of predefinedscores names or a dict mapping the scorer name to the scorer function and/orthe predefined scorer name(s). See Using multiple metric evaluation for more details.

When specifying multiple metrics, the refit parameter must be set to themetric (string) for which the bestparams will be found and used to buildthe bestestimator on the whole dataset. If the search should not berefit, set refit=False. Leaving refit to the default value None willresult in an error when using multiple metrics.

See Demonstration of multi-metric evaluation on cross_val_score and GridSearchCVfor an example usage.

3.2.3.3. Composite estimators and parameter spaces

GridSearchCV and RandomizedSearchCV allow searching overparameters of composite or nested estimators such asPipeline,ColumnTransformer,VotingClassifier orCalibratedClassifierCV using a dedicated<estimator>__<parameter> syntax:

>>>

>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.calibration import CalibratedClassifierCV
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import make_moons
>>> X, y = make_moons()
>>> calibrated_forest = CalibratedClassifierCV(
...    base_estimator=RandomForestClassifier(n_estimators=10))
>>> param_grid = {
...    'base_estimator__max_depth': [2, 4, 6, 8]}
>>> search = GridSearchCV(calibrated_forest, param_grid, cv=5)
>>> search.fit(X, y)
GridSearchCV(cv=5,
             estimator=CalibratedClassifierCV(...),
             param_grid={'base_estimator__max_depth': [2, 4, 6, 8]})

Here, <estimator> is the parameter name of the nested estimator,in this case base_estimator.If the meta-estimator is constructed as a collection of estimators as inpipeline.Pipeline, then <estimator> refers to the name of the estimator,see Nested parameters. In practice, there can be severallevels of nesting:

>>>

>>> from sklearn.pipeline import Pipeline
>>> from sklearn.feature_selection import SelectKBest
>>> pipe = Pipeline([
...    ('select', SelectKBest()),
...    ('model', calibrated_forest)])
>>> param_grid = {
...    'select__k': [1, 2],
...    'model__base_estimator__max_depth': [2, 4, 6, 8]}
>>> search = GridSearchCV(pipe, param_grid, cv=5).fit(X, y)

3.2.3.4. Model selection: development and evaluation

Model selection by evaluating various parameter settings can be seen as a wayto use the labeled data to “train” the parameters of the grid.

When evaluating the resulting model it is important to do it onheld-out samples that were not seen during the grid search process:it is recommended to split the data into a development set (tobe fed to the GridSearchCV instance) and an evaluation setto compute performance metrics.

This can be done by using the train_test_splitutility function.

3.2.3.5. Parallelism

GridSearchCV and RandomizedSearchCV evaluate each parametersetting independently. Computations can be run in parallel if your OSsupports it, by using the keyword n_jobs=-1. See function signature formore details.

3.2.3.6. Robustness to failure

Some parameter settings may result in a failure to fit one or more foldsof the data. By default, this will cause the entire search to fail, even ifsome parameter settings could be fully evaluated. Setting error_score=0(or =np.NaN) will make the procedure robust to such failure, issuing awarning and setting the score for that fold to 0 (or NaN), but completingthe search.

3.2.4. Alternatives to brute force parameter search

3.2.4.1. Model specific cross-validation

Some models can fit data for a range of values of some parameter almostas efficiently as fitting the estimator for a single value of theparameter. This feature can be leveraged to perform a more efficientcross-validation used for model selection of this parameter.

The most common parameter amenable to this strategy is the parameterencoding the strength of the regularizer. In this case we say that wecompute the regularization path of the estimator.

Here is the list of such models:

3.2.4.2. Information Criterion

Some models can offer an information-theoretic closed-form formula of theoptimal estimate of the regularization parameter by computing a singleregularization path (instead of several when using cross-validation).

Here is the list of models benefiting from the Akaike InformationCriterion (AIC) or the Bayesian Information Criterion (BIC) for automatedmodel selection:

3.2.4.3. Out of Bag Estimates

When using ensemble methods base upon bagging, i.e. generating newtraining sets using sampling with replacement, part of the training setremains unused. For each classifier in the ensemble, a different partof the training set is left out.

This left out portion can be used to estimate the generalization errorwithout having to rely on a separate validation set. This estimatecomes “for free” as no additional data is needed and can be used formodel selection.

This is currently implemented in the following classes: