Hyperparameter Tuning: Understanding Grid Search (2024)

Bala Priya C

Posted on • Updated on

Hyperparameter Tuning: Understanding Grid Search (3) Hyperparameter Tuning: Understanding Grid Search (4) Hyperparameter Tuning: Understanding Grid Search (5) Hyperparameter Tuning: Understanding Grid Search (6) Hyperparameter Tuning: Understanding Grid Search (7)

#beginners #machinelearning #datascience #python

In the previous blog post, Role of Cross-Validation, we had looked at how train/test split doesn’t suffice to get a reliable estimate of the out-of-sample accuracy which motivated us to understand the importance of cross-validation.

In this blog post, we shall see how we can do hyperparameter search more conveniently with GridSearchCV in scikit-learn, and learn how to use it to tune multiple hyperparameters.Let’s get started!

Time for a quick recall!

We were looking at classification problem on the iris dataset; We had used 10-fold cross-validation and the mean value of the cross-validated accuracy scores as the estimate of out-of-sample accuracy.☑
Let’s quickly walk through the steps again.

# necessary importsfrom sklearn.datasets import load_irisfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.model_selection import cross_val_scoreimport matplotlib.pyplot as plt%matplotlib inline
# read in the iris datairis = load_iris()# create X (features) and y (response)X = iris.datay = iris.target

# 10-fold cross-validation with K=5 for KNN (the n_neighbors parameter)knn = KNeighborsClassifier(n_neighbors=5)scores = cross_val_score(knn, X, y, cv=10, scoring='accuracy')# use average accuracy as an estimate of out-of-sample accuracyprint(scores.mean())# Output0.9666666666666668

We then wanted to identify the best value of K for the KNN classifier. To find the best K, we looped through a possible set of values for K (n_neighbors) and then picked the value of K that gave the highest cross-validated accuracy.

In KNN classifier, setting a very small value for K will make the model needlessly complex, and a very large value of K would result in a model with high bias, that yields suboptimal performance.

# search for an optimal value of K for KNNk_range = list(range(1, 31))k_scores = []for k in k_range: knn = KNeighborsClassifier(n_neighbors=k) scores = cross_val_score(knn, X, y, cv=10, scoring='accuracy') k_scores.append(scores.mean())# plot the value of K for KNN (x-axis) versus the cross-validated accuracy (y-axis)plt.plot(k_range, k_scores)plt.xlabel('Value of K for KNN')plt.ylabel('Cross-Validated Accuracy')

Hyperparameter Tuning: Understanding Grid Search (8)

As K=13,18 and 20 gave the highest accuracy score, close to 0.98, we decided to choose K=20 as a larger value of K would yield a less complex model. While it is not particularly difficult to write the for loop, we do realize that we may have to do it often.
Therefore, it would be good to have a more convenient way of doing the hyperparameter search, without having to write a loop every time and identify the best parameter through inspection.

Understanding GridSearchCV

Let’s go ahead and import the GridSearchCV class.

from sklearn.model_selection import GridSearchCV

Define the Parameter Grid

We now define the parameter grid (param_grid), a Python dictionary, whose key is the name of the hyperparameter whose best value we’re trying to find and the value is the list of possible values that we would like to search over for the hyperparameter.

# define the parameter values that should be searchedk_range = list(range(1, 31))# create a parameter grid: map the parameter names to the values that should be searchedparam_grid = dict(n_neighbors=k_range)print(param_grid)# param_grid{'n_neighbors': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]}

We now instantiate GridSearchCV . Note that we specify the param_grid instead of n_neighbors argument that we had specified for cross_val_score earlier. Why is this valid?

Remember, the parameter grid, param_grid is a dictionary whose key is n_neighbors and the value is a list of possible values of n_neighbors. Therefore, specifying the param_grid ensures that the value at index i is fetched as the value of n_neighbors in the i_th run.

Instantiate, fit grid and view results

# instantiate the gridgrid = GridSearchCV(knn, param_grid, cv=10, scoring='accuracy', return_train_score=False)

We now go ahead and fit the grid with data, and access the cv_results_ attribute to get the mean accuracy score after 10-fold cross-validation, standard deviation and the parameter values. For convenience, we may store the results in a pandas DataFrame. The mean and standard deviation of the accuracy scores for n_neighbors =1 to 10 are shown below.

# fit the grid with datagrid.fit(X, y)# view the results as a pandas DataFrameimport pandas as pdpd.DataFrame(grid.cv_results_)[['mean_test_score', 'std_test_score', 'params']]# Outputmean_test_score std_test_score params0 0.960000 0.053333 {'n_neighbors': 1}1 0.953333 0.052068 {'n_neighbors': 2}2 0.966667 0.044721 {'n_neighbors': 3}3 0.966667 0.044721 {'n_neighbors': 4}4 0.966667 0.044721 {'n_neighbors': 5}5 0.966667 0.044721 {'n_neighbors': 6}6 0.966667 0.044721 {'n_neighbors': 7}7 0.966667 0.044721 {'n_neighbors': 8}8 0.973333 0.032660 {'n_neighbors': 9}9 0.966667 0.044721 {'n_neighbors': 10}

When using cross_val_score, we tried eyeballing the accuracy scores to identify the best hyperparameters and to make it easier, we plotted the value of hyperparameters vs the respective cross-validated accuracy scores!

Sounds good but doesn’t seem to be a great option!

Once we’ve completed the grid search, the following attributes can be very useful!
We can choose to examine:
☑ the best_score_ , the highest cross-validated accuracy score
☑ the best_params_, the optimal value for the hyperparameters, and
☑ the best_estimator_, which is the best model that has the best hyperparameter.
Let us now examine these for our example.

# examine the best modelprint(grid.best_score_)print(grid.best_params_)print(grid.best_estimator_)# Output0.9800000000000001{'n_neighbors': 13}KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', metric_params=None, n_jobs=None, n_neighbors=13, p=2, weights='uniform')

K=13 has been chosen, remember, K=13 was one of the values of K that gave highest cross-validated accuracy score.✔
So far so good!

Searching for multiple parameters

In this example, the only hyperparameter that we searched for was n_neighbors. What if there were many such hyperparameters?

We may think, “Why not tune each hyperparameter independently?”
Well, we may independently search for the optimal values for each of the hyperparameters; but the model may perform best at some values of the parameters that are very different from the individual best values. So, we have to search for the combination of the parameters that optimizes performance rather than the individual best parameters.

Let us build on the same example of KNNClassifier, in addition to n_neighbors, let us also search for the optimal weighting strategy.
The default weighting option is ‘uniform’ where all points are weighted equally and ‘distance’ option weights points by the inverse of their distance. In this case, closer neighbors of a query point will have a greater influence than neighbors which are further away.

# define the parameter values that should be searchedk_range = list(range(1, 31))weight_options = ['uniform', 'distance']# create a parameter grid: map the parameter names to the values that should be searchedparam_grid = dict(n_neighbors=k_range, weights=weight_options)print(param_grid)# param_grid{'n_neighbors': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30],'weights': ['uniform', 'distance']}

Let us instantiate and fit grid and view results, as before.

# instantiate and fit the gridgrid = GridSearchCV(knn, param_grid, cv=10, scoring='accuracy', return_train_score=False) grid.fit(X, y)# view the resultspd.DataFrame(grid.cv_results_)[['mean_test_score', 'std_test_score', 'params']]# Resultsmean_test_score std_test_score params0 0.960000 0.053333 {'n_neighbors': 1, 'weights': 'uniform'}1 0.960000 0.053333 {'n_neighbors': 1, 'weights': 'distance'}2 0.953333 0.052068 {'n_neighbors': 2, 'weights': 'uniform'}3 0.960000 0.053333 {'n_neighbors': 2, 'weights': 'distance'}4 0.966667 0.044721 {'n_neighbors': 3, 'weights': 'uniform'}5 0.966667 0.044721 {'n_neighbors': 3, 'weights': 'distance'}6 0.966667 0.044721 {'n_neighbors': 4, 'weights': 'uniform'}7 0.966667 0.044721 {'n_neighbors': 4, 'weights': 'distance'}8 0.966667 0.044721 {'n_neighbors': 5, 'weights': 'uniform'}9 0.966667 0.044721 {'n_neighbors': 5, 'weights': 'distance'}10 0.966667 0.044721 {'n_neighbors': 6, 'weights': 'uniform'}11 0.966667 0.044721 {'n_neighbors': 6, 'weights': 'distance'}12 0.966667 0.044721 {'n_neighbors': 7, 'weights': 'uniform'}13 0.966667 0.044721 {'n_neighbors': 7, 'weights': 'distance'}14 0.966667 0.044721 {'n_neighbors': 8, 'weights': 'uniform'}15 0.966667 0.044721 {'n_neighbors': 8, 'weights': 'distance'}16 0.973333 0.032660 {'n_neighbors': 9, 'weights': 'uniform'}17 0.973333 0.032660 {'n_neighbors': 9, 'weights': 'distance'}18 0.966667 0.044721 {'n_neighbors': 10, 'weights': 'uniform'}

Results of Grid Search for two parameters (Truncated View of the DataFrame)
Only the first ten rows of the results have been shown above; We actually have 30*2=60 models (as we had 30 possible values for n_neighbors and 2 possible values for weights) and as we chose 10-fold cross-validation, there will be 60*10=600 predictions made! Time to look at our model’s best score and parameters that yielded the best score.

# examine the best modelprint(grid.best_score_)print(grid.best_params_)# best score and best parameters0.9800000000000001{'n_neighbors': 13, 'weights': 'uniform'}

We obtain the same best cross-validated accuracy score of 0.98, with n_neighbors=13 and weights= ‘uniform’. Now, let us say we have to tune 4 hyperparameters and we have a list of 10 possible values for each of the hyperparameters. This process creates 10*10*10*10 =10,000 models and when we run 10 fold cross-validation, there are 100,000 predictions made. Clearly, things do scale up very quickly and can soon become computationally infeasible.

More efficient hyperparameter searches such as Randomized Search and Informed Search can be very useful in overcoming this drawback. Let us cover them in a subsequent blog post.
Until then, Happy Learning!

References

[1] Here’s the link to the Google Colab notebook for the above used example.
[2] Introduction to Machine Learning in Python with scikit-learn by DataSchool.
[3] Scikit-learn documentation: GridSearchCV
http://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

Cover Image: Photo by Kelly Sikkema on Unsplash

Hyperparameter Tuning: Understanding Grid Search (2024)

References

Top Articles
Latest Posts
Recommended Articles
Article information

Author: Gov. Deandrea McKenzie

Last Updated:

Views: 6107

Rating: 4.6 / 5 (46 voted)

Reviews: 85% of readers found this page helpful

Author information

Name: Gov. Deandrea McKenzie

Birthday: 2001-01-17

Address: Suite 769 2454 Marsha Coves, Debbieton, MS 95002

Phone: +813077629322

Job: Real-Estate Executive

Hobby: Archery, Metal detecting, Kitesurfing, Genealogy, Kitesurfing, Calligraphy, Roller skating

Introduction: My name is Gov. Deandrea McKenzie, I am a spotless, clean, glamorous, sparkling, adventurous, nice, brainy person who loves writing and wants to share my knowledge and understanding with you.