• We will cover sklearn APIs for
  • Specific classification algorithms for least square classification, perceptron, and logistic regression classifier.
    • with regularization
    • multiclass, multilabel and multi-output setting
  • Various classification metrics.

• Cross validation and hyper parameter search for classification works exactly like how it works in regression setting.
  • However there are a couple of CV strategies that are specific to classification

• Logistic regression
• Perceptron
• Ridge classifier (for LSC)
• K-nearest neighbours (KNNs)
• Support vector machines (SVMs)
• Naive Bayes

There are broadly two types of APIs based on their functionality:

Specific

Generic

• SGD classifier

Uses gradient descent for opt

Specialized solvers for opt

Need to specify loss function

All sklearn estimators for classification implement a few common methods for model training, prediction and evaluation.

Model training

fit(X, y[, coef_init, intercept_init, …])

Prediction

predict(X)

decision_function(X)

predicts confidence score for samples.

Evaluation

score(X, y[, sample_weight])

Return the mean accuracy on the given test data and labels.

There a few common miscellaneous methods as follows:

get_params([deep])

converts coefficient matrix to dense array format.

set_params(**params)

densify()

sparsify()

converts coefficient matrix to sparse format.

• treated as multi-output regression
• predicted class corresponds to the output with the highest value

• classifier first converts binary targets to {-1, 1} and then treats the problem as a regression task, optimizing the objective of regressor:

Binary classification:

Multiclass classification:

• ​minimize a penalized residual sum of squares
  • \(\min\limits_\mathbf w ||\mathbf{Xw}-\mathbf{y}||_2^2 + \alpha ||\mathbf w||_2^2\)
    • sklearn provides different solvers for this optimization
    • sklearn uses \(\alpha\) to denote regularization rate
  • predicted class corresponds to the sign of the regressor’s prediction

Step 1: Instantiate a classification estimator without passing any arguments to it. This creates a ridge classifier object.

Step 2: Call fit method on ridge classifier object with training feature matrix and label vector as arguments.

Note: The model is fitted using X_train and y_train.

Set alpha to float value.  The default value is 0.1.

• alpha should be positive.
• Larger alpha values specify stronger regularization.

Using one of the following solvers

svd 

cholesky 

sparse_cg 

lsqr 

sag , saga 

uses a Singular Value Decomposition of the feature matrix to compute the Ridge coefficients.

lbfgs 

uses scipy.linalg.solve  function to obtain the closed-form solution

uses the conjugate gradient solver of scipy.sparse.linalg.cg .

uses the dedicated regularized least-squares routine scipy.sparse.linalg.lsqr  and it is fastest.

uses a Stochastic Average Gradient descent iterative procedure

'saga' is unbiased and more flexible version of 'sag'

uses L-BFGS-B algorithm implemented in scipy.optimize.minimize .

can be used only when coefficients are forced to be positive.

• For large scale data, use 'sparse_cg ' solver.

• When both n_samples  and n_features  are large, use  ‘sag ’ or ‘saga’  solvers.

  • Note that fast convergence is only guaranteed on features with approximately the same scale.

auto 

chooses the solver automatically based on the type of data

Default choice for solver is auto .

If data is already centered, set fit_intercept as false, so that no intercept will be used in calculations.

Default:

Use predict  method to predict class labels for samples

Other classifiers also use the same predict  method.

Step 2: Call predict method on classifier object with feature matrix as an argument.

Step 1: Arrange data for prediction in a feature matrix of shape (#samples, #features) or in sparse matrix format.

• Shares the same underlying implementation with SGDClassifier 

Perceptron() 

 SGDClassifier(loss="perceptron", eta0=1, learning_rate="constant", penalty=None) 

Step 1: Instantiate a Perceptron estimator without passing any arguments to it to create a classifier object.

Step 2: Call fit method on perceptron estimator object with training feature matrix and label vector as arguments.

penalty 

(default = 'l2')

alpha 

(default = 0.0001)

l1_ratio 

(default = 0.15)

fit_intercept 

(default = True)

max_iter 

(default = 1000)

tol 

(default = 1e-3)

eta0 

(default = 1)

• This implementation can fit
  • binary classification
  • one-vs-rest (OVR)
  • multinomial logistic regression

Step 1: Instantiate a classifier estimator without passing any arguments to it. This creates a logistic regression object.

Step 2: Call fit method on logistic regression classifier object with training feature matrix and label vector as arguments

Logistic regression uses specific algorithms for solving the optimization problem in training.  These algorithms are known as solvers.

The choice of the solver depends on the classification problem set up such as size of the dataset, number of features and labels.

solver 

• For small datasets, ‘liblinear’ is a good choice, whereas ‘sag’ and ‘saga ’ are faster for large ones.

• For multiclass problems, only ‘newton-cg’, ‘sag’, ‘saga’ and ‘lbfgs ’ handle multinomial loss.
• ‘liblinear ’ is limited to one-versus-rest schemes

• For  unscaled datasets, ‘liblinear', 'lbfgs' and 'newton-cg ' are robust.

By default, logistic regression uses lbfgs solver.

Regularization is applied by default because it improves numerical stability.  

penalty 

• \(l2\) -  adds a L2 penalty term

• \(l1\) -  adds a L1 penalty term

• elasticnet - both L1 and L2 penalty terms are added

• none - no penalty is added

By default, it uses L2 penalty.

• Not all the solvers supports all the penalties.
• Select appropriate solver for the desired penalty.

  • ​L2 penalty is supported by all solvers

  • L1 penalty is supported only by a few solvers.

LogisticRegression classifier has a class_weight parameter in its constructor.

What purpose does it serve?

Exercise: Read stack overflow discussion on this parameter.

• Handles class imbalanace with differential class weights.
• Mistakes in a class are penalized by the class weight.
  • Higher value here would mean higher emphasis on the class.

This parameter is available in classifier estimators in sklearn.

• This API uses SGD as an optimization technique and can be applied to build a variety of linear classifiers by adjusting the loss parameter.

loss  parameter

'hinge' - (soft-margin) linear Support Vector Machine

'modified_huber' - smoothed hinge loss brings tolerance to outliers as well as probability estimates

'log' - logistic regression

'squared_hinge' - like hinge but is quadratically penalized

'perceptron' - linear loss used by the perceptron algorithm

‘squared_error’, ‘huber’, ‘epsilon_insensitive’, or ‘squared_epsilon_insensitive’ - regression losses

SGDClassifier(loss='log') 

 LogisticRegression(solver='sgd') 

Linear Support vector machine

• An instance of SGDClassifier might have an equivalent estimator in the scikit-learn API.

• SGDClassifier implements a plain stochastic gradient descent learning routine.
  • the gradient of the loss is estimated with one sample at a time and the model is updated along the way with a decreasing learning rate  (or strength) schedule.

Advantages:

• Efficiency.
• Ease of implementation 

Disadvantages:

• Requires a number of hyperparameters.
• Sensitive to feature scaling. 

• It is important
  • to permute (shuffle) the training data before fitting the model.
  • to standardize the features.

Step 1: Instantiate a SGDClassifer estimator by setting appropriate loss parameter to define classifier of interest.  By default it uses hinge loss, which is used for training linear support vector machine.

Step 2: Call fit method on SGD classifier object with training feature matrix and label vector as arguments.

Here we have used `log` loss that defines a logistic regression classifier.

penalty 

• \(l2\) -  adds a L2 penalty term

• \(l1\) -  adds a L1 penalty term

• elasticnet - Convex combination of L2 and L1

​(1 - l1_ratio) * L2 + l1_ratio * L1 

(l1_ratio controls the convex combination of L1 and L2 penalty. default=0.15)

Default:

Default:

max_iter = 1000 

learning_rate 

Stopping criteria

warm_start 

average 

• SGDClassifier also supports averaged SGD (ASGD)

We learnt how to implement the following classifiers with sklearn APIs:

Alternatively we can use SGDClassifier with appropriate loss setting for implementing these classifiers:

• loss = `log` for logistic regression
• loss = `perceptron` for perceptron
• loss = `squared_error` for least square classification

Classification estimators implements a few common methods like fit, score, decision_function, and predict.

​Multilabel

total #labels = 2

We will refer both these models as multi-label classification models, where # of output labels > 1.

Multiclass, multilabel, multioutput problems are referred to as multi-learning problems.

Multiclass classification

(sklearn.multiclass)

Multilabel classification

(sklearn.multioutput)

problem types

meta-estimators

Inherently multiclass

Multiclass as OVO

Multiclass as OVR

Multilabel

Inherently multiclass

Multilabel

LogisticRegression (multi_class = 'multinomial')

RidgeClassifier

LogisticRegressionCV (multi_class = 'multinomial')

RidgeClassifierCV

Multiclass as OVR

Perceptron

LogisticRegression (multi_class = 'ovr')

SGDClassifier

LogisticRegressionCV (multi_class = 'ovr')

RidgeClassifier

RidgeClassifierCV

In Iris dataset,

• There are three class labels: setosa, versicolor and virginica.
• Each example has exactly one label of the three available class labels.
• Thus, this is an instance of a multi-class classification.

In MNIST digit recognition dataset,

• There are 10 class labels: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9.
• Each example has exactly one label of the 10 available class labels.
• Thus, this is an instance of a multi-class classification.

  [[1 0 0]
   [0 0 1]
   [1 0 0]
   [0 1 0]]

target_type

‘multiclass’

‘multiclass-multioutput’

‘unknown’

y

• contains more than two discrete values
• not a sequence of sequences
• 1d or a column vector

• array-like but none of the above, such as a 3d array,
• sequence of sequences, or an array of non-sequence objects.

• 2d array that contains more than two discrete values
• not a sequence of sequences
• dimensions are of size > 1

‘multilabel-indicator’

• label indicator matrix
• an array of two dimensions with at least two columns, and at most 2 unique values.

multiclass

multiclass-multioutput

multilabel-indicator

Apart from these, there are three more types, type_of_target can determine targets corresponding to regression and binary classification.

OneVsRestClassifier

OneVsOneClassifier

• Strategy consists of fitting one classifier per target.

• Allows multiple target variable classifications.

• A multi-label model that arranges binary classifiers into a chain.

• Way of combining a number of binary classifiers into a single multi-label model.

• Able to estimate a series of target functions that are trained on a single predictor matrix to predict a series of responses.

• For a multi-label classification problem with \(k\) classes, \(k\) binary classifiers are assigned an integer between 0 and \(k-1\).
• These integers define the order of models in the chain.

• Capable of exploiting correlations among targets.

So far we learnt how to train classifiers for binary, multi-class and multi-label/output cases. 

We will learn how to evaluate these classifiers with different scoring functions and with cross-validation.

We will also study how to set hyper-parameters for classifiers.  

Many cross-validation and HPT methods discussed in the regression context are also applicable in classifiers.

• We will not repeat that discussion in this topic.
• Instead we will focus on only additional methods that are specific to classifiers.

There may be issues like class imbalance in classification, which tend to impact the cross validation folds. 

The overall class distribution and the ones in folds may be different and this has implications in effective model training.

sklearn.model_selection module provides three stratified APIs to create folds such that the overall class distribution is replicated in individual folds.

• StratifiedKFold

• RepeatedStratifiedKFold

• StratifiedShuffleSplit

Note: Folds obtained via StratifiedShuffleSplit may not be completely different.

cv specifies cross validation iterator

scoring specifies scoring function to use for HPT

cs specifies regularization strengths to experiment with.

refit = True

refit = False

Scores averaged across folds, values corresponding to the best score are selected and final refit with these parameters

 the coefs, intercepts and C that correspond to the best scores across folds are averaged.

accuracy_score

balanced_accuracy_score

top_k_accuracy_score

roc_auc_score

precision_score

recall_score

f1_score

• confusion_matrix  evaluates classification accuracy by computing the confusion matrix with each row corresponding to the true class.

Entry \(i,j\) in a confusion matrix

Example:

number of observations actually in group \(i\), but predicted to be in group \(j\).

Confusion matrix can be displayed with ConfusionMatrixDisplay API in sklearn.metrics.

The classification_report  function builds a text report showing the main classification metrics.

• Treat data as a collection of binary problems, one for each class.
• Then, average binary metric calculations across the set of classes.

• Can be done using average  parameter.

calculates the mean of the binary metrics

computes the average of binary metrics in which each class’s score is weighted by its presence in the true data sample.

gives each sample-class pair an equal contribution to the overall metric

calculates the metric over the true and predicted classes for each sample in the evaluation data, and returns their average

macro 

weighted 

micro 

samples 

None