Featured Photo by Torsten Dettlaff on Pexels
- Handwritten character recognition is one of the practically important issues in pattern recognition applications. Applications of digit recognition include national ID number recognition, postal mail sorting, automatic license plate recognition, bank check processing, form data entry, etc.
- In spite of recent advances in image recognition and computer vision technology, Handwritten Digit Recognition (HDR) remains largely unsolved due to the presence of many ambiguous handwritten digits.
- In this post, we are going to implement the Supervised Machine Learning (ML) and Deep Learning (DL) Scikit-Learn [3, 4] algorithms using the MNIST dataset [5, 6, 9].
- Our goal is to compare various multi-label HDR classifiers based on available ML and DL performance metrics.
The article consists of the following three parts:
- Deep Learning (DL) using (a)
sklearn.neural_network.MLPClassifier = Multi-Layer Perceptron Classifier (MLP) and (b) Convolution Neural Network (CNN) with Keras and TensorFlow [1, 2] (see Appendix A); - Supervised Machine Learning (ML) using the following 12 scikit-learn multi-label classifiers [3, 4, 7, 10, 11]
- Support Vector Machines (SVM)
- Decision Trees (DT)
- Random Forest Classifier (RC)
- Logistic Regression (LR) [8]
- Extra Trees Classifier (ET)
- Gradient Boosting Classifier (GB)
- K-Nearest Neighbors (KNN)
- Naive Bayes (GNB)
- Stochastic Gradient Descent (SGD)
- Perceptron (PERC)
- Linear SVC (LSVC)
- Quadratic Discriminant Analysis (QDA)
3. Unsupervised ML using the Principal Component Analysis (PCA) for the dimensionality reduction within Parts 1 and 2.
Our main goal is to build a text and graphics report comparing the main scikit-learn classification metrics: accuracy_score, classification_report (precision, recall, and f1-score), confusion_matrix, Classification Learning Curve, ROC Curve, Precision-Recall Curve, cohen_kappa_score, and r2_score.
Key Libraries
Let’s set the working directory YOURPATH
import os
os.chdir(‘YOURPATH’)
os. getcwd()
and import the key libraries
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn import datasets
import scikitplot as skplt
import sklearn
from sklearn.datasets import load_digits, load_boston, load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor, GradientBoostingClassifier, ExtraTreesClassifier
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import sys
import warnings
warnings.filterwarnings(“ignore”)
print(“Scikit Plot Version : “, skplt.version)
print(“Scikit Learn Version : “, sklearn.version)
print(“Python Version : “, sys.version)
%matplotlib inline
Scikit Plot Version : 0.3.7 Scikit Learn Version : 1.0.2 Python Version : 3.9.13 (main, Aug 25 2022, 23:51:50)
Input Data Part 1
Let’s load the HDR dataset for DL classifiers [5, 6]
digits = datasets.load_digits()
digits.keys()
dict_keys(['data', 'target', 'frame', 'feature_names', 'target_names', 'images', 'DESCR'])
Let’s get the description key in Appendix C
print(digits.DESCR)
Let’s print out other keys
print(digits.feature_names)
['pixel_0_0', 'pixel_0_1', 'pixel_0_2', 'pixel_0_3', 'pixel_0_4', 'pixel_0_5', 'pixel_0_6', 'pixel_0_7', 'pixel_1_0', 'pixel_1_1', 'pixel_1_2', 'pixel_1_3', 'pixel_1_4', 'pixel_1_5', 'pixel_1_6', 'pixel_1_7', 'pixel_2_0', 'pixel_2_1', 'pixel_2_2', 'pixel_2_3', 'pixel_2_4', 'pixel_2_5', 'pixel_2_6', 'pixel_2_7', 'pixel_3_0', 'pixel_3_1', 'pixel_3_2', 'pixel_3_3', 'pixel_3_4', 'pixel_3_5', 'pixel_3_6', 'pixel_3_7', 'pixel_4_0', 'pixel_4_1', 'pixel_4_2', 'pixel_4_3', 'pixel_4_4', 'pixel_4_5', 'pixel_4_6', 'pixel_4_7', 'pixel_5_0', 'pixel_5_1', 'pixel_5_2', 'pixel_5_3', 'pixel_5_4', 'pixel_5_5', 'pixel_5_6', 'pixel_5_7', 'pixel_6_0', 'pixel_6_1', 'pixel_6_2', 'pixel_6_3', 'pixel_6_4', 'pixel_6_5', 'pixel_6_6', 'pixel_6_7', 'pixel_7_0', 'pixel_7_1', 'pixel_7_2', 'pixel_7_3', 'pixel_7_4', 'pixel_7_5', 'pixel_7_6', 'pixel_7_7']
print(digits.target_names)
[0 1 2 3 4 5 6 7 8 9]
digits.images[0]
array([[ 0., 0., 5., 13., 9., 1., 0., 0.],
[ 0., 0., 13., 15., 10., 15., 5., 0.],
[ 0., 3., 15., 2., 0., 11., 8., 0.],
[ 0., 4., 12., 0., 0., 8., 8., 0.],
[ 0., 5., 8., 0., 0., 9., 8., 0.],
[ 0., 4., 11., 0., 1., 12., 7., 0.],
[ 0., 2., 14., 5., 10., 12., 0., 0.],
[ 0., 0., 6., 13., 10., 0., 0., 0.]])
Input Data Part 2
Let’s work with ML classifiers
import matplotlib.pyplot as plt
from sklearn import datasets
digits = datasets.load_digits()
Our target names remain the same
print(digits.target_names)
[0 1 2 3 4 5 6 7 8 9]
Let’s plot first 5 HDR images
plt.subplot(321)
plt.imshow(digits.images[0], cmap=plt.cm.gray_r,
interpolation=’nearest’)
plt.subplot(322)
plt.imshow(digits.images[1], cmap=plt.cm.gray_r, interpolation=’nearest’)
plt.subplot(323)
plt.imshow(digits.images[2], cmap=plt.cm.gray_r, interpolation=’nearest’)
plt.subplot(324)
plt.imshow(digits.images[3], cmap=plt.cm.gray_r, interpolation=’nearest’)
plt.subplot(325)
plt.imshow(digits.images[4], cmap=plt.cm.gray_r, interpolation=’nearest’)
plt.subplot(326)
plt.imshow(digits.images[5], cmap=plt.cm.gray_r, interpolation=’nearest’)
We can also plot the first n=5 images
for i in range(n):
plt.matshow(digits.images[i],cmap=plt.cm.gray_r)
with the target variable
digits.target[0:5]
array([0, 1, 2, 3, 4])
and the target size
digits.target.size
1797
Let’s display the digit ‘7’
#display digit
%matplotlib inline
def show_digit(index):
plt.imshow(digits.images[index], cmap=plt.cm.gray_r, interpolation=’nearest’)
plt.title(‘The digit is: ‘+ str(digits.target[index]))
plt.show()
Exploratory Data Analysis (EDA)
Let’s split the target and feature variables in our dataset
main_data=digits[‘data’]
targets=digits[‘target’]
Let’s select the SVC classifier and begin with the [:1791] data partitioning
from sklearn import svm
from sklearn.metrics import accuracy_score
svc = svm.SVC(gamma=0.00001,C=100.)
svc.fit(digits.data[1:1791],digits.target[1:1791])
prediction = svc.predict(main_data[1791:])
print(“Predicted values: “, prediction)
print(“Actual values: “, targets[1791:])
ac_1 = accuracy_score(targets[1791:] , prediction)*100
print(“Accuracy is: “,ac_1,”%”)
Predicted values: [4 9 0 8 9 8] Actual values: [4 9 0 8 9 8] Accuracy is: 100.0 %
from sklearn.svm import SVC
svc = SVC()
SVC(C=100., cache_size=200,class_weight=None, coef0=0.0,degree=3,
gamma=0.001, kernel=’rbf’,max_iter=-1,probability=False,
random_state=None, shrinking=True, tol=0.001, verbose=False)
svc = svm.SVC(gamma=0.001 , C=100.)
svc.fit(main_data[:1791], targets[:1791])
SVC(C=100.0, gamma=0.001)
print(“Predicted values: “, prediction)
print(“Actual values: “, targets[1791:])
print(“Accuracy is: “,(accuracy_score(targets[1791:] , prediction)*100),”%”)
Predicted values: [4 9 0 8 9 8] Actual values: [4 9 0 8 9 8] Accuracy is: 100.0 %
Let’s select the [:1347] data partitioning
svc.fit(main_data[:1347], targets[:1347])
prediction = svc.predict(main_data[1347:])
ac_2 = accuracy_score(targets[1347:] , prediction)*100
print(“Accuracy is: “, ac_2,”%”)
Accuracy is: 96.88888888888889 %
Let’s select the [:899] data partitioning
svc.fit(main_data[:899], targets[:899])
prediction = svc.predict(main_data[899:])
ac_3 = accuracy_score(targets[899:] , prediction)*100
print(“Accuracy is: “, ac_3,”%”)
Accuracy is: 96.99331848552339 %
So, we can split data into the 50% train and 50% test subsets without compromising the ML accuracy.
Part 1: Deep Learning
MLP
Let’s prepare our input data
from sklearn import datasets
digits = datasets.load_digits()
digits.images.shape
(1797, 8, 8)
dir(digits)
['DESCR', 'data', 'feature_names', 'frame', 'images', 'target', 'target_names']
Let’s plot a few selected training images
import matplotlib.pyplot as plt
plt.imshow(digits.images[0],cmap=’binary’)
plt.show()
def plot_multi(i):
”’Plots 16 digits, starting with digit i”’
nplots = 16
fig = plt.figure(figsize=(15,15))
for j in range(nplots):
plt.subplot(4,4,j+1)
plt.imshow(digits.images[i+j], cmap=’binary’)
plt.title(digits.target[i+j])
plt.axis(‘off’)
plt.show()
plot_multi(0)
Let’s split our input data
y = digits.target
x = digits.images.reshape((len(digits.images), -1))
x.shape
(1797, 64)
into the training and test sets
x_train = x[:1000]
y_train = y[:1000]
x_test = x[1000:]
y_test = y[1000:]
Let’s train the MLP model
from sklearn.neural_network import MLPClassifier
mlp = MLPClassifier(hidden_layer_sizes=(15,), activation=’logistic’, alpha=1e-4,
solver=’sgd’, tol=1e-4, random_state=1,
learning_rate_init=.1, verbose=True)
mlp.fit(x_train,y_train)
Iteration 1, loss = 2.22958289 Iteration 2, loss = 1.91207743 Iteration 3, loss = 1.62507727 Iteration 4, loss = 1.32649842 Iteration 5, loss = 1.06100535 Iteration 6, loss = 0.83995513 Iteration 7, loss = 0.67806075 Iteration 8, loss = 0.55175832 Iteration 9, loss = 0.45840445 Iteration 10, loss = 0.39149735 Iteration 11, loss = 0.33676351 Iteration 12, loss = 0.29059880 Iteration 13, loss = 0.25437208 Iteration 14, loss = 0.22838372 Iteration 15, loss = 0.20200554 Iteration 16, loss = 0.18186565 Iteration 17, loss = 0.16461183 Iteration 18, loss = 0.14990228 Iteration 19, loss = 0.13892154 Iteration 20, loss = 0.12833784 Iteration 21, loss = 0.12138920 Iteration 22, loss = 0.11407971 Iteration 23, loss = 0.10677664 Iteration 24, loss = 0.10037149 Iteration 25, loss = 0.09593187 Iteration 26, loss = 0.09250135 Iteration 27, loss = 0.08676698 Iteration 28, loss = 0.08356043 Iteration 29, loss = 0.08209789 Iteration 30, loss = 0.07649168 Iteration 31, loss = 0.07410898 Iteration 32, loss = 0.07126869 Iteration 33, loss = 0.06926956 Iteration 34, loss = 0.06578496 Iteration 35, loss = 0.06374913 Iteration 36, loss = 0.06175492 Iteration 37, loss = 0.05975664 Iteration 38, loss = 0.05764485 Iteration 39, loss = 0.05623663 Iteration 40, loss = 0.05420966 Iteration 41, loss = 0.05413911 Iteration 42, loss = 0.05256140 Iteration 43, loss = 0.05020265 Iteration 44, loss = 0.04902779 Iteration 45, loss = 0.04788382 Iteration 46, loss = 0.04655532 Iteration 47, loss = 0.04586089 Iteration 48, loss = 0.04451758 Iteration 49, loss = 0.04341598 Iteration 50, loss = 0.04238096 Iteration 51, loss = 0.04162200 Iteration 52, loss = 0.04076839 Iteration 53, loss = 0.04003180 Iteration 54, loss = 0.03907774 Iteration 55, loss = 0.03815565 Iteration 56, loss = 0.03791975 Iteration 57, loss = 0.03706276 Iteration 58, loss = 0.03617874 Iteration 59, loss = 0.03593227 Iteration 60, loss = 0.03504175 Iteration 61, loss = 0.03441259 Iteration 62, loss = 0.03397449 Iteration 63, loss = 0.03326990 Iteration 64, loss = 0.03305025 Iteration 65, loss = 0.03244893 Iteration 66, loss = 0.03191504 Iteration 67, loss = 0.03132169 Iteration 68, loss = 0.03079707 Iteration 69, loss = 0.03044946 Iteration 70, loss = 0.03005546 Iteration 71, loss = 0.02960555 Iteration 72, loss = 0.02912799 Iteration 73, loss = 0.02859103 Iteration 74, loss = 0.02825959 Iteration 75, loss = 0.02788968 Iteration 76, loss = 0.02748725 Iteration 77, loss = 0.02721247 Iteration 78, loss = 0.02686225 Iteration 79, loss = 0.02635636 Iteration 80, loss = 0.02607439 Iteration 81, loss = 0.02577613 Iteration 82, loss = 0.02553642 Iteration 83, loss = 0.02518749 Iteration 84, loss = 0.02484300 Iteration 85, loss = 0.02455379 Iteration 86, loss = 0.02432480 Iteration 87, loss = 0.02398548 Iteration 88, loss = 0.02376004 Iteration 89, loss = 0.02341261 Iteration 90, loss = 0.02318255 Iteration 91, loss = 0.02296065 Iteration 92, loss = 0.02274048 Iteration 93, loss = 0.02241054 Iteration 94, loss = 0.02208181 Iteration 95, loss = 0.02190861 Iteration 96, loss = 0.02174404 Iteration 97, loss = 0.02156939 Iteration 98, loss = 0.02119768 Iteration 99, loss = 0.02101874 Iteration 100, loss = 0.02078230 Iteration 101, loss = 0.02061573 Iteration 102, loss = 0.02039802 Iteration 103, loss = 0.02017245 Iteration 104, loss = 0.01997162 Iteration 105, loss = 0.01989280 Iteration 106, loss = 0.01963828 Iteration 107, loss = 0.01941850 Iteration 108, loss = 0.01933154 Iteration 109, loss = 0.01911473 Iteration 110, loss = 0.01905371 Iteration 111, loss = 0.01876085 Iteration 112, loss = 0.01860656 Iteration 113, loss = 0.01848655 Iteration 114, loss = 0.01834844 Iteration 115, loss = 0.01818981 Iteration 116, loss = 0.01798523 Iteration 117, loss = 0.01783630 Iteration 118, loss = 0.01771441 Iteration 119, loss = 0.01749814 Iteration 120, loss = 0.01738339 Iteration 121, loss = 0.01726549 Iteration 122, loss = 0.01709638 Iteration 123, loss = 0.01698340 Iteration 124, loss = 0.01684606 Iteration 125, loss = 0.01667016 Iteration 126, loss = 0.01654172 Iteration 127, loss = 0.01641832 Iteration 128, loss = 0.01630111 Iteration 129, loss = 0.01623051 Iteration 130, loss = 0.01612736 Iteration 131, loss = 0.01590220 Iteration 132, loss = 0.01582485 Iteration 133, loss = 0.01571372 Iteration 134, loss = 0.01560349 Iteration 135, loss = 0.01557688 Iteration 136, loss = 0.01534420 Iteration 137, loss = 0.01527883 Iteration 138, loss = 0.01517545 Iteration 139, loss = 0.01503663 Iteration 140, loss = 0.01501192 Iteration 141, loss = 0.01482535 Iteration 142, loss = 0.01471388 Iteration 143, loss = 0.01463948 Iteration 144, loss = 0.01454059 Iteration 145, loss = 0.01441742 Iteration 146, loss = 0.01431741 Iteration 147, loss = 0.01428414 Iteration 148, loss = 0.01416364 Iteration 149, loss = 0.01406742 Iteration 150, loss = 0.01402651 Iteration 151, loss = 0.01389720 Iteration 152, loss = 0.01381412 Iteration 153, loss = 0.01371300 Iteration 154, loss = 0.01362465 Iteration 155, loss = 0.01357048 Iteration 156, loss = 0.01348760 Iteration 157, loss = 0.01339543 Iteration 158, loss = 0.01331941 Iteration 159, loss = 0.01320812 Iteration 160, loss = 0.01315415 Iteration 161, loss = 0.01308279 Iteration 162, loss = 0.01302708 Iteration 163, loss = 0.01290042 Iteration 164, loss = 0.01289267 Iteration 165, loss = 0.01277558 Iteration 166, loss = 0.01277238 Iteration 167, loss = 0.01261308 Iteration 168, loss = 0.01260611 Iteration 169, loss = 0.01248789 Iteration 170, loss = 0.01239662 Iteration 171, loss = 0.01231743 Iteration 172, loss = 0.01227346 Iteration 173, loss = 0.01223136 Iteration 174, loss = 0.01217211 Iteration 175, loss = 0.01208682 Iteration 176, loss = 0.01204707 Iteration 177, loss = 0.01200225 Iteration 178, loss = 0.01188677 Iteration 179, loss = 0.01184993 Iteration 180, loss = 0.01175130 Iteration 181, loss = 0.01171178 Iteration 182, loss = 0.01166052 Iteration 183, loss = 0.01163843 Iteration 184, loss = 0.01154892 Iteration 185, loss = 0.01147629 Iteration 186, loss = 0.01142365 Iteration 187, loss = 0.01136608 Iteration 188, loss = 0.01128053 Iteration 189, loss = 0.01128869 Training loss did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.
Out[14]:
MLPClassifier(activation='logistic', hidden_layer_sizes=(15,),
learning_rate_init=0.1, random_state=1, solver='sgd',
verbose=True)
Let’s plot the above loss values vs 0<epochs<190
loss_values = mlp.loss_curve_
import matplotlib.pyplot as plt
plt.plot(loss_values)
plt.savefig(‘mlploss.png’)
Let’s predict our test values with mlp
predictions = mlp.predict(x_test)
and look at the first 50 samples
predictions[:50]
array([1, 4, 0, 5, 3, 6, 9, 6, 1, 7, 5, 4, 4, 7, 2, 8, 2, 2, 5, 7, 9, 5,
4, 4, 9, 0, 8, 9, 8, 0, 1, 2, 3, 4, 5, 6, 7, 8, 3, 0, 1, 2, 3, 4,
5, 6, 7, 8, 5, 0])
y_test[:50]
array([1, 4, 0, 5, 3, 6, 9, 6, 1, 7, 5, 4, 4, 7, 2, 8, 2, 2, 5, 7, 9, 5,
4, 4, 9, 0, 8, 9, 8, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4,
5, 6, 7, 8, 9, 0])
Let’s check the accuracy
from sklearn.metrics import accuracy_score
accuracy_score(y_test, predictions)
0.9146800501882058
Let’s print out the classification report
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
print(classification_report(predictions, y_test))
precision recall f1-score support
0 0.95 0.96 0.96 78
1 0.89 0.88 0.88 81
2 0.91 0.93 0.92 75
3 0.82 0.86 0.84 76
4 0.95 0.98 0.96 81
5 0.96 0.89 0.92 89
6 0.99 0.98 0.98 81
7 0.95 0.99 0.97 77
8 0.83 0.90 0.86 70
9 0.89 0.81 0.85 89
accuracy 0.91 797
macro avg 0.91 0.92 0.91 797
weighted avg 0.92 0.91 0.91 797
The f1-score per label is given by
from sklearn.metrics import f1_score
f1mlp=f1_score(predictions, y_test, average=None)
print(f1mlp)
[0.95541401 0.88198758 0.92105263 0.83870968 0.96341463 0.92397661 0.98136646 0.96815287 0.8630137 0.84705882]
Let’s plot these values
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1mlp[0], f1mlp[1], f1mlp[2],
f1mlp[3], f1mlp[4], f1mlp[5],
f1mlp[6], f1mlp[7],f1mlp[8],f1mlp[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score”)
plt.savefig(‘f1scoremlp.png’)
Let’s look at the precision score
from sklearn.metrics import precision_score
precisionmlp=precision_score(predictions, y_test, average=None)
print(precisionmlp)
[0.94936709 0.8875 0.90909091 0.82278481 0.95180723 0.96341463 0.9875 0.95 0.82894737 0.88888889]
and plot this attribute
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionmlp[0], precisionmlp[1], precisionmlp[2],
precisionmlp[3], precisionmlp[4], precisionmlp[5],
precisionmlp[6], precisionmlp[7],precisionmlp[8],precisionmlp[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score”)
plt.savefig(‘precisionscoremlp.png’)
Let’s calculate the recall score
from sklearn.metrics import recall_score
recallmlp=recall_score(predictions, y_test, average=None)
print(recallmlp)
[0.96153846 0.87654321 0.93333333 0.85526316 0.97530864 0.88764045 0.97530864 0.98701299 0.9 0.80898876]
and plot this attribute
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallmlp[0], recallmlp[1], recallmlp[2],
recallmlp[3], recallmlp[4], recallmlp[5],
recallmlp[6], recallmlp[7],recallmlp[8],recallmlp[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score”)
plt.savefig(‘recallscoremlp.png’)
Let’s calculate the Cohen’s kappa
from sklearn.metrics import cohen_kappa_score
cohmlp=cohen_kappa_score(predictions, y_test)
print(cohmlp)
0.9051851126141099
and the R2-score
import sklearn
r2mlp=sklearn.metrics.r2_score(predictions, y_test)
print(r2mlp)
0.7560938940479025
Let’s proceed further with the scikit-learn ML performance plots
import scikitplot as skplt
import sklearn
from sklearn.datasets import load_digits, load_boston, load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor, GradientBoostingClassifier, ExtraTreesClassifier
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import sys
import warnings
warnings.filterwarnings(“ignore”)
print(“Scikit Plot Version : “, skplt.version)
print(“Scikit Learn Version : “, sklearn.version)
print(“Python Version : “, sys.version)
%matplotlib inline
Scikit Plot Version : 0.3.7 Scikit Learn Version : 1.0.2 Python Version : 3.9.13 (main, Aug 25 2022, 23:51:50)
Let’s plot the MLP confusion matrix
skplt.metrics.plot_confusion_matrix(y_test, predictions,
title=”MLP Confusion Matrix”,figsize=(10,7), title_fontsize=”small”, text_fontsize=”small”,
cmap=”Oranges”)
The normalized MLP confusion matrix is as follows
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predictions)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘MLP Confusion Matrix’)
plt.savefig(‘confusionmatrixmlp.png’)
Let’s plot the MLP Classification Learning Curve
skplt.estimators.plot_learning_curve(mlp, x_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”MLP Classification Learning Curve”);
plt.savefig(‘learningcurvemlp.png’)
Let’s plot the MLP ROC Curve
Y_test_probs = mlp.predict_proba(x_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”MLP ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurvemlp.png’)
Let’s plot the MLP Precision-Recall Curve
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”MLP Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurvemlp.png’)
Part 2: Supervised ML
SVM 50:50 Split
Let’s import the dataset, classifiers and performance metrics
from sklearn import datasets, svm, metrics
from sklearn.model_selection import train_test_split
digits = datasets.load_digits()
_, axes = plt.subplots(nrows=1, ncols=4, figsize=(10, 3))
for ax, image, label in zip(axes, digits.images, digits.target):
ax.set_axis_off()
ax.imshow(image, cmap=plt.cm.gray_r, interpolation=”nearest”)
ax.set_title(“Training: %i” % label)
Let’s flatten the images, create a classifier, and split data into the 50% train and 50% test subsets
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))
clf = svm.SVC(gamma=0.001)
X_train, X_test, y_train, y_test = train_test_split(
data, digits.target, test_size=0.5, shuffle=False
)
Learn the digits on the train subset
clf.fit(X_train, y_train)
Predict the value of the digit on the test subset
predicted = clf.predict(X_test)
Let’s plot the first 4 predicted images
_, axes = plt.subplots(nrows=1, ncols=4, figsize=(10, 3))
for ax, image, prediction in zip(axes, X_test, predicted):
ax.set_axis_off()
image = image.reshape(8, 8)
ax.imshow(image, cmap=plt.cm.gray_r, interpolation=”nearest”)
ax.set_title(f”Prediction: {prediction}”)
Let’s print out the classification report
print(
f”Classification report for classifier {clf}:\n”
f”{metrics.classification_report(y_test, predicted)}\n”
)
Classification report for classifier SVC(gamma=0.001):
precision recall f1-score support
0 1.00 0.99 0.99 88
1 0.99 0.97 0.98 91
2 0.99 0.99 0.99 86
3 0.98 0.87 0.92 91
4 0.99 0.96 0.97 92
5 0.95 0.97 0.96 91
6 0.99 0.99 0.99 91
7 0.96 0.99 0.97 89
8 0.94 1.00 0.97 88
9 0.93 0.98 0.95 92
accuracy 0.97 899
macro avg 0.97 0.97 0.97 899
weighted avg 0.97 0.97 0.97 899
Let’s check the confusion matrix for this report
disp = metrics.ConfusionMatrixDisplay.from_predictions(y_test, predicted)
disp.figure_.suptitle(“Confusion Matrix”)
print(f”Confusion matrix:\n{disp.confusion_matrix}”)
plt.show()
Confusion matrix: [[87 0 0 0 1 0 0 0 0 0] [ 0 88 1 0 0 0 0 0 1 1] [ 0 0 85 1 0 0 0 0 0 0] [ 0 0 0 79 0 3 0 4 5 0] [ 0 0 0 0 88 0 0 0 0 4] [ 0 0 0 0 0 88 1 0 0 2] [ 0 1 0 0 0 0 90 0 0 0] [ 0 0 0 0 0 1 0 88 0 0] [ 0 0 0 0 0 0 0 0 88 0] [ 0 0 0 1 0 1 0 0 0 90]]
This matrix stores all the misclassified images, their predicted labels, and their true labels, in a list called misclassified. It is clear that true label 3 is misclassified as 5, 7, and 8 (3/79 ~ 3.8%, 4/79 ~ 5%, and 5/79 ~ 6.3%, respectively), whereas 4 and 5 are misclassified as 9 (4/90 ~ 4.4% and 2/90 ~ 2.2%, respectively).
70:30 Train/Test Data Split
SVM
Let’s split the input digits.data with test_size=0.3
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(digits.data, digits.target,test_size=0.3)
X_train.shape, X_test.shape
((1257, 64), (540, 64))
Let’s fit the svm.SVC classifier
from sklearn import svm
svc = svm.SVC(C=100.0, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.001, kernel=’rbf’, max_iter=-1, probability=True,
random_state=None, shrinking=True, tol=0.001, verbose=False)
svc.fit(X_train, y_train)
SVC(C=100.0, gamma=0.001, probability=True)
Our SVC predictions are as follows
predsvc = svc.predict(X_test)
predsvc
array([6, 5, 5, 9, 5, 1, 7, 3, 8, 2, 4, 0, 0, 8, 7, 5, 2, 6, 6, 9, 8, 4,
2, 1, 0, 1, 0, 3, 4, 3, 3, 4, 1, 7, 6, 3, 2, 8, 6, 9, 3, 2, 1, 9,
8, 8, 8, 3, 5, 2, 3, 2, 2, 1, 5, 9, 2, 2, 8, 7, 1, 7, 4, 5, 2, 5,
8, 9, 6, 0, 3, 8, 7, 7, 7, 9, 9, 6, 2, 4, 1, 6, 8, 6, 3, 1, 6, 1,
2, 8, 1, 0, 5, 6, 3, 2, 8, 4, 6, 4, 5, 2, 7, 9, 2, 6, 6, 5, 3, 0,
3, 6, 9, 3, 8, 1, 6, 6, 0, 4, 1, 4, 4, 6, 3, 7, 1, 3, 1, 4, 9, 5,
7, 6, 4, 3, 0, 6, 7, 6, 9, 0, 6, 9, 6, 7, 6, 6, 1, 2, 8, 4, 1, 3,
6, 8, 4, 8, 8, 1, 9, 4, 1, 3, 6, 3, 0, 6, 5, 8, 6, 7, 1, 1, 3, 4,
5, 7, 7, 3, 2, 5, 2, 9, 4, 8, 6, 8, 7, 4, 5, 5, 1, 9, 2, 9, 3, 7,
4, 7, 0, 2, 9, 2, 0, 2, 0, 9, 4, 1, 2, 1, 6, 3, 9, 0, 4, 1, 5, 8,
3, 4, 2, 9, 7, 0, 0, 1, 4, 1, 8, 8, 5, 3, 5, 3, 5, 0, 9, 8, 8, 8,
6, 4, 8, 2, 8, 9, 4, 4, 4, 2, 8, 9, 7, 4, 2, 8, 6, 1, 3, 6, 3, 9,
1, 6, 7, 0, 1, 1, 4, 6, 8, 4, 6, 0, 5, 0, 8, 4, 9, 1, 3, 3, 0, 5,
1, 7, 2, 1, 0, 4, 1, 1, 3, 7, 5, 2, 4, 6, 2, 2, 4, 4, 0, 6, 6, 3,
0, 3, 5, 3, 7, 9, 9, 5, 8, 2, 3, 0, 9, 4, 3, 0, 0, 2, 2, 3, 8, 9,
8, 2, 1, 7, 7, 8, 6, 4, 4, 5, 7, 4, 0, 7, 9, 9, 0, 7, 4, 4, 4, 5,
2, 4, 9, 2, 0, 6, 6, 3, 7, 4, 9, 8, 5, 2, 9, 5, 1, 2, 7, 4, 9, 5,
1, 0, 8, 9, 6, 7, 7, 1, 5, 1, 7, 8, 9, 9, 9, 5, 6, 1, 5, 7, 8, 6,
5, 1, 8, 9, 4, 8, 1, 6, 6, 4, 5, 5, 1, 3, 3, 1, 3, 7, 6, 7, 5, 7,
1, 9, 2, 2, 6, 5, 8, 6, 1, 8, 8, 3, 2, 8, 9, 6, 4, 7, 0, 2, 2, 7,
4, 2, 7, 1, 6, 6, 1, 0, 1, 2, 7, 3, 8, 5, 2, 0, 2, 8, 1, 8, 4, 3,
5, 2, 8, 9, 6, 7, 5, 0, 0, 2, 7, 9, 3, 7, 4, 1, 6, 8, 7, 8, 6, 1,
1, 0, 3, 4, 4, 1, 8, 3, 5, 5, 9, 9, 9, 4, 1, 8, 9, 4, 4, 4, 3, 1,
9, 0, 6, 2, 4, 7, 3, 0, 8, 6, 2, 0, 0, 8, 1, 7, 4, 6, 1, 0, 8, 3,
9, 9, 6, 9, 7, 8, 4, 0, 5, 2, 1, 3])
to be compared with the actual test data
y_test
array([6, 5, 5, 9, 5, 1, 7, 3, 8, 2, 4, 0, 0, 8, 7, 5, 2, 6, 6, 9, 8, 4,
2, 1, 0, 1, 0, 3, 4, 3, 3, 4, 1, 7, 6, 3, 2, 8, 6, 9, 3, 2, 1, 9,
8, 8, 8, 3, 5, 2, 3, 2, 2, 1, 5, 9, 2, 2, 8, 7, 1, 7, 4, 5, 2, 5,
8, 9, 6, 0, 3, 8, 7, 7, 7, 9, 9, 6, 2, 4, 1, 6, 8, 6, 3, 1, 6, 1,
2, 8, 1, 0, 5, 6, 3, 2, 8, 4, 6, 4, 5, 2, 7, 9, 2, 6, 6, 5, 3, 0,
3, 6, 9, 3, 8, 1, 6, 6, 0, 4, 1, 4, 4, 6, 3, 7, 1, 3, 1, 4, 9, 5,
7, 6, 4, 3, 0, 6, 7, 6, 9, 0, 6, 9, 6, 7, 6, 6, 1, 2, 8, 4, 1, 3,
6, 8, 4, 8, 8, 1, 9, 4, 1, 3, 6, 3, 0, 6, 5, 8, 6, 7, 1, 1, 3, 4,
5, 7, 7, 3, 2, 5, 2, 9, 4, 8, 6, 8, 7, 4, 5, 5, 1, 9, 2, 9, 3, 7,
4, 7, 0, 2, 9, 2, 0, 2, 0, 9, 4, 1, 2, 1, 6, 3, 9, 0, 4, 1, 5, 8,
3, 4, 2, 9, 7, 0, 0, 1, 4, 1, 8, 8, 5, 3, 5, 3, 5, 0, 9, 8, 8, 8,
6, 4, 8, 2, 8, 9, 4, 4, 4, 2, 8, 9, 7, 4, 2, 8, 6, 1, 3, 6, 3, 9,
1, 6, 7, 0, 1, 1, 4, 6, 8, 4, 6, 0, 5, 0, 8, 4, 5, 1, 3, 3, 0, 5,
1, 7, 2, 1, 0, 4, 1, 1, 3, 7, 5, 2, 4, 6, 2, 2, 4, 4, 0, 6, 6, 3,
0, 3, 5, 3, 7, 9, 9, 5, 8, 2, 3, 0, 9, 4, 3, 0, 0, 2, 2, 3, 8, 9,
8, 2, 1, 7, 7, 8, 6, 4, 4, 5, 7, 4, 0, 7, 9, 9, 0, 7, 4, 4, 4, 5,
2, 4, 9, 2, 0, 6, 6, 3, 7, 4, 9, 8, 5, 2, 9, 5, 1, 2, 7, 4, 9, 5,
1, 0, 8, 9, 6, 7, 7, 1, 5, 1, 7, 8, 9, 9, 9, 5, 6, 1, 5, 7, 8, 5,
5, 1, 8, 9, 4, 8, 1, 6, 6, 4, 5, 5, 1, 3, 3, 1, 3, 7, 6, 7, 5, 7,
1, 9, 2, 2, 6, 5, 8, 6, 1, 8, 8, 3, 2, 8, 9, 6, 4, 7, 0, 2, 2, 7,
4, 2, 7, 1, 6, 6, 1, 0, 1, 2, 7, 3, 8, 5, 2, 0, 2, 8, 1, 8, 4, 3,
5, 2, 8, 9, 6, 7, 5, 0, 0, 2, 7, 9, 3, 7, 4, 1, 6, 8, 7, 8, 6, 1,
1, 0, 3, 4, 4, 1, 8, 3, 5, 5, 9, 9, 9, 4, 1, 8, 9, 4, 4, 4, 3, 1,
9, 0, 6, 2, 4, 7, 3, 0, 8, 6, 2, 0, 0, 8, 1, 7, 4, 6, 1, 0, 8, 3,
9, 9, 6, 9, 7, 8, 4, 0, 5, 2, 1, 3])
Let’s plot real test data (horizontal axis) vs predictions (vertical axis) as a scatter plot
plt.scatter(y_test,predsvc)
In this example, the digit ‘5’ is misinterpreted as ‘6’ and ‘9’, whereas ‘8’ is misclassified as ‘1’.
The SVC pioeline looks as follows
from sklearn import svm
svc = svm.SVC(C=100.0, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.001, kernel=’rbf’, max_iter=-1, probability=False,
random_state=None, shrinking=True, tol=0.001, verbose=False)
svc.fit(X_train, y_train)
SVC(C=100.0, gamma=0.001)
predictions1 = svc.predict(X_test)
The histogram of difference y_test-predictions1 looks good
plt.hist(y_test-predictions1)
Let’s check the acccuracy_score
from sklearn.metrics import accuracy_score
accuracy_score(y_test, predictions1)
0.9944444444444445
Let’s look at the SVC confusion matrix and the classification report
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predictions1, y_test))
print(classification_report(predictions1, y_test))
[[52 0 0 0 0 0 0 0 0 0]
[ 0 52 0 0 0 0 0 0 1 0]
[ 0 0 50 0 0 0 0 0 0 0]
[ 0 0 0 58 0 0 0 0 0 0]
[ 0 0 0 0 42 0 0 0 0 0]
[ 0 0 0 1 0 48 0 0 0 0]
[ 0 0 0 0 0 0 56 0 0 0]
[ 0 0 0 0 0 0 0 65 0 0]
[ 0 0 0 0 0 0 0 0 52 0]
[ 0 0 0 1 0 0 0 0 0 62]]
precision recall f1-score support
0 1.00 1.00 1.00 52
1 1.00 0.98 0.99 53
2 1.00 1.00 1.00 50
3 0.97 1.00 0.98 58
4 1.00 1.00 1.00 42
5 1.00 0.98 0.99 49
6 1.00 1.00 1.00 56
7 1.00 1.00 1.00 65
8 0.98 1.00 0.99 52
9 1.00 0.98 0.99 63
accuracy 0.99 540
macro avg 0.99 0.99 0.99 540
weighted avg 0.99 0.99 0.99 540
Let’s plot the SVC classification learning curve for this test example
import scikitplot as skplt
import sklearn
from sklearn.datasets import load_digits, load_boston, load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor, GradientBoostingClassifier, ExtraTreesClassifier
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import sys
import warnings
warnings.filterwarnings(“ignore”)
print(“Scikit Plot Version : “, skplt.version)
print(“Scikit Learn Version : “, sklearn.version)
print(“Python Version : “, sys.version)
%matplotlib inline
Scikit Plot Version : 0.3.7 Scikit Learn Version : 1.0.2 Python Version : 3.9.13 (main, Aug 25 2022, 23:51:50)
skplt.estimators.plot_learning_curve(svc, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”SVC Classification Learning Curve”);
DT
Let’s look at the Decision Tree (DT) Classifier
from sklearn.tree import DecisionTreeClassifier
dt = DecisionTreeClassifier(criterion=’gini’)
dt.fit(X_train, y_train)
predictions2 = dt.predict(X_test)
accuracy_score(y_test, predictions2)
0.8314814814814815
The DT classification report is as follows
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predictions2, y_test))
print(classification_report(predictions2, y_test))
[[55 0 0 0 2 1 0 2 0 0]
[ 0 48 1 0 1 0 0 1 4 1]
[ 0 4 48 0 1 1 0 1 0 2]
[ 1 0 7 49 0 2 0 0 5 2]
[ 3 7 0 0 44 1 1 2 1 0]
[ 0 0 0 0 0 38 0 0 0 1]
[ 0 0 0 0 0 1 45 0 1 0]
[ 0 1 1 1 3 0 0 41 0 3]
[ 0 1 2 3 1 0 0 3 32 2]
[ 0 2 0 2 1 2 0 4 2 49]]
precision recall f1-score support
0 0.93 0.92 0.92 60
1 0.76 0.86 0.81 56
2 0.81 0.84 0.83 57
3 0.89 0.74 0.81 66
4 0.83 0.75 0.79 59
5 0.83 0.97 0.89 39
6 0.98 0.96 0.97 47
7 0.76 0.82 0.79 50
8 0.71 0.73 0.72 44
9 0.82 0.79 0.80 62
accuracy 0.83 540
macro avg 0.83 0.84 0.83 540
weighted avg 0.83 0.83 0.83 540
The DT Classification Learning Curve is given by
skplt.estimators.plot_learning_curve(dt, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”DT Classification Learning Curve”);
RC
Let’s look at the Random Forest Classifier (RF aka RC)
from sklearn.ensemble import RandomForestClassifier
rc = RandomForestClassifier(n_estimators=150)
rc.fit(X_train, y_train)
predictions3 = rc.predict(X_test)
accuracy_score(y_test, predictions3)
0.9814814814814815
The RC classification report is
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predictions3, y_test))
print(classification_report(predictions3, y_test))
[[59 0 0 0 0 0 0 0 0 0]
[ 0 63 0 1 0 0 0 0 0 0]
[ 0 0 58 0 0 0 0 0 0 0]
[ 0 0 0 52 0 0 0 0 1 1]
[ 0 0 0 0 53 0 0 0 0 0]
[ 0 0 0 0 0 45 0 0 0 1]
[ 0 0 0 0 0 0 45 0 0 0]
[ 0 0 0 1 0 0 0 54 0 1]
[ 0 0 1 1 0 0 1 0 44 0]
[ 0 0 0 0 0 1 0 0 0 57]]
precision recall f1-score support
0 1.00 1.00 1.00 59
1 1.00 0.98 0.99 64
2 0.98 1.00 0.99 58
3 0.95 0.96 0.95 54
4 1.00 1.00 1.00 53
5 0.98 0.98 0.98 46
6 0.98 1.00 0.99 45
7 1.00 0.96 0.98 56
8 0.98 0.94 0.96 47
9 0.95 0.98 0.97 58
accuracy 0.98 540
macro avg 0.98 0.98 0.98 540
weighted avg 0.98 0.98 0.98 540
The RF/RC Classification Learning Curve is
skplt.estimators.plot_learning_curve(rc, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”RF/RC Classification Learning Curve”);
LR
let’s consider the Logistic Regression (LR) Classifier
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
lr.fit(X_train, y_train)
predictions3 = lr.predict(X_test)
accuracy_score(y_test, predictions3)
0.9629629629629629
with the LR classification report given by
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predictions3, y_test))
print(classification_report(predictions3, y_test))
[[59 0 0 0 0 1 0 0 1 0]
[ 0 60 0 0 0 1 0 1 1 0]
[ 0 0 59 0 0 0 0 0 0 0]
[ 0 0 0 51 0 1 0 0 0 0]
[ 0 1 0 0 53 0 0 1 0 0]
[ 0 0 0 0 0 42 0 0 0 1]
[ 0 0 0 0 0 0 45 0 0 0]
[ 0 0 0 0 0 0 0 52 0 1]
[ 0 2 0 4 0 0 1 0 43 2]
[ 0 0 0 0 0 1 0 0 0 56]]
precision recall f1-score support
0 1.00 0.97 0.98 61
1 0.95 0.95 0.95 63
2 1.00 1.00 1.00 59
3 0.93 0.98 0.95 52
4 1.00 0.96 0.98 55
5 0.91 0.98 0.94 43
6 0.98 1.00 0.99 45
7 0.96 0.98 0.97 53
8 0.96 0.83 0.89 52
9 0.93 0.98 0.96 57
accuracy 0.96 540
macro avg 0.96 0.96 0.96 540
weighted avg 0.96 0.96 0.96 540
The LR Classification Learning Curve is
skplt.estimators.plot_learning_curve(lr, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”LR Classification Learning Curve”);
LR + Scaler
Let’s examine the impact of available scalers on the LR accuracy score
from sklearn import preprocessing
import numpy as np
scaler = preprocessing.StandardScaler().fit(X_train)
X_scaled = scaler.transform(X_train)
lr.fit(X_scaled, y_train)
predictions3 = lr.predict(X_test)
accuracy_score(y_test, predictions3)
0.7814814814814814
from sklearn import preprocessing
import numpy as np
scaler = preprocessing.MinMaxScaler().fit(X_train)
X_scaled = scaler.transform(X_train)
lr.fit(X_scaled, y_train)
predictions4 = lr.predict(X_test)
accuracy_score(y_test, predictions4)
0.8666666666666667
from sklearn import preprocessing
import numpy as np
scaler = preprocessing.RobustScaler().fit(X_train)
X_scaled = scaler.transform(X_train)
lr.fit(X_scaled, y_train)
predictions5 = lr.predict(X_test)
accuracy_score(y_test, predictions5)
0.6814814814814815
It is clear that scalers do not improve the accuracy of LR.
ET
Let’s invoke the Extra Trees (ET) Classifier
et=ExtraTreesClassifier()
et.fit(X_train, y_train)
predictions3 = et.predict(X_test)
accuracy_score(y_test, predictions3)
0.9888888888888889
with the ET classification report
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predictions3, y_test))
print(classification_report(predictions3, y_test))
[[59 0 0 0 0 0 0 0 0 0]
[ 0 63 0 0 0 0 0 0 0 0]
[ 0 0 59 0 0 0 0 0 0 0]
[ 0 0 0 54 0 0 0 0 1 1]
[ 0 0 0 0 53 0 0 0 0 1]
[ 0 0 0 0 0 45 0 0 0 0]
[ 0 0 0 0 0 0 45 0 0 0]
[ 0 0 0 0 0 0 0 54 0 0]
[ 0 0 0 1 0 0 1 0 44 0]
[ 0 0 0 0 0 1 0 0 0 58]]
precision recall f1-score support
0 1.00 1.00 1.00 59
1 1.00 1.00 1.00 63
2 1.00 1.00 1.00 59
3 0.98 0.96 0.97 56
4 1.00 0.98 0.99 54
5 0.98 1.00 0.99 45
6 0.98 1.00 0.99 45
7 1.00 1.00 1.00 54
8 0.98 0.96 0.97 46
9 0.97 0.98 0.97 59
accuracy 0.99 540
macro avg 0.99 0.99 0.99 540
weighted avg 0.99 0.99 0.99 540
and the ET Classification Learning Curve given by
skplt.estimators.plot_learning_curve(et, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”ET Classification Learning Curve”);
GB
Let’s invoke the Gradient Boosting Classifier (GBC)
gbc=GradientBoostingClassifier()
gbc.fit(X_train, y_train)
predictions3 = gbc.predict(X_test)
accuracy_score(y_test, predictions3)
0.9648148148148148
The GBC classification report is
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predictions3, y_test))
print(classification_report(predictions3, y_test))
[[59 0 0 0 0 1 0 0 1 0]
[ 0 61 1 0 0 0 0 0 0 0]
[ 0 2 58 0 0 0 0 1 0 0]
[ 0 0 0 51 0 0 0 0 0 1]
[ 0 0 0 0 53 1 0 1 0 0]
[ 0 0 0 0 0 42 0 0 0 1]
[ 0 0 0 0 0 0 45 0 0 0]
[ 0 0 0 1 0 0 0 52 0 2]
[ 0 0 0 3 0 0 1 0 44 0]
[ 0 0 0 0 0 2 0 0 0 56]]
precision recall f1-score support
0 1.00 0.97 0.98 61
1 0.97 0.98 0.98 62
2 0.98 0.95 0.97 61
3 0.93 0.98 0.95 52
4 1.00 0.96 0.98 55
5 0.91 0.98 0.94 43
6 0.98 1.00 0.99 45
7 0.96 0.95 0.95 55
8 0.98 0.92 0.95 48
9 0.93 0.97 0.95 58
accuracy 0.96 540
macro avg 0.96 0.97 0.96 540
weighted avg 0.97 0.96 0.96 540
The GBC Classification Learning Curve is
skplt.estimators.plot_learning_curve(gbc, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”GBC Classification Learning Curve”);
KNN
Let’s look at the KNeighborsClassifier (KNN)
import numpy as np
import pandas as pd
from sklearn.neighbors import KNeighborsClassifier
For selecting the K value, we calculate the KNN error rate vs K
error_rate = []
for i in range(1,40):
knn = KNeighborsClassifier(n_neighbors=i)
knn.fit(X_train,y_train)
pred_i = knn.predict(X_test)
error_rate.append(np.mean(pred_i != y_test))
and plot the result as follows
import matplotlib.pyplot as plt
plt.figure(figsize=(10,6))
plt.plot(range(1,40),error_rate,color=’blue’, linestyle=’dashed’, marker=’o’,
markerfacecolor=’red’, markersize=10)
plt.title(‘Error Rate vs. K Value by Elbow Method’)
plt.xlabel(‘K’)
plt.ylabel(‘Error Rate’)
It is clear that 5 is the optimal value of K.
According to the above plot, let’s run KNN with n_neighbors=5
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
predictions3 = knn.predict(X_test)
accuracy_score(y_test, predictions3)
0.9925925925925926
The KNN classification report is as follows
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predictions3, y_test))
print(classification_report(predictions3, y_test))
[[58 0 0 0 0 0 0 0 0 1]
[ 0 48 3 0 0 0 1 0 3 2]
[ 0 1 42 1 0 0 0 0 0 2]
[ 0 0 0 43 0 3 0 0 0 2]
[ 0 0 0 0 48 0 0 0 0 2]
[ 0 0 0 1 1 39 0 0 2 0]
[ 0 2 1 0 3 0 45 0 0 0]
[ 1 2 0 2 0 2 0 54 1 9]
[ 0 8 13 8 1 2 0 0 39 5]
[ 0 2 0 0 0 0 0 0 0 37]]
precision recall f1-score support
0 0.98 0.98 0.98 59
1 0.76 0.84 0.80 57
2 0.71 0.91 0.80 46
3 0.78 0.90 0.83 48
4 0.91 0.96 0.93 50
5 0.85 0.91 0.88 43
6 0.98 0.88 0.93 51
7 1.00 0.76 0.86 71
8 0.87 0.51 0.64 76
9 0.62 0.95 0.75 39
accuracy 0.84 540
macro avg 0.85 0.86 0.84 540
weighted avg 0.86 0.84 0.84 540
The KNN Classification Learning Curve is
Even though both KNN learning curve and accuracy score look good, the f1-scores for labels ‘8’ and ‘9’ are 0.64 and 0.75, respectively.
GNB
Let’s look at the GaussianNB (GNB) Classifier
from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
gnb.fit(X_train, y_train)
predictions3 = gnb.predict(X_test)
accuracy_score(y_test, predictions3)
0.8388888888888889
The classification report is
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predictions3, y_test))
print(classification_report(predictions3, y_test))
[[59 0 0 0 0 0 0 0 0 0]
[ 0 63 0 0 0 0 0 0 1 0]
[ 0 0 59 0 0 0 0 0 0 0]
[ 0 0 0 52 0 0 0 0 1 2]
[ 0 0 0 0 53 0 0 0 0 0]
[ 0 0 0 0 0 42 0 0 0 0]
[ 0 0 0 0 0 0 45 0 0 0]
[ 0 0 0 1 0 0 0 54 0 1]
[ 0 0 0 2 0 1 1 0 43 0]
[ 0 0 0 0 0 3 0 0 0 57]]
precision recall f1-score support
0 1.00 1.00 1.00 59
1 1.00 0.98 0.99 64
2 1.00 1.00 1.00 59
3 0.95 0.95 0.95 55
4 1.00 1.00 1.00 53
5 0.91 1.00 0.95 42
6 0.98 1.00 0.99 45
7 1.00 0.96 0.98 56
8 0.96 0.91 0.93 47
9 0.95 0.95 0.95 60
accuracy 0.98 540
macro avg 0.97 0.98 0.97 540
weighted avg 0.98 0.98 0.98 540
The GNB Classification Learning Curve is
skplt.estimators.plot_learning_curve(gnb, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”GNB Classification Learning Curve”);
SGD
Let’s look at the SGDClassifier
from sklearn.linear_model import SGDClassifier
sgd = SGDClassifier(max_iter=150, tol=None)
sgd.fit(X_train, y_train)
predsgd = sgd.predict(X_test)
acc_sgd=accuracy_score(y_test, predsgd)
print(acc_sgd)
0.95
The classification report is
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predsgd, y_test))
print(classification_report(predsgd, y_test))
[[59 0 0 0 0 1 1 0 1 0]
[ 0 57 0 0 0 0 0 0 3 0]
[ 0 0 57 0 0 0 0 0 0 0]
[ 0 0 1 52 0 0 0 0 1 0]
[ 0 4 0 0 52 0 0 2 0 0]
[ 0 1 0 1 0 42 0 0 0 2]
[ 0 1 0 0 0 0 45 0 1 0]
[ 0 0 0 0 0 0 0 52 0 0]
[ 0 0 1 2 1 0 0 0 39 0]
[ 0 0 0 0 0 3 0 0 0 58]]
precision recall f1-score support
0 1.00 0.95 0.98 62
1 0.90 0.95 0.93 60
2 0.97 1.00 0.98 57
3 0.95 0.96 0.95 54
4 0.98 0.90 0.94 58
5 0.91 0.91 0.91 46
6 0.98 0.96 0.97 47
7 0.96 1.00 0.98 52
8 0.87 0.91 0.89 43
9 0.97 0.95 0.96 61
accuracy 0.95 540
macro avg 0.95 0.95 0.95 540
weighted avg 0.95 0.95 0.95 540
The SGD Classification Learning Curve is
skplt.estimators.plot_learning_curve(sgd, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”SGD Classification Learning Curve”);
Perceptron
Let’s look at the Perceptron
from sklearn.linear_model import Perceptron
perceptron = Perceptron(max_iter=150)
perceptron.fit(X_train, y_train)
predperc = perceptron.predict(X_test)
acc_perc=accuracy_score(y_test, predperc)
print(acc_perc)
0.9388888888888889
The Perceptron classification report is
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predperc, y_test))
print(classification_report(predperc, y_test))
[[58 0 0 0 0 0 1 0 1 0]
[ 0 58 0 0 0 0 0 0 2 0]
[ 0 1 59 1 0 1 0 0 0 1]
[ 0 0 0 50 0 0 0 0 2 1]
[ 0 2 0 0 53 1 0 2 0 0]
[ 1 1 0 1 0 44 0 0 0 3]
[ 0 0 0 0 0 0 45 0 2 0]
[ 0 0 0 1 0 0 0 52 0 2]
[ 0 1 0 2 0 0 0 0 38 3]
[ 0 0 0 0 0 0 0 0 0 50]]
precision recall f1-score support
0 0.98 0.97 0.97 60
1 0.92 0.97 0.94 60
2 1.00 0.94 0.97 63
3 0.91 0.94 0.93 53
4 1.00 0.91 0.95 58
5 0.96 0.88 0.92 50
6 0.98 0.96 0.97 47
7 0.96 0.95 0.95 55
8 0.84 0.86 0.85 44
9 0.83 1.00 0.91 50
accuracy 0.94 540
macro avg 0.94 0.94 0.94 540
weighted avg 0.94 0.94 0.94 540
The Perceptron Classification Learning Curve is
skplt.estimators.plot_learning_curve(perceptron, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”Perceptron Classification Learning Curve”);
LSVC
Let’s look at the LinearSVC (LSVC) Classifier
from sklearn.svm import SVC, LinearSVC
linear_svc = LinearSVC()
linear_svc.fit(X_train, y_train)
predlsvc = linear_svc.predict(X_test)
acc_lsvc=accuracy_score(y_test, predlsvc)
print (acc_lsvc)
0.9296296296296296
The LSVC classification report is
[[59 0 0 0 0 0 0 0 0 0]
[ 0 50 0 0 0 0 0 1 0 0]
[ 0 0 57 0 0 0 0 0 0 0]
[ 0 0 0 49 0 1 0 0 0 1]
[ 0 0 0 0 52 1 0 2 0 0]
[ 0 0 0 0 0 40 0 0 0 1]
[ 0 1 0 0 0 1 45 0 0 0]
[ 0 0 0 0 0 0 0 51 0 1]
[ 0 12 2 6 1 1 1 0 45 3]
[ 0 0 0 0 0 2 0 0 0 54]]
precision recall f1-score support
0 1.00 1.00 1.00 59
1 0.79 0.98 0.88 51
2 0.97 1.00 0.98 57
3 0.89 0.96 0.92 51
4 0.98 0.95 0.96 55
5 0.87 0.98 0.92 41
6 0.98 0.96 0.97 47
7 0.94 0.98 0.96 52
8 1.00 0.63 0.78 71
9 0.90 0.96 0.93 56
accuracy 0.93 540
macro avg 0.93 0.94 0.93 540
weighted avg 0.94 0.93 0.93 540
The LSVC Classification Learning Curve is
skplt.estimators.plot_learning_curve(linear_svc, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”LSVC Classification Learning Curve”);
QDA
Let’s look at the Quadratic Discriminant Analysis (QDA)
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
qda = QuadraticDiscriminantAnalysis()
qda.fit(X_train, y_train)
predqda = qda.predict(X_test)
acc_qda=accuracy_score(y_test, predqda)
print(acc_qda)
0.8685185185185185
Let’s check the QDA classification report
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
print(confusion_matrix(predqda, y_test))
print(classification_report(predqda, y_test))
[[58 0 0 0 0 0 0 0 0 0]
[ 0 52 0 0 0 0 1 0 1 0]
[ 0 0 49 1 1 0 0 0 0 0]
[ 0 0 0 36 0 1 0 0 0 4]
[ 0 0 0 0 51 0 0 0 0 1]
[ 0 0 0 0 0 45 7 0 0 1]
[ 0 0 0 0 0 0 34 0 0 0]
[ 1 2 0 0 0 0 0 54 0 5]
[ 0 9 10 18 0 0 3 0 44 3]
[ 0 0 0 0 1 0 1 0 0 46]]
precision recall f1-score support
0 0.98 1.00 0.99 58
1 0.83 0.96 0.89 54
2 0.83 0.96 0.89 51
3 0.65 0.88 0.75 41
4 0.96 0.98 0.97 52
5 0.98 0.85 0.91 53
6 0.74 1.00 0.85 34
7 1.00 0.87 0.93 62
8 0.98 0.51 0.67 87
9 0.77 0.96 0.85 48
accuracy 0.87 540
macro avg 0.87 0.90 0.87 540
weighted avg 0.89 0.87 0.86 540
The QDA Classification Learning Curve is
skplt.estimators.plot_learning_curve(qda, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(6,4), title_fontsize=”large”, text_fontsize=”large”,
title=”QDA Classification Learning Curve”);
Selected Cross-Validation
Let’s import the key libraries
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import Pipeline
and introduce the following two functions
from functools import reduce
def get_model_name(model): “”” Returns a string with the name of a sklearn model model: Sklearn stimator class “”” if isinstance(model, Pipeline): estimator = model.steps[-1][1] name = “Pipeline” + str(estimator)[:str(estimator).find(“(“)]
else:
name = str(model)[:str(model).find(“(“)]
return name
def plot_cv_score(X, y, models_list, cv = 5, scoring_list = None, refit = True, return_scores = False):
names, mean_score = list(), list()
ldf = list()
names = list()
for i, model in enumerate(models_list):
name = _get_model_name(model)
if refit:
model.fit(X, y)
for metric in score_list:
score = cross_val_score(model, X, y, cv = cv, scoring = metric, n_jobs= -1)
mean_score.append(np.mean(score))
tmp = pd.DataFrame({name: mean_score}, index = score_list)
ldf.append(tmp)
mean_score = list()
frame_scores = reduce(lambda x,y: pd.merge(x,y, left_index = True, right_index = True), ldf).T
fig, ax = plt.subplots(1,1, figsize = (10,5))
frame_scores.plot.bar(ax = ax, cmap = 'RdYlBu', edgecolor = "black")
ax.legend(loc = 'best')
ax.set_xlabel("Score")
ax.set_title("Cross validation model benchmark")
if return_scores:
return frame_scores
let’s create the list of selected models
models_list =[LogisticRegression(random_state= 42),
SVC(probability= True),
RandomForestClassifier(random_state = 42),
GaussianNB()]
and the score
score_list = [“accuracy”]
Let’s plot the accuracy for these models as a single bar plot
t = plot_cv_score(X = X_train, y = y_train, models_list = models_list, cv = 5, scoring_list = score_list, refit = True)
Let’s consider the following ML algorithms
rom sklearn import linear_model
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import Perceptron
from sklearn.linear_model import SGDClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC, LinearSVC
from sklearn.naive_bayes import GaussianNB
and compute the accuracy score for these algorithms:
- stochastic gradient descent (SGD) learning
sgd = linear_model.SGDClassifier(max_iter=5, tol=None)
sgd.fit(X_train, y_train)
Y_pred = sgd.predict(X_test)
sgd.score(X_train, y_train)
acc_sgd = round(sgd.score(X_train, y_train) * 100, 2)
print(round(acc_sgd,2,), “%”)
93.48 %
- Random Forest
random_forest = RandomForestClassifier(n_estimators=100)
random_forest.fit(X_train, y_train)
Y_prediction = random_forest.predict(X_test)
random_forest.score(X_train, y_train)
acc_random_forest = round(random_forest.score(X_train, y_train) * 100, 2)
print(round(acc_random_forest,2,), “%”)
100.0 %
- Logistic Regression
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
Y_pred = logreg.predict(X_test)
acc_log = round(logreg.score(X_train, y_train) * 100, 2)
print(round(acc_log,2,), “%”)
100.0 %
- KNN
knn = KNeighborsClassifier(n_neighbors = 5)
knn.fit(X_train, y_train)
Y_pred = knn.predict(X_test)
acc_knn = round(knn.score(X_train, y_train) * 100, 2)
print(round(acc_knn,2,), “%”)
99.12 %
- Gaussian Naive Bayes
gaussian = GaussianNB()
gaussian.fit(X_train, y_train)
Y_pred = gaussian.predict(X_test)
acc_gaussian = round(gaussian.score(X_train, y_train) * 100, 2)
print(round(acc_gaussian,2,), “%”)
82.74 %
- Perceptron
perceptron = Perceptron(max_iter=5)
perceptron.fit(X_train, y_train)
Y_pred = perceptron.predict(X_test)
acc_perceptron = round(perceptron.score(X_train, y_train) * 100, 2)
print(round(acc_perceptron,2,), “%”)
95.62 %
- Linear SVC
linear_svc = LinearSVC()
linear_svc.fit(X_train, y_train)
Y_pred = linear_svc.predict(X_test)
acc_linear_svc = round(linear_svc.score(X_train, y_train) * 100, 2)
print(round(acc_linear_svc,2,), “%”)
99.05 %
- Decision Tree
decision_tree = DecisionTreeClassifier()
decision_tree.fit(X_train, y_train)
Y_pred = decision_tree.predict(X_test)
acc_decision_tree = round(decision_tree.score(X_train, y_train) * 100, 2)
print(round(acc_decision_tree,2,), “%”)
100.0 %
Let’s combine these score values into a single bar plot
results = pd.DataFrame({
‘Model’: [‘SVM’, ‘KNN’, ‘LR’,
‘RF’, ‘NB’, ‘Perc’,
‘SGD’,
‘DT’],
‘Score’: [acc_linear_svc, acc_knn, acc_log,
acc_random_forest, acc_gaussian, acc_perceptron,
acc_sgd, acc_decision_tree]})
result_df = results.sort_values(by=’Score’, ascending=False)
result_df = result_df.set_index(‘Score’)
result_df.reset_index(inplace=True)
print(result_df)
Score Model 0 100.00 LR 1 100.00 RF 2 100.00 DT 3 99.12 KNN 4 99.05 SVM 5 95.62 Perc 6 93.48 SGD 7 82.74 NB
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Model’], result_df[‘Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Model”)
plt.ylabel(“Score”)
plt.title(“Score”)
plt.show()
Random Forest HPO
Let’s invoke Random Forest hyper-parameter tuning using GridSearchCV
param_grid = { “criterion” : [“gini”, “entropy”],
“min_samples_leaf” : [1, 5, 10, 25, 50, 70],
“min_samples_split” : [2, 4, 10, 12, 16, 18, 25, 35],
“n_estimators”: [100, 400, 700, 1000, 1500]}
from sklearn.model_selection import GridSearchCV, cross_val_score
rf = RandomForestClassifier(n_estimators=100, max_features=’auto’, oob_score=True, random_state=1, n_jobs=-1)
clf = GridSearchCV(estimator=rf, param_grid=param_grid,
n_jobs=-1)
clf.fit(X_train, y_train)
clf.best_params_
{'criterion': 'entropy',
'min_samples_leaf': 1,
'min_samples_split': 4,
'n_estimators': 100}
Let’s run the Classifier with the above optimized hyper-parameters
random_forest = RandomForestClassifier(criterion = “gini”,
min_samples_leaf = 1,
min_samples_split = 2,
n_estimators=1000,
max_features=’auto’,
oob_score=True,
random_state=1,
n_jobs=-1)
random_forest.fit(X_train, y_train)
Y_prediction = random_forest.predict(X_test)
random_forest.score(X_train, y_train)
print(“oob score:”, round(random_forest.oob_score_, 4)*100, “%”)
oob score: 97.3 %
Recall that the out-of-bag (OOB) error is the average error for each calculated using predictions from the trees that do not contain in their respective bootstrap sample. This allows the RandomForestClassifier to be fit and validated whilst being trained.
Let’s run cross-validation tests
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix
predictions = cross_val_predict(random_forest, X_train, y_train, cv=3)
confusion_matrix(y_train, predictions)
array([[120, 0, 0, 0, 2, 0, 0, 0, 0, 0],
[ 0, 122, 0, 0, 0, 0, 1, 0, 0, 0],
[ 1, 0, 122, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 125, 0, 1, 0, 1, 0, 0],
[ 0, 0, 0, 0, 117, 0, 0, 2, 0, 1],
[ 0, 0, 0, 0, 1, 127, 1, 0, 1, 1],
[ 2, 2, 0, 0, 1, 0, 119, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 119, 2, 1],
[ 0, 4, 1, 1, 1, 0, 1, 0, 118, 0],
[ 0, 1, 0, 1, 0, 2, 0, 4, 2, 129]], dtype=int64)
Let’s plot the confusion matrix
from sklearn.metrics import confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay
y_pred = random_forest.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
cm_display = ConfusionMatrixDisplay(cm).plot()
The cross_val_score is as follows
from sklearn.model_selection import cross_val_score
scores = cross_val_score(random_forest, X_train, y_train, cv=10, scoring = “accuracy”)
print(“Scores:”, scores)
print(“Mean:”, scores.mean())
print(“Standard Deviation:”, scores.std())
Scores: [0.98412698 0.96825397 0.97619048 0.98412698 0.96031746 0.95238095 0.97619048 0.976 0.96 0.992 ] Mean: 0.9729587301587301 Standard Deviation: 0.011897384380696487
The precision, recall, and f1 scores are given by
from sklearn.metrics import precision_score, recall_score
print(“Precision:”, precision_score(y_train, predictions,average=None))
print(“Recall:”,recall_score(y_train, predictions,average=None))
Precision: [0.97560976 0.94573643 0.99186992 0.98425197 0.95901639 0.97692308 0.97540984 0.94444444 0.95934959 0.97727273] Recall: [0.98360656 0.99186992 0.99186992 0.98425197 0.975 0.96946565 0.95967742 0.97540984 0.93650794 0.92805755]
precision=precision_score(y_train, predictions,average=None)
print(“Precision Scores:”, precision)
print(“Mean:”, precision.mean())
print(“Standard Deviation:”, precision.std())
Precision Scores: [0.97560976 0.94573643 0.99186992 0.98425197 0.95901639 0.97692308 0.97540984 0.94444444 0.95934959 0.97727273] Mean: 0.9689884149053591 Standard Deviation: 0.01519412377578287
from sklearn.metrics import f1_score
f1_score(y_train, predictions,average=None)
array([0.97959184, 0.96825397, 0.99186992, 0.98425197, 0.96694215,
0.97318008, 0.96747967, 0.95967742, 0.94779116, 0.95202952])
f1=f1_score(y_train, predictions,average=None)
print(“F1 Scores:”, f1)
print(“Mean:”, f1.mean())
print(“Standard Deviation:”, f1.std())
F1 Scores: [0.97959184 0.96825397 0.99186992 0.98425197 0.96694215 0.97318008 0.96747967 0.95967742 0.94779116 0.95202952] Mean: 0.9691067696685891 Standard Deviation: 0.013103537173650717
recall=recall_score(y_train, predictions,average=None)
print(“Recall Scores:”, recall)
print(“Mean:”, recall.mean())
print(“Standard Deviation:”, recall.std())
Recall Scores: [0.98360656 0.99186992 0.99186992 0.98425197 0.975 0.96946565 0.95967742 0.97540984 0.93650794 0.92805755] Mean: 0.9695716758019506 Standard Deviation: 0.020940400443025423
Let’s summarize our QC scores:
sklearn.metrics.precision_score(y_train, predictions, average=’macro’)
0.9689884149053591
sklearn.metrics.recall_score(y_train, predictions, average=’micro’)
0.9689737470167065
sklearn.metrics.f1_score(y_train, predictions, average=’micro’)
0.9689737470167065
sklearn.metrics.fbeta_score(y_train, predictions, average=’micro’,beta=0.5)
0.9689737470167065
sklearn.metrics.matthews_corrcoef(y_train, predictions)
0.965563685896965
In-Depth Scikit Plot Analysis
Let’s get back to our original input data
y = digits.target
x = digits.images.reshape((len(digits.images), -1))
x.shape
(1797, 64)
The train/test subsets are
x_train = x[:1000]
y_train = y[:1000]
x_test = x[1000:]
y_test = y[1000:]
MLPClassifier
Let’s begin with the MLP Classifier
from sklearn.neural_network import MLPClassifier
mlp = MLPClassifier(hidden_layer_sizes=(15,), activation=’logistic’, alpha=1e-4,
solver=’sgd’, tol=1e-4, random_state=1,
learning_rate_init=.1, verbose=True)
mlp.fit(x_train,y_train)
predictions = mlp.predict(x_test)
Let’s check the accuracy
from sklearn.metrics import accuracy_score
acc_mlp=accuracy_score(y_test, predictions)
print(acc_mlp)
0.9146800501882058
and the classification report
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
print(classification_report(predictions, y_test))
precision recall f1-score support
0 0.95 0.96 0.96 78
1 0.89 0.88 0.88 81
2 0.91 0.93 0.92 75
3 0.82 0.86 0.84 76
4 0.95 0.98 0.96 81
5 0.96 0.89 0.92 89
6 0.99 0.98 0.98 81
7 0.95 0.99 0.97 77
8 0.83 0.90 0.86 70
9 0.89 0.81 0.85 89
accuracy 0.91 797
macro avg 0.91 0.92 0.91 797
weighted avg 0.92 0.91 0.91 797
Let’s compute the f1-score
from sklearn.metrics import f1_score
f1mlp=f1_score(predictions, y_test, average=None)
print(f1mlp)
[0.95541401 0.88198758 0.92105263 0.83870968 0.96341463 0.92397661 0.98136646 0.96815287 0.8630137 0.84705882]
Let’s print out this array as the following bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1mlp[0], f1mlp[1], f1mlp[2],
f1mlp[3], f1mlp[4], f1mlp[5],
f1mlp[6], f1mlp[7],f1mlp[8],f1mlp[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score”)
plt.savefig(‘f1scoremlp.png’)
Let’s compute the precision score
from sklearn.metrics import precision_score
precisionmlp=precision_score(predictions, y_test, average=None)
print(precisionmlp)
[0.94936709 0.8875 0.90909091 0.82278481 0.95180723 0.96341463 0.9875 0.95 0.82894737 0.88888889]
and plot this array as the bar chart
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionmlp[0], precisionmlp[1], precisionmlp[2],
precisionmlp[3], precisionmlp[4], precisionmlp[5],
precisionmlp[6], precisionmlp[7],precisionmlp[8],precisionmlp[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score”)
plt.savefig(‘precisionscoremlp.png’)
Let’s compute the recall score
from sklearn.metrics import recall_score
recallmlp=recall_score(predictions, y_test, average=None)
print(recallmlp)
[0.96153846 0.87654321 0.93333333 0.85526316 0.97530864 0.88764045 0.97530864 0.98701299 0.9 0.80898876]
and plot the result
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallmlp[0], recallmlp[1], recallmlp[2],
recallmlp[3], recallmlp[4], recallmlp[5],
recallmlp[6], recallmlp[7],recallmlp[8],recallmlp[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score”)
plt.savefig(‘recallscoremlp.png’)
Let’s check the Cohen’s kappa
from sklearn.metrics import cohen_kappa_score
cohmlp=cohen_kappa_score(predictions, y_test)
print(cohmlp)
from sklearn.metrics import cohen_kappa_score
cohmlp=cohen_kappa_score(predictions, y_test)
print(cohmlp)
0.9051851126141099
Let’s look at the R2-coefficient too
import sklearn
r2mlp=sklearn.metrics.r2_score(predictions, y_test)
print(r2mlp)
0.7560938940479025
Let’s plot the normalized MLP confusion matrix
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predictions)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘MLP Confusion Matrix’)
plt.savefig(‘confusionmatrixmlp.png’)
Let’s plot the MLP Classification Learning Curve
skplt.estimators.plot_learning_curve(mlp, x_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”MLP Classification Learning Curve”);
plt.savefig(‘learningcurvemlp.png’)
The MLP ROC Curve is
Y_test_probs = mlp.predict_proba(x_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”MLP ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurvemlp.png’)
The MLP Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”MLP Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurvemlp.png’)
SVM/SVC
Let’s continue with other classifiers using the 70:30 digits data splitting
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(digits.data, digits.target,test_size=0.3)
X_train.shape, X_test.shape
((1257, 64), (540, 64))
Let’s train the SVC model
from sklearn import svm
svc = svm.SVC(C=100.0, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.001, kernel=’rbf’, max_iter=-1, probability=True,
random_state=None, shrinking=True, tol=0.001, verbose=False)
svc.fit(X_train, y_train)
SVC(C=100.0, gamma=0.001, probability=True)
and make test data predictions
predsvc = svc.predict(X_test)
The accuracy score is
from sklearn.metrics import accuracy_score
acc_svc =accuracy_score(y_test, predsvc)
print(acc_svc)
0.9962962962962963
with the SVC classification report given by
print(classification_report(predsvc, y_test))
precision recall f1-score support
0 1.00 1.00 1.00 44
1 1.00 1.00 1.00 61
2 1.00 1.00 1.00 54
3 1.00 1.00 1.00 52
4 1.00 1.00 1.00 61
5 0.96 1.00 0.98 45
6 1.00 0.98 0.99 61
7 1.00 1.00 1.00 50
8 1.00 1.00 1.00 59
9 1.00 0.98 0.99 53
accuracy 1.00 540
macro avg 1.00 1.00 1.00 540
weighted avg 1.00 1.00 1.00 540
where the f1-score is
from sklearn.metrics import f1_score
f1svc=f1_score(predsvc, y_test, average=None)
print(f1svc)
[1. 1. 1. 1. 1. 0.97826087 0.99173554 1. 1. 0.99047619]
The f1-score plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1svc[0], f1svc[1], f1svc[2],
f1svc[3], f1svc[4], f1svc[5],
f1svc[6], f1svc[7],f1svc[8],f1svc[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score SVM”)
plt.savefig(‘f1scoresvc.png’)
The recall score is
from sklearn.metrics import recall_score
recallsvc=recall_score(predsvc, y_test, average=None)
print(recallsvc)
[1. 1. 1. 1. 1. 1. 0.98360656 1. 1. 0.98113208]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallsvc[0], recallsvc[1], recallsvc[2],
recallsvc[3], recallsvc[4], recallsvc[5],
recallsvc[6], recallsvc[7],recallsvc[8],recallsvc[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score SVM”)
plt.savefig(‘recallscoresvc.png’)
The SVC precision score is
from sklearn.metrics import precision_score
precisionsvc=precision_score(predsvc, y_test, average=None)
print(precisionsvc)
[1. 1. 1. 1. 1. 0.95744681 1. 1. 1. 1. ]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionsvc[0], precisionsvc[1], precisionsvc[2],
precisionsvc[3], precisionsvc[4], precisionsvc[5],
precisionsvc[6], precisionsvc[7],precisionsvc[8],precisionsvc[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score SVM”)
plt.savefig(‘precisionscoresvc.png’)
The Cohen’s kappa is
from sklearn.metrics import cohen_kappa_score
cohsvc=cohen_kappa_score(predsvc, y_test)
print(cohsvc)
0.9958792781105728
and the R2-coefficient is
import sklearn
r2svc=sklearn.metrics.r2_score(predsvc, y_test)
print(r2svc)
0.9961039936611382
Let’s plot the normalized SVM/SVC confusion matrix
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predsvc)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘SVM Confusion Matrix’)
plt.savefig(‘confusionmatrixsvc.png’)
The SVM/SVC Classification Learning Curve is
skplt.estimators.plot_learning_curve(svc, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”SVM Classification Learning Curve”);
plt.savefig(‘learningcurvesvc.png’)
The SVM/SVC ROC Curve is given by
Y_test_probs = svc.predict_proba(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”SVM ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurvesvc.png’)
whereas the SVM/SVC Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”SVM Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurvesvc.png’)
Decision Tree (DT)
Let’s import the DT Classifier
from sklearn.tree import DecisionTreeClassifier
and train the DT model
dt = DecisionTreeClassifier(criterion=’gini’)
dt.fit(X_train, y_train)
Our prediction on test data is
predtc = dt.predict(X_test)
while the accuracy score is
acc_dt=accuracy_score(y_test, predtc)
print(acc_dt)
0.8351851851851851
and the DT classification report is
print(classification_report(predtc, y_test))
precision recall f1-score support
0 0.95 1.00 0.98 42
1 0.89 0.82 0.85 66
2 0.80 0.86 0.83 50
3 0.79 0.91 0.85 45
4 0.85 0.78 0.81 67
5 0.81 0.79 0.80 48
6 0.93 0.88 0.90 64
7 0.82 0.80 0.81 51
8 0.76 0.76 0.76 59
9 0.75 0.81 0.78 48
accuracy 0.84 540
macro avg 0.84 0.84 0.84 540
weighted avg 0.84 0.84 0.84 540
The DT f1-score is
from sklearn.metrics import f1_score
f1dt=f1_score(predtc, y_test, average=None)
print(f1dt)
[0.97674419 0.8503937 0.82692308 0.84536082 0.8125 0.8 0.90322581 0.81188119 0.76271186 0.78 ]
The corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1dt[0], f1dt[1], f1dt[2],
f1dt[3], f1dt[4], f1dt[5],
f1dt[6], f1dt[7],f1dt[8],f1dt[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score DT”)
plt.savefig(‘f1scoredt.png’)
The DT recall score is
from sklearn.metrics import recall_score
recalldt=recall_score(predtc, y_test, average=None)
print(recalldt)
[1. 0.81818182 0.86 0.91111111 0.7761194 0.79166667 0.875 0.80392157 0.76271186 0.8125 ]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recalldt[0], recalldt[1], recalldt[2],
recalldt[3], recalldt[4], recalldt[5],
recalldt[6], recalldt[7],recalldt[8],recalldt[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score DT”)
plt.savefig(‘recallscoredt.png’)
The DT precision score is
from sklearn.metrics import precision_score
precisiondt=precision_score(predtc, y_test, average=None)
print(precisiondt)
[0.95454545 0.8852459 0.7962963 0.78846154 0.85245902 0.80851064 0.93333333 0.82 0.76271186 0.75 ]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisiondt[0], precisiondt[1], precisiondt[2],
precisiondt[3], precisiondt[4], precisiondt[5],
precisiondt[6], precisiondt[7],precisiondt[8],precisiondt[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score DT”)
plt.savefig(‘precisionscoredt.png’)
The DT Cohen’s kappa is
from sklearn.metrics import cohen_kappa_score
cohdt=cohen_kappa_score(predtc, y_test)
print(cohdt)
0.8165480807402204
and the DT R2-coefficient is
import sklearn
r2dt=sklearn.metrics.r2_score(predtc, y_test)
print(r2dt)
0.5866539322676103
Let’s plot the DT Confusion Matrix
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predtc)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘DT Confusion Matrix’)
plt.savefig(‘confusionmatrixdt.png’)
The DT Classification Learning Curve is
skplt.estimators.plot_learning_curve(dt, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”DT Classification Learning Curve”);
plt.savefig(‘learningcurvedt.png’)
The DT ROC Curve is
Y_test_probs = dt.predict_proba(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”DT ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurvedt.png’)
The DT Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”DT Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurvedt.png’)
Random Forest Classifier (RFC aka RC)
Let’s invoke RandomForestClassifier
from sklearn.ensemble import RandomForestClassifier
rc = RandomForestClassifier(n_estimators=150)
rc.fit(X_train, y_train)
predrc = rc.predict(X_test)
acc_rc=accuracy_score(y_test, predrc)
print(acc_rc)
0.9740740740740741
with the following classification report
print(classification_report(predrc, y_test))
precision recall f1-score support
0 1.00 1.00 1.00 44
1 1.00 0.97 0.98 63
2 1.00 1.00 1.00 54
3 0.98 1.00 0.99 51
4 0.98 0.97 0.98 62
5 0.96 0.96 0.96 47
6 0.97 0.97 0.97 60
7 0.98 0.91 0.94 54
8 0.95 1.00 0.97 56
9 0.92 0.98 0.95 49
accuracy 0.97 540
macro avg 0.97 0.97 0.97 540
weighted avg 0.97 0.97 0.97 540
Here, the f1-score is
from sklearn.metrics import f1_score
f1rc=f1_score(predrc, y_test, average=None)
print(f1rc)
[1. 0.98387097 1. 0.99029126 0.97560976 0.95744681 0.96666667 0.94230769 0.97391304 0.95049505]
The bar chart of this array is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1rc[0], f1rc[1], f1rc[2],
f1rc[3], f1rc[4], f1rc[5],
f1rc[6], f1rc[7],f1rc[8],f1rc[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score RC”)
plt.savefig(‘f1scorerc.png’)
The recall score is given by
from sklearn.metrics import recall_score
recallrc=recall_score(predrc, y_test, average=None)
print(recallrc)
[1. 0.96825397 1. 1. 0.96774194 0.95744681 0.96666667 0.90740741 1. 0.97959184]
The corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallrc[0], recallrc[1], recallrc[2],
recallrc[3], recallrc[4], recallrc[5],
recallrc[6], recallrc[7],recallrc[8],recallrc[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score RC”)
plt.savefig(‘recallscorerc.png’)
The precision score is
from sklearn.metrics import precision_score
precisionrc=precision_score(predrc, y_test, average=None)
print(precisionrc)
[1. 1. 1. 0.98076923 0.98360656 0.95744681 0.96666667 0.98 0.94915254 0.92307692]
The bar plot of this array is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionrc[0], precisionrc[1], precisionrc[2],
precisionrc[3], precisionrc[4], precisionrc[5],
precisionrc[6], precisionrc[7],precisionrc[8],precisionrc[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score RC”)
plt.savefig(‘precisionscorerc.png’)
The Cohen’s kappa and the R2-coefficient are as follows
from sklearn.metrics import cohen_kappa_score
cohrc=cohen_kappa_score(predrc, y_test)
print(cohrc)
0.9711571477623898
import sklearn
r2rc=sklearn.metrics.r2_score(predrc, y_test)
print(r2rc)
0.9599345010188922
Let’s plot the normalized confusion matrix
The RC Classification Learning Curve is
skplt.estimators.plot_learning_curve(rc, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”RC Classification Learning Curve”);
plt.savefig(‘learningcurverc.png’)
The RC ROC Curve is
Y_test_probs = rc.predict_proba(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”RC ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurverc.png’)
The RC Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”RC Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurverc.png’)
Logistic Regression (LR)
Let’s consider LogisticRegression (LR)
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
lr.fit(X_train, y_train)
predlr = lr.predict(X_test)
acc_lr=accuracy_score(y_test, predlr)
print(acc_lr)
0.9611111111111111
The LR classification report is
precision recall f1-score support
0 1.00 1.00 1.00 44
1 1.00 0.91 0.95 67
2 1.00 0.96 0.98 56
3 0.94 0.98 0.96 50
4 0.95 0.98 0.97 59
5 0.94 0.92 0.93 48
6 0.98 0.98 0.98 60
7 0.96 0.96 0.96 50
8 0.90 0.93 0.91 57
9 0.94 1.00 0.97 49
accuracy 0.96 540
macro avg 0.96 0.96 0.96 540
weighted avg 0.96 0.96 0.96 540
with the f1-score
from sklearn.metrics import f1_score
f1lr=f1_score(predlr, y_test, average=None)
print(f1lr)
[1. 0.953125 0.98181818 0.96078431 0.96666667 0.92631579 0.98333333 0.96 0.9137931 0.97029703]
to be displayed as the following bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1lr[0], f1lr[1], f1lr[2],
f1lr[3], f1lr[4], f1lr[5],
f1lr[6], f1lr[7],f1lr[8],f1lr[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score LR”)
plt.savefig(‘f1scorelr.png’)
The recall score is
from sklearn.metrics import recall_score
recalllr=recall_score(predlr, y_test, average=None)
print(recalllr)
[1. 0.91044776 0.96428571 0.98 0.98305085 0.91666667 0.98333333 0.96 0.92982456 1. ]
to be visualized as the bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recalllr[0], recalllr[1], recalllr[2],
recalllr[3], recalllr[4], recalllr[5],
recalllr[6], recalllr[7],recalllr[8],recalllr[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score LR”)
plt.savefig(‘recallscorelr.png’)
Similarly, the precision score is computed and displayed as follows
from sklearn.metrics import precision_score
precisionlr=precision_score(predlr, y_test, average=None)
print(precisionlr)
[1. 1. 1. 0.94230769 0.95081967 0.93617021 0.98333333 0.96 0.89830508 0.94230769]
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionlr[0], precisionlr[1], precisionlr[2],
precisionlr[3], precisionlr[4], precisionlr[5],
precisionlr[6], precisionlr[7],precisionlr[8],precisionlr[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score LR”)
plt.savefig(‘precisionscorelr.png’)
The Cohen’s kappa and the R2-coefficient are as follows
from sklearn.metrics import cohen_kappa_score
cohlr=cohen_kappa_score(predlr, y_test)
print(cohlr)
0.9567319248951685
import sklearn
r2lr=sklearn.metrics.r2_score(predlr, y_test)
print(r2lr)
0.9243960907246911
The LR Confusion Matrix is
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predlr)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘LR Confusion Matrix’)
plt.savefig(‘confusionmatrixlr.png’)
The LR Classification Learning Curve is
skplt.estimators.plot_learning_curve(lr, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”LR Classification Learning Curve”);
plt.savefig(‘learningcurvelr.png’)
The LR ROC Curve is
Y_test_probs = lr.predict_proba(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”LR ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurvelr.png’)
The LR Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”LR Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurvelr.png’)
Extra Trees (ET) Classifier
Let’s look at ExtraTreesClassifier
from sklearn.ensemble import ExtraTreesClassifier
et=ExtraTreesClassifier()
et.fit(X_train, y_train)
predet = et.predict(X_test)
acc_et=accuracy_score(y_test, predet)
print(acc_et)
0.9851851851851852
with the classification report
print(classification_report(predet, y_test))
precision recall f1-score support
0 1.00 1.00 1.00 44
1 1.00 0.98 0.99 62
2 1.00 1.00 1.00 54
3 1.00 1.00 1.00 52
4 0.98 0.98 0.98 61
5 0.98 0.98 0.98 47
6 0.98 0.98 0.98 60
7 1.00 0.93 0.96 54
8 0.97 1.00 0.98 57
9 0.94 1.00 0.97 49
accuracy 0.99 540
macro avg 0.99 0.99 0.99 540
weighted avg 0.99 0.99 0.99 540
where the f1-score is given by
from sklearn.metrics import f1_score
f1et=f1_score(predet, y_test, average=None)
print(f1et)
[1. 0.99186992 1. 1. 0.98360656 0.9787234 0.98333333 0.96153846 0.98275862 0.97029703]
The bar plot of this array is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1et[0], f1et[1], f1et[2],
f1et[3], f1et[4], f1et[5],
f1et[6], f1et[7],f1et[8],f1et[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score ET”)
plt.savefig(‘f1scoreet.png’)
The recall score is
from sklearn.metrics import recall_score
recallet=recall_score(predet, y_test, average=None)
print(recallet)
[1. 0.98387097 1. 1. 0.98360656 0.9787234 0.98333333 0.92592593 1. 1. ]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallet[0], recallet[1], recallet[2],
recallet[3], recallet[4], recallet[5],
recallet[6], recallet[7],recallet[8],recallet[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score ET”)
plt.savefig(‘recallscoreet.png’)
The precision score is
from sklearn.metrics import precision_score
precisionet=precision_score(predet, y_test, average=None)
print(precisionet)
[1. 1. 1. 1. 0.98360656 0.9787234 0.98333333 1. 0.96610169 0.94230769]
The bar plot of this array is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionet[0], precisionet[1], precisionet[2],
precisionet[3], precisionet[4], precisionet[5],
precisionet[6], precisionet[7],precisionet[8],precisionet[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score ET”)
plt.savefig(‘precisionscoreet.png’)
The Cohen’s kappa and the R2-coefficient are
from sklearn.metrics import cohen_kappa_score
cohet=cohen_kappa_score(predet, y_test)
print(cohet)
0.9835190618073333
import sklearn
r2et=sklearn.metrics.r2_score(predet, y_test)
print(r2et)
0.9795101759227319
The ET Confusion Matrix is
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predet)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘ET Confusion Matrix’)
plt.savefig(‘confusionmatrixet.png’)
The ET Classification Learning Curve
skplt.estimators.plot_learning_curve(et, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”ET Classification Learning Curve”);
plt.savefig(‘learningcurveet.png’)
The ET ROC Curve is
Y_test_probs = et.predict_proba(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”ET ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurveet.png’)
The ET Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”ET Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurveet.png’)
Gradient Boosting (GB) Classifier
Let’s look at GradientBoostingClassifier
from sklearn.ensemble import GradientBoostingClassifier
gbc=GradientBoostingClassifier()
gbc.fit(X_train, y_train)
predgbc = gbc.predict(X_test)
acc_gbc=accuracy_score(y_test, predgbc)
print(acc_gbc)
0.9611111111111111
The GB classification report is
print(classification_report(predgbc, y_test))
precision recall f1-score support
0 0.95 0.98 0.97 43
1 1.00 0.97 0.98 63
2 0.98 0.98 0.98 54
3 0.96 0.98 0.97 51
4 0.92 0.98 0.95 57
5 0.98 0.94 0.96 49
6 0.95 0.97 0.96 59
7 0.94 0.90 0.92 52
8 0.98 0.95 0.97 61
9 0.94 0.96 0.95 51
accuracy 0.96 540
macro avg 0.96 0.96 0.96 540
weighted avg 0.96 0.96 0.96 540
where the f1-score is
from sklearn.metrics import f1_score
f1gbc=f1_score(predgbc, y_test, average=None)
print(f1gbc)
[0.96551724 0.98387097 0.98148148 0.97087379 0.94915254 0.95833333 0.95798319 0.92156863 0.96666667 0.95145631]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1gbc[0], f1gbc[1], f1gbc[2],
f1gbc[3], f1gbc[4], f1gbc[5],
f1gbc[6], f1gbc[7],f1gbc[8],f1gbc[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score GB”)
plt.savefig(‘f1scoregbc.png’)
The recall score is
from sklearn.metrics import recall_score
recallgbc=recall_score(predgbc, y_test, average=None)
print(recallgbc)
[0.97674419 0.96825397 0.98148148 0.98039216 0.98245614 0.93877551 0.96610169 0.90384615 0.95081967 0.96078431]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallgbc[0], recallgbc[1], recallgbc[2],
recallgbc[3], recallgbc[4], recallgbc[5],
recallgbc[6], recallgbc[7],recallgbc[8],recallgbc[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score GB”)
plt.savefig(‘recallscoregbc.png’)
The precision score is
from sklearn.metrics import precision_score
precisiongbc=precision_score(predgbc, y_test, average=None)
print(precisiongbc)
[0.95454545 1. 0.98148148 0.96153846 0.91803279 0.9787234 0.95 0.94 0.98305085 0.94230769]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisiongbc[0], precisiongbc[1], precisiongbc[2],
precisiongbc[3], precisiongbc[4], precisiongbc[5],
precisiongbc[6], precisiongbc[7],precisiongbc[8],precisiongbc[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score GB”)
plt.savefig(‘precisionscoregbc.png’)
The Cohen’s kappa score and the R2-coefficient are as follows
from sklearn.metrics import cohen_kappa_score
cohgbc=cohen_kappa_score(predgbc, y_test)
print(cohgbc)
0.9567383624669052
import sklearn
r2gbc=sklearn.metrics.r2_score(predgbc, y_test)
print(r2gbc)
0.9224255324212992
The GB Confusion Matrix is
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predgbc)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘GB Confusion Matrix’)
plt.savefig(‘confusionmatrixgbc.png’)
The GB Classification Learning Curve is
The GB ROC Curve is
The GB Precision-Recall Curve is
K-Neighbors Classifier (KNN)
Let’s consider KNeighborsClassifier
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
predknn = knn.predict(X_test)
acc_knn=accuracy_score(y_test, predknn)
print(acc_knn)
0.9851851851851852
with the following classification report
print(classification_report(predknn, y_test))
precision recall f1-score support
0 1.00 1.00 1.00 44
1 1.00 0.94 0.97 65
2 1.00 1.00 1.00 54
3 1.00 1.00 1.00 52
4 0.97 1.00 0.98 59
5 0.98 0.98 0.98 47
6 1.00 0.97 0.98 62
7 1.00 0.98 0.99 51
8 0.95 1.00 0.97 56
9 0.96 1.00 0.98 50
accuracy 0.99 540
macro avg 0.99 0.99 0.99 540
weighted avg 0.99 0.99 0.99 540
where the f1-score is given by
from sklearn.metrics import f1_score
f1knn=f1_score(predknn, y_test, average=None)
print(f1knn)
[1. 0.96825397 1. 1. 0.98333333 0.9787234 0.98360656 0.99009901 0.97391304 0.98039216]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1knn[0], f1knn[1], f1knn[2],
f1knn[3], f1knn[4], f1knn[5],
f1knn[6], f1knn[7],f1knn[8],f1knn[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score KNN”)
plt.savefig(‘f1scoreknn.png’)
The recall score is
from sklearn.metrics import recall_score
recallknn=recall_score(predknn, y_test, average=None)
print(recallknn)
[1. 0.93846154 1. 1. 1. 0.9787234 0.96774194 0.98039216 1. 1. ]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallknn[0], recallknn[1], recallknn[2],
recallknn[3], recallknn[4], recallknn[5],
recallknn[6], recallknn[7],recallknn[8],recallknn[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score KNN”)
plt.savefig(‘recallscoreknn.png’)
The precision score is
from sklearn.metrics import precision_score
precisionknn=precision_score(predknn, y_test, average=None)
print(precisionknn)
[1. 1. 1. 1. 0.96721311 0.9787234 1. 1. 0.94915254 0.96153846]
and the precision bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionknn[0], precisionknn[1], precisionknn[2],
precisionknn[3], precisionknn[4], precisionknn[5],
precisionknn[6], precisionknn[7],precisionknn[8],precisionknn[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score KNN”)
plt.savefig(‘precisionscoreknn.png’)
The Cohen’s kappa score and the R2-coefficient are as follows
from sklearn.metrics import cohen_kappa_score
cohknn=cohen_kappa_score(predknn, y_test)
print(cohknn)
0.9835175526618007
import sklearn
r2knn=sklearn.metrics.r2_score(predknn, y_test)
print(r2knn)
0.9535258324694893
The KNN Confusion Matrix is
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predknn)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘KNN Confusion Matrix’)
plt.savefig(‘confusionmatrixknn.png’)
The KNN Classification Learning Curve is
skplt.estimators.plot_learning_curve(knn, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”KNN Classification Learning Curve”);
plt.savefig(‘learningcurveknn.png’)
The KNN ROC Curve is
The KNN Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”KNN Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurveknn.png’)
Naive Bayes – Gaussian NB (GNB)
Let’s invoke GNB
from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
gnb.fit(X_train, y_train)
predgnb = gnb.predict(X_test)
acc_gnb=accuracy_score(y_test, predgnb)
print(acc_gnb)
0.8462962962962963
with the classification report
print(classification_report(predgnb, y_test))
precision recall f1-score support
0 1.00 1.00 1.00 44
1 0.85 0.78 0.81 67
2 0.50 0.93 0.65 29
3 0.81 0.98 0.88 43
4 0.90 0.92 0.91 60
5 0.94 0.94 0.94 47
6 0.98 0.98 0.98 60
7 0.98 0.74 0.84 66
8 0.88 0.59 0.71 88
9 0.63 0.92 0.75 36
accuracy 0.85 540
macro avg 0.85 0.88 0.85 540
weighted avg 0.88 0.85 0.85 540
The f1-score is
from sklearn.metrics import f1_score
f1gnb=f1_score(predgnb, y_test, average=None)
print(f1gnb)
[1. 0.8125 0.65060241 0.88421053 0.90909091 0.93617021 0.98333333 0.84482759 0.70748299 0.75 ]
with the bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1gnb[0], f1gnb[1], f1gnb[2],
f1gnb[3], f1gnb[4], f1gnb[5],
f1gnb[6], f1gnb[7],f1gnb[8],f1gnb[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score GNB”)
plt.savefig(‘f1scoregnb.png’)
The recall score is
from sklearn.metrics import recall_score
recallgnb=recall_score(predgnb, y_test, average=None)
print(recallgnb)
[1. 0.7761194 0.93103448 0.97674419 0.91666667 0.93617021 0.98333333 0.74242424 0.59090909 0.91666667]
with the bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallgnb[0], recallgnb[1], recallgnb[2],
recallgnb[3], recallgnb[4], recallgnb[5],
recallgnb[6], recallgnb[7],recallgnb[8],recallgnb[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score GNB”)
plt.savefig(‘recallscoregnb.png’)
The precision score is
from sklearn.metrics import precision_score
precisiongnb=precision_score(predgnb, y_test, average=None)
print(precisiongnb)
[1. 0.85245902 0.5 0.80769231 0.90163934 0.93617021 0.98333333 0.98 0.88135593 0.63461538]
with the bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisiongnb[0], precisiongnb[1], precisiongnb[2],
precisiongnb[3], precisiongnb[4], precisiongnb[5],
precisiongnb[6], precisiongnb[7],precisiongnb[8],precisiongnb[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score GNB”)
plt.savefig(‘precisionscoregnb.png’)
The Cohen’s kappa is
from sklearn.metrics import cohen_kappa_score
cohgnb=cohen_kappa_score(predgnb, y_test)
print(cohgnb)
0.8288934191538585
The R2-coefficient is
import sklearn
r2gnb=sklearn.metrics.r2_score(predgnb, y_test)
print(r2gnb)
0.5725061683672357
The GNB Confusion Matrix is
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predgnb)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘GNB Confusion Matrix’)
plt.savefig(‘confusionmatrixgnb.png’)
The GNB Classification Learning Curve is
skplt.estimators.plot_learning_curve(gnb, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”GNB Classification Learning Curve”);
plt.savefig(‘learningcurvegnb.png’)
The GNB ROC Curve is
Y_test_probs = gnb.predict_proba(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”GNB ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurvegnb.png’)
The GNB Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”GNB Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurvegnb.png’)
Stochastic Gradient Descent (SGD) Learning
Let’s look at SGD
sgd = SGDClassifier(max_iter=150, tol=None)
sgd.fit(X_train, y_train)
predsgd = sgd.predict(X_test)
acc_sgd=accuracy_score(y_test, predsgd)
print(acc_sgd)
0.9444444444444444
The SGD classification report is given by
print(classification_report(predsgd, y_test))
precision recall f1-score support
0 0.98 1.00 0.99 43
1 0.97 0.94 0.95 63
2 0.94 0.96 0.95 53
3 0.96 0.91 0.93 55
4 0.95 0.98 0.97 59
5 0.94 0.96 0.95 46
6 0.97 0.98 0.97 59
7 0.92 0.96 0.94 48
8 0.97 0.83 0.89 69
9 0.85 0.98 0.91 45
accuracy 0.94 540
macro avg 0.94 0.95 0.95 540
weighted avg 0.95 0.94 0.94 540
with the following multi-label scores and their corresponding bar plots:
- f1-score
from sklearn.metrics import f1_score
f1sgd=f1_score(predsgd, y_test, average=None)
print(f1sgd)
[0.98850575 0.9516129 0.95327103 0.93457944 0.96666667 0.94623656
0.97478992 0.93877551 0.890625 0.90721649]
The bar plot:
import pandas as pd
results = pd.DataFrame({
'Label': ['0', '1', '2',
'3', '4', '5',
'6',
'7','8','9'],
'F1-Score': [f1sgd[0], f1sgd[1], f1sgd[2],
f1sgd[3], f1sgd[4], f1sgd[5],
f1sgd[6], f1sgd[7],f1sgd[8],f1sgd[9]]})
result_df = results.sort_values(by='F1-Score', ascending=False)
result_df = result_df.set_index('F1-Score')
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({'font.size':20})
plt.bar(result_df['Label'], result_df['F1-Score'], color ='maroon',
width = 0.4)
plt.xlabel("Label")
plt.ylabel("F1-Score")
plt.title("F1-Score SGD")
plt.savefig('f1scoresgd.png')
- recall-score
from sklearn.metrics import recall_score
recallsgd=recall_score(predsgd, y_test, average=None)
print(recallsgd)
[1. 0.93650794 0.96226415 0.90909091 0.98305085 0.95652174 0.98305085 0.95833333 0.82608696 0.97777778]
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallsgd[0], recallsgd[1], recallsgd[2],
recallsgd[3], recallsgd[4], recallsgd[5],
recallsgd[6], recallsgd[7],recallsgd[8],recallsgd[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score SGD”)
plt.savefig(‘recallscoresgd.png’)
- precision-score
from sklearn.metrics import precision_score
precisionsgd=precision_score(predsgd, y_test, average=None)
print(precisionsgd)
[0.97727273 0.96721311 0.94444444 0.96153846 0.95081967 0.93617021 0.96666667 0.92 0.96610169 0.84615385]
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionsgd[0], precisionsgd[1], precisionsgd[2],
precisionsgd[3], precisionsgd[4], precisionsgd[5],
precisionsgd[6], precisionsgd[7],precisionsgd[8],precisionsgd[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score SGD”)
plt.savefig(‘precisionscoresgd.png’)
The Cohen’s kappa and the R2-coefficient are
from sklearn.metrics import cohen_kappa_score
cohsgd=cohen_kappa_score(predsgd, y_test)
print(cohsgd)
0.9381752540729914
import sklearn
r2sgd=sklearn.metrics.r2_score(predsgd, y_test)
print(r2sgd)
0.9144340844863577
The SGD Confusion Matrix is
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predsgd)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘SGD Confusion Matrix’)
plt.savefig(‘confusionmatrixsgd.png’)
The SGD Classification Learning Curve
skplt.estimators.plot_learning_curve(sgd, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”SGD Classification Learning Curve”);
plt.savefig(‘learningcurvesgd.png’)
The SGD ROC Curve is
sgd = SGDClassifier(max_iter=150, tol=None,loss=’modified_huber’)
sgd.fit(X_train, y_train)
predsgd = sgd.predict(X_test)
Y_test_probs = sgd.predict_proba(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”SGD ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurvesgd.png’)
The SGD Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”SGD Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurvesgd.png’)
Perceptron (PERC)
Let’s look at PERC
perceptron = Perceptron(max_iter=150)
perceptron.fit(X_train, y_train)
predperc = perceptron.predict(X_test)
acc_perc=accuracy_score(y_test, predperc)
print(acc_perc)
0.9277777777777778
with the classification report
print(classification_report(predperc, y_test))
precision recall f1-score support
0 0.98 1.00 0.99 43
1 0.98 0.86 0.92 70
2 1.00 0.90 0.95 60
3 0.85 1.00 0.92 44
4 0.98 0.95 0.97 63
5 0.98 0.84 0.90 55
6 0.97 0.98 0.97 59
7 0.94 0.94 0.94 50
8 0.69 0.98 0.81 42
9 0.92 0.89 0.91 54
accuracy 0.93 540
macro avg 0.93 0.93 0.93 540
weighted avg 0.94 0.93 0.93 540
Here, the f1-score is
from sklearn.metrics import f1_score
f1perc=f1_score(predperc, y_test, average=None)
print(f1perc)
[0.98850575 0.91603053 0.94736842 0.91666667 0.96774194 0.90196078 0.97478992 0.94 0.81188119 0.90566038]
with the corresponding bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1perc[0], f1perc[1], f1perc[2],
f1perc[3], f1perc[4], f1perc[5],
f1perc[6], f1perc[7],f1perc[8],f1perc[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score PERC”)
plt.savefig(‘f1scoreperc.png’)
The recall score is
from sklearn.metrics import recall_score
recallperc=recall_score(predperc, y_test, average=None)
print(recallperc)
[1. 0.85714286 0.9 1. 0.95238095 0.83636364 0.98305085 0.94 0.97619048 0.88888889]
with the corresponding bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallperc[0], recallperc[1], recallperc[2],
recallperc[3], recallperc[4], recallperc[5],
recallperc[6], recallperc[7],recallperc[8],recallperc[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score PERC”)
plt.savefig(‘recallscoreperc.png’)
The precision score is
from sklearn.metrics import precision_score
precisionperc=precision_score(predperc, y_test, average=None)
print(precisionperc)
[0.97727273 0.98360656 1. 0.84615385 0.98360656 0.9787234 0.96666667 0.94 0.69491525 0.92307692]
with the corresponding bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionperc[0], precisionperc[1], precisionperc[2],
precisionperc[3], precisionperc[4], precisionperc[5],
precisionperc[6], precisionperc[7],precisionperc[8],precisionperc[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score PERC”)
plt.savefig(‘precisionscoreperc.png’)
The Cohen’s kappa score and the R2-coefficient are
from sklearn.metrics import cohen_kappa_score
cohperc=cohen_kappa_score(predperc, y_test)
print(cohperc)
0.9196661529776163
import sklearn
r2perc=sklearn.metrics.r2_score(predperc, y_test)
print(r2perc)
0.7983361280900092
The PERC Confusion Matrix is
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predperc)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘PERC Confusion Matrix’)
plt.savefig(‘confusionmatrixperc.png’)
The PERC Classification Learning Curve is
skplt.estimators.plot_learning_curve(perceptron, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”PERC Classification Learning Curve”);
plt.savefig(‘learningcurveperc.png’)
The PERC ROC Curve is
perceptron = Perceptron(max_iter=150)
perceptron.fit(X_train, y_train)
predperc = perceptron.predict(X_test)
Y_test_probs = perceptron.decision_function(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”PERC ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurveperc.png’)
The PERC Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”PERC Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurveperc.png’)
Linear SVC (LSVC)
Let’s look at the LSVC classifier
from sklearn.svm import SVC, LinearSVC
linear_svc = LinearSVC()
linear_svc.fit(X_train, y_train)
predlsvc = linear_svc.predict(X_test)
acc_lsvc=accuracy_score(y_test, predlsvc)
print (acc_lsvc)
0.9351851851851852
The LSVC classification report is
print(classification_report(predlsvc, y_test))
precision recall f1-score support
0 0.98 1.00 0.99 43
1 0.98 0.86 0.92 70
2 0.98 0.93 0.95 57
3 0.92 0.96 0.94 50
4 0.95 0.98 0.97 59
5 0.94 0.92 0.93 48
6 0.98 0.95 0.97 62
7 0.92 0.90 0.91 51
8 0.81 0.92 0.86 52
9 0.90 0.98 0.94 48
accuracy 0.94 540
macro avg 0.94 0.94 0.94 540
weighted avg 0.94 0.94 0.94 540
The f1-score is
from sklearn.metrics import f1_score
f1lsvc=f1_score(predlsvc, y_test, average=None)
print(f1lsvc)
[0.98850575 0.91603053 0.95495495 0.94117647 0.96666667 0.92631579 0.96721311 0.91089109 0.86486486 0.94 ]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1lsvc[0], f1lsvc[1], f1lsvc[2],
f1lsvc[3], f1lsvc[4], f1lsvc[5],
f1lsvc[6], f1lsvc[7],f1lsvc[8],f1lsvc[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score LSVC”)
plt.savefig(‘f1scorelsvc.png’)
The recall score is
from sklearn.metrics import recall_score
recalllsvc=recall_score(predlsvc, y_test, average=None)
print(recalllsvc)
[1. 0.85714286 0.92982456 0.96 0.98305085 0.91666667 0.9516129 0.90196078 0.92307692 0.97916667]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recalllsvc[0], recalllsvc[1], recalllsvc[2],
recalllsvc[3], recalllsvc[4], recalllsvc[5],
recalllsvc[6], recalllsvc[7],recalllsvc[8],recalllsvc[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score LSVC”)
plt.savefig(‘recallscorelsvc.png’)
The precision score is
from sklearn.metrics import precision_score
precisionlsvc=precision_score(predlsvc, y_test, average=None)
print(precisionlsvc)
[0.97727273 0.98360656 0.98148148 0.92307692 0.95081967 0.93617021 0.98333333 0.92 0.81355932 0.90384615]
and the corresponding bar plot is
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionlsvc[0], precisionlsvc[1], precisionlsvc[2],
precisionlsvc[3], precisionlsvc[4], precisionlsvc[5],
precisionlsvc[6], precisionlsvc[7],precisionlsvc[8],precisionlsvc[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score LSVC”)
plt.savefig(‘precisionscorelsvc.png’)
The Cohen’s kappa score and the R2-coefficient are
from sklearn.metrics import cohen_kappa_score
cohlsvc=cohen_kappa_score(predlsvc, y_test)
print(cohlsvc)
0.9299426491294344
import sklearn
r2lsvc=sklearn.metrics.r2_score(predlsvc, y_test)
print(r2lsvc)
0.8658915877097091
The LSVC Confusion Matrix is
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predlsvc)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘LSVC Confusion Matrix’)
plt.savefig(‘confusionmatrixlsvc.png’)
The LSVC Classification Learning Curve is
skplt.estimators.plot_learning_curve(linear_svc, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”LSVC Classification Learning Curve”);
plt.savefig(‘learningcurvelsvc.png’)
The LSVC ROC Curve is
LinearSVC_classifier = SVC(kernel=’linear’,probability=True)
from sklearn.svm import SVC, LinearSVC
linear_svc = LinearSVC_classifier
linear_svc.fit(X_train, y_train)
predlsvc = linear_svc.predict(X_test)
Y_test_probs = linear_svc.predict_proba(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”LSVC ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurvelsvc.png’)
The LSVC Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”LSVC Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurvelsvc.png’)
Quadratic Discriminant Analysis (QDA)
Let’s examine QDA
qda = QuadraticDiscriminantAnalysis()
qda.fit(X_train, y_train)
predqda = qda.predict(X_test)
acc_qda=accuracy_score(y_test, predqda)
print(acc_qda)
0.8870370370370371
with the classification report
print(classification_report(predqda, y_test))
precision recall f1-score support
0 0.80 1.00 0.89 35
1 0.97 0.74 0.84 80
2 0.91 0.98 0.94 50
3 0.92 1.00 0.96 48
4 0.82 0.98 0.89 51
5 0.98 0.78 0.87 59
6 0.83 0.98 0.90 51
7 0.96 0.74 0.83 65
8 0.93 0.90 0.92 61
9 0.75 0.97 0.85 40
accuracy 0.89 540
macro avg 0.89 0.91 0.89 540
weighted avg 0.90 0.89 0.89 540
The f1-score is
from sklearn.metrics import f1_score
f1qda=f1_score(predqda, y_test, average=None)
print(f1qda)
[0.88607595 0.83687943 0.94230769 0.96 0.89285714 0.86792453 0.9009009 0.83478261 0.91666667 0.84782609]
with the bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘F1-Score’: [f1qda[0], f1qda[1], f1qda[2],
f1qda[3], f1qda[4], f1qda[5],
f1qda[6], f1qda[7],f1qda[8],f1qda[9]]})
result_df = results.sort_values(by=’F1-Score’, ascending=False)
result_df = result_df.set_index(‘F1-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘F1-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“F1-Score”)
plt.title(“F1-Score QDA”)
plt.savefig(‘f1scoreqda.png’)
The recall score is
from sklearn.metrics import recall_score
recallqda=recall_score(predqda, y_test, average=None)
print(recallqda)
[1. 0.7375 0.98 1. 0.98039216 0.77966102 0.98039216 0.73846154 0.90163934 0.975 ]
with the bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Recall-Score’: [recallqda[0], recallqda[1], recallqda[2],
recallqda[3], recallqda[4], recallqda[5],
recallqda[6], recallqda[7],recallqda[8],recallqda[9]]})
result_df = results.sort_values(by=’Recall-Score’, ascending=False)
result_df = result_df.set_index(‘Recall-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Recall-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Recall-Score”)
plt.title(“Recall-Score QDA”)
plt.savefig(‘recallscoreqda.png’)
The precision score is
from sklearn.metrics import precision_score
precisionqda=precision_score(predqda, y_test, average=None)
print(precisionqda)
[0.79545455 0.96721311 0.90740741 0.92307692 0.81967213 0.9787234 0.83333333 0.96 0.93220339 0.75 ]
with the bar plot
import pandas as pd
results = pd.DataFrame({
‘Label’: [‘0’, ‘1’, ‘2’,
‘3’, ‘4’, ‘5’,
‘6’,
‘7’,’8′,’9′],
‘Precision-Score’: [precisionqda[0], precisionqda[1], precisionqda[2],
precisionqda[3], precisionqda[4], precisionqda[5],
precisionqda[6], precisionqda[7],precisionqda[8],precisionqda[9]]})
result_df = results.sort_values(by=’Precision-Score’, ascending=False)
result_df = result_df.set_index(‘Precision-Score’)
result_df.reset_index(inplace=True)
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Label’], result_df[‘Precision-Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Label”)
plt.ylabel(“Precision-Score”)
plt.title(“Precision-Score QDA”)
plt.savefig(‘precisionscoreqda.png’)
The Cohen’s kappa score and the R2-coefficient are
from sklearn.metrics import cohen_kappa_score
cohqda=cohen_kappa_score(predqda, y_test)
print(cohqda)
0.8743280518558931
import sklearn
r2qda=sklearn.metrics.r2_score(predqda, y_test)
print(r2qda)
0.7428227983353953
The QDA Confusion Matrix is
from sklearn.metrics import confusion_matrix
import seaborn as sns
cm = confusion_matrix(y_test, predqda)
target_names=[‘0′,’1′,’2′,’3′,’4′,’5′,’6′,’7′,’8′,’9’]
cmn = cm.astype(‘float’) / cm.sum(axis=1)[:, np.newaxis]
fig, ax = plt.subplots(figsize=(20,20))
sns.heatmap(cmn, annot=True, fmt=’.2f’, xticklabels=target_names, yticklabels=target_names,)
plt.ylabel(‘Actual’)
plt.xlabel(‘Predicted’)
plt.title(‘QDA Confusion Matrix’)
plt.savefig(‘confusionmatrixqda.png’)
The QDA Classification Learning Curve is
skplt.estimators.plot_learning_curve(qda, X_train, y_train,
cv=7, shuffle=True, scoring=”accuracy”,
n_jobs=-1, figsize=(10,10), title_fontsize=”small”, text_fontsize=”small”,
title=”QDA Classification Learning Curve”);
plt.savefig(‘learningcurveqda.png’)
The QDA ROC Curve is
Y_test_probs = qda.predict_proba(X_test)
skplt.metrics.plot_roc_curve(y_test, Y_test_probs,
title=”QDA ROC Curve”, figsize=(20,10));
plt.savefig(‘rocurveqda.png’)
The QDA Precision-Recall Curve is
skplt.metrics.plot_precision_recall_curve(y_test, Y_test_probs,
title=”QDA Precision-Recall Curve”, figsize=(20,10));
plt.savefig(‘precisionrecallcurveqda.png’)
Final QC Comparisons
Let’s incorporate the following 11 accuracy scores acc_* into a single DataFrame
import pandas as pd
results = pd.DataFrame({
‘Model’: [‘SVM’, ‘DT’, ‘RF’,
‘LR’, ‘ET’, ‘GB’,
‘KNN’,
‘GNB’,’SGD’,’PER’,’MLP’],
‘Score’: [acc_svc, acc_dt, acc_rc,
acc_lr, acc_et, acc_gbc,
acc_knn, acc_gnb,acc_sgd,acc_perc,acc_mlp]})
result_df = results.sort_values(by=’Score’, ascending=False)
result_df = result_df.set_index(‘Score’)
This yields the table
result_df.reset_index(inplace=True)
print(result_df)
Score Model 0 0.994444 SVM 1 0.988889 KNN 2 0.987037 ET 3 0.972222 RF 4 0.962963 LR 5 0.940741 PER 6 0.938889 GB 7 0.931481 SGD 8 0.914680 MLP 9 0.833333 DT 10 0.818519 GNB
creating the bar plot
import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Model’], result_df[‘Score’], color =’maroon’,
width = 0.4)
plt.xlabel(“Model”)
plt.ylabel(“Score”)
plt.title(“Accuracy Score”)
plt.savefig(‘accuracyscore.png’)
Let’s expand the list of models
import pandas as pd
results = pd.DataFrame({
‘Model’: [‘SVM’, ‘DT’, ‘RF’,
‘LR’, ‘ET’, ‘GB’,
‘KNN’,
‘GNB’,’SGD’,’PER’,’MLP’,’LSVM’,’QDA’],
‘Score’: [acc_svc, acc_dt, acc_rc,
acc_lr, acc_et, acc_gbc,
acc_knn, acc_gnb,acc_sgd,acc_perc,acc_mlp,acc_lsvc,acc_qda]})
result_df = results.sort_values(by=’Score’, ascending=False)
result_df = result_df.set_index(‘Score’)
result_df.reset_index(inplace=True)
print(result_df)
Score Model 0 0.996296 SVM 1 0.985185 ET 2 0.985185 KNN 3 0.974074 RF 4 0.961111 LR 5 0.961111 GB 6 0.951852 LSVM 7 0.944444 SGD 8 0.927778 PER 9 0.914680 MLP 10 0.887037 QDA 11 0.846296 GNB 12 0.835185 DT
creating the following bar plot
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Model’], result_df[‘Score’], color =’red’,
width = 0.4)
plt.xlabel(“Model”)
plt.ylabel(“Score”)
plt.title(“Accuracy Score”)
plt.savefig(‘accuracyscore13models.png’)
Let’s create the Cohen’s kappa bar plot
import pandas as pd
results = pd.DataFrame({
‘Model’: [‘SVM’, ‘DT’, ‘RF’,
‘LR’, ‘ET’, ‘GB’,
‘KNN’,
‘GNB’,’SGD’,’PER’,’MLP’,’LSVM’,’QDA’],
‘Score’: [cohsvc, cohdt, cohrc,
cohlr, cohet, cohgbc,
cohknn, cohgnb,cohsgd,cohperc,cohmlp,cohlsvc,cohqda]})
result_df = results.sort_values(by=’Score’, ascending=False)
result_df = result_df.set_index(‘Score’)
result_df.reset_index(inplace=True)
print(result_df)
Score Model 0 0.995879 SVM 1 0.983519 ET 2 0.983518 KNN 3 0.971157 RF 4 0.956738 GB 5 0.956732 LR 6 0.938175 SGD 7 0.929943 LSVM 8 0.919666 PER 9 0.905185 MLP 10 0.874328 QDA 11 0.828893 GNB 12 0.816548 DT
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Model’], result_df[‘Score’], color =’green’,
width = 0.4)
plt.xlabel(“Model”)
plt.ylabel(“Score”)
plt.title(“Cohen’s Kappa Score”)
plt.savefig(‘cohenkappascore13models.png’)
Let’s create the R2-coefficient bar plot
import pandas as pd
results = pd.DataFrame({
‘Model’: [‘SVM’, ‘DT’, ‘RF’,
‘LR’, ‘ET’, ‘GB’,
‘KNN’,
‘GNB’,’SGD’,’PER’,’MLP’,’LSVM’,’QDA’],
‘Score’: [r2svc, r2dt, r2rc,
r2lr, r2et, r2gbc,
r2knn, r2gnb,r2sgd,r2perc,r2mlp,r2lsvc,r2qda]})
result_df = results.sort_values(by=’Score’, ascending=False)
result_df = result_df.set_index(‘Score’)
result_df.reset_index(inplace=True)
print(result_df)
Score Model 0 0.996104 SVM 1 0.979510 ET 2 0.959935 RF 3 0.953526 KNN 4 0.924396 LR 5 0.922426 GB 6 0.914434 SGD 7 0.865892 LSVM 8 0.798336 PER 9 0.756094 MLP 10 0.742823 QDA 11 0.586654 DT 12 0.572506 GNB
fig = plt.figure(figsize = (20, 5))
plt.rcParams.update({‘font.size’:20})
plt.bar(result_df[‘Model’], result_df[‘Score’], color =’black’,
width = 0.4)
plt.xlabel(“Model”)
plt.ylabel(“Score”)
plt.title(“R2 Score”)
plt.savefig(‘r2score13models.png’)
Summary
The HDR ML/AI multi-label classification using MNIST data has been an area of interest for many years. In this work, the scikit-learn algorithms’ accuracy, precision, recall, F1-score, Cohen’s kappa, and R2 score are considered for supervised ML and DL based algorithms such as
- Support Vector Machines (SVM)
- Decision Trees (DT)
- Random Forest Classifier (RC)
- Logistic Regression (LR) [8]
- Extra Trees Classifier (ET)
- Gradient Boosting Classifier (GB)
- K-Nearest Neighbors (KNN)
- Naive Bayes (GNB)
- Stochastic Gradient Descent (SGD)
- Perceptron (PERC)
- Linear SVC (LSVC)
- Quadratic Discriminant Analysis (QDA)
In addition to the above metrics, the scikit ML performance plots in the form of the normalized confusion matrix image, classification learning, precision-recall and ROC curve have been compared.
The impact of various train/test data split ratios, data scaling, PCA, Keras CNN models and hyper-parameter optimization (HPO) have been examined.
Results show that the 4 best performing multi-label classifiers are SVM, ET, KNN, and RF; 4, 5, 8, and 9 are among the most misclassified labels.
Explore More
GIS ML/AI: Multi-Label Classification of Satellite Images with Fast.AI
Multi-Label Keras CNN Image Classification of MNIST Fashion Clothing
ML/AI Prediction of Wine Quality
99% Accurate Breast Cancer Classification using Neural Networks in TensorFlow 2.11.0
The Power of AIHealth: Comparison of 12 ML Breast Cancer Classification Models
A Comparison of Binary Classifiers for Enhanced ML/AI Breast Cancer Diagnostics – 1. Scikit-Plot
HPO-Tuned ML Diabetes-2 Prediction
ML/AI Image Classifier for Skin Cancer Detection
Supervised ML/AI Stock Prediction using Keras LSTM Models
References
[1] Deep Learning Project – Handwritten Digit Recognition using Python
[2] Tuning a Deep Convolutional Network for Image Recognition, with keras and TensorFlow
[3] Handwritten Digit Recognition with scikit-learn
[4] Recognizing Handwritten Digits in Python with scikit-learn
[5] MNIST Dataset: Digit Recognizer
[6] MNIST Handwritten Digit Recognition
[7] Github: Recognizing-Handwritten-Digits-with-scikit-learn
[8] Logistic Regression using Python (scikit-learn)
[9] Handwritten digit recognition on MNIST dataset using python
[10] Recognizing hand-written digits
[11] Recognizing Handwritten Digits with Scikit-learn
Appendix A: TensorFlow/Keras CNN (Part 1b)
Following the DL CNN tutorial, let’s load the train and test MNIST data
import tensorflow as tf
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
print (x_train.shape, y_train.shape, x_test.shape, y_test.shape)
(60000, 28, 28) (60000,) (10000, 28, 28) (10000,)
Let’s plot the selected image
import matplotlib.pyplot as plt
def plot_img(i):
# plot the image and the target for sample i
plt.imshow(x_train[i])
plt.title(y_train[i])
plt.axis(‘off’)
plot_img(2)
Let’s plot the selected training sets
plt.hist(x_train[0])
(array([[28., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[28., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[28., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[28., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[26., 0., 1., 0., 0., 1., 0., 0., 0., 0.],
[26., 0., 0., 0., 0., 0., 1., 0., 0., 1.],
[26., 0., 0., 0., 0., 0., 0., 0., 1., 1.],
[24., 1., 0., 0., 0., 0., 1., 0., 0., 2.],
[21., 1., 0., 1., 0., 0., 0., 0., 3., 2.],
[20., 1., 1., 0., 0., 1., 1., 0., 0., 4.],
[20., 0., 0., 1., 1., 1., 0., 0., 1., 4.],
[18., 0., 0., 0., 1., 1., 2., 0., 0., 6.],
[15., 2., 0., 0., 0., 1., 1., 1., 1., 7.],
[15., 0., 0., 2., 0., 1., 0., 2., 1., 7.],
[16., 2., 1., 1., 0., 0., 0., 1., 2., 5.],
[18., 0., 0., 0., 0., 1., 1., 3., 0., 5.],
[15., 1., 0., 2., 2., 0., 0., 1., 0., 7.],
[16., 0., 0., 0., 1., 1., 1., 1., 0., 8.],
[19., 0., 0., 2., 0., 1., 1., 0., 1., 4.],
[20., 2., 0., 1., 0., 0., 1., 2., 1., 1.],
[24., 0., 1., 1., 0., 0., 1., 0., 0., 1.],
[25., 0., 1., 0., 0., 0., 0., 0., 0., 2.],
[25., 1., 0., 0., 0., 0., 0., 1., 0., 1.],
[26., 0., 1., 0., 1., 0., 0., 0., 0., 0.],
[28., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[28., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[28., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[28., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]),
Let’s normalize the data
x_train = x_train/255.
x_test = x_test/255.
This is how the image of zero looks like
import numpy as np
import seaborn as sns
zero = np.array([[0,0,0,1,0,0,0,0],
[0,1,2,3,2,1,0,0],
[0,4,3,1,3,2,0,0],
[1,5,2,0,0,5,1,0],
[2,4,0,0,0,6,2,0],
[1,3,0,0,0,4,1,0],
[0,2,3,2,1,3,0,0],
[0,0,3,4,3,1,0,0]])
sns.heatmap(zero, annot=True)
Let’s check that the pooled values are what you expect for both the max pooling and the average pooling operations for the above image
from skimage.util import view_as_blocks
pooling_window_shape = (2,2)
view = view_as_blocks(zero, pooling_window_shape)
flatten_view = view.reshape(view.shape[0], view.shape[1], -1)
mean_view = np.mean(flatten_view, axis=2)
max_view = np.max(flatten_view, axis=2)
sns.heatmap(max_view, annot=True)
sns.heatmap(mean_view, annot=True)
Let’s create the following multi-layered CNN model
from keras import models
from keras import layers
model = models.Sequential()
model.add( layers.Conv2D(10, 4, input_shape=(28,28,1), activation=’relu’) )
model.summary()
Model: "sequential_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_3 (Conv2D) (None, 25, 25, 10) 170
=================================================================
Total params: 170
Trainable params: 170
Non-trainable params: 0
Let’s add layers
model.add( layers.Flatten() )
model.add( layers.Dense(10, activation=’relu’) )
model.add( layers.Dense(10, activation=’softmax’) )
model.add( layers.Dense(10, activation=’softmax’) )
model.summary()
Model: "sequential_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_3 (Conv2D) (None, 25, 25, 10) 170
flatten_6 (Flatten) (None, 6250) 0
dense_9 (Dense) (None, 10) 62510
dense_10 (Dense) (None, 1) 11
dense_11 (Dense) (None, 1) 2
flatten_7 (Flatten) (None, 1) 0
dense_12 (Dense) (None, 10) 20
dense_13 (Dense) (None, 10) 110
dense_14 (Dense) (None, 10) 110
=================================================================
Total params: 62,933
Trainable params: 62,933
Non-trainable params: 0
Let’s compile the model
from keras.optimizers import RMSprop
model.compile(loss=’categorical_crossentropy’,
optimizer=RMSprop(lr=0.1),
metrics=[‘acc’])
kx_train10 = x_train10.reshape(len(x_train10),28,28,1)
kx_test10 = x_test10.reshape(len(x_test10),28,28,1)
kx_test10.shape
(1000, 28, 28, 1)
Let’s fit the model
history = model.fit(kx_train10, y_train10, validation_data=(kx_test10,y_test10),
batch_size=5, epochs=10)
We’re getting an accuracy larger than 98% from this DL example. The further DL hyperparameter tuning of the above CNN model yields the classification accuracy of 99.8%.
Appendix B: PCA Dimensionality Reduction (Part 3)
Let’s apply the PCA decomposition to our training dataset X_train
pca = PCA(random_state=1)
pca.fit(X_train)
skplt.decomposition.plot_pca_component_variance(pca, figsize=(8,6));
plt.savefig(‘pcavariance.png’)
It is clear that the explained variance ratio for first 11 components is equal to 0.762.
Let’s plot the PCA 2-D projection
skplt.decomposition.plot_pca_2d_projection(pca, X_train, y_train,
figsize=(10,10),
cmap=”tab10″);
plt.savefig(‘pca2dprojection.png’)
We can see a clear separation boundary between the two clusters of digits {0, 4, 6} and {1, 2, 3, 5, 7, 8, 9}.
Appendix C: MNIST Description Key
.. _digits_dataset:
Optical recognition of handwritten digits dataset
--------------------------------------------------
**Data Set Characteristics:**
:Number of Instances: 1797
:Number of Attributes: 64
:Attribute Information: 8x8 image of integer pixels in the range 0..16.
:Missing Attribute Values: None
:Creator: E. Alpaydin (alpaydin '@' boun.edu.tr)
:Date: July; 1998
This is a copy of the test set of the UCI ML hand-written digits datasets
https://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits
The data set contains images of hand-written digits: 10 classes where
each class refers to a digit.
Preprocessing programs made available by NIST were used to extract
normalized bitmaps of handwritten digits from a preprinted form. From a
total of 43 people, 30 contributed to the training set and different 13
to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of
4x4 and the number of on pixels are counted in each block. This generates
an input matrix of 8x8 where each element is an integer in the range
0..16. This reduces dimensionality and gives invariance to small
distortions.
For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G.
T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C.
L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469,
1994.
.. topic:: References
- C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their
Applications to Handwritten Digit Recognition, MSc Thesis, Institute of
Graduate Studies in Science and Engineering, Bogazici University.
- E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika.
- Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin.
Linear dimensionalityreduction using relevance weighted LDA. School of
Electrical and Electronic Engineering Nanyang Technological University.
2005.
- Claudio Gentile. A New Approximate Maximal Margin Classification
Algorithm. NIPS. 2000.
Make a one-time donation
Make a monthly donation
Make a yearly donation
Choose an amount
Or enter a custom amount
Your contribution is appreciated.
Your contribution is appreciated.
Your contribution is appreciated.
DonateDonate monthlyDonate yearly
Digital High Science 