SVM in Machine Learning

Introduction

SVM in Machine Learning is an abbreviation for Support Vector Machine. It is one of the significant machine learning algorithms used for supervised learning. It can solve Regression as well as Classification problems. However, we mostly use it for solving classification problems only.

Before this algorithm, we have covered other algorithms of machine learning as well. It is recommended from our side that you go through them also. You can find the link to the same below.

Logistic Regression: https://ainewgeneration.com/logistic-regression/

Decision Tree: https://ainewgeneration.com/decision-tree-in-machine-learning/

Random Forest: https://ainewgeneration.com/random-forest-in-machine-learning/

Table of Contents

1. What is SVM? Why is it called so?
2. Types of SVM
3. Hyperplane and Support Vectors in SVM
4. Kernel Function
5. How to implement SVM

What is SVM? Why is it called so?

In Support Vector Machine algorithm or SVM as it is popularly called, we plot all the data points belonging to different classes available to us in an n- dimensional space. Here, n is actually same as the number of features or attributes present in our data set. The Support vector machine then draws a decision boundary or a hyperplane(we will study about this as we proceed) such that the boundary divides the data points into different classification groups. The chosen decision boundary is such that it divides the classes in the best possible manner.

You might be thinking why is this algorithm given the name Support Vector Machine. Well, the reason to this lies in the fact that while choosing the best optimal decision boundary or hyperplane, this algorithm makes use of lines which are parallel to the decision boundary and passing through the nearest data points of the classes. These lines are known as Support Vectors, and so the name Support Vector Machine. We will study about Support Vectors in detail in this blog as we proceed.

Types of SVM

SVM is of two major types. Depending on the dimension of the decision boundary or the  hyperplane, the types of SVM are:

1. Linear SVM: When the data is linearly separable i.e. a straight line can be drawn to classify the data points into two classes , then the SVM classifier used to classify this type of data is known as Linear SVM classifier. This type of SVM is mostly used when the data space is two dimensional i.e. only two classes are involved in classification.
2. Non- Linear SVM: When the data is non- linearly separable i.e. a straight line cannot be drawn to classify the data points into classes, then the SVM classifier used to classify this type of data is known as Non- Linear SVM classifier. This type of SVM is mostly used when the data space is multi- dimensional i.e. more than two classes are involved and there are multiple attributes involved in the classification.

Hyperplane and Support Vectors in SVM

Hyperplane : This is actually the boundary which divides the data points into different classes, thus also known as a decision boundary. It can be a straight line when only two classes are involved in the classification. However, when many classes are involved and the data space is complex, the decision boundary also become three- dimensional and takes the form of a hyper plane or sometimes a non- linear profile.

Also, we see that multiple boundaries can be drawn to separate the data points into different classes. However, the one which is most optimal is known as a hyperplane. This is done by creating a hyperplane that has a maximum margin, which means the maximum distance between the data points.

Support Vector : The lines passing through the data points in different classes and parallel to the hyperplane are known as Support Vectors. As these vectors support the hyperplane so the name Support Vector.

Kernel Function

A kernel function takes the data as input, transforms it into the required form and provides the output. Different kernel functions are used by different SVM algorithms. Some of the kernel functions are as follows:

1. Linear kernel
2. Polynomial kernel
3. Gaussian kernel
4. Gaussian radial basis function (rbf)
5. Hyperbolic tangent kernel
6. Sigmoid kernel

Out of these, linear kernel and Gaussian radial basis function (rbf) are some of the most popularly used ones.

How to implement SVM

Here we will use the Wine quality data set to carry out our prediction by implementing SVM. You can find the data set below for your reference.

Wine Quality dataset: https://drive.google.com/file/d/1M6KzRK-Vh9Y5BB2X_5k9b_bdKydHT09-/view?usp=sharing

Starting with importing the libraries and then some data processing as we do in every prediction problem.

import numpy as np
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score,mean_squared_error 

Post the import of necessary libraries, we will read our dataset which is a comma- separated value form(csv file) and will convert that into a panda data frame named df.

df=pd.read_csv(r"D:\wine quality dataset-SVM\winequality-red.csv")

Now, we will see our data frame by printing the first five entries of it. You can the output in the image given below the code.

df.head()

Now, we will use the describe() function in order to analyze the data set in a better way. The output of this function can be seen in the image below.

df.describe()

Now, we will use the info() function to check for the null values in our data set and also the data type of each column. As you can see the output, our data set does not contain any null value.

df.info()

output:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1599 entries, 0 to 1598
Data columns (total 12 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   fixed acidity         1599 non-null   float64
 1   volatile acidity      1599 non-null   float64
 2   citric acid           1599 non-null   float64
 3   residual sugar        1599 non-null   float64
 4   chlorides             1599 non-null   float64
 5   free sulfur dioxide   1599 non-null   float64
 6   total sulfur dioxide  1599 non-null   float64
 7   density               1599 non-null   float64
 8   pH                    1599 non-null   float64
 9   sulphates             1599 non-null   float64
 10  alcohol               1599 non-null   float64
 11  quality               1599 non-null   int64  
dtypes: float64(11), int64(1)
memory usage: 150.0 KB

Next, we call the value_counts() function to see the different types of wine qualities present in the data set.

df["quality"].value_counts()

output:
5    681
6    638
7    199
4     53
8     18
3     10
Name: quality, dtype: int64

Then, we carry out the following code to remove the discrepancy arising due to several types of wine qualities. If the quality is greater than 5, we are assigning a value of 1, else a 0. This will increase the accuracy of our model.

qua=[]

for i in df["quality"]:
 if i>5:
    qua.append(1)
 
 else:
    qua.append(0)
    
df["quality"]=qua   

df.head()

Next, we are defining our independent categorical variable as shown below. Here, we are including all features except our target(or dependent) variable.

X_train, X_test, y_train, y_test= train_test_split(X, y, test_size=0.2, random_state=42)

After this we are investigating the shape i.e. no. of rows and columns of our test and train dataset.

X_train.shape, X_test.shape

output:
((1279, 11), (320, 11))

Next, we are scaling our training and test data in the range of zero to one for increasing the accuracy otherwise machine learning will consider, greater value to be of higher importance.

from sklearn.preprocessing import StandardScaler
sc=StandardScaler()

X_train=sc.fit_transform(X_train)
X_test=sc.fit_transform(X_test)

In this dataset, quality is our target variable as you might have understood. So, now we are training our model on the dataset by using SVM as you can see in the code snippet below.

from sklearn.svm import SVC
cls=SVC(kernel="rbf")
cls.fit(X_train, y_train)

output:
SVC()

After, training our model our next task is to predict the test values as we do in every machine learning problem.

y_pred=cls.predict(X_test)
y_pred

output:
array([0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0,
       1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0,
       1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0,
       1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1,
       1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0,
       1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0,
       0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1,
       1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1,
       1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0,
       1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0,
       0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0,
       1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1,
       1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1,
       0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1,
       1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0], dtype=int64)

After carrying out the prediction, we are now evaluating the accuracy of our model by using the MSE metric.

mean_squared_error(y_test, y_pred)

output:
0.234375

Finally, we are finding out the accuracy score of the prediction done by our model.

accuracy_score(y_test, y_pred)

output:
0.765625

End Notes

I hope this blog was helpful in making you familiar with SVM in machine learning. Many more such blogs will be covered in the future.