In this blog, we will focus on various types of algorithms Boosting in machine learning you can also explore previous machine learning blogs https://ainewgeneration.com/category/machine-learning/. As boosting can solve complex problems with good accuracy It is a technique of ensemble learning which combines weak models to produces a strong model.
Table of Contents
- Why is Boosting used?
- What is Ensemble Learning?
- How the Boosting algorithm works?
- Types of Boosting in Machine Learning.
Why is Boosting used?
Let’s suppose you have a dataset that contains images of cats and dogs Now you have to build the model to classify between cats and dogs which is a binary classifier model. So based on features of cats and dogs model will be able to learn from images during the training phase. Some features below on basis of that model will be able to classify between cats or dogs.
- The Image has bigger limbs – “DOG”
- The Image has pointy ears – “CAT”
- The Image has sharpened claws – “CAT”
- The Image has wider mouth structure – “DOG”
- The Image has smaller eye curved shape – “CAT”
From the above features, the single model is predicting output based on the feature so they are called a weak learner to generate a strong model the combination of all the weak models and based on the majority of voting in above i.e. cats predicted as 3 times and dogs as 2 times which is final output will be an image of a cat by the strong model is all combination of weak models that will give you more accurate or precious predictions with higher accuracy.
What is Ensemble Learning ?
Ensemble learning is a technique in which a combination of the weak learners produces a strong learner To increase the accuracy of the model.
In the below image weak learner 1, weak learner 2, weak learner 3 ……… weak learner n are weak machine learning algorithms and respective there outputs as output 1, output 2, output 3, …. output n. to produce a strong model is a combination of all weak machine learning algorithms and based on the majority of voting output by the weak models that will be respective output by the strong model.
Ensemble Learning is of 2 types :
- Bagging (Parallel) : Bagging is a technique in which a combination of the weak learners produces a strong learner In order to increase the accuracy of the model. It works in parallel way Every weak model is connected in parallel. The performace of weak learner increase by parallel changing bootstrap datasets like random forest https://ainewgeneration.com/random-forest-in-machine-learning/.
- Boosting (Sequential) : Boosting is a technique in which a combination of the weak learners produces a strong learner In order to increase the accuracy of the model. It works in Sequentially way Every weak model is connected in sequential as weak learner are sequentially produces during the training phase the performance of model is increases by assiging higher weightage to previous incorrect classify.
How the Boosting algorithm works?
In boosting weak learners are combined to form strong learners in order to get high accuracy by the model.
Let’s understand with an example below the dataset contains the square and circle. Now make the model with the boosting algorithm for predictions.
- Step 1: the base learner takes all the distributions and assign same weight or attention to each remark.
- Step 2 : If there may be any prediction errors due to first base studying algorithm, then we pay higher attention to observations having prediction blunders. Then, we observe the subsequent base learning algorithm.
- Step 3: Iterate Step 2 till the restriction of base gaining knowledge of set of rules is reached or better accuracy is executed.
Types of Boosting in Machine Learning
- AdaBoost (Adaptive Boosting)
- Gradient Tree Boosting
AdaBoost (Adaptive Boosting)
AdaBoost or Adaptive Boosting is one of the ensemble boosting classifiers proposed by means of Yoav Freund and Robert Schapire in 1996.
It combines multiple weak classifiers to increase the accuracy of classifiers.
AdaBoost is an iterative ensemble method. AdaBoost classifier builds a strong classifier via combining multiple poorly performing classifiers so that you will get excessive accuracy strong classifier.
The basic idea of Adaboost is to set the weights of the classifiers and to train a sample of data in each iteration to ensure accurate predictions for unusual views.
Any machine learning algorithm can be used as a base learner but by default decision tree is used.
- Assign every observation Xi an initial weights value , Wi = 1/n where n is total no. of observation.
- Train a weak moel by default will be “Decision Tree”
- For each observation preiction by the model:
- if preiction is incorrectly “Wi” is increased.
- if prediction is correctly “Wi” is decreased.
- Train a new weak model where observation with greater weights are given more priority.
- Repeat step 3 an 4 untill observation perfectily predicted or a preset number of tree are predicted.
I hope boosting algorithms in machine learning was clearly understandable from basic to advanced. In the next blog of the Machine Learning Series, we will do some more Boosting algorithms with their implementations.