🎓 Advanced Machine Learning Series – Part 2: Boosting Techniques

🎓 Advanced Machine Learning Series – Part 2: Boosting Techniques

In this second part of the Advanced Machine Learning series on Darchumstech, we explore Boosting — a powerful family of ensemble techniques that convert weak learners into strong ones through sequential training.

Boosting is an ensemble method that builds models sequentially. Each model tries to correct the errors made by the previous ones.

It is especially useful for improving accuracy and reducing bias.

• AdaBoost: Adjusts weights on training examples based on prediction errors.
• Gradient Boosting: Uses gradient descent to minimize a loss function.
• XGBoost: A regularized and optimized version of gradient boosting.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

model = AdaBoostClassifier(n_estimators=50)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print("Accuracy:", accuracy_score(y_test, y_pred))

This simple AdaBoost model improves performance by focusing on misclassified points.

Gradient Boosting uses the gradients of a loss function to minimize prediction error in a sequence of models.

Popular libraries include sklearn, xgboost, and lightgbm.

import xgboost as xgb
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

model = xgb.XGBClassifier(use_label_encoder=False, eval_metric='mlogloss')
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print("Accuracy:", accuracy_score(y_test, y_pred))

XGBoost is fast, efficient, and often wins Kaggle competitions due to its regularization features.