InterviewsVector

How to use XGBoost in Python?

Pythonmachine-learning

Quick answer

Use XGBoost in Python by installing xgboost, then either its scikit-learn API (XGBClassifier / XGBRegressor with fit/predict) or the native API with xgb.DMatrix and xgb.train. Key knobs are n_estimators, max_depth, learning_rate, and subsample; use early stopping with an evaluation set to avoid overfitting. It is a gradient-boosted decision tree library known for strong tabular performance.

Short answer: pip install xgboost, then use either the scikit-learn API (XGBClassifier/XGBRegressor with fit/predict) — the one most people want — or the native API (xgb.DMatrix + xgb.train). The knobs that matter are n_estimators, max_depth, learning_rate, and subsample; use early stopping with an eval set so you don't overfit.

XGBoost (eXtreme Gradient Boosting) is a gradient-boosted decision tree library that's a perennial top performer on tabular data. Gradient boosting builds trees sequentially, each one correcting the errors of the ensemble so far. XGBoost adds regularisation, native missing-value handling, and a fast implementation on top.

Install it:

pip install xgboost

The scikit-learn API (start here)

XGBClassifier and XGBRegressor behave like any scikit-learn estimator — fit/predict, and they slot into Pipeline, cross_val_score, and GridSearchCV. This is the API to reach for by default:

from xgboost import XGBClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
 
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
 
model = XGBClassifier(
    n_estimators=300,       # number of boosting rounds (trees)
    max_depth=3,            # tree depth — the main complexity knob
    learning_rate=0.1,      # shrinkage; lower needs more trees
    subsample=0.8,          # row sampling per tree — fights overfitting
    colsample_bytree=0.8,   # column sampling per tree
    eval_metric="logloss",
)
 
model.fit(X_train, y_train)
preds = model.predict(X_test)
print(f"Test accuracy: {accuracy_score(y_test, preds):.3f}")

Early stopping — don't overfit

Give it an eval set and stop once the validation metric stops improving. In modern XGBoost (2.x) early_stopping_rounds is a constructor argument:

model = XGBClassifier(
    n_estimators=2000,          # an upper bound — early stopping picks the real number
    learning_rate=0.05,
    early_stopping_rounds=50,   # stop after 50 rounds with no val improvement
    eval_metric="logloss",
)
 
model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)
print("Best iteration:", model.best_iteration)

Set n_estimators high and let early stopping find the right number of trees, rather than guessing it.

Feature importance

import pandas as pd
importances = pd.Series(model.feature_importances_).sort_values(ascending=False)
print(importances.head())

For trustworthy importances on correlated features, prefer SHAP values or sklearn.inspection.permutation_importance over the built-in gain.

The native API (DMatrix)

The native API wraps data in a DMatrix and gives you the lowest-level control. It's worth knowing for custom objectives and when squeezing performance:

import xgboost as xgb
 
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)
 
params = {"max_depth": 3, "eta": 0.1, "objective": "binary:logistic"}
booster = xgb.train(
    params, dtrain, num_boost_round=1000,
    evals=[(dtest, "eval")], early_stopping_rounds=50, verbose_eval=False,
)
proba = booster.predict(dtest)   # probabilities — threshold at 0.5 for labels

Note eta here is the native alias for learning_rate, and predict returns probabilities, not class labels.

Which API to use

  • scikit-learn API — default choice: pipelines, grid search, and familiar fit/predict.
  • Native API — custom objectives/metrics, fine-grained control, or interop with the wider XGBoost ecosystem.

Related guides

Sources

Key takeaways

  • •Install xgboost, then use either its scikit-learn API (XGBClassifier/XGBRegressor with fit/predict) or the native API (DMatrix + xgb.train).
  • •Key hyperparameters are n_estimators, max_depth, learning_rate, and subsample.
  • •Use early_stopping_rounds with an evaluation set to avoid overfitting.
  • •XGBoost is a gradient-boosted decision tree library known for strong tabular performance.

Frequently asked questions

What are the two ways to use XGBoost in Python?

The scikit-learn API (XGBClassifier/XGBRegressor) or the native API with xgb.DMatrix and xgb.train.

How do I stop XGBoost from overfitting?

Use early_stopping_rounds with an eval set, and tune max_depth, learning_rate, and subsample.

By Mohammad Wasi

Software Engineering Leader & Technical Author · Updated September 9, 2026

See all posts

Related Posts