Classification with Python: Logistic Regression and Decision Trees

Part 3 of the Python for AI/ML track. Last updated: September 2026.

Regression predicts numbers; classification predicts categories: spam or not, churned or retained, which species this flower is. It is the most commercially used branch of ML, and two algorithms cover most real cases — logistic regression and decision trees.

The intuition

A classifier draws boundaries between categories. Logistic regression (despite the name, it is a classifier) draws one straight boundary and outputs probabilities — "87% chance this is versicolor." A decision tree asks a series of yes/no questions — "petal length > 2.5? → …" — carving the data into boxes. Straight boundary vs boxy boundaries; both are useful.

Head-to-head on the Iris dataset

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=1)

logreg = LogisticRegression(max_iter=200).fit(X_train, y_train)
tree = DecisionTreeClassifier(random_state=1).fit(X_train, y_train)

print(f"Logistic regression accuracy: {accuracy_score(y_test, logreg.predict(X_test)):.4f}")  # 0.9778
print(f"Decision tree accuracy:       {accuracy_score(y_test, tree.predict(X_test)):.4f}")  # 0.9556

The confusion matrix: where exactly it fails

Accuracy hides which mistakes happen. The confusion matrix shows them — rows are the truth, columns are the prediction:

from sklearn.metrics import confusion_matrix

print(confusion_matrix(y_test, tree.predict(X_test)))
# [[14  0  0]
#  [ 0 17  1]   ← one versicolor called virginica
#  [ 0  1 12]]  ← one virginica called versicolor

Setosa is never confused; the two similar species mix up occasionally. That pattern — not the single accuracy number — is what you act on.

Precision, recall, and when accuracy lies

From the confusion matrix come the two metrics that matter in production:

• Precision — of everything the model called positive, how many truly were? ("When it says fraud, is it right?") Low precision = false alarms.
• Recall — of all actual positives, how many did it catch? ("How much fraud slipped through?") Low recall = missed cases.
• F1 — the harmonic mean of the two, a single balanced number.

from sklearn.metrics import classification_report

print(classification_report(y_test, tree.predict(X_test),
                            target_names=iris.target_names, digits=2))
#               precision    recall  f1-score   support
#       setosa       1.00      1.00      1.00        14
#   versicolor       0.94      0.94      0.94        18
#    virginica       0.92      0.92      0.92        13
#     accuracy                           0.96        45

Warning: on imbalanced data (99% legitimate, 1% fraud), a model that always says "legitimate" scores 99% accuracy and is useless. Always check precision/recall, not just accuracy.

Which classifier to reach for

• Logistic regression — start here. Fast, calibrated probabilities, works well when classes are roughly separable. Great baseline.
• Decision tree — handles non-linear boundaries, needs no scaling, and you can literally print its rules for stakeholders. Prone to overfitting if left unconstrained (Part 4 fixes that).
• Both expose predict_proba — use probabilities with a threshold you choose, not just hard labels.

Key takeaways

• Classification predicts categories by drawing boundaries between them.
• Logistic regression = linear boundary + probabilities; decision tree = yes/no rules, interpretable but overfits.
• The confusion matrix shows exactly which classes get mixed up.
• Precision = trust in positive predictions; recall = share of positives caught; accuracy lies on imbalanced data.
• Use predict_proba and tune the decision threshold for your cost of errors.

Next in this series: Model Evaluation Done Right: Train-Test Split and Cross-Validation.