Machine Learning with scikit-learn

When Traditional ML Still Wins

• Tabular data with structured features
• Small datasets (hundreds to thousands of samples)
• Explainability requirements
• Low-latency inference without GPU
• Classification, regression, clustering tasks

Text Classification Pipeline

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

# Build pipeline
pipe = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=5000, ngram_range=(1, 2))),
    ("clf", LogisticRegression(max_iter=1000)),
])

# Train
pipe.fit(train_texts, train_labels)

# Evaluate
predictions = pipe.predict(test_texts)
print(classification_report(test_labels, predictions))

Combining ML with LLMs

# Use embeddings as features for sklearn
from openai import OpenAI

client = OpenAI()

def get_embeddings(texts):
    response = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [e.embedding for e in response.data]

embeddings = get_embeddings(texts)
clf = LogisticRegression()
clf.fit(embeddings, labels)

Key Algorithms

Common Pitfall: Data Leakage in Preprocessing Pipelines

Data leakage occurs when information from the test set influences the training process. A classic example is fitting a TfidfVectorizer on the entire dataset (including test data) before splitting. Always split first, then fit transformers only on training data:

# Wrong: leaks test data into vocabulary
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(all_texts)  # includes test data!
X_train, X_test = train_test_split(X, test_size=0.2)

# Correct: fit only on train, transform test separately
X_train_raw, X_test_raw, y_train, y_test = train_test_split(texts, labels, test_size=0.2)
vectorizer = TfidfVectorizer()
X_train = vectorizer.fit_transform(X_train_raw)
X_test = vectorizer.transform(X_test_raw)  # transform only — no fit!

Using scikit-learn's Pipeline object avoids this mistake automatically by handling fit/transform sequencing correctly.

Best Practice: Use cross_val_score for Reliable Accuracy Estimates

A single train/test split can give misleading results depending on which samples land in each set. Use k-fold cross-validation for a more robust accuracy estimate:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(pipe, texts, labels, cv=5, scoring="f1_macro")
print(f"Mean F1: {scores.mean():.3f} (+/- {scores.std():.3f})")

Five-fold cross-validation uses 80% of data for training and 20% for testing across five different splits, giving a much better estimate of real-world performance.

Named Product Reference: XGBoost for Tabular AI Data

For classification tasks on tabular data — the kind common in business AI applications — XGBoost consistently outperforms most sklearn algorithms. Install it with pip install xgboost and use it inside a sklearn pipeline exactly like LogisticRegression. XGBoost is used by Airbnb, Uber, and Kaggle competition winners for structured data tasks and handles missing values natively without imputation.