Data Foundations for AI

NumPy: Numerical Computing

import numpy as np

# Embeddings are NumPy arrays
embedding = np.array([0.1, 0.5, -0.3, 0.8])

# Cosine similarity
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

sim = cosine_similarity(embedding1, embedding2)

# Matrix operations for attention
attention_scores = np.matmul(queries, keys.T) / np.sqrt(d_k)
attention_weights = softmax(attention_scores)

Pandas: Data Preparation

import pandas as pd

# Load and explore training data
df = pd.read_csv("training_data.csv")
print(df.describe())
print(df.isnull().sum())

# Clean and prepare
df = df.dropna(subset=["text", "label"])
df["text"] = df["text"].str.lower().str.strip()
df["text_length"] = df["text"].apply(len)

# Split for training
from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.2, random_state=42)

Common AI Data Tasks

• Tokenization stats: Analyze token distributions with Pandas
• Embedding analysis: Compute similarities with NumPy
• Dataset balancing: Sample or augment underrepresented classes
• Feature engineering: Create numeric features from text
• Evaluation: Calculate metrics across test sets

Common Pitfall: Using Lists Instead of NumPy Arrays for Batch Operations

A common beginner mistake is accumulating embedding vectors in a Python list and then calling similarity functions in a loop. This is orders of magnitude slower than batching with NumPy:

# Slow: Python loop
similarities = [cosine_similarity(query, doc) for doc in embeddings_list]

# Fast: NumPy vectorized — computes all similarities at once
import numpy as np
query_arr = np.array(query)
docs_arr = np.array(embeddings_list)  # shape: (n_docs, embedding_dim)

# Vectorized cosine similarity
norms = np.linalg.norm(docs_arr, axis=1, keepdims=True)
docs_normalized = docs_arr / norms
query_normalized = query_arr / np.linalg.norm(query_arr)
similarities = np.dot(docs_normalized, query_normalized)

For 10,000 documents, the vectorized version is typically 100–500x faster.

Best Practice: Use Pandas pipe for Readable Data Pipelines

Chain transformations on DataFrames using pipe to make preprocessing readable:

def remove_nulls(df, col):
    return df.dropna(subset=[col])

def lowercase_text(df, col):
    df[col] = df[col].str.lower().str.strip()
    return df

def add_word_count(df, col):
    df["word_count"] = df[col].str.split().str.len()
    return df

clean_df = (
    df
    .pipe(remove_nulls, "text")
    .pipe(lowercase_text, "text")
    .pipe(add_word_count, "text")
)

This pattern makes each step reusable and testable independently.

Named Product Reference: Polars as a Pandas Alternative

For datasets over 10 million rows, consider Polars instead of Pandas. Polars is a Rust-backed DataFrame library with a lazy evaluation API that parallelizes operations across CPU cores automatically. Its syntax is similar to Pandas but it typically runs 5–10x faster on large datasets. The polars package is a drop-in complement — use Pandas for smaller exploratory work and Polars for large-scale AI data preprocessing pipelines.