Your fine-tuned model is only as good as the data you train it on. Dataset preparation is where most fine-tuning projects succeed or fail.

Dataset Formats

Most fine-tuning frameworks expect data in a conversational format:

{
  "messages": [
    {"role": "system", "content": "You are a medical coding assistant."},
    {"role": "user", "content": "Patient presents with acute bronchitis..."},
    {"role": "assistant", "content": "ICD-10 Code: J20.9 — Acute bronchitis, unspecified..."}
  ]
}

How Much Data Do You Need?

Surprisingly little for many tasks:

• Style/format transfer: 50-200 high-quality examples
• Domain adaptation: 500-2,000 examples
• New capability: 2,000-10,000 examples
• Complex reasoning: 10,000+ examples with chain-of-thought

Quality always trumps quantity. 100 expertly crafted examples outperform 10,000 noisy ones.

Data Collection Strategies

1. Manual curation: Have domain experts write ideal input-output pairs. Highest quality but most expensive.
2. Synthetic generation: Use a stronger model (GPT-4, Claude) to generate training data. Fast but requires careful quality filtering.
3. Distillation: Run a large model on your production queries, filter the best responses, and use those to train a smaller model.
4. Existing logs: If you have production chat logs with human feedback, these are gold.

Data Quality Checklist

• ✅ Consistent formatting across all examples
• ✅ No contradictory examples (same input, different correct outputs)
• ✅ Representative of real-world distribution
• ✅ Balanced across categories/labels
• ✅ Free of PII and sensitive data
• ✅ Validated by domain experts

Train/Validation Split

Always hold out 10-20% of your data for validation. This lets you detect overfitting — when training loss drops but validation loss increases, you've trained too long.

Common Pitfalls

• Contamination: Training data that leaks into your evaluation set gives falsely optimistic results
• Distribution shift: Training on formal text but deploying on casual queries
• Label noise: Inconsistent or incorrect labels in your training data poison the model