AI Model Wiki

    Skip to main content
    24 min
    Lesson 1 of 7

    Data Preparation for Fine-Tuning

    From the course AI Fine-Tuning Pipelines

    Prepare high-quality datasets for fine-tuning language models.

    Data Collection Strategies

    Dataset Requirements

    Quality > Quantity

    Minimum viable dataset:

    • Simple tasks: 50-100 examples
    • Complex tasks: 500-1000 examples
    • Domain expertise: 1000-5000 examples
    • Production quality: 5000+ examples

    Data Formats

    Chat format (recommended):

    {"messages": [
      {"role": "system", "content": "You are a helpful legal assistant."},
      {"role": "user", "content": "What is a tort?"},
      {"role": "assistant", "content": "A tort is a civil wrong..."}
    ]}
    

    Instruction format:

    {"instruction": "Summarize this contract", "input": "...", "output": "..."}
    

    Data Quality Pipeline

    class DataQualityPipeline:
        def process(self, raw_data):
            # Step 1: Deduplication
            data = self.deduplicate(raw_data, threshold=0.85)
            
            # Step 2: Quality filtering
            data = self.filter_quality(data, min_score=0.7)
            
            # Step 3: Format validation
            data = self.validate_format(data)
            
            # Step 4: PII removal
            data = self.remove_pii(data)
            
            # Step 5: Length filtering
            data = self.filter_length(data, min_tokens=10, max_tokens=4096)
            
            return data
    

    Synthetic Data Generation

    def generate_training_data(seed_examples, target_count):
        synthetic_data = []
        
        for seed in seed_examples:
            variations = llm.generate(
                f"Create 5 variations of this example: {seed}",
                temperature=0.8
            )
            synthetic_data.extend(variations)
        
        filtered = quality_filter(synthetic_data, threshold=0.8)
        return filtered[:target_count]
    

    Key Takeaways

    • •Quality trumps quantity: 100 high-quality examples often outperform 10,000 low-quality ones
    • •Chat format with system/user/assistant messages is the modern standard
    • •Data quality pipeline: deduplicate, filter, validate format, remove PII, check length
    • •LLM-generated synthetic data can expand datasets when filtered for quality
    0 of 7 lessons marked complete

    Progress is kept in this browser only.