AI Model Wiki

    Skip to main content
    45 min
    Lesson 4 of 5

    Human-in-the-Loop & Production

    From the course AutoGen: Multi-Agent Conversations

    Human-AI Collaboration

    Human Input Modes

    # Always ask human before acting
    user_proxy = UserProxyAgent(human_input_mode="ALWAYS")
    
    # Ask only when termination condition met
    user_proxy = UserProxyAgent(human_input_mode="TERMINATE")
    
    # Fully automated
    user_proxy = UserProxyAgent(human_input_mode="NEVER")
    

    Termination Conditions

    Control when conversations end:

    user_proxy = UserProxyAgent(
        max_consecutive_auto_reply=5,
        is_termination_msg=lambda msg: "TASK COMPLETE" in msg.get("content", ""),
    )
    

    Production Architecture

    State Management:

    • Save conversation state for resuming interrupted workflows
    • Log all agent messages for auditing and debugging
    • Track token usage and costs per conversation

    Error Handling:

    • Catch and retry failed API calls
    • Handle code execution failures gracefully
    • Set global timeouts for entire workflows
    • Implement fallback agents for critical tasks

    Scaling Considerations

    • Use async agents for concurrent conversations
    • Implement request queuing for rate-limited APIs
    • Cache agent responses for repeated queries
    • Monitor conversation quality with automated evaluations

    AutoGen vs. Alternatives

    FeatureAutoGenCrewAILangGraph
    FocusConversationRole-basedState machines
    Code executionBuilt-inVia toolsVia tools
    Human-in-loopNativeLimitedNative
    ComplexityMediumLowHigh

    Common Pitfall: Not Logging Agent Messages for Debugging

    Multi-agent conversations are hard to debug without full logging. AutoGen conversations can be inspected via groupchat.messages after they complete, but this is lost if the process crashes. Log every message to a database or file during the run:

    def log_message(sender, recipient, message):
        db.insert("agent_logs", {
            "sender": sender.name,
            "recipient": recipient.name if recipient else "group",
            "content": message.get("content", ""),
            "timestamp": datetime.now().isoformat(),
        })
    
    # Register the logging hook
    for agent in [assistant, user_proxy]:
        agent.register_hook("process_message_before_send", log_message)
    

    Best Practice: Budget Tokens per Conversation

    In production, set a token budget per conversation to prevent runaway API costs. AutoGen supports per-model token tracking through the llm_config cache mechanism. Monitor the total_tokens_used from the LLM config after each run and alert when it exceeds a threshold:

    result = user_proxy.initiate_chat(assistant, message="Build a data pipeline")
    
    # Check token usage after conversation ends
    usage = user_proxy.chat_messages[assistant][-1].get("usage", {})
    print(f"Total tokens used: {usage.get('total_tokens', 'unknown')}")
    

    Named Product Reference: AutoGen v0.4 AgentChat

    Microsoft's AutoGen v0.4 introduced a new agentchat package with a redesigned async-first API. If you are starting a new project, use pip install autogen-agentchat (v0.4) rather than the older pyautogen package. The new API supports async agents, better streaming, and a more modular architecture — the legacy package is in maintenance mode.

    Key Takeaways

    • •Three human input modes: ALWAYS, TERMINATE, and NEVER
    • •Set termination conditions with message patterns and reply limits
    • •Log all agent interactions for auditing and debugging
    • •AutoGen excels at conversational multi-agent patterns with human oversight
    0 of 5 lessons marked complete

    Progress is kept in this browser only.