Product teams, problem statements, and AI workflows that work (Experimentation Elite - Winter 24)

12/11/2024 · By Iqbal Ali

My talk at Experimentation Elite 2024 was about using AI to solve a specific process problem at a major company that I was consulting at. Seeing there was a disconnect between the experiment ideas and the user problems they were trying to solve.

Here’s an AI summary of the talk (don’t worry, I used a small energy efficient model):

Talk Summary: Fixing the “Empty Box” Problem

By Iqbal

Core Problem

Teams blame stats/tech for failed experiments, but the real issue is upstream failures:

  • Poor problem definition
  • Solution bias (jumping to answers)
  • “Hemorrhaging relevance” (losing focus as ideas move from problem → execution)

Key Insights

  1. Root Cause Chain

    • Experiments fail because teams skip critical steps: problem framing → idea generation → design → execution.
    • Solution bias is the core issue: Humans default to regurgitating old ideas (C-suite, LinkedIn swipes) or jumping to solutions too fast.
    • Einstein’s rule: Spend 55 minutes on the problem, not solutions.

  2. The Challenge

    • PMs resist “more process,” even when they know frameworks (e.g., TOSCA).
    • Time pressure is the real barrier—needed a scalable fix.

Solution: AI + Human Collaboration

  • AI’s role: Accelerate divergent thinking (where humans excel but lack speed), and also ensuring relevance between stages of the process is maintained.
  • Workshop structure (45 min):
    1. Root Cause ID
      • Manual sticky notes → AI prompts to expand questions/insights (e.g., TOSCA framework tables).
      • Result: Deeper exploration in minutes.
    2. Problem Statements
      • AI generates drafts; PMs validate/iterate.
      • Result: Less stress, higher quality.
    3. Ideation
      • Manual brainstorm → AI prompts (e.g., “crazy ideas”) → diverse, high-quality ideas.
      • Result: More ideas, reduced bias, level playing field for all PMs.

Why It Worked

  • AI as a “thinking partner”: Not a replacement, but a tool to unblock creativity and expose blind spots.
  • Kolb’s learning cycle: Hands-on, reflective workshops made it practical.
  • PMs in control: Validated AI outputs like a “boss”—no rigid processes.

Visual Update

  • Old metaphor: “Empty box” (passive).
  • New metaphor: PMs build the box (toolbox = active creation).