Conversion Hotel is an annual experimentation and conversion optimization conference held in Texel, Netherlands. It’s constantly referred to as the premier event for experimentation, and one that the experts visit frequently. Being invited to speak at CH25 was an absolute honour, a probable career highlight, and absolutely terrifying.
The topic was the Jungle of AI in Experimentation. And my experience with the topic of AI had gone on a journey, from one of excitement to one of realisation of how dangerously and unsustainably the Big Tech company were appraoching the development of the technology.
I wanted to not only tell the story of my journey, but I wanted to raise the sustainability issue, along with practical solutions of what we can do about it. So, the talk had to be valuable. The content of my talk was at times highly technical but I had to try and explain it to techies and non-techies alike.
You can read more about my talk here, along with a short excerpt.
Overall, I thought it went well. Some feedback (including the negative ones):
“A bit too deep and technical for basic users”
“A fun presentation with some wonderful artwork. Iqbal’s take on how we could (And should) optimise our AI usage to minimise its impact on the planet was refreshing.”
“A very welcome breath of fresh air. I’m so over people (vendors, experts) telling me that AI is this magical tool that can solve all my problems.
“With Iqbal, you can see that he’s not all talk, he’s doing that stuff also. I love how he raised awareness for the ecological impact of AI. Probably the most important keynote of the weekend for me.”
“How brilliant were these slides! The drawings! Color contrast was a bit problematic for the circles, maybe not relying on light gray (for the energy comparison). Iqbal is such a sympatric guy. Loved how he also empowered everyone to take action.”
“I didn’t really get the point”
Overall, I was the third highest rated speaker at the event! Which was a huge relief as not only was this a technical topic, but a sometimes unpopular AI-realist take that I realise not everyone was ready to hear.
Here’s an AI-written summary of my talk (nervous laugh—pretty sure they used small models for this 😅):
Challenging the Trajectory of Large Language Models (LLMs)
A Presentation by Iqbal Ali
Large AI systems are consuming unsustainable energy and pushing technology toward a “cliff edge.” This talk advocates for a mindset shift—prioritizing small, energy-efficient models in carefully architected workflows over massive, resource-hungry LLMs.
1. The Unsustainable Cost of Large AI
Energy Consumption & Environmental Impact
- Training flagship LLMs (e.g., ChatGPT) requires energy equivalent to powering 43,680 homes for a day.
- Annual energy demand grows 2.4x—an unsustainable trajectory.
- Daily operational costs (e.g., OpenAI serving 2.5B prompts/day) equal powering 1.6M homes.
Diminishing Returns & Benchmark Collapse
- Performance gains are questionable: Latest models suffer from “accuracy collapse” due to benchmarks leaking into training data.
- Pressure to adopt: Large tech companies may be safe, but others risk being dragged into “uncertain terrain” to stay relevant.
2. Downsizing Expectations with “Simple Intelligences”
The Ant Colony Analogy
- Complexity emerges from simplicity: Like ants working in unison, small models can achieve high performance when coordinated effectively.
- Shift from “mega models” to “simple intelligences”:
- Small models = distilled versions of LLMs, trained on reasoning/rationales (not raw data).
- Energy-efficient: Training and inference costs are “really small”.
- Task decomposition: Breaking problems into smaller steps eliminates accuracy gaps between small/large models.
3. Architecting Workflows with Validation
Three Rules of the Jungle
- Think Small
- Use small models for simple, discrete tasks (e.g., sentiment analysis, data extraction).
- Think in Workflows
- Design interactions like an architect, not a one-off prompt.
- Think in Validation
- Introduce breakpoints to check AI outputs at each step.
Case Study: Small Model vs. GPT-5
| Approach | Model | Task | Result | Energy Use | Privacy |
|---|---|---|---|---|---|
| Single Prompt | GPT-5 | Extract insight-sentiment pairs | ”Acceptable” | High | Cloud |
| Single Prompt | Tiny 500MB | Same task | Poor performance | Low | Local |
| Workflow | Tiny 500MB | Step-by-step (validate at each stage) | Outperformed GPT-5 | Minimal | Local |
Key Insight:
- Validation breakpoints transform small models into highly accurate, controllable tools.
- Faster, cheaper, and private—no reliance on cloud-based LLMs.
4. Takeaways for Your Work
Actionable Principles
✅ Think Small
- Use small models for repetitive, simple tasks to:
- Slash energy costs
- Improve privacy (local deployment)
- Reduce financial burden
✅ Be an Architect
- Break tasks into workflows (not one “mega prompt”).
- Example: Instead of one complex query, use:
- Extract negatives
- Extract positives
- Structure results
✅ Implement Validation (Breakpoints)
- Pause and check AI outputs at critical stages.
- Regain control over the “black box.”
✅ Automate Workflows
- Turn validated workflows into repeatable, scalable processes.
🔬 Experiment First
- Test small models against LLMs on your repetitive tasks.
- Measure speed, cost, and accuracy—you may be surprised!
5. Final Thought
“We’re in the jungle, and the best way to understand what we can do in this landscape is to do what we do best: experiment.”