LLMs can teach themselves to better predict the future

The talk explains how self‑play and DPO fine‑tuning on ranked Polymarket predictions improves 14‑billion‑parameter models to GPT‑4o level by 7‑10% accuracy using real‑world outcomes.

Overview

In our first paper from Lightning Rod Labs (https://lightningrod.ai), we explore if AI can improve its forecasts via self-play and real-world outcomes:

12,100 questions and outcomes from Polymarket (politics, sports, crypto, science, etc)
Base model generates multiple distinct reasoning traces and predictions per question
Rank predictions by how close they were to the actual outcome
Fine-tune with DPO on the ranked traces & predictions

Result: +7-10% accuracy over control, bringing two small (14B) models on par with GPT-4o (over 10x larger).

Arxiv Link: https://arxiv.org/abs/2502.05253

Links

Tech stack