Welcome to session 48! This week Jyoti Pravat Senapati takes us through a July 2026 paper on skills and models that improve together, and shows what happened when he rebuilt the idea to improve skill files instead of model weights.
Jyoti is a Product Manager at Enphase Energy in Bengaluru and the creator of skillkeeper. He holds an MTech from IIT Madras and a PGP from the Indian School of Business.
Session Overview
Most agent skills are written once and never revisited. The models keep getting better; the skills sit frozen. Skill Self-Play describes a loop that fixes that — a proposer that invents tasks, a solver that attempts them, a validator that scores the result, and a skill controller that reads what happened and updates the library.
The paper points that loop at the model. Jyoti points it at the skill file instead: the model stays fixed, the skill evolves, and every promotion is Git-verified so you can roll it back.
Key Takeaways
- A test beats another prompt tweak: A prompt fix is fast, free, and invisible — it gets buried as the context window fills. A deterministic test (grep, link check, or LLM-as-judge) survives.
- The loop needs four parts: Proposer, solver, validator, skill controller. Each can run on a different model tier, and only the proposer and controller need to be frontier class.
- Hold back the answer key: If the solver ever sees the held-out tasks, it optimises for passing rather than being right, and results degrade over time.
- One judge is not enough: A council of validators — consumer trust, product thinking, reader clarity — catches what a single rubric misses, and each returns JSON so promotion can be automated.
- Promotion should be reversible: Skills are version-controlled and promoted only when they score better, so a change made for one industry can be rolled back before you write for the next.
- This is loop engineering, and loops rot: Left running, a self-improving loop starts grading its own output and drifts. Verifiable skill changes are what keep it honest.
Topics Covered
Why Skills Are the Unit Worth Improving
Most people improve skills by hand: run a session, spot a gap, ask Claude or Codex to patch the SKILL.md at the end. Jyoti opens with why that's ad hoc, and what a deterministic version would look like.
The Exam Analogy
A question setter writes a paper, a student attempts it without seeing it in advance, and a validator holds both the paper and the answers. Evolution happens on the question, the skill, or the model — and the student must never see the paper.
Proposer, Solver, Validator, Skill Controller
The proposer is a frontier model carrying a memory of who you are, so the tasks it invents match your actual work rather than a generic user's. The solver is whatever tool runs your skill. The validator can be a swarm. The controller writes the gap back into the skill file.
What the Paper Reports
Tool calling for Mistral moved from 2.8 to 42.9, and logic puzzles from 1 to 12, across five open-source models in the 35B–200B range. Jyoti's read: thinking models already do this invisibly, and the value of doing it explicitly is being able to drop to a much cheaper model without losing quality.
Two Demos
First, improving a stock Anthropic PRD skill: the profile snapshot, three generated tasks, three validator rubrics, and a scored A/B where the improvement was a modest 0.208 — small enough that a human should make the call. Second, a deliberately weak presentation skill rebuilt into something worth showing.
Q&A Highlights
- Won't the context get corrupted after five iterations of prompt patches?
- When is this worth the setup cost, and when is it too much for a one-off task?
- Does every model need its own skill file?
- How much hallucination can a skill actually fix, versus a tool call, versus the model?
- Where do you put an explicit instruction — the skill file, or the validation layer?
- Why do skill files only ever grow, and never get cut back to the essentials?
- Where do guardrails and PII checks fit in the loop?
Here's the entire recording of the session.