Making local SLMs viable in agent loops: A hybrid approach with Jev

Originally published on LinkedIn.

While building an AI browser project recently, one of the biggest practical bottlenecks I ran into was per-step latency and reliability in multi-turn workflows.

Running a local model (like MiniCPM or Qwen) to inspect the full page and output a structured tool call on every single turn typically takes 4–5 seconds per step. On top of that, smaller models often struggle with the cognitive load of parsing dozens of element IDs at once. For tasks that mostly involve simple clicks and navigation, this delay and instability quickly add up.

With discrete decision models like Jev gaining traction recently, I tested an alternative runtime pattern: decoupling discrete choice from text generation.

How I set it up

Instead of treating every turn as an open-ended generative task, I split the execution loop:

1. Discrete routing first: On each turn, the runtime formats the current goal, recent action history, and candidate target elements (@ref), then asks Jev two simple questions: which tool to use, and which target reference to act on.
2. The @none option: I explicitly provide @none in the candidate list. Without an abstention option, classifiers tend to hallucinate clicks when no element matches.
3. Direct execution: If the tool is a simple click or completion, the runtime executes it directly. The generative model is not called at all.
4. Targeted generation: If the action is a text input, Jev identifies the target field, and the local SLM only has to generate the text value for that specific field, rather than parsing the entire page structure.
5. Fallback: If Jev selects @none, loops on the same action, or encounters an issue, the runtime falls back to the local model.
6. Action Ledger: A single history list tracks executed actions across steps so the context stays consistent between both engines.

What I observed

• Noticeable speedup on UI actions: For tasks dominated by clicks and navigation, latency dropped from several seconds per step to around 500ms.
• Smaller local models become viable: A 2B–7B local model often struggles if it has to parse dozens of element IDs, verify state, and generate calls all at once. Handing off target selection to a fast router makes local models much more reliable for the parts they actually need to do.
• Data stays local: The routine navigation logic can be handled quickly, while actual text generation runs entirely on-premise without routing sensitive data through external APIs.

I am also looking into Laya, an open-source model exploring similar discrete decision ideas, and plan to test it as an open-source alternative.

Curious if others working on agent loops are exploring similar fast-router patterns. Would love to hear how you handle latency in multi-step automation.