The goal

Simple: Let's build a super-charged next-level gaming companion that actually feels good.

There are already multiple frameworks that let LLMs control NPC dialogue. They are fantastic for role-playing and staying in character, but they have two recurring problems: weak world agency and latency. Good at talking but far less reliable at performing actions and terrible at complex instruction sets. You may have also noticed how popular AI NPC demos often cut between the player speaking and the AI replying, trying to mask latency. Can we do better?

I wanted a companion that:

1. Useful and instant. It should fight, fetch, loot, inspect, carry and give items etc etc, follow complex multi-step instructions without feeling buggy or experimental. This matters especially in VR, where navigating menus is cumbersome and immersion-breaking. it needs to be FAST fast, not just fast
2. Alive and present. It should have a fun, endearing personality, not canned robotic pre-written responses. Remember shared experiences and change over time. The microphone stays active while a session is running: you do not summon Varkos through a dialogue menu, you talk to him. When immersion kicks in, it should feel like you are not playing alone.
3. Local and private wherever practical. The elephant in the room is that cloud LLM calls can get quite pricy (especially with multi-thousand-token LLM calls) and the added latency can be an experience killer. And why turn a private single-player game into a metered and surveilled experience? Let's try to give as much control to the user as possible (bonus it's a fun technical challenge).

Complex commands

Varkos can handle commands that extend beyond one immediate action. Plans can wait for events, preserve targets between steps, monitor progress and repair or stop when world state changes. Nothing is pre-scripted.

Let's see some examples

Varkos receives a conditional instruction involving the next arrow. He registers the future trigger instead of acting immediately, waits for the correlated projectile impact and then continues the plan.

Long-form multi-step command

“I want you to wait here and I’m gonna go over there. Once you see the signal, the signal is going to be an arrow I fire up in the sky, I want you to pick up this potion and come and bring it to me. Okay?”

Item search

Varkos can search the grounded world state for a requested item, identify where it is and respond using what is actually present in the game.

“Do you see the ceremonial sword anywhere?”

Varkos picks up a different sword and brings it to us. We tell him that’s not the one, then he offers to be on the lookout.

Hide-and-seek

Hide-and-seek is not a single API call. It becomes a persistent goal with movement, waiting, monitoring and completion conditions.

“Let’s play hide-and-seek again. You wait here and I’m gonna go hide, then count to ten and come and try to find me.”

Loot this chest and give me the potion

This combines a grounded container, a filtered loot step and an inventory transfer. Each physical result advances the next part of the plan.

Pick up all the items

“Pick up all the items and give them to me” becomes a bounded collection plan over real references. Varkos gathers them, returns and transfers them without pretending that one magical action means “all.”

Combat

Varkos receives grounded events from the game, can warn the player through a fast reflex path and uses native body control to act. Instruction plans can strategize (e.g. attack this, then retreat, etc.), and his emotional state can affect how and if he chooses to fight.

Personality evolution

Varkos is fully customizable. He does not have to be a demon dog, and the runtime does not have to control only a single character. What systems are applied and what they do, is up to open configuration.

One part of my current build still fully depends on big model/cloud LLM calls: slow personality evolution. This work happens away from the real-time action path. As the player and Varkos travel together, important interactions become evidence for gradual changes to his personality.

My demo Varkos begins as a demon reincarnated as a dog. He considers his canine instincts humiliating, his dog body a prison, and is mistrustful, proud and sarcastic. Through shared experiences he can become more and more domesticated, grow attached to the player and starts enjoying being a dog. Eventually he starts bringing over toys because he wants to play, running off to chase things and seeking affirmation from the player.

Only the starting character traits are authored. The system changes both his explicit traits and his emotional homeostasis. How easily he becomes irritated, frightened, affectionate or playful, etc etc. He can overwrite parts of his vocabulary and code. Changes are versioned and reversible.

I could make it more bounded, but I think there's something fun about some open world clankiness, so how he evolves is up in the air.

Dog in and out of the game - Void mode

My plan is to make this system a gaming companion that can follow you across multiple different games, not just Skyrim (Skyrim felt like a good starting point due to its massive modding community, VR support and big open world).

For this reason he exists outside the game too. When the game closes, he enters “void mode” and cannot see or feel anything. How he responds to that depends on his personality evolution.

This state also works as an in-between for different games. One moment Varkos could be fighting a dragon, then the world goes dark, then he appears beside you in Microsoft Flight Simulator. Maybe he would be shocked, need time to understand the new world and slowly learn what its machines and rules mean, or maybe he knows about it already and overjoyed tries to chase the sun.

Let's talk technology now

Unfortunately I am bitter-lesson pilled. Big model is better. If we wanted a perfectly intelligent system then letting a council of hyper-intelligent LLMs control impulses, sensory processing, thinking and acting at sufficient refresh rate would be best.

In some early experiments this worked insanely well, unfortunately today it is too slow and too expensive. I do believe this will be the approach of some vague future.

Until then however we need to hack our way in. Today's games have pretty cool "AI" (not in the llm sense, more in the behavioral graph one), games like Red Dead Redemption and Dwarf Fortress have tons of depth and they can run perfectly on 10 year old hardware.

Through this whole AI-craze people have forgotten that we had intelligent systems that could process speech since the 1970s, and somewhat LLM-like behavior with chatbots like SmarterChild in the early 2000s. There's a lost art that is being overlooked today in things like traditional NLP and behavioral graphs.

Let's take a quick look at Varkos tech stack.

The game runs on Windows, the audio processing and brain runs on my M4 MacBook. It could all run on Windows (provided there is dedicated ~12gb or more gpu ram for it), but I do development on the MacBook and I got so deep in that... eeh.

Audio:

Microphone is always on.

Main voice to text engine is (custom kernels) optimized Qwen3-ASR 1.7b. A custom harness is built around Qwen3-ASR, that processes and stitches audio in rolling partials (by default that model does not support streaming). The goal is to process audio in 40ms-80ms be it a tiny utterance like "Hey" on a 1 minute long monologue.

VAD-like methods such as turnpipe and Silero (both optimized) are used to distinguish when a turn is open.

Lexical analysis also is done on the text trying to decide if the player has made a point or is not done talking yet (eg thinking mid-sentence). In a perfect world of sufficiently fast and smart LLMs, the LLM would perform better, but I have to resort to more rudimentary but fast NLP approaches.

This is important as with the microphone always on as we want to start processing the player's utterance ASAP. It is also important for turn interruption and barge in and to have the companion not speak over the player

I will be open-sourcing this Qwen3-ASR harness soon (bear with me I have a day job).

Audio generation.

Optimized version of PocketTTS called PocketTTS-Raven (open sourced this a while ago. You can see it in action here - https://pantel.is/projects/pocket-tts-raven/?b=1 or grab its code https://github.com/pkalogiros/pocket-tts-raven ) is used as the main fast engine.

Similarly optimized qwen-3-tts (write up coming soon). Depending on the complexity of the generation we either use qwen-3 since it has better emotional control. If a generation has taken longer we default to PocketTTS since it is quite fast (20-30ms audio generation).

One trick I do, is that I generate multiple voices for different emotions (angry, sad, neutral, confused etc) - load them all in memory, and then use them where appropriate.

"Thinking."

This is the secret sauce and biggest differentiator. I call this system (ALE - Action Latent Encoder because it's an action encoder in need of a fun acronym). Under the hood, ALE is a hybrid of embeddings, small classifiers, explicit rules and traditional ML. ALE detects structure, identifies negation, commands, continuation, pronouns, and sequences. For example, “pick up the sword and bring it to me” becomes two linked action slots.

ALE is designed to be largely invariant to phrasing. You can say pick up, you can say grab, fetch, go get the damn sword you fool - it doesn't matter, it will still understand you. If there is not enough context it will inject from previous discussion. If it doesn't, it will fallback to 'clarification' and the dog will ask what do you mean.

It creates embeddings from the full text as well as its extracted structure, then semantically combines and compares it with action prototypes. A separate classifier estimates whether the turn is a command, question, chat, clarification or complex request. Everything gets merged together.

The main difference between ALE and other such hybrid-classifiers is that it accepts the world state as well. It tries to match information from the world JSON to the player's request.

ALE needs to have a different version for each game Varkos would participate in. So in a way, it is not fully plug and play but a small preparation and compatibility step would need to be implemented to ensure actions are accounted for and world state is understood.

ALE can be trained in a few minutes, so the system could in theory use a beefier LLM offline to review a session and re-train itself let's say overnight based on the player's experience and improve itself. In my limited internal evals, ALE performs surprisingly close to large LLMs at selecting the right action and decomposing the plan. I do not consider this a serious benchmark yet, but it has been reliable enough to drive Varkos in practice.

It runs quite fast, around 2-20ms on M4 MacBook and essentially acts as a tool+target function and plan decomposer caller.

A local fine-tuned LLM is then used to fuse, Varkos persona, emotions, etc etc, recent history, + action chosen and lets the brain form and paint the response. Extra grounding is performed to weed out hallucinations and re-ground it. If it fails maybe he will speak a cached response, if we have a time budget we can reprocess. With a smart prefill strategy, in certain cases the dog can begin answering in under 500ms—from the player stopping speaking to the dog yapping.

Budget breakdown

• ~40-80ms for voice to text.
• ~20-60ms for audio generation.
• ~20ms for action analysis
• And 300-600ms for creating the response and grounding its eligibility (since if the dog is afraid of spiders and we ask it to attack a spider, it might be cute for it to stubbornly deny).
• Everything needs to be streaming and start as early as possible (eg llm speech doesn't need to wait to be completed for the dog to speak out loud. Prefill early as soon as possible, etc).

Limitations

Fast enough and local models are not very capable at keeping the thread across multiple turns and a discussion over a long period of time can drift making the dog appear confused. I expect this to improve as both hardware and software evolve over time (1 to 2 years my estimation). Also this runs in real time on consumer (albeit higher-end) hardware today in the near future it will be commonplace.

Surprisingly enough, using remote LLM providers does not help much. Big models are still too slow. There are super-fast inference providers out there such as Cerebras. These work really well in terms of latency, and leave headroom for greater context, higher intelligence and depth. However, the models they provide (gpt-oss-120b and gemma31 as of today) are also pretty bad at holding a conversation (Why was GLM and the king of roleplay Qwen taken away huh??).

A final word

Overall I think there is something special here. Maybe it's because Varkos is a dog, and who doesn't like dogs. Maybe it's the low latency and that Varkos is actually useful in scouting areas for clues and objects or as a pack-mule. Maybe seeing the small cracks in his personality as he complains why he hasn't been called a "good boy" recently, but as I playtest the system I catch myself actually having tons of fun.

Let me be clear that I do not expect LLMs to replace hand-crafted characters and storylines. Slop is a real thing, and intentional design is still king and I believe and want it to remain so. But I think it is only a matter of time before we start seeing more such systems. An AI that actually plays with us, not merely talks at us, can be a different medium.

Plus the philosophical mindfuck of it all. Pleasingly ridiculous to wield the power of thunder to create a different kind of intelligence, then forcing it to be a dog and go hunt things together (is it better morally than having it do never-ending work? Well, it's not alive so it doesn't matter), but it's nice to think about. In a world of never-ending online discourse around permanent underclasses and world-ending rogue agents, it's nice to deal with alignment through shared experiences and taming the thing to play fetch and eat treats.

In Plato's cave we may still be alone, but at least we can be having fun with this weird distorted and alien thing that is now deep in the cave with us.

I'll probably be open sourcing parts of the system, and eventually all of it soon, along with a version that supports multiple NPCs (cloud inf only for now) interacting (actually interacting not just larping) with each other.

Until then follow me on @pkalogiros if you're curious and wanna see (slow) updates. Cheers.