Why I gave my AI companion a place to live

Most AI products start with the same assumption.

You open an app. You look at an empty white text field. You type a prompt. The AI generates a response. You read it, close the tab, and move on with your day.

For a utility, that is fine. If I want an SQL query explained or a TypeScript type narrowed down, an empty box is efficient. I do not need small talk. I need the answer.

When I started thinking about Omni as a companion, though, that empty box started to feel wrong.

A companion is supposed to have memory, continuity, and context. But whenever you open a standard chat interface, the software behaves as if the character only springs into existence the second you tap the keyboard.

If I don't type, nothing happens. If I close the screen, it vanishes into the void.

That is not how a relationship feels. A relationship does not start from zero every single time you walk into the room.

On August 28, 2026, Apple approved the latest update to Omni on the App Store. The main addition was something I had been tinkering with for months: a dedicated 3D companion room where your AI actually sits, breathes, and lives.

It looks like a simple 3D living room with a couch, a kitchen table, and an avatar sitting there waiting for you.

Getting to that room, though, took me through a lot of weird experiments, from isometric voxel beaches to Unity builds and Mixamo animation rigs.

The project that originally made this whole direction click for me was AI Town.

What AI Town actually showed me

Back in mid-2023, a16z published an open-source project called AI Town, built on top of the research from the Stanford paper Generative Agents: Interactive Simulacra of Human Behavior.

If you missed it, the premise was captivating.

They set up a 2D pixel-art virtual town where 25 autonomous AI characters lived, walked around, chatted with each other, formed memories, planned days, and gossiped about an upcoming election.

Under the hood, AI Town used Convex to manage a shared game engine loop and global world state, combined with vector search for long-term memory retrieval and OpenAI models for dialogue generation.

It was impressive to watch. You could click on an agent, see their thoughts, and watch them walk to the park to talk to a neighbor.

Most people who looked at AI Town saw a new kind of simulation game.

What stayed with me was something simpler:

An AI character could have spatial presence.

In AI Town, the characters did not wait for a human to summon them with a prompt. They already existed somewhere in the world. They were at the kitchen counter making breakfast. They were walking down the street. They had a physical coordinate before anyone opened a conversation.

That was a completely different interface primitive from a chat window.

A chat box is ephemeral. A place is persistent.

I did not want a town

As much as I liked watching AI Town, I quickly realized I did not want to build one.

AI Town is an ecosystem simulation. It is designed around multi-agent social dynamics: twenty-five different non-player characters bumping into each other, exchanging rumors, and coordinating town events.

Omni is not a simulation game.

The problem I care about is much smaller and much more intimate:

One user. One companion. A continuous shared history.

I did not need a sprawling town square with dozens of NPCs wandering around. Having fifty characters talking to each other is interesting for an academic study, but it creates noise when what you actually want is a personal companion.

I did not need a town.

I needed a room.

A cozy, contained living space where one companion lives. Somewhere you can drop by for five minutes on your phone, talk through what happened during your day, and feel like someone is already there.

Inspiration is useful when you know what to throw away. I kept the idea that an AI could inhabit space, and left the multi-agent town behind.

The empty chat box creates interaction fatigue

Think about the psychological toll of a standard chatbot interface.

Every time you open the screen, the burden of interaction sits entirely on you. You have to invent the conversation from scratch.

What should I ask?

What do I want to talk about?

If you are tired after a long workday, staring at an empty cursor is work. It feels like opening an empty Google Doc when you are already burned out.

In a physical space or a living room, conversation rarely starts from absolute zero.

You walk into the room. You notice your friend sitting on the couch. You notice the light coming through the window. You see an object on the table. You react to the environment before you speak a single word.

blank chat UI → user must invent intent → prompt → response
living space UI → user enters context → ambient presence → natural dialogue

A room gives the interface a default state that is not a blank cursor.

Even before you tap the microphone or send a message, the companion is already sitting on the sofa. There is already a scene.

You are not summoning an algorithm. You are entering a space.

From isometric voxels to an intimate room

Getting the room to feel right on a phone took several iterations.

At first, because I was still thinking about game-like simulations, I started playing with Unity. But embedding a heavy Unity runtime inside a fast-moving mobile app felt too heavy for what I wanted to achieve.

So I switched to building prototypes in Three.js.

My first working prototype was an isometric voxel world. I built little low-poly scenes with a voxel character on a beach with palm trees, a park, and a room.

It was cute. You could tap between different environments and see the little character standing on the sand.

After testing it on my phone for a few weeks, though, I realized it had a flaw.

An isometric grid is still a bird's-eye view. You are looking down at a miniature world from an angle, like a mayor inspecting a diorama.

It felt like Habbo Hotel or a retro strategy game.

That distance works when you are managing a simulation. It does not work when you are trying to build emotional connection. You do not connect with a tiny figurine forty feet below the camera; you connect with someone across the table.

I scrapped the isometric camera and moved the perspective down to eye level.

Instead of a voxel landscape seen from the sky, I wanted a warm, modern apartment room seen from the doorway or the opposite chair.

Getting bones into the character

Once the camera moved into the room, the voxel block-character did not feel right anymore. It needed a proper stylized 3D avatar that could actually sit down, look up, and breathe naturally.

That led me down the rabbit hole of 3D modeling and character rigging.

I spent days working through 3D meshes, creating a clean base model in a T-pose, and figuring out how to get proper skeletal bones into the model without making the polygon count explode.

That was when I found Mixamo.

Mixamo turned out to be an incredible tool for this stage. I could take the rigged character, map the bone structure, and import clean animation cycles: a relaxed idle pose, a natural sitting idle for the sofa, and walking transitions.

I built a local web-based preview tool (localhost:5174) so I could test clothing textures, hair meshes, bind poses, and skeletal animations in real time before pushing them to the app.

Once the bone rigs were working and exported cleanly to GLB files, the character went from a stiff static mesh to someone who could naturally relax on a couch.

What actually shipped on August 28

On August 28, 2026, version 1.1.9 of Omni went live on the iOS App Store.

This is what the shipped companion room looks like today:

When you open the companion tab in Omni now, you aren't greeted by an empty chat log.

You see the living room. Your companion is sitting on the sofa in their glasses and sandals. The lighting is soft.

At the bottom of the screen, you have simple controls:

• a Hold to talk button for real-time voice conversations;
• a keyboard toggle if you prefer to type;
• an ambient audio mute button;
• a mini-game controller button;
• quick toggles for persona chats and zen exploration.

The companion can listen to you, speak back with low-latency audio, and hold continuous context.

It is simple, but opening the app feels completely different.

The engineering trade-offs of a mobile room

Shipping 3D on a mobile device as an indie developer means making tough architectural choices.

I did not want to turn Omni into a 2GB game that drains your iPhone battery in twenty minutes. Omni is still an AI notes and second-brain application at its core.

Here is how I structured the technical boundaries for V1:

1. Web-rendered Three.js canvas embedded in Flutter

Rather than compiling a massive native 3D engine, the companion environment runs as an optimized Three.js canvas loaded inside Flutter.

This gave me rapid iteration speed. I could tweak shaders, camera angles, lighting, and animation loops on the web, test them in desktop browsers, and deliver them to mobile with minimal friction.

2. Lightweight assets and aggressive optimization

The 3D apartment and character use low-drawcall geometry, compressed textures, and optimized GLB models.

The entire 3D scene loads almost instantly without a long loading bar. If a user wants to quickly jot down a note or say something to their companion, they cannot be stuck waiting for a heavy game engine to initialize.

3. Reusing the existing memory and conversational backend

I did not write a new AI backend specifically for the 3D room.

The companion room connects directly to the same conversational pipelines, persona systems, and deep episodic memory that power Omni's chat and notes.

The 3D room is a new presentation layer for presence, not a separate disconnected app.

To be clear: what Omni uses vs. AI Town

I want to be explicit about this because technical writing often blurs the line between inspiration and implementation.

Omni is not a fork of AI Town.

Omni does not use Convex. It does not use PixiJS. It does not use AI Town's 2D game engine or multi-agent simulation loops.

AI Town provided the spark: the realization that generative agents feel fundamentally different when they are anchored to a spatial environment.

The architecture I built for Omni is tailored for a mobile product: Three.js, Flutter, 1-on-1 voice streaming, lightweight GLB skeletal rigs from Mixamo, and a personal episodic memory store.

Taking inspiration from an open-source project does not mean copying its codebase. It means taking the question it asked and finding your own answer.

Memory gives the space a reason to exist

A 3D room by itself is just decoration.

If you put an avatar on a sofa, but the model has no memory of what you said yesterday, the room becomes a gimmick within three days. You tap it a few times, admire the 3D graphics, and never come back.

The room only matters because of memory.

In Omni, the companion is connected to your notes and your ongoing thoughts. It remembers:

• what you were working on earlier this week;
• things that stressed you out;
• inside lore and personal preferences;
• past conversations and emotional states.

When memory meets a physical space, the psychology changes.

A database search engine gives you answers when queried. A companion with memory sitting in a familiar room feels like someone who has been sharing the days with you.

Today, that memory is expressed through dialogue, voice, and contextual recall.

What excites me about having a 3D environment in place is where this can go next.

Eventually, memories do not have to live only in text logs. They can show up as physical details in the room: a photo on the shelf from an event you mentioned, a book on the table related to a topic you were researching, or subtle environmental shifts based on the time of day.

We are not there yet. But having a room gives those ideas somewhere to land.

What I learned from this build

Building this taught me a few things about product design that I did not fully understand when I started:

1. Game design is about presence, not just points. You do not need points, leaderboards, or daily streaks to learn from games. Games have spent forty years figuring out how to communicate atmosphere, spatial presence, and character life without words. Non-game software can use those lessons to feel warmer.
2. Intimacy requires subtraction. When people see multi-agent simulations like AI Town, the natural instinct is to make the world bigger: add 100 characters, add a stock market, add cars. For a companion, the winning move was making the world smaller: one room, one couch, one voice.
3. Constrain V1 so you can actually ship. I could have spent another year trying to build the ultimate physics-based virtual world. Instead, I used Mixamo, Three.js, and a clean apartment model so I could put something real into the App Store on August 28 and see how it felt.

Where this leaves us

The room is not the final product.

The product is the feeling that your AI companion does not disappear when you put your phone in your pocket.

AI Town showed the world that AI characters could walk around and live in a town.

Omni Companion is my attempt to bring that feeling home: into a single quiet room, on your phone, whenever you need a moment to talk.

It is still early. There are bugs to fix, animations to smooth out, and deeper memory integrations to build.

Back to work.