Scene Inference Lab (SCIL)

This project is an experimental NodeJS based web application, that allows you to explore AI assisted story telling or role-playing. I imagine you could create AI players with custom prompting to help you learn foreign languages, too.

(If I remember correctly, the download package has a modification I added for someone learning Korean, with a button on the dialog screen that pops up a Korean keyboard and which keeps a “glossary”. You can highlight words in the player dialog and have them be translated into the glossary for you with sample phrases.)

The app uses an SQLite database with an embedding vector extension for semantic queries. It is made to use local LLM models on your computer, rather than online APIs.

In order to test it, you need to have either LM Studio installed locally (API mode) and start its inbuilt server, or have a locally deployed GGUF files (LLM and vector model) on your computer. Edit the .env file in the source tree (see Downloads below) to specify which model files you wish to use and where they are.

Once you run the Node app, you can access its port using your browser.

General Description

Scene Inference Lab is scene-based. You define a “world” and its players (called agents), and then craft interactions between them in chunks called “scenes”. Scenes consist of a number of dialogs or turns between characters (you do or say something, then the agent replies and does something, and so on).

When a scene is finished, you can create an AI generated synopsis. Characters are “smart” in the sense that their AI prompts evolve and grow over time, and prior dialogs are injected into your player’s prompt so their behaviour can be surprisingly dynamic.

The synopses are helpful to reduce the “context window” that the AI models needs to support, as the worlds get larger. SCIL keeps a tally of the total number of tokens for each dialog in the scene, and when that token exceeds the model’s capabilites, scene and character synopses are used instead of the full dialogs.

The App is divided into multiple “tool” screens.

Generator

Use the Generator first to create your story “world”. This information becomes part of each character’s prompt as “common knowledge”. If you prefer not to specify your own characters (next step), the AI can randomly generate characters for your, based on your world description.

Scene Editor

A scene is based on the (theatrical) description of a place or situation. A scene consists of dialogs. A dialog is one exchange (prompt-response) between two characters - typically you and an AI agent, but you can simulate multi-player parties in the “game” screen (Basic Inference).

Agent Editor

Next, you can specify your characters in the Agent Editor. SCIL dates back a few years, when agentic workflows where not the norm yet. The idea then and now is to run multiple passes of the LLM per turn, refining the output in steps.

Each Character has a Play prompt, which defines how it behaves. It also has a Forward prompt. The Forward prompt evaluates the dialogs and scenes so far, and injects new plans or motivations into the “planning” part of your character, and actually modifies the “Play” prompt. There is also a checker prompt, which constrains the “style” of the characters output.

Basic Inference

This module is the actual “game”. It lets you step through and add to the dialogs in the current scene. Dialogs can be edited, or “replayed”.

Style Tuner

The style tuner allows you to step through the dialogs of a scene, and give the AI feedback on how good they remain “in character”. You can do this by checking a box, “great”, or by editing what an agent said and telling what was wrong with the response. This will be stored with your checker prompt for the next rounds of dialog.

Vector Expert

SCIL uses an SQLITE plugin for maintaining an (embedding) vector database. What this means is that you can store and retrieve “knowledge” based on similar meaning. For example if you feed the dialogs into the DB, you can then ask about the age of one of the characters, and the retrieval query might pick up on fact that at some point, the agent was described as walking with a cane. So it might suggest that they are perhaps elderly. Things like that.

The Vector Export lets you add “lore” to your world, which the characters can explore (i.e. include in their “state”) when you hit the “Plan” or “Wonder” buttons in the game screen. So for example when you hit “Wonder”, the agent will come up with a piece of knowledge they might want to know, and then that question gets passed to the vector/knowledge database. If an answer is found, it will be added to the agent’s memory.

Downloads

Package includes the NodeJS app, but you will need to download LM Studio or a GGUF model file from HuggingFace for instance. The SQLite DB file and a few built-in prompts for the system are created when you start the server. This app is just a proof-of-concept though, and not user-friendly.