On this article, you’ll find out how an agent’s method to managing state — stateless or stateful — shapes each its implementation and the deployment structure constructed round it.
Matters we are going to cowl embrace:
- What separates stateless from stateful brokers, and the tradeoffs every design imposes on scaling.
- How one can implement a stateless agent that relies upon fully on the consumer to provide dialog historical past.
- How one can implement a stateful agent that manages its personal reminiscence by a database layer.
Introduction
A earlier article laid out a complete architectural roadmap for AI agent deployment, analyzing the infrastructure wanted to deliver brokers into manufacturing settings.
As a follow-up, we now flip to a elementary, sensible query that needs to be answered earlier than any load balancer is configured: the place does the agent’s reminiscence reside? Brokers could deal with their state (the context gained to this point and the dialog historical past) in numerous methods, and this code-level choice can considerably affect your complete deployment structure.
This text breaks down the 2 main paradigms for dealing with an agent’s state: stateless and stateful design. A simplified model of a real-world implementation, utilizing open language fashions served by the quick Groq API, will illustrate these concepts in follow.
Preliminary Setup
If that is the primary time you might be utilizing language fashions from Groq in a Python program, you’ll want to put in the required library: pip set up groq.
After that, we import it and set our Groq API key within the code under:
|
import os from groq import Groq
# Get an API key in https://console.groq.com/keys and set it right here os.environ[“GROQ_API_KEY”] = “PASTE_YOUR_GROQ_API_KEY_HERE”
# Initializing the consumer consumer = Groq()
# Utilizing an environment friendly mannequin from Groq: Llama 3.1 8B Instantaneous MODEL_ID = “llama-3.1-8b-instant” |
An necessary setup choice right here is the selection of a particular mannequin. llama-3.1-8b-instant is a extremely cost-efficient mannequin that’s, on the time of writing, generously supported on Groq’s 2026 free tier: it permits as much as 14,400 requests per day. That makes it an excellent selection for illustrating the stateless and stateful agent paradigms under.
Stateless Brokers: Fireplace and Neglect
Stateless brokers deal with every request as utterly remoted and unbiased. The agent reads the person immediate, invokes the LLM inference engine, and delivers the output. As soon as that execution cycle ends, every little thing is forgotten.
The Tradeoff
Architectures based mostly on stateless brokers may be scaled horizontally with exceptional ease. Since no person reminiscence is saved on a backend server, incoming requests may be forwarded to any accessible occasion. There may be, nevertheless, an necessary limitation in multi-turn conversations: the frontend should re-send the entire dialog historical past alongside each new request. Consequently, the context window grows with a snowballing impact, rapidly driving up token utilization.
Illustrative Instance
This runnable code illustrates, by a primary state of affairs, how a stateless agent usually interacts with a Groq language mannequin.
First, we outline a stateless_agent operate that emulates an agent’s interplay with our chosen mannequin. Importantly, no state or reminiscence of the dialog is saved internally. As a substitute, the earlier dialog historical past can optionally be handed in as a parameter and appended to the present immediate. The API name to the Groq mannequin takes place in consumer.chat.completions.create().
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
def stateless_agent(immediate: str, provided_history: record = None) -> str: “”“ The agent depends utterly on the consumer to supply context. It retains no info from previous interactions in native reminiscence. ““” # Initializing with a system immediate messages = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# Appending no matter historical past the consumer supplied if provided_history: messages.prolong(provided_history)
# Appending the brand new immediate messages.append({“position”: “person”, “content material”: immediate})
# The LLM processes your complete chain of messages response = consumer.chat.completions.create( mannequin=MODEL_ID, messages=messages, max_tokens=100 )
return response.selections[0].message.content material.strip() |
To grasp the restrictions of a stateless agent, we simulate a easy user-model dialog by it:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# — Testing the Stateless Agent —
print(“— Flip 1 —“) prompt_1 = “Hello, my title is Alice and I’m studying about API infrastructure.” response_1 = stateless_agent(prompt_1) print(f“Agent: {response_1}”)
print(“n— Flip 2 (With out Shopper Context) —“) # The agent fails right here as a result of it retained no reminiscence of Flip 1 prompt_2 = “What’s my title and what am I studying about?” response_2 = stateless_agent(prompt_2) print(f“Agent: {response_2}”)
print(“n— Flip 2 (With Shopper Context) —“) # The frontend MUST inject the historical past into the payload for the agent to succeed frontend_payload = [ {“role”: “user”, “content”: prompt_1}, {“role”: “assistant”, “content”: response_1} ] response_3 = stateless_agent(prompt_2, provided_history=frontend_payload) print(f“Agent: {response_3}”) |
Output:
|
—– Flip 1 —– Agent: Hiya Alice, good to meet you. Studying about API infrastructure can be a fascinating and rewarding subject. What particular features of API infrastructure would you like to discover or talk about? Are you wanting for info on API administration, safety, deployment, or one thing else?
—– Flip 2 (With out Shopper Context) —– Agent: Sadly, I don‘t have any details about you, together with your title. Our dialog simply began, so I’m right here to assist you with any questions or matters you‘d prefer to study. Please be at liberty to share your title and a subject you’re in studying about.
—– Flip 2 (With Shopper Context) —– Agent: Your title is Alice, and you are studying about API infrastructure. |
The implementation is straightforward, however and not using a consumer or frontend that sends the total dialog historical past to the agent on each flip, the agent’s LLM lacks the context it must reply sure questions correctly.
Stateful Brokers: Context-driven Continuity
Below this method, the agent takes on the reminiscence burden itself. The consumer, in the meantime, solely must ship the latest person immediate along with a singular identifier, usually related to the present session. The agent then retrieves the session historical past or context from a database and appends the brand new message to it. As soon as the LLM inference has been processed, the agent updates the context within the database.
The Tradeoff
This can be a a lot neater expertise from the consumer facet. It additionally facilitates advanced and asynchronous workflows wherein brokers could must pause their execution and look forward to instruments, app responses, or human approval. Nevertheless it all comes with a price: scaling this answer turns into a lot tougher, beginning with the necessity for a persistent database layer within the structure. In infrastructures that scale horizontally, methods corresponding to centralized reminiscence caching with Redis can also change into essential to keep away from “localized amnesia”, the place a session’s historical past is stranded on the only occasion that occurred to serve the sooner turns.
Illustrative Instance
We illustrate the essential concepts behind a stateful agent by incorporating a “persistent” database layer. For simplicity, we use a tiny SQLite database. The hot button is to have the agent handle its personal dialog reminiscence as a substitute of relying on a frontend to supply it externally:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
import sqlite3 import json
# Initializing an in-memory SQLite database for pocket book testing conn = sqlite3.join(‘:reminiscence:’) cursor = conn.cursor() cursor.execute(”‘CREATE TABLE IF NOT EXISTS agent_memory (session_id TEXT PRIMARY KEY, historical past TEXT)’”) conn.commit()
def stateful_agent(session_id: str, new_prompt: str) -> str: “”“ The agent manages its personal state utilizing a database. The consumer solely sends the brand new immediate and their session ID. ““” # 1. Retrieving current state from the database cursor.execute(“SELECT historical past FROM agent_memory WHERE session_id=?”, (session_id,)) row = cursor.fetchone()
if row: conversation_history = json.hundreds(row[0]) else: # Initializing with system immediate for brand new classes conversation_history = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# 2. Appending the brand new person immediate conversation_history.append({“position”: “person”, “content material”: new_prompt})
# 3. Processing the LLM name utilizing the retrieved historical past response = consumer.chat.completions.create( mannequin=MODEL_ID, messages=conversation_history, max_tokens=100 ).selections[0].message.content material.strip()
# 4. Updating the state with the assistant’s reply conversation_history.append({“position”: “assistant”, “content material”: response})
# 5. Saving the brand new state again to the database cursor.execute(”‘ INSERT INTO agent_memory (session_id, historical past) VALUES (?, ?) ON CONFLICT(session_id) DO UPDATE SET historical past=excluded.historical past ‘”, (session_id, json.dumps(conversation_history))) conn.commit()
return response |
Discover how the session identifier is used to question the related info from previous interactions within the dialog at hand.
Now let’s attempt all of it in a dialog much like the earlier one, however this time with the person asking the agent to recall the person’s personal title:
|
# — Testing the Stateful Agent —
print(“— Flip 1 —“) print(f“Agent: {stateful_agent(‘user_123’, ‘Hello, I’m Bob and I wish to scale my AI app.’)}”)
print(“n— Flip 2 —“) # Discover how the consumer NO LONGER sends the context payload. Simply the session ID. print(f“Agent: {stateful_agent(‘user_123’, ‘What was my title once more?’)}”) |
Output:
|
—– Flip 1 —– Agent: Hiya Bob, scaling an AI app can be a advanced course of. Might I ask:
1. What kind of AI expertise is your app constructed on? (e.g., machine studying, pure language processing, pc imaginative and prescient) 2. Are you utilizing any cloud companies like AWS, Google Cloud, or Azure? 3. What are your scalability targets (e.g., enhance person depend, scale back latency, enhance response occasions)?
This info will assist me higher perceive your necessities and present extra efficient help.
—– Flip 2 —– Agent: Your title is Bob. |
This instance is, in fact, a good distance from a scaled-up manufacturing structure, nevertheless it serves to make clear the important thing distinction between how stateful and stateless brokers work.
Wrapping Up: The Tradeoffs
The selection between a stateful and a stateless architectural design boils right down to correctly matching the infrastructure to the workflow:
- Stateless brokers are most popular in easy pipelines oriented to very particular duties, like textual content extraction, summarization, or single-turn classification chatbots. They maintain the structure light-weight, which is often sufficient in such use circumstances, avoiding database bottlenecks and permitting seamless horizontal scaling.
- Stateful brokers make rather more sense if we intend to develop long-running assistants, coding assistants, or multi-turn bots in functions like customer support. As a result of the agent owns the historical past, the consumer payload stays small on each flip, and the dialog may be trimmed or summarized server-side as a substitute of being resent in full because it grows.
