# What is AgentLabs? Frontend as a service for building chat-based AI Agents. AgentLabs is an [open-source](https://github.com/agentlabs-inc/agentlabs) platform to create rich chat-based AI Assistant applications in minutes. [AgentLabs.dev ](https://agentlabs.dev)frontend can be controlled using our [Node](https://www.npmjs.com/package/@agentlabs/node-sdk) and [Python SDKs](https://pypi.org/project/agentlabs-sdk/) and allows you to stream content in real time between your users and your backend. The solution provides: :man\_scientist: User management and Authentication :speech\_balloon: Full-featured Chat as a service :man\_technologist: Support for Markdown, Images, and File Transfer :fireworks: Asynchronous and real-time streaming SDKs :b: Backend Agnostic: you can use AgentLabs with any backend framework or tool ## How it looks {% embed url="" %} Basic remote code execution example {% endembed %} {% embed url="" %} Support for markdown showcase {% endembed %} # Get Started with AgentLabs Self Hosting & Cloud Version Depending on your need, AgentLabs provides a self-hostable and cloud version. ## Cloud version The cloud version is the most convenient way to start using AgentLabs. You only have to [register](https://console.agentlabs.dev) and follow the wizard to configure your account without worrying about infrastructure management. {% hint style="info" %} Keep in mind that AgentLabs is still in Alpha, so the cloud version is probably convenient especially if you don't want to update it every week. :smile: {% endhint %} ## Self-hosted version The cloud version of the plugin uses exactly the same version as the self-hosted one. It means you can choose to self-host AgentLabs on your own infra and keep the exact same feature set. At this moment, we only support docker-compose to run it at a small scale. However, we'll support helm-chart for Kubernetes as soon as we roll out the beta version. ## Which one should I choose? At the moment, choosing the Cloud Version is probably the safest solution unless you have some very specific infrastructure or security constraints. # Pricing Free self-hosted version and freemium cloud version **Agentlabs is currently in alpha and free to use for everyone.** We're going to offer both free and premium tiers in the future for our cloud version. Our self-hosted version will remain free, and we'll provide optional enterprise-grade support for it. # Installation Installing the Python or Node SDK You can use both our Python or NodeJS SDK based on your preferences. Our SDK will help you to stream content from your backend to the CharUI. {% tabs %} {% tab title="Python" %} Using pip ```sh pip install agentlabs-sdk ``` Or using poetry ```bash poetry add agentlabs-sdk ``` {% endtab %} {% tab title="Nodejs" %} ```bash npm install --save @agentlabs/node-sdk ``` {% endtab %} {% endtabs %} # Secret Key How to get your API / Secret Key You can generate the secret key from your admin console. After you create your first project, open the `Secrets` section and click on `Generate secret`. {% hint style="info" %} For some security reasons, your secret key will never be shown twice. Keep it secure. {% endhint %} {% embed url="" %} How to generate a secret key {% endembed %} # Using the SDK Getting started with our SDK ## Basic example Since we provide the entire UI and streaming for you, you only have to listen for messages from the front end and send messages from your backend.\ \ Here's a dead simple example of a ping-pong app. Indeed, you can make much more complex applications with multiple agents streaming in real time. {% tabs %} {% tab title="Typescript" %} ```typescript import { Project } from 'agentlabs-sdk'; const project = new Project({ projectId: 'you-project-id', secret: 'your-secret', url: 'provided-in-your-console' }); const agent = project.agent('agent-id'); project.onChatMessage((message) => { if (message.text === 'ping') { agent.send({ conversationId: message.conversationId, text: 'pong' }) } else { agent.send({ conversationId: message.conversationId, text: 'I do not understand' }) } }); project.connect() ``` {% endtab %} {% tab title="Python" %} ```python from agentlabs.project import IncomingChatMessage, MessageFormat, Project def handle_message(message: IncomingChatMessage): if message.text == 'ping': agent.send( conversation_id=message.conversation_id, text="pong" ) else: agent.send( conversation_id=message.conversation_id, text="I did not get that" ) project = Project( project_id='your-project-id', secret='your-secret', url='provided-in-your-console' ) agent = project.agent('agent-id') project.on_chat_message(handle_message) project.connect() project.wait(); ``` {% endtab %} {% endtabs %} **Next** Learn more about our core concepts and follow some recipes to get started. {% content-ref url="/pages/R21c8oRX2G4mrfFHm89J" %} [Core Concepts](/core-concepts/frontend-as-a-service) {% endcontent-ref %} {% content-ref url="/pages/ni06RdZYDOeTtrhHPfQA" %} [Recipes](/recipes/before-we-start) {% endcontent-ref %} # Frontend as a service Here is why AgentLabs is a frontend as a service ## Why we're building this As a developer, every time you start a new project (especially a chat-based UI) you have to handle many boring things frontend-wise: :expressionless: Authentication :expressionless: User management :expressionless: Real-time streaming and Async I/O management :expressionless: Analytics :expressionless: Payment :expressionless: And even more... All these problems take a long time to solve, especially if you want to create a product that scales. However, as an AI Agent developer, you must have enough things on your plate already. [AgentLabs](http://agentlabs.dev/) aims to provide a built-in solution that covers all the above points without writing any single front-end code. ## What is it? It's simple: we provide everything that is frontend-related and we give you the possibility to control it via our backend SDK. Everything you can see in the following video is provided by [AgentLabs](http://agentlabs.dev/) and requires 0 lines of code. This includes: :white\_check\_mark:The authentication portal :white\_check\_mark:The conversation chat UI :white\_check\_mark:The realtime connection :white\_check\_mark:Even the dark and light theme :smile: {% embed url="" %} Frontend as a service example {% endembed %} # User Authentication What you must know about authentication As we said previously, AgentLabs allows you to manage and authenticate your users. ## Authentication methods AgentLabs lets you configure the authentication methods you want to register and sign in your users. You can configure basic email methods and different oauth2 providers. The methods you configure will be available to your users. You can configure them from your [admin console](https://console.agentlabs.dev/).

Configuring authentication methods

{% hint style="info" %} The current auth methods set is limited but we'll rollout more methods as we go :rocket: {% endhint %} ## Authentication request Once you've configured your auth methods, you can request authentication from your users simply using our SDK. {% tabs %} {% tab title="Python" %} ```python agent.request_login( conversation_id="your-conversation", text="Please login to access this feature" ) ``` {% endtab %} {% tab title="TypeScript" %} ```python agent.requestLogin({ conversationId: "your-conversation", text: "Please login to access this feature" }) ``` {% endtab %} {% endtabs %} \ The snippet above will trigger the following in-chat authentication component.

Asking for in-chat authentication

## Managing your users Then, you can manage all your application users from your admin console.\

User management dashboard

# Project What you have to know about a project During the onboarding, the first thing we ask you to do is to create a Project. You can think of a project as a Tenant or as an Application you want to build. ## Project slug and domain When you create a project, AgentLabs attributes it a `slug` which is a unique identifier but also a subdomain where your frontend application will be available to your users. Let's say your project slug is `gpt-chat`, then your application will be available at `https://gpt-chat.agentlabs.dev` , and your users will be able to register and use your Chat UI there.

Process of creating a project in your console

# Agents Your project can contain one or many agents. An agent is simply an entity that allows you to communicate with a user through a Conversation in the Chat UI. When a user starts a new conversation, all available agents of your project will be able to handle and send messages to the user through our SDK. UI-wise, a good way to think about an agent is to imagine them as the Sender of a message. Every agent can programmatically send a message from your backend using our SDK.
## Technical implications The agents can receive every user's real-time input through a socket connection. They can also send real-time and asynchronous message times to the conversation. This means your backend must persist a real-time connection with the frontend UI. No worries, with our SDK, persisting a real-time connection is as simple as writing `project.connect()`. Moreover, AgentLabs will handle the persistent of every conversation for you so you don't have to worry about your user being offline for example.

Schema of a peristent connection between your backend and the frontend

## Creating agents You can create as many agents as you need from your [admin console](https://console.agentlabs.dev/). Creating an agent from the console allows you to attribute helpful metadata such as a name, an avatar, and many other things. ## How many agents should I create? It depends on what you want to achieve. If you want to build something as simple as ChatGPT, then you only need to create one agent that will represent ChatGPT. If you want to handle more complex multi-agent conversations or plug systems like [AutoGen](https://github.com/microsoft/autogen), then you may want to create more agents. # Messages Messages are the building blocks of a chat-based app. AgentLabs allows you to handle and send messages in many different ways. {% content-ref url="/pages/dBvzNbh4cKmxEtjf623f" %} [Messages Format](/core-concepts/messages/messages-format) {% endcontent-ref %} {% content-ref url="/pages/FXBHZv5aUW0t7GD20hqh" %} [One-off vs Stream](/core-concepts/messages/one-off-vs-stream) {% endcontent-ref %} {% content-ref url="/pages/XZtqo0Ls6muYVkFlIWjT" %} [Broken mention](broken://pages/XZtqo0Ls6muYVkFlIWjT) {% endcontent-ref %} # Messages Format About markdown and PlainText ## Plain text By default when you send a message using the SDK, the UI shows it in plain text. Plain text messages are great for chatting or sending messages that don't contain formatting to the user.

Plain text response example

Sending plain text is the default behavior of the SDK, so you just have to run the following code: {% tabs %} {% tab title="Python" %} ```python agent.send({ conversation_id: 'your-conversation-id', text: 'Your plain text value', }); ``` {% endtab %} {% tab title="TypeScript" %} ``` agent.send({ conversationId: 'your-conversation-id', text: 'Your plain text value', }); ``` {% endtab %} {% endtabs %} ## Markdown Often, you may want your agent to return more than just a simple text. In that situation, you most likely want to use the markdown format. Markdown is great for applying **some** *styling,* [or to display a link](https://agentlabs.dev), generating a table, displaying `code snippet` , images, and more. You can see a [full markdown cheatsheet here.](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) {% embed url="" %} Example of a message containing markdown {% endembed %} Send indicate the interpreter you are sending text in Markdown, you just have to use the `MARKDOWN` format option. {% tabs %} {% tab title="Python" %} ```python from agentlabs.chat importMessageFormat agent.send( text="**My Text**", format=MessageFormat.MARKDOWN, conversation_id=self.conversation_id, ) ``` {% endtab %} {% tab title="TypeScript" %} ``` agent.send({ conversationId: 'your-conversation-id', text: 'Your plain text value', format: 'Markdown' }); ``` {% endtab %} {% endtabs %} # One-off vs Stream ## One-off messages A one-off message is a standard message that you can send entirely as you would do when you're chatting with a friend and hit `send` . The entire message will be sent at once. One-off messages are great for small content that you know in advance.3 For example, in our [Ping-Pong](/recipes/ping-pong) recipe, we answer "pong" using a one-off message every time the user says "ping". ## Streamed messages Streamed messages are different from one-off messages. When you send a streamed message, the user immediately sees a message box popping up. However, every time you send a new stream on the same channel, it will aggregate to the same message box. {% hint style="info" %} You can think of them as having this "typewriter" look and feel as you could have for example in [ChatGPT](https://chat.openai.com/). {% endhint %} Sending a stream works in 3 steps: :blue\_circle: **Open a stream channel using the** `createStream()` **method** ```typescript const channel = agent.createStream(conversationId) ``` {% hint style="info" %} Note that starting from `v0.0.51` openeing a stream triggers an event that will displays a smooth animation in the frontend (see [Typewriter animation](/core-concepts/messages/typewriter-animation)) {% endhint %} :blue\_circle: **Write content to the stream as many times as you want** ```typescript channel.write("this") channel.write("is") channel.write("a") channel.write("streamed") channel.write("message") ``` :blue\_circle: **When you're done, close the channel** ```typescript channel.streamEnd() ``` Et voilà :tada:\ \ \ Here's an example of how a streamed response looks {% embed url="" %} Example streamed message with code execution {% endembed %} ## Which one to choose? It's pretty rare to hesitate between both, but if you're in that situation, here's a quick recap that might help. | Solution | Info | Warning | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | | One-off |

✅ great when you know the content of the message upfront
✅ suited when you need to inform the user quickly about something

| :x:Generally not suited to send multiple messages in a row | | Stream |

✅ feel more natural / human-friendly
✅ suited for long-running tasks in the background

| :x:Be careful with very very long answers because it can become borring for the user to wait, especially if you know the whole response upfront. | {% hint style="info" %} Indeed, you have to test each solution on your own to make sure you decide which one is best suited to your specific use case. In general, you want to use a mix of both in your application anyway. {% endhint %} # Typewriter animation How animate your messages properly with Typewrite We explained in the previous section that you can use either the `send` method (for one-off messages) or the `write` method (for sending streams). But sometimes, you want to send some one-off messages that look like streams. Since `v0.0.51` the Node and Python SDKs provide a `typewrite()` method that allows you to simulate a stream with a smooth animation, even if you know the content in advance. {% embed url="" %} Showcasing typewriter animation using typewrite() {% endembed %} {% hint style="info" %} The full code of this video is [available here](https://github.com/agentlabs-inc/agent-examples/tree/main/nodejs-examples/ts-typewrite-vs-send). {% endhint %} ### How it works? {% tabs %} {% tab title="TypeScript" %} ```typescript await agent.typewrite({ conversationId: "conversationId", text: "Here's your message", intervalMs: 60, // optional initialDelayMs: 1500, // optional }); ``` {% endtab %} {% tab title="Python" %} ```python agent.typewrite( conversationId="conversationId", text="Here's your message", interval_ms=60, # optional initial_delay_ms=1600 # optional ) ``` {% endtab %} {% endtabs %} Behind the scenes, AgentLabs will instantiate a new stream and start sending the stream with a nice typewriter effect. # Attachments ## Introduction Sending attachments is particularly useful if you want your agent backend to send some images or any other files to your users. AgentLabs SDK allows you to effortlessly send attachments along with your messages. Some attachments like Images will be displayed with a lightbox preview, others like zip files will be displayed as simple downloadable files.

Example of 3 attachments sent in a message

Example of zip files

## How to send an attachment Sending attachments is super easy. You only have to instantiate an Attachment and then pass it to the `send` method as an argument. {% tabs %} {% tab title="Python" %} ```python from agentlabs.attachment import Attachment attachment = Attachment.from_local_file("path-to-your-file") agent.send( conversation_id=message.conversation_id, text="Here is your zip file", attachments=[attachment], ) ``` {% endtab %} {% tab title="Typescript" %} ```typescript import {Attachment} from "@agentlabs/node-sdk"; const attachment = Attachment.fromLocalFile('path-to-your-file') await agent.send({ text: 'Here is an image for you...', conversationId: '', attachments: [attachment], }); ``` {% endtab %} {% endtabs %} {% hint style="info" %} You can pass as many attachments as you want. We also support all mime types as long as each file does not exceed 10MB. {% endhint %} ## Going further If you want to go further with attachments, you can check[ this example](https://github.com/agentlabs-inc/agent-examples/tree/main/nodejs-examples/ts-midjourney-clone-basic) where we show you how to create a basic midjourney clone. Here's the result: {% embed url="" %} A basic midjourney clone {% endembed %} # Before we start What you must know before using our recipes This section shows you many examples of what you can build with AgentLabs. These examples will focus on the code you have to write using the SDK, but they assume you already know how to access your admin dashboard and configure a project using the admin wizard. As a quick reminder, here are the steps you have to follow to setup your admin dashboard (at least for the cloud version). 1. Login to the [admin console](https://console.agentlabs.dev) 2. Create a project and follow the wizard 3. Create your first agent 4. You're all set! # Ping-Pong Let's start with a ping pong ## What are we building? Let's start with a dead simple example to understand how AgentLabs works. In this recipe, we'll create a basic server with one single agent that listens for an incoming message. Every time a user sends a message through the Frontend UI, the agent will evaluate it and answer. If the message is equal to `ping` the agent will reply with `pong`. Otherwise, the agent will say he did not get the message. The full example is available in [our example repository](https://github.com/agentlabs-inc/agent-examples/tree/main/nodejs-examples/ts-ping-pong). ## Requirements To follow this tutorial, you must have: * some basic programming skills in Python or Typescript * created a project and an agent in [your console](https://console.agentlabs.dev) ## Let's code ### Initializing the SDK First, we must initialize the SDK with the project ID and the secret key. You can find this information on your [admin console](https://console.agentlabs.dev/admin/). {% tabs %} {% tab title="Typescript" %} {% code fullWidth="false" %} ```typescript import { Project } from 'agentlabs-sdk'; const project = new Project({ projectId: 'you-project-id', secret: 'your-secret', }); const agent = project.agent('agent-id'); ``` {% endcode %} {% endtab %} {% tab title="Python" %} ```python from agentlabs.chat import IncomingChatMessage, MessageFormat from agentlabs.project import Agent, Project project = Project( project_id='your-project-id', secret='your-secret' ) agent = project.agent('agent-id') ``` {% endtab %} {% endtabs %} ### Listen for messages Now, we can use the `onChatMessage` method to apply some logic every time the user sends a message in the chat. The `onChatMessage` accepts a callback as an argument, this callback will handle our logic. Then, we'll use the `send` method exposed on the agent instance to send a message back to the chat on behalf of the agent. {% hint style="info" %} Note the message argument contains some useful data, such as the conversation ID or the message's text. {% endhint %} {% tabs %} {% tab title="Typescript" %} ```typescript import { Project } from 'agentlabs-sdk'; const project = new Project({ projectId: 'you-project-id', secret: 'your-secret', }); const agent = project.agent('agent-id'); project.onChatMessage((message) => { if (message.text === 'ping') { agent.send({ conversationId: message.conversationId, text: 'pong' }) } else { agent.send({ conversationId: message.conversationId, text: 'I did not get that' }) } }) ``` {% endtab %} {% tab title="Python" %} ```python from agentlabs.chat import IncomingChatMessage, MessageFormat from agentlabs.project import Agent, Project project = Project( project_id='your-project-id', secret='your-secret' ) agent = project.agent('agent-id') def handle_message(message: IncomingChatMessage): if message.text == 'ping': agent.send( conversation_id=message.conversation_id, text="pong" ) else: agent.send( conversation_id=message.conversation_id, text="I did not understand" ) project.on_chat_message(handle_message) ``` {% endtab %} {% endtabs %} ### Init the connection So far we instantiated our project and agent instances. We also defined the behavior of the agent. Now, we need to initiate the connection between your server and ours. No worries, it's the simplest part. We just have to use the `project.connect()` method. {% hint style="info" %} Note in Python you will have to use the `project.wait()` in addition to `project.connect()`. {% endhint %} {% tabs %} {% tab title="Typescript" %} ```typescript import { Project } from 'agentlabs-sdk'; const project = new Project({ projectId: 'you-project-id', secret: 'your-secret', }); const agent = project.agent('agent-id'); project.onChatMessage((message) => { if (message.text === 'ping') { agent.send({ conversationId: message.conversationId, text: 'pong' }) } else { agent.send({ conversationId: message.conversationId, text: 'I did not get that' }) } }) project.connect() ``` {% endtab %} {% tab title="Python" %} ```python from agentlabs.chat import IncomingChatMessage, MessageFormat from agentlabs.project import Agent, Project project = Project( project_id='your-project-id', secret='your-secret' ) agent = project.agent('agent-id') def handle_message(message: IncomingChatMessage): if message.text == 'ping': agent.send( conversation_id=message.conversation_id, text="pong" ) else: agent.send( conversation_id=message.conversation_id, text="I did not understand" ) project.on_chat_message(handle_message) project.connect() project.wait(); ``` {% endtab %} {% endtabs %} ### :tada:Congrats You did it! Here's the result you must have :relaxed: {% embed url="" %} # ChatGPT with LangChain {% hint style="info" %} You can retrieve the [full example of this recipe here](https://github.com/agentlabs-inc/agent-examples/tree/main/python-examples/chatgpt-clone-no-memory-langchain) {% endhint %} ## What we're going to cook In this recipe, we will build a simple version of ChatGPT using the [LangChain](https://www.langchain.com/) framework and their ChatOpenAI model. To keep it simple, we won't add memory to this Chat Model. However, you will be able to find a full example with a basic memory in our [examples repository](https://github.com/agentlabs-inc/agent-examples/tree/main/python-examples). Here's the final result: {% embed url="" %} Final result of what you're going to build {% endembed %} ## Let's code ### Init AgentLabs First, we'll init the AgentLabs SDK, our agent and to open the connection with the server. {% tabs %} {% tab title="Python" %} 
from agentlabs.agent import Agent
from agentlabs.chat import IncomingChatMessage, MessageFormat
from agentlabs.project import Project

if __name__ == "__main__":
    env = parse_env_or_raise()
    project = Project(
            project_id=env.project_id,
            agentlabs_url=env.agentlabs_url,
            secret=env.secret,
    )

    agent = project.agent(id=env.agent_id)

    project.connect()
    project.wait()
{% endtab %} {% endtabs %} {% hint style="info" %} Looking at the [full example](https://github.com/agentlabs-inc/agent-examples/tree/main/python-examples/chatgpt-clone-simple-memory-langchain), you will see we created a `parse_env_or_raise()` method. But you can handle the configuration variable the way you want. {% endhint %} {% hint style="warning" %} Don't forget your `OPENAI_API_KEY` environment variable if you want everything to work. {% endhint %} ### Prepare LangChain Then, we'll init [LangChain](https://www.langchain.com/) and the ChatOpenAI model.\ \ Let's import every dependency we need:
{% tabs %} {% tab title="Python" %} ```python from langchain.callbacks.base import BaseCallbackHandler from langchain.chat_models import ChatOpenAI from langchain.schema.messages import BaseMessage, HumanMessage, SystemMessage from langchain.schema.output import LLMResult ``` {% endtab %} {% endtabs %} Now we have imported our dependencies, let's init our model by adding the following line.
{% tabs %} {% tab title="Python" %} ```python llm = ChatOpenAI(streaming=True) ``` {% endtab %} {% endtabs %} {% hint style="info" %} Setting streaming to True allows us to get fragments of the response as they arrive and not wait for the entire response to be available. You can find more info about streaming in the [LangChain docs](https://python.langchain.com/docs/modules/model_io/models/chat/streaming). {% endhint %} Now, we'll create a class that extends the `BaseCallbackHandler` of [LangChain](https://www.langchain.com/) to handle the stream fragments as they arrive. What we want is to process every incoming stream and forward it to the client.
{% tabs %} {% tab title="Python" %} ```python class AgentLabsStreamingCallback(BaseCallbackHandler): def __init__(self, agent: Agent, conversation_id: str): super().__init__() self.agent = agent self.conversation_id = conversation_id def on_llm_start( self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any ) -> Any: self.stream = self.agent.create_stream(format=MessageFormat.MARKDOWN, conversation_id=self.conversation_id) def on_llm_new_token(self, token: str, **kwargs: Any) -> Any: self.stream.write(token) def on_llm_end(self, response: LLMResult, **kwargs: Any) -> Any: self.stream.end(); ``` {% endtab %} {% endtabs %} This handler is pretty straightforward: * On LLM start, we create a stream for our agent * When we receive a token, we stream it using our agent * On LLM end, we close the stream for our agent ### Handle incoming messages We're mostly done! We initiated AgentLabs and configured [LangChain](https://www.langchain.com/), now we need to handle the incoming messages. To do so, we'll use the `on_chat_message` method provided by AgentLabs. {% tabs %} {% tab title="Python" %} ```python if __name__ == "__main__": env = parse_env_or_raise() project = Project( project_id=env.project_id, agentlabs_url=env.agentlabs_url, secret=env.secret, ) llm = ChatOpenAI(streaming=True) agent = project.agent(id=env.agent_id) project.on_chat_message(handle_task) # ADDED THIS LINE project.connect() project.wait() ``` {% endtab %} {% endtabs %} This method takes a handler function as an argument. Let's define it. {% tabs %} {% tab title="Python" %} 
def handle_task(message: IncomingChatMessage):
    print(f"Handling message: {message.text} sent by {message.member_id}")
    messages: List[BaseMessage] = [
            SystemMessage(content="You are a general assistant designed to help people with their daily tasks. You should format your answers in markdown format as you see fit."),
            HumanMessage(content=message.text)
    ]
    callback = AgentLabsStreamingCallback(agent=agent, conversation_id=message.conversation_id)
    llm(messages, callbacks=[callback])
{% endtab %} {% endtabs %} In this function, we handle the incoming message from the user and them we pass it to the LLM. We also pass it a first message to give it some context so it knows how to handle the user's input. As a second argument, you can see we give it an instance of our callback class that we previously created. Now, every time a user sends a message, the LLM will receive it, and we'll stream the LLM responses back to the user. ### Et voilà :tada: Congrats, you created your own version of ChatGPT! \ \ You can retrieve the [full example of this recipe here](https://github.com/agentlabs-inc/agent-examples/tree/main/python-examples/chatgpt-clone-no-memory-langchain) :relaxed: ### # Code Interpreter How to build an AI capable of generating and running code {% hint style="info" %} You can retrieve the [full example of this recipe here](https://github.com/agentlabs-inc/agent-examples/tree/main/nodejs-examples/ts-code-interpretor-e2b) :pancakes: It was also inspired by [this tutorial](https://e2b.dev/docs/guide/simple-gpt4-code-interpreter) made by E2B. {% endhint %} ## What we're cooking today :cook: In this recipe, we will build a simple version of ChatGPT Code Interpreter using [OpenAI GPT-4](https://platform.openai.com/docs/guides/gpt) and [E2B](https://e2b.dev/). Here's the final result: {% embed url="" %} Example of code executor chat built with AgentLabs {% endembed %} ## What's the plan? \ Building a code interpreter is pretty straightforward; we'll proceed as follows: 1. We'll use the AgentLabs SDK to listen for user's input in the Chat. 2. We'll send the request to[ GPT-4 ](https://platform.openai.com/docs/guides/gpt)and ask it to generate some code snippets if asked by the user. 3. GPT is returning some code to run; then, we'll use [E2B](https://e2b.dev/) to run the code in a sandboxed cloud environment. 4. We'll use the AgentLabs SDK again to display the result to the user {% hint style="info" %} Using E2B is not mandatory, but it's a great solution to run code safely in a sandboxed environment, so we highly recommend you check this out. {% endhint %} ## Let's code ### Prepare the environment variables In this tutorial, we'll need many environment variables. If you look at our example, you'll see we verify every variable is present before running the app, but you're free to manage your environment variables as you want. We'll need the following variables: * AgentLabs Project ID, Agent ID, URL and Secret (that you can find on your console) * Open AI Api Key * E2B Api Key ### Handle user requests \ Once our variables are ready, we'll import and init the AgentLabs SDK. Here, we instantiate the project and the agent. We start listening for user messages using the `.onChatMessage()` method, we'll complete it later in this recipe. We don't forget to use the `.connect()` method to open the socket connection. {% tabs %} {% tab title="TypeScript" %} ```typescript import { Project } from "@agentlabs/node-sdk"; const project = new Project({ projectId: 'your-project-id', secret: 'your-secret', url: 'agentlabs-url' }); const agent = project.agent(agentId); project.onChatMessage((userMessage) => { // We'll fill it out later }); project.connect(); ``` {% endtab %} {% endtabs %} ### Talk to GPT-4 Whenever a user sends a message, we'll forward it to GPT-4 using the [ChatCompletionAPI](https://platform.openai.com/docs/guides/gpt/chat-completions-api). We expect GPT-4 to send us a response with potentially the code to execute. We'll provide GPT with two kinds of information: * The context: a list of fake history messages so GPT understands what we expect, plus in the end the message sent by the user. * The function calling config: we leverage the [function call API](https://platform.openai.com/docs/guides/gpt/function-calling) so ChatGPT will hopefully output some code we can directly execute in the sandbox. {% tabs %} {% tab title="TypeScript" %} ```typescript import OpenAI from 'openai'; // ...other parts // We define the options for the function call API. const functions = [ { name: 'exec_code', description: 'Executes the passed JavaScript code using Nodejs and returns the stdout and stderr', parameters: { type: 'object', properties: { code: { type: 'string', description: 'The JavaScript code to execute.', }, }, required: ['code'], }, }, ] project.onChatMessage(async (userMessage) => { const conversationId = userMessage.conversationId; // We call the ChatCompletionAPI with some context // and our function call API config const chatCompletion = await openai.chat.completions.create({ model: 'gpt-4', messages: [ { role: 'system', content: 'You are a senior developer that can code in JavaScript. Always produce valid JSON.', }, { role: 'user', content: 'Write hello world', }, { role: 'assistant', content: '{"code": "print("hello world")"}', name: 'exec_code', }, { role: 'user', // Here, we send the message of the user content: userMessage.text, } ], functions, }); }); project.connect(); ``` {% endtab %} {% endtabs %} {% hint style="warning" %} Don't forget the OpenAI SDK needs an environment variable named `OPENAI_API_KEY` to be set. {% endhint %} ### ### Parsing GPT's response If you look at the chatCompletion result, you will see it contains something like this:
```json { ... "choices": [ { "index": 0, "message": { "role": "assistant", "content": null, "function_call": { "name": "exec_code", "arguments": "{\n \"code\": \"\n console.log('something') \n \"\n}" } }, "finish_reason": "function_call" } ], ... } ``` From this result we want to: * Know if the `choices[0].function_call.name` equals `exec_code.` If yes, it means we have to execute the code. * Parse the arguments of `choices[0].function_call.arguments` in order to extract the value of the code to execute. {% tabs %} {% tab title="TypeScript" %} ```typescript // Inside the onChatMessage handler const message = chatCompletion.choices[0].message; const func = message["function_call"]; // If function_call is undefined, we don't handle this. if (!func) { return; } const funcName = func.name; // Get rid of newlines and leading/trailing spaces in the raw function arguments JSON string. // This sometimes help to avoid JSON parsing errors. let args = func["arguments"]; args = args.trim().replace(/\n|\r/g, ""); // Parse the cleaned up JSON string. const funcArgs = JSON.parse(args); // We don't handle that case in this example if (funcName !== "exec_code") { return; } // Here we have our code ready to be executed const code = funcArgs["code"]; ``` {% endtab %} {% endtabs %} ### Executing the code Now we have extracted the code to run; we can use E2B to execute it safely in a sandboxed environment. We will use their `process API` to start a new process that will run our code. Here's the plan: 1. Initiate an E2B session using `Session.create()` method. 2. Write the code we want to execute in a file named `index.js` in the sandbox. 3. Create a new process, running the `node index.js` command to run our code remotely. 4. We'll keep the `onStdout` and `onStderr` callbacks empty for now but we'll use them later. 5. We'll wait for the process execution to be completed. {% tabs %} {% tab title="TypeScript" %} ```typescript import { Session } from '@e2b/sdk' // The following part will still run inside the onChatMessage function. const session = await Session.create({ id: 'Nodejs', apiKey: e2bApiKey, }); await session.filesystem.write('/index.js', code); const proc = await session.process.start({ cmd: 'node /index.js', onStdout: (data) => { // we'll use it later }, onStderr: (data) => { // we'll use it later } }); await proc.finished; ``` {% endtab %} {% endtabs %} The code above will run our code until the execution is completed. Every output in Stdout or Stderr will trigger a call of one of the given callbacks. ### Sending results and feedback to the user in real time So far we're able to listen for messages, ask completions to OpenAI, and eventually execute the code. We still need to implement some real-time feedback to the users so they know what's going on, and they will view the output generated by the code interpreter. To do so, we'll leverage the `agent.createSteam()` method provided by AgentLabs. We can add some feedback whenever we want. Let's create a stream if the ChatCompletion contains a function\_call.
{% tabs %} {% tab title="TypeScript" %} ```typescript const func = message["function_call"]; // If function_call is undefined, we don't handle this. if (!func) { return; } // We open a stream const stream = agent.createStream({ conversationId, }, { format: 'Markdown', }); ``` {% endtab %} {% endtabs %} Here, we open a stream channel, and we indicate we'll send some Markdown, so the Chat UI knows how to display the code output. Now we can use `stream.write` to write on the chat every time we need to. For example, we can indicate we received some code to execute. {% tabs %} {% tab title="TypeScript" %} ```typescript // This is the code we wrote before let args = func["arguments"]; args = args.trim().replace(/\n|\r/g, ""); const funcArgs = JSON.parse(args); stream.write(`Here is the code I have to execute:\n`); stream.write(`\`\`\`js\n${funcArgs["code"]}\n\`\`\`\n\n`); ``` {% endtab %} {% endtabs %} {% hint style="info" %} Note we stream in markdown format, so we use \`\`\` in front of the code so it's shown in a code block. {% endhint %} Then, we want to notify the user we'll start executing the code, and crucially we want to output the result of the code interpreter in real time. To do so, we'll write to our stream before executing the code, and every time the `onStdout` or `onStderr` callbacks are called. And once we're done, we'll `end` the stream. {% tabs %} {% tab title="TypeScript" %} ```typescript await session.filesystem.write('/index.js', code); stream.write(`Executing the code...\n\n`); // This is to open a code block stream.write(`\`\`\`\n`); const proc = await session.process.start({ cmd: 'node /index.js', onStdout: (data) => { stream.write(data.line + '\n'); }, onStderr: (data) => { stream.write(data.line + '\n'); } }); await proc.finished; // We close the code block stream.write(`\n\`\`\`\n`); stream.end(); ``` {% endtab %} {% endtabs %} ### Et voilà :tada: Congrats, you just rebuilt your own code interpreter! Feel free to check out [the full example here](https://github.com/agentlabs-inc/agent-examples/tree/main/nodejs-examples/ts-code-interpretor-e2b). # Mutli-Agent with AutoGen {% hint style="info" %} This recipe is cooking :rice\_cracker: {% endhint %} # Github Everything that we build is open-source. Any PR is welcome. Even if you don't want to ship code, giving a start :star: will help us a lot. Join on on Github now :arrow\_down: # Discord You can join our Discord server here to Chat with other people crafting amazing agents.\ \ Join our community :arrow\_down: # Support team The easiest way to contact us is probably to join our Discord, but if you prefer sending a request by email, you can reach out to us using `support[at]agentlabs.dev` and we'll be happy to help.