AI Observability and Self-Driving Bug Fixes at PostHog

Hi, everyone. My name is radu. I work at STT. I'm a product engineer. I've been at PostHog for about a year. I lead the AI observability team where we build tools to observe and to test your AI in production. But before I show you a bit more of that, who knows what PostHog does? Can you please raise raise your hand? You don't count. So 2 people. Great. So it's perfect because now I can tell you what we do. So STT is a remote company, and our Fillion, it started out as a product analytics company. So, analytics for you to understand what your users do, and it became a company that wants to build every single dev tool, that you need to make a successful product. And so we try literally to build them all. So we have obviously product analytics. Can you read? You can't really read. Yep. We have obviously product analytics. We have air tricks, logs, session replay, which gives you a video of everything your News saw and did with their Matter movements. We have AI observability, evals for AI, feature tricks, experiments, and so on and so on. We are a open source company, and that means that all of our code Date is open source. So you can also, self host this, and you can also contribute to City. But we obviously also have a managed solution. And when I say open source, I'm I truly mean it. We're radically open source because our whole company is open source, and that means that we don't have a confluence or internal documentation everything's on our website and you can go see how we operate so that means how we hire people how we fire people our salaries our secrets they're all here and you can also ask your LLM your agent because it's it's been trained on this it's public AI? AI News going back to what I do like I said AI team is in charge of building tools for you to observe your AI STT you can see every single Jun call, how much your users are costing, and so on. But we also build evals. Evals are tests for your LLMs because LLMs cannot be, tested, you know, deterministically anymore, at least for the most part. And so Doors example, Home, we have online evals. And these are evaluations that you run on live traces. So if you have users that are having conversations with, let's say, a chatbot, you track all these conversations Orin post talk, and then you can run online evals on those conversations and see test whatever you wanna STT. For example, did they say bad things to the user? Did they, use the wrong the the wrong words Doors example? Did they hallucinate Jun so on? News, this will help you catch bad things that are happening in your agent or your AI features. STT LLM as a whole, like I said Hall many products Jun you can see Doors example if you have an exception in your Location. It's all tracked here. Logs, customer support tricks. And as you grow bigger and bigger and have more users, you don't really have time to look at every single 1 of these insights that can provide you with an insight when something goes E4B, for example, an error or a customer support ticket. And so it happened to us at PostHog as well, and we decided to create this thing that we called the signals pipeline. A signal being a piece of information that could represent potentially something that is bad and should be improved. For example, a failing eval, an exception, a customer support ticket. So let me show you the end result of what the signal pipeline does, and then I'll show you exactly how we built it. So I have an example here. Let's see if it if you can see anything. So this is on Slack. This is from a week and a half ago. So I woke up with getting this notification, as you can see Date 5AM, saying that our clusters feature has a bug and that users cannot see clusters that are older in 7 days TTS telling me that 4.7 k distinct users are being affected by it including a high priority onboarding customer Jun then it went ahead and also created a PR for the fix with the fix. Now I didn't like the tricks, so I talk the agent to it was being a bit too aggressive. You know, sometimes they try to fix too much. And so I told it that I want some changes. And then at the end, it made these changes. I approved the PR. We shipped it to production, and the bug was fixed. This would have taken us much longer to notice that we have a bug And then also to actually prioritize it Jun then to fix it. This was great because I woke up in the morning Android was already fixed for me. I just had to review the code. Alright. Now, like I said, we have all the tools or we try to have all the tools for us for you to build a successful product, and that obviously includes a coding harness. So something like codex, lead code, open code, and so on. This is the same problem that we saw Orin Slack in the coding harness. And you don't need to read all the details. At least you should be able to see the titles. Here's a summary of the this is what we saw on MAC. And here's the more in-depth summary of what's happening a root cause analysis and how to resolve City. But more interestingly here Qween have evidence evidence of this bug happening Qween have 2 pieces of evidence 1 of them is a failing evaluation so we have an agent where a customer or multiple customers can ask questions and some of the customers were complaining that this is a bug, and then we have another piece of evidence which was a support ticket from a STT. Like, you saw that was mentioned, high file onboarding customer. These are 2 pieces of evidence that were grouped together because somehow, and I'll explain how after, they represent the same problem that your customer is having. And then they were grouped together, and I guess when it's zoomed in like this, I can't see the bottom of the right side. Let me just refresh. That doesn't matter. Here, there's a UI bug, obviously. Maybe they moved City, and then we we have a new update. But there was a research agent that ran, to find this root cause and now to make this root cause analysis and how to resolve the bug. And then there was an implementation agent that actually looked at this, looked at your code, and then made a PR with a fix. Now lead look at how this actually works in the background. Like I said earlier, we have many pieces of data scattered in different Matter, support tickets, exceptions, evaluations failing, and we call them signals. Now these signals are emitted into the signals pipeline via 2 ways. There's a can you see? No. There's a push and a pull way. So the 1st 1 is the push way. So we have a function in the Home, which you can call, like STT, give it a bunch of data. This is the example that led to the eval the failing evaluation to be, tracked as a signal. And this is great. This is our main way, but the problem is if you don't actually make this call or you don't make it in the right time, then you might be missing some bugs that are deploying, and you're just not tracking them. So this is why we made a 2nd approach to emit signals, which are scouts. These are AI agents that scout your code, scout your data, and find signal for you. So this is on a call approach. Now once these signals are emitted in the signals pipeline, there are 2 main workflows. I I AI this a bit for the presentation, but there are 2 big workflows. 1st is to group related signals together AI you saw the evaluation and the customer support tickets representing the same error. It was grouped together. And the 2nd flow is to actually run the research agent and the implementation agent that fixes that. So let's look at the 1st step. So when a new signal comes into this pipeline, we look at that organization's other signals that they have ingested, to get an example from every single different type of signal. We do this because when they come from different sources, they have different formats. And so in order for us to group them together, we need to have kinda speak the same language. We take an example from each AI, so an example from Abror tracking, an example for STT support ticket, and so on. And we make an LLM call to generate 1 to 3 natural language queries that will help us find signals related to this new signal that we are ingesting. Once we get these 3 queries, we LLM 1 to 3 queries, we embed these queries, in vector space, and we do a semantic search against all the previously ingested signals that you have for your organization. And so the result of that is sorted by, the lowest cosine distance, which basically means that we can return, let's say, here are the top 10 previously ingested signals that are possibly related to this new signal that you're ingesting. Then we make another LLM call asking is this new signal related to any of these top signals that we have in the STT? And it either says yes or no. Now these groupings that we have of signals that are related together are called signal reports. So it's a report with multiple signals. So if the new signal that we're ingesting is related to any of them, we put it in that signal report. If it's not, well, we create a new signal report for that particular signal. Each report has a weight which mostly represents do we have enough data for us to actually start a research and fix City. We update that weight Jun now we're done with the 1st workflow which is the grouping. Now if that signal report is past a certain threshold of weight, then we have enough data and we can start researching it and fixing it. If it doesn't, then we just wait for more signals to be to be ingested. Now let's say it Doors. And so we promote the signal report Jun we go to the 2nd workflow. We make an AI call to detect any prompt injection attempt or some other security issues because these signals are coming from error tracking Date. So this is stuff that people can try to inject, bad prompts. Then we select which repositories to act upon. So sometimes you have 1 repository with a big Location. Sometimes you have multiple. So perhaps your fix needs to happen in Multi repositories. Then we run that research agent that generates this big root cause analysis and and how to resolve City. And then it also outputs a state whether it's actionable or not or if it needs human input. If it's not actionable, we reset the Date, start this whole process Owner. We're just waiting for more signals for us to, consider it actionable. It could require human input, which is usually when we need to make a product design. Or if it's actionable, then we run the implementation agent that outputs the PR. We think that this is, a flow that will change considerably how we work as software engineers. Jun we have been using this for months internally, and we're launching it next week. You don't need to use it, but you can get inspired in your own company or own application, to replicate this. Like I said, it's all open source. All this code is open source, and we'd love to hear your feedback for it. On this QR code, you can it's gonna take you to this website, which is going to explain this whole vision that we have. We call it self driving, because it finds the bugs Android also fixes them for you. And we think you're gonna be able to concentrate on the fun stuff instead of fixing bug reports. If you have any questions, I'm here. You can send me an email. You can ask me now. We also have a Discord channel. Thank you for listening.