Sarathy & me!

Hello. Yeah. So, myself, Vishwa. So I'm here to talk about my open source project that I built for myself called Sarathi, which is, an open class jail assistant that I just built for myself, and I'm here to share my, you know, experience, motivation, you know, what you can learn about it. So just a customer slide about who I am. So I'm working in, Dell, Canada as a distinguished tech leader. So my domain is completely on telecom, 5 g, network intelligence, etcetera. So I do have a website, and I have a YouTube channel. If you speak my language, Tamar, you might enjoy some tech content in Tamar. So let's start with, motivation. So fast forward a few months back when Peter Steinbarger introduced, open claw in, in the x side, it went crazy. And then, you know, lot lot more copycats came and, there is lot of clause, like, you know, nano cloud, pico cloud, like, any anything cloud would be there. And, as you might imagine, we thinkers, we want to jump something that is new and viral. Right? So I went ahead and downloaded the open plot. I tried to set things up. Some things worked, you know, out of the box. Like, you know, I was able to connect to Telegram and Bouchard. It worked. But then as soon as I started to use it mainstream, like, you know, creating some tasks or, you know, writing a cron job, it failed miserably. And, when I tried to learn something like how do I configure x versus y, the the documentation was not clear. And there were, like, too many competing projects, on the Internet. And, even if I tried to so I'm a developer. So naturally, I tried to understand the code. I tried to see what was what doesn't work if something failed. And it was simply not, possible for me to kind of keep up with all of the modules, you know, that that these personalized, chatbot sorry. These personalized, open classes and seems to have. And, because it became too popular, it also invited a lot of vulnerabilities. And, you know, every single night, there used to be some updates and, you know, used to be paranoid whether, you know, did I insult something that is vulnerable or should I keep on patching it and and stuff like that? And the main thing for me is that if I kind of don't understand how it works, I kind of don't like it because then it means that it's totally black box. I'm kind of blind to what's happening. You know, being a developer, it's it's a kind of side effect that we always want to know how things work that gives us extra confidence to use it. So along the same time, there used to be a I mean, there's still a project called, I think it is, nano claw, where the the user's ethos is that we use cloud code to kind of personalize the open claw such that, you know, it it, whatever that you don't know or you don't need, the he had the author actually asked Cloud Code to kind of remove the code and then kind of personalized it, to just his, way of working. And which also seems to like it, and then he kind of popularized it. So I want to I want to experiment that. So I know Python a lot. So I kind of, for the nanobot, which is the, you know, Python implementation of, OpenClark. And then I started then removing the stuff that I don't need and then, you know, copying the stuff that I like within OpenClaw and, another, you know, another claw like, project called Hermes. Hermes is getting popular right now. So if you are totally new to this, you have you have no idea what is OpenClaw Hermes. I would recommend to start with Hermes because it is much more manageable than OpenClub. So I forked the nanobot, and then I removed stuff I don't want. For example, I'm not going to use, WeChat. I'm not going to use Slack. So I removed all of those stuff. And then I like certain aspects like, you know, automatic skill building. And then, you know, things like, when you create a skill file, it automatically open automatically reloads so that, you don't have to restart your gateway, etcetera. So it's like, an a la carte meal where I I kind of shop their own different competing projects and whichever stuff that I I like, I kind of took it. And then I created the for me. So this is the, project link which you can use, and then you could go over the stuff that I that I created for myself. But, again, if you look at the read me, I I made a clear statement that I'm not looking for contributors or I don't I don't care that, you know, something is there or not. I'm just building this for myself. Right? So I'm just sharing it on GitHub so that people might get inspired and motivated, and then they kind of fool my level and then customize it for themselves. But I'm not really looking for people to contribute or I'm not here to build something that other people like. It is 100% my own personal projects. And I believe that's how a personal assistant should be. Right? It it is dealing with your own sensitive user data. You don't wanna give it to some external company. At the same time, you don't want something that, that, that another person would need. Right? It should be 100% yours alone. Hence, it's called personal assistant. So I kind of went with that ethos. And then how I deployed it? So I have a a very old, you know, m 1 Mac studio with, 32 GB of RAM. So I'm a paranoid person, and I don't wanna give the complete host access to my, to my agent. At the same time, I don't wanna waste my time configuring the the do's and don'ts for my agent. So I kind of want the middle field where the agent is is behaving in the root access, so I don't have to worry about configuring something. At the same time, it is still sandbox so that it doesn't touch my host. Second of being with this approach where, on the main host, I have the coin model, 9,000,000,000 model with, 206 k context. I'll talk about the context length, next. And then I have a docker where I have Saradi running as 1 container with root access. So I I I just have to bring him up. Like, I don't have to configure anything. And then whichever host services that I want to provide. For example, I locally host the search sir, XNG, search engine. I have a local crawler for, call crawl for AI to do the crawling. Whatever I want to give the ability for the agent to use from my host, I kind of packed it as a container. And then using the docker host networking, I'm able to achieve this without Internet access. Right? The only thing that talks to Internet is my local host, which is having socket connection to, let's say, Telegram or mail or, Discord, but then the agent itself is completely unaware of it. It simply calls the local host, port of, LM studio or or Olauma. And then the model is able to talk as if it is it is sitting, you know, right next to next to each other, you know, within the host name space. So this way, I don't have to run tail scale or I don't have to open an SSH channel going through Internet. I'm still able to contact my agent on the go. At the same time, I'm not really doing anything, anything super, crazy or fancy that might involve lot of security bottleneck too. So I thought I'll just settle for this 1. So, this is the first contact that I sorry. Where I'm I was able to get it up and running, and, I have 3 channels. 1 is Discord, Telegram, and mail. All 3 are, working correctly. And once the first step is done, and I started to experiment, right, where I use Sardi to extend Sardi. Right? Because Sardi is also a coding agent. So for certain things like bug fixes or certain new features that doesn't, that doesn't involve greater design, I simply, you know, chat via Telegram. It updates its code. It restarts automatically and then comes back. So this part, I kind of copied from open clock. And then I once I have the search engine hooked up, I can I started asking, him to come to do a compare comparative analysis, you know, curated newsletters and even understanding the search paper in a TLTR format because it has it has now got access to the, mail? Because I own my own website domain, so I kind of created 1 other email domain for for Saturday, and it's hosted within my iCloud. So I don't wanna use Gmail, etcetera, because I wanted to have all of the agent data within myself, and I kind of went with Apple's ecosystem for this because I'm anyways using a lot of Apple devices. And whenever I kind of see new stuff, that is that is happening, on the Internet, for example, I copied the, head misses, auto, skill create. Sorry. Auto create skills, which is very useful because, you know, the way it works is that if I ask if I give a task to Saturday and then Saturday fails, I kind of instruct him, educate him how to do it. And And then every 10 prompts or so, they'll be under the system prompt that is running within salary to kind of go look back last 10 conversation. And then if the user has, helped you, instructed you to do something, then you try to create your own skill. That is how Hermes work. So I didn't copy Hermes's code. I didn't really clone Hermes. Instead, I simply asked Sarathi to go and check out that particular feature and then self create a skill for it. So all of this, Sarathy was able to do. You know, mind you, it is a 9,000,000,000 model. It is not it is not such a big model. But for its for its parameter, it is actually punching, you know, a higher weight than I expected. It was able to create all of these skills on demand without external code change just by prompting it to go and read the, GitHub read me and then try to create a skill on demand. Then I went with the LLM Wiki that, Bouchard you should all know that from Tabati where, you know, whenever I, I can I can show this? I have it here. So so whenever I put something on the inbox, it automatically starts a heartbeat job, and then it will ask the agent to parse the inbox and then create the put something on the raw folder, and then it will then try to create a synthesized Wiki. And it was also able to use, man made JS to kind of, you know, provide an analytics view of of what it is ingesting. And all of this is done locally by the same 9,000,000,000 model. Mhmm. Also happens to be my my assistant. Like, I'm not using, you know, multiple models or multiple assistants. It is the same framework that is that is that that that I'm trying to make it work for all of my needs, which is what I think the future, you know, version of assistant, should be. Right. Yeah. And finally, I also I'm also experimenting with the self general approach, because I wanted to kind of create a dataset for fine tuning. Let's say today, I'm using out of box model. So as I continue my history with Sarathy and then I ask Sarathy to kind of write a journal for himself everyday night at 11PM. And once they have enough data set, I might wanna tune the the base model to be even more personalized to me. I mean, that is the experiment I'm doing. I don't know how far I'm gonna get with it, but this is something that I'm doing currently. So in retrospect, not everything is processed because I experimented with sub agents. It failed, exclusively for my setup because, the the models cannot scale beyond, beyond a single turn approach. And then, the memory dot m d bloat is actually real, especially for smaller models, because even though I explicitly asked the agent not to bloat memory dot m d, keep it within 2,000 characters, you know, have a structure like, you know, user profile, Bouchard lessons learned, etcetera. All of this is not really, you know, practically possible with the 9,000,000,000 model. And, you know, because I'm a paranoid person, I'm actually, sandboxing my agent, but it also means that I'm actually limiting my agent by not providing more access. Right? So I'm kind of, experimenting with another project called Saree OS, which I will talk in the next slide, where I'm trying to see, like, how can I solve this approach? Like, I want to expose my host capabilities, but without giving it access to access. So so that is what I'm trying to experiment now. And, because I have this 32 GB of memory, I cannot do much. Right? Anything, I do is is kind of, defined by the kv cache length. And because I want longer context, I have to settle with anything, greater than 32 k, which means even though technically I can run a 24,000,000,000 parameter model, I am limited to use 9,000,000,000 model because I need to give more room for the k b cash for for my use case. So that is where I am. And, in future, I might wanna try with, local models, you know, try to use top of quant, you know, implementation whenever that is available, and then make SADA the OS work. So that is a queue to understand SADA the OS. So this is what I'm planning where I would run, an agent framework within the host, which would then safely expose the host capability like the file system, the mail, or even a coding agent. Like, I might have a coding agent like cloud code or open code on my host, which might have access to, let's say, s export and I want to, let's say, create an iOS app, then I might prompt it via Telegram that goes to Saudi, that goes to the host networking. And then there there has to be some kind of, safe interface like HTTPS or something or WebSocket without giving the shell access, but have some kind of a read or write or specific execute access where I might be able to control whatever that is there within my host. So that is again another experiment that I'm trying, and this is also I'm making it open source. So if you guys are interested, you know, you could you could check out. And, finally, I'm also doing and because this whiteboarding is kind of making, you know, everyone just create apps without much, much, needed development. I have also experimented some on device, you know, AI powered apps like, you know, French learning app called Appandra and, you know, on device, LLM, based VLM based, receipt scanner app, which you can find it here. And maybe in next, in some of the upcoming, you know, events, I might I might, talk about this if if there's an interest. But, yeah, that's pretty much I wanted to cover today.