Eric Chat: Offline Local AI

Eric is an open source Python package I just released that allows you to run large language models locally, securely, and offline on Macs with Apple Silicon. You can generate up to 70 tokens per second with a 117,000,000,000 parameter model right from your MacBook Pro. But before I talk about the technical details behind EricChat, let me just introduce a bit about myself and my background. In early 20 20, I launched a Python package called Happy Transformer, while I was studying computer engineering at Queens. This Python package has been downloaded 266,000 times and it abstracts fine tuning large language models. So with just a few lines of code, you can fine tune models for a variety of use cases. Within 1 day, I trained a model to correct spelling and grammar, and this model has been downloaded close to 20,000,000 times, 19,600,000 times. So with Happy Transformer, you can create cutting edge AI models, whether you're a beginner or an expert. But I wanted to do something more than just focus on a single topic within AI. I wanted to create an ecosystem of AI tools. And so I created the Eric ecosystem, which includes ERICsearch, which is a vector database that scales to millions of documents fast and efficiently. We have Eric Transformer for fine tuning, pretraining, inference and retrieval augmented generation. Both Eric and Eric Transformer use pure PyTorch for much of the code, so it's highly optimized. And then, of course, we have EricChat. Given it's an AI Tinkers presentation, I want to explain some of the technical details behind EricChat and how it's able to achieve over 70 tokens per second on a MacBook Pro. So Eric uses Eric Transformer, which then uses Apple's MLXLM Python package, which is built on Apple's MLX framework. Apple's MLX framework is an array framework similar to PyTorch or NumPy, but built specifically for Apple silicon. So it's software by Apple made specifically for Apple hardware, and I believe it has great performance. And long term, I trust that Apple's going to maintain it to achieve fast inference on Apple silicon. Then for the graphical user interface, I use Toga, which is a Python package that allows you to create native apps with pure Python. Here's the inference code behind it. Well, just a simple toy example. So from Eric transformer, we're importing Eric Chat MLX. We're then instantiating a model, calling the object we created, and that's all there is to performing inference with Eric Transformer. And this isn't the focus of the presentation, but, of course, my background is in training models. So let me just quickly show you an example of training a 20,000,000,000 parameter model on a single GPU. So with Air Transformer, you can train OpenAI's GPT OSS 20 b on 1 GPU, an h 200 or a b 200, with the following code. So instantiating a class, we're defining training arguments, we call a method dot train. We can save the model locally or push it to Hugging Face. That's all there is to training a model after you've formatted your data into a JSONL file. And Eric generates charts like this locally, so you don't have to connect to a service like WNB. These are generated locally as you train models with the Eric transformer. To use EricChat, you simply pip install it and then run these 2 lines of Python code and that will then open up the app like this. Here it is running live. Going to press submit. After asking it, tell me about Ottawa, the model's thinking, and it's now generating 83 per second with a 117,000,000,000 parameter model right from this MacBook Pro. But this MacBook Pro has 128 gigabytes of memory. Perhaps you just have a MacBook Air with 8 gigabytes. And if that's the case, you can still use Eric. Instead of using the 117,000,000,000 parameter model, you can instead use a 3,000,000,000 parameter model, which also works quite well. Let me just switch over to the 3,000,000,000 parameter model, and I'll clear the chat here. Here it is. 163 tokens per second right from this Mac without being connected to the Internet. Well, I'm connected to the Internet now, but you don't have to be connected to the Internet. It runs completely offline locally and securely. Okay. So in conclusion, I recommend you all give Eric a try if you have a Mac with Apple Fillion. But if you don't have a Mac, you can also consider building your own AI applications with Eric Transformer and Eric Search. And keep on tinkering and let me know if you have any questions.