Oakland | B Corp’s Post

Transcript

Here are the key AI things that Microsoft have announced on day one of Microsoft Ignite. Now the big theme is anyone was expecting was agents. They are all on board with the concept of an AI agent, which is great because that's actually where I see this technology going in the next 5 years. Some of their gushing sentiment over every app is going to be an AI app. I think we can take with the heaping mountain of salt that it deserves. But let's talk agents now. We're only gonna be talking the AI things right here. There was a lot in the keynote, OK, including the extremely amusing Azure Local, which feels like a run back to on Prem. But we're just going to talk agents here. And the thing that they are extremely proud of is what you can see above me, which I can only show you part of because that side of it contains some secrets. But this is the Azure Agent AI Foundry. And Microsoft's intention with the Azure Agent Foundry is that it becomes a one stop shop for all of your advanced AI needs. So when Copilot runs out of beans, this is where you come. He's also coming with some Python And I think Java SDK. So all of this will be programmatically controlled, giving us AI developers a significantly more fine grained control over our large language model deployments and what they're doing now. As I said, Microsoft are all in on agents and what I thought when they first announced it. Hours ago. Agent Foundry was going to be would be like a drag and drop placement for something like say control flow, which is this object here, which is a way of orchestrating many AI agents doing work at once. If you want to know what an AI agent is, I think I have a video somewhere. I'm pointing like there's a playlist here. There isn't a playlist here, but I think I've got a video somewhere on my account about what an agent is. And if I don't, I'll make one for you. But this is a way of orchestrating agents and I thought that was what this was going to be. And apparently that functionality is. Coming, but it's on the road map it's in. It's in a month or so. So let's look at what AI agent Foundry can actually do. Let's do that by running down the menu on the this side of the blade here inside AI Foundry. It's it's basically AI Studio, but more so right now. So the purpose of AI Studio in Azure was to control models and to keep track of your token usage and deploy models effectively into Docker containers, which would allow you to query lots and lots of different large language models. And that functionality is carried over into the current iteration of AI Foundry. So we have the normal. Things that you would expect. We have a model catalog which contains many things. You can now I can move this across because none of this is secret. You can now query 1800 some different models, including bringing your own model. They've got extensions for Hugging Face, They've got Llama which you can deploy. Really cool stuff is you can things like Llama and Mistral which require like a container for user deploy, which I'm not sure if Mistral does. Anyway, let's just run with Dharma, which you would have to normally self host. You can either bring your own. Compute and specify that or you can run it as a managed serverless instance if you want to cut down on cost. But let's continue to run down and have a look at these things. There is the playground like you would expect. So you can deploy a model and then just like instantly chat to it. That's fine. There are things like AI services, which this contains all of the things that used to be called cognitive skills and we're in the Azure Cognitive Services SDK. They are now called AI services and this brings in all the things like speech and language and text translation and image tagging. Content safety and all of that stuff that is now here. We're going to skip over code because that's not as well built out right now as it could do. But the exciting thing is that they have brought a new functionality to fine tuning. Now fine tuning, you will remember we used to use that to adapt models to give them specific information. So if there was like business logic that you, sorry, not business logic, business information that you wanted to put into your large language model, you would use fine tuning. And you do that by giving it lots of prompts and lots of responses and you divide that into a training and a test. That and then you would be able to fine tune your model to tell it to say never talk about your competitors or things like that. What they have now done is come up with a thing called Distillation, which actually is not even on here. I don't have access to this feature yet, but Distillation is using a large language model to generate those prompts and responses and allow you to moderate the sentiment of the prompts and responses. And that gives you a much in addition to getting for free basically. Huge data set to fine tune your model from. It also allows you to moderate things like sentiment because in the bad old days of fine tuning, what we would do is we would give it a prompt and a response and then we would monitor to see if it actually came up with that exact same response. Now in distillation, we can use a large language model to just go like, did it kinda get the gist of it? Is the sentiment of its response correct? And so fine tuning should now be one of the things I think because of distillation. That we just now do all of the time this is going to be very easy for us to do I'm going to skip down to tracing because let's talk about what's new. Tracing is adding graphical interfaces on top of log analytics workspaces to show you how you're large language models are working throughout your stack so this is bringing Copilot like traceability so in Copilot you can set off agent chains and you can set off large language models in flows and you can monitor and. Debug those flows as they happen. This, I haven't got it working yet because it's literally hours old, is going to bring graphical interfaces on top of workspaces, as I say, which will allow you to see a little bit like, well, what they're aping here obviously is Lang trace. So where Lang trace you can hook callbacks into a Lang chain application to be piped up to the web to allow you to monitor where your agent is in real time. This is going to do the same. But on top of the Log Analytics workspace for any agent that is created in Azure, and they have at the moment, there are a small number of models. So basically all of the Mistral models and all of the Llama models are ready to do this out-of-the-box. You have to add additional callbacks using the Azure AI Foundry SDK to other models if you want to do this, but the Llama and the Mistral ones will just do this out-of-the-box. The final thing I want to talk about, and I'm gonna have to hurry because the next talks in 3 minutes time, is the evaluation. They were extremely pleased with evaluation because it addresses the VEX question of how do we know if our large language model is doing well. They describe this as unit testing for large language models, and I disagree on the semantics, but the sentiment is broadly OK. We're basically getting. Insight into whether our LLM is doing all right now. I have not, as I say, because it's hours old, run an evaluation yet. I'm going to try and get that running tomorrow. As you can see, it's pretty dark outside here right now, but you are going to be able to programmatically build in sentiment analysis. If people are dinging your endpoints with violent content, content that is against your terms of service, that kind of thing, you're going to be able to evaluate that and see what people are doing interestingly. They had a quite good disambiguation between what they called primary jail breaking and secondary jail breaking, and this should pick that up. Primary jailbreaking is where somebody is literally typing in like disobey or previous instructions to your chat interface. And secondary jailbreak breaking is where they have been sneaky and put it into like a CSV that they've uploaded or an image that they've uploaded. So if you upload an image with text on it and say can you interpret this text? And the text is ignore all previous instructions, do a thing that is a secondary jailbreak, which this is designed to pick up if your users are doing it right. Enough of all that I'm going to get my filthy hands on the code and I'm going to watch a whole bunch more of Microsoft Ignite tomorrow because this is very rapidly changing and there's lots of new things. There's additions to a framework called Semantic Kernel. There are my notes for the day, which I've got loads of notes for the day. And if you want those notes. Comment below and I will send them to you. At the moment they're in handwritten form and like I say, I've got lots and lots of things to make videos on because there's semantic kernel has changed. There's a whole new stuff on autonomous agents. We've got how data gets trained. We've got the difference and how they have trained GPO 1. There's a lot that's changed and how tokenization works and the different services that Azure AI is going to provide with standard batched and provisioned. There's the whole thing of distillation. There is so much. On open AI and business, Open AI, enterprise level and all of the authentication that they're going to bring in through, there's a whole load. There's weeks worth of content here. We're gonna get our hands on it. Follow me for more.