The endpoint security sensor I've been building has an optional feature where an AI model looks at an alert and gives a second opinion: is this actually suspicious, or does it look like a false positive. It never blocks anything on its own, it just adds a plain language explanation next to the alert. You can point it at Anthropic, OpenAI, or Azure OpenAI, or you can run it fully local with Ollama so nothing about your machine's activity ever leaves it.
That last option is the one I care most about, and it's also the one where the choice of model actually matters. Cloud providers hide the model size from you. Local means picking one, and picking one for a security sensor comes with a real constraint: it runs quietly in the background on somebody's machine, and it can't compete with everything else running for memory and CPU.
At first I just grabbed the default, something in the 3 billion parameter range. Then I looked at what the model actually has to do here and swapped it for something smaller.
It isn't holding a conversation. It's reading a small, fixed set of facts, things like which process triggered the alert, what command line it ran, what file it touched, and producing one of a few possible verdicts along with a short explanation. That's a narrow classification task with some writing on top, not open ended reasoning. A model good enough to have a wide ranging conversation is more model than this job needs.
So I went smaller. Qwen2.5 at 1.5 billion parameters turned out to be a good fit, and not just because of size. It's Apache 2.0 licensed, which matters if you want to actually redistribute a product with a model attached rather than just use one yourself. The memory footprint is small enough to run comfortably even on a machine that's already tight on RAM, which describes plenty of real security appliances and older laptops. And for the specific job of reading structured facts and producing a structured verdict, it holds up fine. I tested it against a real alert my sensor generated, a PowerShell process launched with an execution policy bypass flag, and it came back with a correct, sensibly worded assessment in about a second once the model was warm.
The bigger lesson generalizes past this one feature. A lot of "we need an AI feature" problems in real products are actually narrow classification or extraction jobs wearing a chatbot costume. Once you notice that, you stop reaching for the biggest model you can afford and start asking what the smallest model is that can still do the specific, narrow thing you're asking of it. For alert triage, that turned out to be a lot smaller than I expected going in.
The Ollama configuration with the model choice lives in vk-ai-sensor's default config.