One Mac on my home network does nothing except run language models. It answers on a local address, speaks the same API the cloud services do, and nothing I send it leaves the house.
I use cloud models every day and they are better than anything I can run myself. So the local box has to earn its place, and it does so in three ways.
The first is privacy that does not depend on a policy. Some of what I build reads my own financial data. I am comfortable sending a coding question to a vendor. I am less comfortable sending account balances and transaction history, whatever the terms of service say. A model on my own machine removes the question.
The second is cost that does not scale with curiosity. Batch jobs that would run up a bill, such as summarizing a long archive or classifying a few thousand records, cost electricity and patience when they run locally.
The third is the one I value most: I learn how the thing works. A hosted model is a text box. A local one makes you choose a model size, a quantization level and a context length, and then shows you what each choice costs in memory and speed. You find out quickly that context is not free and that a model which fits in memory is not the same as a model that answers well.
The limits are real. Smaller models lose the thread on long tasks. Calling tools, where the model has to produce a structured request for a program to run, was the first thing to break in my setup and the last thing to become reliable. For agent work I still reach for the large hosted models.
There is one design rule I had to bend. Everything else I run at home lives in containers, so that no project can leave runtimes and libraries scattered across the machine. Inference is the exception, because macOS does not pass the GPU through to a container. The model server therefore runs directly on the host, under its own account, on a machine I do not use for anything personal. If that server is ever compromised, there is nothing else on the box to find.
After thirty years of telling organizations to know where their data goes, running my own models is the same advice applied at home. Use the cloud for what it is best at, and keep a place where the answer to “who else saw this?” is nobody.