Local AI for the organisations that cannot use the cloud
What runs on premises now, what does not yet, and how to tell which you are being sold. Written from a cluster that has run continuously for over a year.
Some organisations cannot send their data to a cloud AI service, and they have good reasons. A school holds pupil records. A regulated business holds client files under obligations that were written long before cloud services existed and that still apply to them. Many organisations settled their confidentiality position years ago and have never had cause to revisit it, and that position does not permit the routine transmission of internal documents to a third party. For all of them the question is not which AI product to buy but whether AI can be run inside the building at all, and what would be given up by doing so.
We can answer that question from experience rather than from a brochure, because the company runs its own system. Our cluster is four machines, on our own premises, working together as a single system, and it has run continuously for over a year. It performs inference, which is the step in which a model actually produces an answer; it searches and answers questions over our own documents; it does the work of an assistant with a memory that the operator controls; and it enforces the controls that decide what a model is permitted to act on. Nothing leaves our network unless we have decided that it may. What follows is drawn from running that system rather than from reading about one.
What runs on premises today
More than most buyers are led to believe. Models whose weights are openly published, up to around 120 billion parameters, will run on hardware that an organisation can buy, own and house. At that scale the hardware is a set of workstations and small servers rather than a data centre, and it will sit in a locked cupboard.
Retrieval over private documents runs locally. That means a system that indexes your own files, answers questions about them, and shows you the passage it relied on, so that an answer can be checked against its source rather than taken on trust. An assistant with memory runs locally, with that memory held on your own machines and under your operator’s control, so that what the system remembers and what it forgets is a decision your organisation makes rather than a default a vendor has set.
The controls run locally as well, and in our experience they matter more than the choice of model. A gateway decides what the model may act on. Permission tiers separate reading from doing, so that a model which may read a record cannot also change it without a separate permission. Every action is logged, and the log is kept where you can read it. Without those three things a local model is simply a private way of making the same mistakes a cloud one would make.
For record-heavy work, which is most of the work in most organisations, this is sufficient: drafting, summarising, classifying, finding documents, and answering questions about your own material. To give one example, a school office can ask which letters went to a particular family last term and receive the answer with each letter shown, without any of those letters leaving the building. It is the kind of work our own cluster has been doing every day for over a year.
What does not run locally yet
A buyer should be told this part directly, because it is the part that a sales conversation tends to leave out.
The strongest hosted models are still ahead on some tasks. On the hardest reasoning problems and on long, open-ended work, the largest models available through the major providers remain better than anything that runs on a workstation. Whether that gap matters depends entirely on the work. For record-heavy tasks it rarely does. For research at the edge of what these systems can do, it can.
Work involving images and speech is thinner locally. Reading images, transcribing speech and generating pictures can all be done on premises, but the quality and the range are behind the hosted services, and the gap closes unevenly from one month to the next.
Anything a vendor keeps behind its own interface cannot be run locally at all, whatever the label on the product says. If the model is only available as a service, the data goes to the service. That is not a criticism of the vendor; it is a statement about where the computation happens.
How to tell what you are being sold
“On premises” has become a label rather than a description, and a number of products carry the label while sending the actual work to a cloud service. Five questions separate a local deployment from a hosted one with a local badge, and a vendor who is selling the former will be glad to answer them.
Where does inference run? Inference is the moment at which the model produces its answer. If that happens on a machine in your building, the deployment is local. If the machine in your building sends the request elsewhere and displays the reply, what you have is a terminal.
What leaves the network? The honest answer is a list: either nothing, or these specific things for these specific purposes. Telemetry, usage statistics, crash reports and licence checks all count. Ask for the list in writing, and ask how you would know if it changed.
Is there telemetry, and can it be turned off? Many products report home by default. Ask whether the product keeps working with reporting disabled, and whether the vendor will confirm that in the contract.
Who holds the model weights? The weights are the model itself, the large file of numbers that the software runs. If they are on your machines, you can run the system without the vendor. If the vendor holds them and licenses your access to them, the deployment is local only for as long as the licence is.
What happens when the vendor goes? Insolvency, acquisition and a change of strategy are all ordinary events in this market. A local system should carry on running on the day the vendor disappears. If it would stop, the system was never under your control, whatever the label said.
Two phrases deserve a careful reading. “Private cloud” usually means someone else’s cloud with a partition in it. “Your data is not used for training” is a promise about use rather than about location; the data still travels.
What it costs
Running AI on your own premises is a capital purchase rather than a subscription, and it carries three costs that a subscription conceals.
The first is hardware. Machines with the memory and the graphics capacity to run useful models are a real purchase, though of the kind an organisation can buy, house and insure like any other equipment. They also age, and the models improve faster than the machines do, so the purchase will be repeated.
The second is electricity. A machine running a model under load draws several hundred watts, sometimes more, for as long as it is working, and that shows on the bill. Our cluster is metered and we know what it costs to run; an organisation that is serious about this should meter its own.
The third is an operator, and it is the cost most often overlooked. Someone on your side has to be able to restart the system, update it, read its logs and recognise when something is wrong. We train that person as part of every deployment and leave an operator guide that has been checked against the system, because a system that only we can run is not a system you own.
Against those three costs sits what a subscription would cost over the same number of years, and, for the organisations this piece is written for, the cost of a data boundary they could not otherwise draw.
Where to start
Not with hardware. Start by writing down the boundary: which information may leave the organisation, which may not, and who decides. Most organisations have never written it down, and a local deployment is only justified where the boundary says the data has to stay. The boundary tells you what to build, the five questions above tell you what you are being offered, and a year of running our own system is what tells us, every day, what actually works.
Written by the people who do the work at EGM AI. All Insights · Write to the company