On-prem AI hardware: what to actually buy

The short answer

Buy one workstation with one graphics card that has 24 to 32 GB of memory. Everything else in the box is secondary. That single machine, new or used, lands around $1,500 to $4,000 and runs the mid-size open models that handle back-office work for a small team.

We run our own consulting work on exactly this: a used HP workstation with an eight-core processor, a 1,000 watt power supply, and workstation graphics cards with 32 GB each. It started with one card. The second one came later, when the first ran out of room, which is the order we recommend to everyone.

Before you buy anything, make sure you have the reason to be on-prem at all. If your data can go to a cloud provider, you probably want the cloud, and we lay out that decision in frontier vs private on-prem AI. This guide is for the businesses that have already decided the AI has to come to the data.

The one number that matters

An AI model has to fit inside the memory on the graphics card to run at a useful speed. Bigger models are smarter, and bigger models need more memory. So the memory on the card sets the ceiling on what your machine can ever become, and no amount of processor or system RAM raises it.

To make that concrete from our own setup: a model in the 27-billion-parameter class, compressed the way everyone runs them locally, takes roughly 15 to 18 GB of card memory before you add the working space it needs to read a long contract or a month of emails. That is why 24 GB is the floor and 32 GB is comfortable. Spend on memory first and let everything else follow.

Three configurations that make sense

One workstation, one card

This is the default, and for most small businesses it is the whole answer. A tower workstation, often bought used from the business refurb market, with one modern graphics card in the 24 to 32 GB range.

Your card choices come down to a short list. NVIDIA’s top consumer card, the GeForce RTX 5090, ships with 32 GB and is the path with the widest software support. NVIDIA’s previous-generation 24 GB cards are plentiful on the used market and still do the job. AMD’s Radeon AI PRO workstation cards also carry 32 GB, tend to cost less per gigabyte, and run the mainstream open-source tools well now, which is why they are what we bought. Either brand works. Memory and price should decide it.

This is the configuration behind the numbers in how much it costs to self-host an LLM, so read that for the full cash picture.

A small unified-memory box

The alternative is a compact machine where the processor and graphics share one big pool of memory. Apple’s Mac Studio can be configured with up to 512 GB of unified memory on the top model. NVIDIA’s DGX Spark packs 128 GB into a desktop the size of a book. Mini PCs built around AMD’s Ryzen AI Max chips offer up to 128 GB the same way.

The appeal is real: these boxes run models that would need two or three discrete cards, they are nearly silent, and they draw a fraction of the power. The trade is speed. Shared memory is slower than a dedicated card, and the gap is widest when the machine has to read a lot of text at once, which is most of what business AI does. A unified-memory box is the right call for one or two people in a quiet office, and the wrong call for a team of ten firing requests at it all day.

Two cards or a server

When one machine is not enough, the next step is a second card in the same workstation, which is what we did. Two 32 GB cards give you 64 GB, enough to run larger models or serve more people at once. Watch for the practical limits: the case needs physical room for two full-length cards, the power supply needs real headroom, and the motherboard needs two proper slots.

A dedicated server with four or more cards is the step after that. Almost no small business starts there, and you should treat anyone who quotes one on day one with suspicion. Start with one card. The machine will tell you when it is out of room, and you will have a year of real usage to size the upgrade against.

What not to buy

• A laptop. Even the expensive ones have too little graphics memory and throttle under sustained load. It will run a small model for demos and disappoint you the first real week.
• A gaming PC with a small card. Cards with 8 to 16 GB run small models that are useful for narrow jobs like tagging and sorting. They do not run the models that feel close to ChatGPT, and that is what people expect.
• An “AI PC” on the strength of its NPU. The neural chips in new laptops are built for small on-device features. They are not a substitute for a graphics card and will not run a business-grade model.
• A rack server without a graphics card. Plenty of businesses already own a server and assume it can take on AI. Without a card in it, it cannot, and most rack servers have neither the slot nor the power supply to add one.
• The biggest card just in case. Memory you do not use is money spent on the wrong year. A 32 GB card today, and a second one later if needed, beats one enormous card bought on speculation.

The rest of the machine

The card gets the budget. The rest of the box just needs to not get in the way.

• Processor: any recent eight-core chip. It loads the model and hands work to the card. It is not the bottleneck.
• System memory: at least as much as the card, ideally double. 64 GB is a sensible target. It gives the machine room to stage models and keeps a second card viable.
• Storage: a fast NVMe drive of 1 to 2 TB. Models are 10 to 20 GB apiece and you will keep several, plus whatever documents the machine indexes.
• Power supply: 850 to 1,000 watts for a single card, more for two. Modern cards draw several hundred watts under load and a marginal supply is the most common cause of mysterious crashes.
• Case and slots: a full-size tower with a proper x16 slot and room for a long, thick card. Used business workstations from HP, Dell, and Lenovo tick every box, which is why they are the sweet spot.
• Network: wired Ethernet. The machine serves the rest of the office, and Wi-Fi is the wrong link for that.
• Operating system: Linux, for the best support from the open-source AI tools. If the person maintaining it lives in Windows, that works too, at the cost of some friction.

Power, heat, and where it lives

Under load a single-card machine draws a few hundred watts. We cap our cards at 225 watts each, which costs a little speed and buys a lot of stability and quiet. Over a normal business month that comes to an electric bill in the range of ten to forty dollars at average commercial rates.

The machine needs airflow and a place nobody minds a fan. A closet with a door that closes and a vent that works is fine. A sealed cabinet is not. Put it on a small battery backup so a power blip does not corrupt a model download or a database in the middle of a write.

New or used

Buy the workstation used and the graphics card new, or from a seller with a real return window. Used business towers are cheap, well built, and come with the power and slots you need. Cards are the part that fails, that gets abused, and that carries the warranty worth having. Software is free: the models come from the open-weight libraries, and tools like llama.cpp and Ollama run them without a license fee.

What this means for you

Decide the on-prem question first, then buy in this order: the card with 24 to 32 GB of memory, a used tower that can power and hold it, 64 GB of system memory, a fast drive, and a battery backup. Skip the laptop, the small-card gaming PC, and the rack server without a card. Choose a Mac Studio or similar unified-memory box only if the machine has to be silent and will serve a couple of people.

Then run it for a year before buying anything else. The models will keep getting better for free, and the first machine will tell you exactly what the second one needs to be. Our private AI page covers what we set up and support, and if you want a straight answer on whether you need any of this yet, that is what the assessment is for.