A private LLM for law firms: what it does for privilege and confidentiality, and what it does not

The short answer

If your firm wants AI to read client files, the cleanest way to stay inside Rule 1.6 is to make sure the files never reach a third party. A private LLM does that. The model runs on a workstation in your office or in a cloud tenant that only your firm can reach, the client documents stay where they already live, and there is no vendor with terms of service standing between you and your own data.

That is the confidentiality problem, and it is most of what lawyers worry about when they ask this question. Privilege is a different problem, and a private LLM neither helps nor hurts it beyond removing one common way to lose it. The rest of this guide separates the two, walks through what the bar opinions actually require, and describes what a private setup looks like at a small firm without pretending it fixes everything.

Confidentiality and privilege are two different problems

They get said in one breath, so it is worth pulling them apart.

Confidentiality is an ethics duty. Model Rule 1.6, adopted in some form by every state, says a lawyer shall not reveal information relating to the representation and shall make reasonable efforts to prevent unauthorized disclosure of or access to it. It covers everything about a matter, not just what the client told you, and it applies whether or not anyone ever litigates over it. This is the rule a consumer chatbot runs into. If the terms let the vendor retain your inputs, have staff review them, or use them to train a model, then putting a client’s file into the tool is a disclosure to a third party, and the question becomes whether it was authorized and whether your efforts to prevent it were reasonable.

Privilege is a rule of evidence. It protects communications between a lawyer and a client made for the purpose of getting legal advice, and it can be waived by sharing the communication outside that circle. Work product protection covers the lawyer’s own notes and analysis prepared for litigation. Neither is created by a piece of software, and neither is waived by using one, unless using it amounts to disclosure. Federal Rule of Evidence 502 is the useful reference here: it says an inadvertent disclosure does not waive privilege when the holder took reasonable steps to prevent it and to fix it. A written AI policy, a tool that does not send data anywhere, and a record of both are what reasonable steps look like on paper.

So the honest framing is this. A private LLM keeps you on the right side of confidentiality because there is no third party. It keeps privilege intact for the same reason, but it adds nothing to privilege that you did not already have. Anyone selling you a private model as a privilege shield is overselling it.

What the bar opinions require

Since 2023 the guidance has become consistent across jurisdictions. The table below is a summary, and the opinions themselves are linked in the sources. Read the one for your state before you rely on any of this.

The Florida opinion is worth calling out because it draws the line this guide is built on. It notes that a lawyer using a third-party generative AI service faces a disclosure question that a lawyer using an in-house tool does not, while making clear the in-house tool still has to be secured. California’s guidance says the same thing from the other direction: do not input confidential client information into any tool that lacks adequate confidentiality and security protections, and anonymize what you can before you do.

Two rows in that table say “nothing,” and they matter. The reason lawyers get in trouble with AI is almost never a data leak. It is a brief with citations that do not exist. A private model does not fix that. Only reading the output does.

Where a private LLM helps

• No vendor in the chain. The data path is your document store to your model and back. There is no privacy policy to reread every quarter, no update to the terms that quietly changes what the vendor may do, and no support engineer who could see a prompt.
• Nothing is retained or trained on unless you decide it is. Prompts, outputs, and logs are firm records under your retention policy, not a vendor’s.
• Matters with restrictions become workable. A client whose outside counsel guidelines forbid third-party AI, a protective order that limits who may see a production, a matter where the other side is a technology company: the private setup lets you use the tool on those matters at all.
• Consent gets simpler. The ABA’s informed consent trigger is aimed at tools that learn from inputs or share them. A private model does neither, so for most matters the consent conversation collapses to your general engagement terms and your written policy. Client-specific restrictions still apply.
• Reasonable efforts are easy to show. If a disclosure question ever comes up, a machine that provably sends nothing anywhere is a short conversation.

Where it does not help

• The output is still the lawyer’s. Summaries can miss the one clause that matters. Drafts can assert facts nobody gave the model. Citations can be fiction. Rule 1.1 and Rule 3.3 are unchanged, and the review step is not optional.
• Inside the firm, walls still stand. A model that can search every matter file is a conflict-screen problem waiting to happen. Access has to follow the same rules as your document management system, so a screened lawyer cannot ask the model about the matter they are screened from.
• Security is now your job. A vendor with a SOC 2 audit and a signed agreement has done work you now have to do yourself: named accounts, encrypted disks, backups, patching, and a log of who asked what. If the box sits under a desk with a shared password, you have traded a confidentiality risk for a security one.
• Logs are discoverable in principle. A prompt and an answer about a litigation matter are firm records. Much of it will be work product, but decide up front what you keep, for how long, and who can see it, rather than discovering the answer during a production.
• A client can still say no. Private or not, if the engagement letter or outside counsel guidelines prohibit AI on the matter, that governs.

Three setups, ranked

Consumer chatbot on a personal account. Not for client work. The consumer terms of the major providers allow training on inputs unless the user opts out, and chat history is retained by default. This is the setup the bar opinions are warning about. Fine for drafting a generic client-facing FAQ, never for anything with a name in it. We cover how the tiers differ in is your business data safe with ChatGPT.

Business or API tier from a major provider. OpenAI, Anthropic, and the big clouds sell business plans and API access under terms that exclude customer content from training and let the customer control retention. Several bar opinions treat this as acceptable after due diligence: read the terms, document that you read them, confirm retention settings, and keep the practice of stripping names and details the task does not need. For a firm whose matters allow it, this is the practical middle path, and it is where most small firms should start. The plan question for OpenAI specifically is in ChatGPT Team vs Enterprise for a small business.

Private LLM on hardware you control. The model runs on a workstation in your office or in a tenant only your firm can reach. No third party, no terms, no retention outside your policy. This is the right answer when your matters include restricted material, when a client asks for it, or when the partners simply do not want client files leaving the building as a matter of policy. It costs more up front and you own the operations. We priced it in how much it costs to self-host an LLM and the general case for it is in frontier AI vs private on-prem AI.

Plenty of firms end up with both: the business tier for general drafting and the private model for the sensitive corner. That is a normal place to land.

What a private LLM looks like at a small firm

Concretely, one workstation with one professional graphics card holding 24 to 32 gigabytes of memory runs an open-weight model that is capable enough for the work below, for a firm of a few lawyers and staff. It sits in the office, on the office network, with no inbound access from outside. If the firm already runs a server closet, it goes there. We wrote up the shopping list in what on-prem AI hardware to actually buy.

What it is good at:

• Reading a deposition transcript or a document production and producing a summary, a timeline, or a list of every place a topic comes up
• First drafts of routine documents from the firm’s own templates and prior work, in the firm’s own voice
• Answering questions across the firm’s matter files, with the source document cited so the lawyer can check
• Comparing two versions of an agreement and explaining what changed
• Sorting intake email by matter type and drafting the acknowledgement

What it is not:

• A research tool. The model does not know current law. Citations come from a citator and get checked by a person.
• A decision maker. It drafts, a lawyer decides. That line is the whole point of the supervision rules.

The checklist before the first client file goes in

1. Write the policy. One page: approved tools, what may and may not go into each, who reviews output, retention for prompts and logs. Our small business AI policy template is a starting point; add the confidentiality and citation-check lines.
2. Map access to your conflict walls. Whatever controls who can open a matter in your document system controls who can ask the model about it.
3. Lock the box. Named accounts, encrypted disks, backups, patch schedule, and no remote access that bypasses the firm’s normal login. In Massachusetts, fold it into the written information security program that 201 CMR 17.00 already requires of any business holding residents’ personal information.
4. Decide retention. Keep prompts and outputs as long as the matter file and no longer, or keep nothing beyond the session. Write down which.
5. Update the engagement letter. A plain sentence that the firm uses AI tools that do not share client information with third parties, and that a lawyer reviews all AI-assisted work. Check your state’s opinion for what it expects here.
6. Train the team on the failure mode. Show them a fabricated citation. Then show them the citator.

Where this fits

The technology decision is the smaller half of this. The bigger half is deciding which matters the tool touches, who is responsible for the output, and what the policy says, and that is firm-by-firm work. If you want the private setup, local AI is what we build. If you want to know whether you need it at all, that is what the assessment is for.