Case study · For builders

Building a private contract analyst on one GPU: accuracy, cost and code

If your customers won't let their documents go to a public AI service, can you still build the feature? We reviewed 225 contracts with an open model on a single RTX 5090: under a cent each, every answer backed by a quote we could check, then improved without any training. The code is open.

ComputeNexaOctober 2026Model: Qwen 2.5 14B Instruct (4-bit) GPU: one RTX 5090

Who this is for

Teams who build document AI: startups making a review tool for law firms, insurers or HR; agencies building for clients whose contracts forbid third-party AI; in-house teams whose legal department said no to public chatbots. Your customers want a finished product. You want to know if a private version is good enough, what it costs to run, and how to improve it. That's what this measures.

What we set out to test

The alternative to a public AI service is an open model on a GPU only you use, in the country you choose. We tested four claims:

We used contracts as the example because they're sensitive, long, and have public expert-labelled test sets. The same setup applies to any document review.

The setup

• Model: Qwen 2.5 14B Instruct, 4-bit (AWQ), Apache 2.0. Served by vLLM through an OpenAI-compatible API, so the analyst is ordinary Python calling /v1/chat/completions.
• GPU: one RTX 5090 (32 GB), launched as a Model API on ComputeNexa and stopped afterwards.
• Method: the contract is split into large parts (about 10,000 tokens each); the model fills in a fixed list of fields as JSON. Every answer must come with a quote from the contract, and we check the quote is really there. An answer whose quote can't be found doesn't count.

The test

• Commercial contracts: CUAD's official test set, 102 contracts from public SEC filings, annotated by lawyers. We score 19 fields: parties, dates, renewal, governing law, and risk flags such as exclusivity, non-compete, change of control and uncapped liability.
• NDAs: ContractNLI's test set, 123 NDAs. For each of 17 statements ("the receiving party may share information with its employees") the answer is yes, no or not mentioned.
• Fair play: instructions were developed on the separate dev set (61 NDAs) and checked against the training labels, never tuned on test. We looked at 30 test NDAs during an earlier pilot, so we also report the 93 never looked at.

Results

NDAs (ContractNLI, 123 test NDAs)

On the dev set the same change moved accuracy from 66.9% to 74.2%. Accuracy is the average over the 17 statements. One NDA in the guided run failed (the model's JSON answer was malformed three times) and counts as wrong. The reference column is Claude Sonnet 5.5 through Anthropic's API, given the same plain questions, as a yardstick (it doesn't accept temperature 0, so its answers can vary slightly between runs; three documents hit a setup error on our side and were re-run). The documents are public, so sending them to an API is fine here.

Commercial contracts (CUAD, 102 test contracts)

Where the base model goes wrong

Most NDA mistakes came from three habits, not random errors:

1. It reads "not allowed, except…" as "no"

"…shall not disclose to any third party, except its professional advisers…"
Question: may the receiving party share with third parties? Lawyers' answer: yes (some sharing is allowed). Model: no.

2. It misses the standard list of what isn't confidential

Almost every NDA excludes information "received from a third party without restriction". The model found the "independently developed" item in that list but rarely connected the "third party" item to its question.

3. It doesn't know the dataset's conventions

ContractNLI counts "return or destroy" only when it's triggered by the end of the agreement; a return on request counts as not mentioned. A human might read it the model's way, but every firm has conventions like this, and teaching them is the point.

Better instructions: one reading rule per question

We added one plain sentence per question ("an exception that allows some sharing means yes") and checked each rule against the training labels. On the dev set accuracy rose from 66.9% to 74.2%; on the test set from 67.1% to 71.8%. The test gain is smaller than on dev, which is normal: dev is where the rules were written. No training, no extra cost.

Private versus a frontier API

Hosted cost: $8.34 for both test sets on Anthropic's usage page, split by the amount of text in each. Private cost: GPU time on one RTX 5090 at $1.49 an hour.

Two very different pictures. On commercial contracts the private model is within 2.4 points of a frontier model, at about a twenty-seventh of the cost. For risk-flag review, private is already a strong choice. On NDAs the hosted model leads by 19 points: NDA questions hinge on careful reading ("not allowed, except…"), where a much larger model shines. Better instructions closed about a quarter of that gap, at roughly a ninth of the hosted cost.

Part of the gap is discipline rather than understanding. The private model paraphrased its quote in about 150 NDA answers, which our check then rejected; the hosted model quoted word for word every time. Quoting exactly is the kind of habit fine-tuning teaches well, and that's the next experiment.

Why the NDA gap is so large

Commercial-contract questions are mostly "is there a clause about X?": find it and quote it. NDA questions ask for an interpretation ("may the receiving party share with its advisers?"), often hinging on an exception in one sentence and a definition elsewhere, with three possible answers. That's where size matters. We broke the 14.5-point gap (private with better instructions, 71.8%, against 86.3%) into its parts:

• About 4 points are quoting, not understanding. If we accepted the private model's answers even when its quote was reworded, it would score 75.7%. The hosted model never reworded a quote.
• About 6 points come from three of the 17 questions, all about exceptions and lists:

Where the private model was wrong and the hosted model right, the mistakes were: saying yes when the NDA doesn't address the point (120 times), the right answer with a reworded quote (96), reading "not allowed, except…" as no (65), and missing a clause that was there (86).

Two caveats keep this in proportion. Some questions are hard for every model because of the dataset's own conventions: on "return or destroy" the hosted model scores 46%, the same as the private one. And instructions are blunt: they helped overall but made one question worse ("independent development" fell from 81% to 59%). Fine-tuning learns from labelled examples instead of rules, and it's well suited to the two biggest causes above: quoting exactly, and how these exception clauses should be read. Our next experiment measures how much of the gap that closes.

Planning numbers

One RTX 5090 at $1.49 an hour, four documents at a time, billed by the minute:

GPU time only; add your own storage and app costs. The 4-bit model scored within a point of the full-precision 14B on the dev set (66.9% vs 66.2%) at about a third of the cost per document.

How it's built

About 1,000 lines of Python: the field list (one file, so you can add your own document types), the prompts, the quote check, and a scorer for both datasets. It calls the model through the standard openai client, so the same code runs against your own vLLM, a ComputeNexa Model API, or a hosted API for comparison.

Where this leaves the four claims

• Good enough to be useful: on commercial contracts, yes: 88% on risk flags, 2.4 points behind a frontier model. On NDAs, partly: 71.8%, against 86.3% for the frontier model. Either way, as a first pass that a person checks.
• Cheap and fast: 3–4 seconds and about $1.50 per 1,000 NDAs.
• Checkable: every answer comes with a verified quote, and rewording was caught rather than shown.
• Better by yourself: +4.7 points from instructions alone. Next, we'll fine-tune the model on ContractNLI's 423 training NDAs, so it learns these conventions from examples, and measure the same test set again.