Jev vs Qwen: Better Constituent Matching at 3% of the Cost

At Trellis, we sync donors, attendees, and purchasers into our customers' Blackbaud Raiser's Edge NXT CRM via our integration, that means every time someone buys a ticket, donation, bids on an auction item, or performs a number of other actions we have to answer one question before we can write anything: is this person already a constituent in this CRM?

Get that wrong in one direction, and a gift lands on a stranger's record because they happen to share a phone number. Get it wrong in the other direction, and the CRM fills up with duplicates that a development officer has to clean up by hand (and potentially pay extra for!). Neither is acceptable to a charity that lives and dies by the quality of its donor data.

Most of the time this is not a hard problem. Our deterministic matching system compares names, emails, phone numbers, and addresses and either finds an exact match or correctly concludes there is nobody to match. The hard cases are the leftovers: a candidate with the same name but a different email, two records that both look right, a nickname on one side and a legal name on the other. For those we hand the person and the candidate list to an AI model and ask it to decide.

For the last four months that model has been Qwen3 235B running on Amazon Bedrock. This post is about what happened when we replayed every one of those decisions through Jev, TypeSafe's flagship model and the first of what they call System One models.

Why we looked at Jev at all

Qwen was doing a decent job, but "decent" for a matching system means a few percentage points of decisions are wrong, and every wrong decision is a real person's data and more work for the already resource-constrained charities. We also had the usual pains of running a general purpose LLM for a structured task: a long (not too long mind you as we process thousands of these records) prompt full of rules, JSON we had to validate and retry, four second round trips, and a bill that scaled with LLM input/output token pricing (Qwen is a lot cheaper than the closed sourced frontier models, but still).

A System One model is a different shape of thing. It is not an LLM you prompt in the same way. You hand it a state (our person and the candidate constituents) and a set of typed questions, and it evaluates each question against that state and returns a typed answer with a probability distribution. For us that is a Choice question (which candidate, if any, is this person) and a Score question (how confident are you). No text generation, no hoping that the structured JSON requested gets parsed correctly from the LLMs output, less prompting to babysit, and each question is evaluated independently, so adding more of them does not degrade the others. That structure is what made me want to know if it could also make better decisions, not just cheaper ones.

How we tested it

I am not interested in a benchmark on made-up data. We have four months of real decisions tracked in our database for this exact reason, so we used those.

• 2,024 decisions. Every case where our deterministic matcher could not decide and Qwen was asked, from mid-May to mid-September 2026, replayed through Jev with exactly the candidate pool production saw at the time.
• 1,908 held out. 116 of the decisions were used while tuning the questions we ask Jev. The other 1,908 were never looked at until the final run, so the headline numbers are not the result of tuning to the test.
• Same definition of correct for both. The right answer is the record of the same real person among the candidates, or "no match" when there is none. When Qwen and Jev agreed, we assumed they were both right. When they disagreed, a deterministic classifier read the evidence (which identifiers each candidate actually shares, how the given names relate including short forms and preferred names) and bucketed the case. Every case that classifier charged against Jev, and every case it could not bucket, was then read by a human.

That last point matters. It would be straightforward to grade a new model by asking the new model. We did not do that.

Note: This may not be perfectly scientific and should not be taken as "Qwen bad, Jev good", or have these results took verbatim. There is still a lot of benchmarking and manual checking/tuning to be done before we switch our implementations in production. For now this serves as a great test case for a new type of model that I believe will have a huge number of use cases, including many more within Trellis.

The results

Qwen got 109 of 2,024 decisions wrong. Jev got 23 wrong, plus 31 "policy" cases explained below.

Where Qwen was wrong

• 80 times it said "ambiguous" when a candidate had the exact same name and an exact shared email, phone number or street address. These are the easy ones. Qwen's reasoning usually flagged a differing secondary phone or address and refused to commit. Jev linked them with confidence of 0.92 or higher.
• 15 times it created a duplicate when the record already existed with the exact name and an exact shared identifier. In a few cases there were already two identical records and Qwen added a third.
• 12 times it linked a relative. A spouse or child sharing the household email or phone, with a clearly different first name. This is the worst kind of error because it puts one person's gift on another person's record. Jev declined every one of them.
• 2 times it picked the weaker of two duplicate records, the one sharing only a phone number over the one sharing email and address.

Where Jev was wrong

• 8 nickname and preferred name gaps. Larry vs Laurence, Chuck vs Charles with a "Deacon" in front, a preferred name that only appeared in the record's email address. Jev hedged and said ambiguous where Qwen correctly linked.
• 9 organization normalization misses. The same company with and without an "Inc.", or the same street written two ways. Jev declined; Qwen linked.
• 4 tie breaks. Two records with identical evidence where Jev picked the other one.
• 2 genuine misses where it linked a different person on a shared address.

Every one of these has a plausible cause and a fix, mostly on our side of the API: a bigger nickname table (or some other instructions around how humans nickname), comparing against the CRM's preferred name field, normalizing organization suffixes and street formats before the model ever sees them. None of those fixes were applied before the final run, which is the point of a held-out set.

The 31 policy cases

There is one bucket that is not a model error in either direction: a candidate with the exact same full name and no shared identifier at all. No email, no phone, no address in common. Qwen links these. Jev declines them.

Whether to link two "John Smith" records that share nothing else is a product decision and not an accuracy one. It is one we have debated internally, but generally leave up to how our matching criteria is set up as it has configurable flexibility built in. I counted every one of those 31 against Jev in the table above, because that is how production behaves today. If we decide Jev's behaviour is the right one, its accuracy on the full set goes to 98.8%.

Cost, tokens, and latency

Jev used less than half the input tokens for the same person and candidates because the state and typed questions replace the wall of rules in our prompt. The cost difference is bigger than the token difference because of pricing, and the latency difference is bigger still. Four seconds per decision is fine in a background sync. 151 ms is incredibly fast and will help the system scale as we continue to grow as a company, ensuring that we do not end up with a huge backlog of syncs waiting on Bedrock calls.

What happens next

The numbers are good enough that Jev is now running in shadow next to Qwen, making the same decisions on the same inputs without writing anything, so we can watch the two disagree on live traffic before we flip the implementation. In parallel, we are continuing to gather data on the nickname and normalization gaps above and making the call on the exact-name-no-identifier policy.

If you are running a general purpose LLM for a narrow, structured decision like this one, I would encourage you to run the same experiment. Take the real decisions you have already made, replay them, and grade both models against the evidence rather than against each other. The result for us was a model that is more accurate on the decisions that matter most, 25 times faster, and costs 3% as much.