Tool calling reliability: DeepSeek, Qwen, Kimi open weights and what to check before you build

Tool calling reliability in DeepSeek, Qwen, Kimi open weights is four separate properties wearing one name. Whether the model decides to call a tool at all. Whether it names a tool that exists. Whether the arguments it produces validate against your schema. Whether it accepts the result you hand back instead of thrashing. A model can be strong at one and weak at another, and almost every number you will read online flattens all four into a single percentage.

There is a second problem, and it is the one that actually costs you a weekend. Very little of this is a property of the weights. Between your JSON Schema and the function you eventually execute sit a chat template, a sampler, a surface token format specific to the model family, a server-side parser chosen by a command line flag and a client library that assumes OpenAI semantics. Any of those can be the wrong version. When it is, the weights behave exactly as trained and your agent still fails.

Moonshot understood this well enough to ship a public verifier for its own model, because the same K2 weights served by different vendors produced measurably different tool call behaviour. That repository is the single most useful artefact in this whole area, and it exists as an admission.

The model cards are marketing documents with numbers in them. The useful material sits in issue trackers, in vendor discussion threads on Hugging Face and in the Jinja files nobody reads. I follow those as they land, in English and Chinese, and the pattern repeats with every release. What follows is what to check before you commit an agent architecture to one of these families.

The three families speak three different tool-call dialects

Start with the surface form, because everything downstream depends on it. These models do not emit OpenAI JSON. They emit family-specific token sequences that a parser converts into OpenAI JSON, and the conversion is where things go wrong.

Kimi K2 wraps its calls in explicit special tokens. A section opens with a tool calls section marker, each call is delimited individually, and a further token separates the tool identifier from its arguments. The tool identifier itself is structured as functions.{name}:{index}, so the call id encodes a position rather than being an opaque string. Moonshot's own guidance notes that the finish_reason you get when calls end varies by inference engine, so any check you write against it has to be adjusted per backend. That is a documented portability warning in the vendor's own repository, and most client code ignores it.

DeepSeek changed dialect at V3.2. Tool calls moved into a markup form the templates call DSML, with distinct markers for the calls block, for each invoke by name and for each parameter, including an explicit attribute recording whether a parameter is a string. The same template revision introduced a developer role used for search agent scenarios. If you upgraded a DeepSeek deployment across that boundary and kept the old parser, nothing about your logs would tell you why tool calls stopped appearing.

Qwen recommends Hermes-style tool use, which means a JSON object inside tool_call tags rather than dedicated vocabulary tokens. Qwen's documentation is unusually candid about the consequence: function calling here is prompt engineering, and the docs concede that generation is not guaranteed to follow the protocol even with correct prompting and templates. The coder-oriented Qwen models use a different form again, an XML structure with a function element and nested parameter elements.

That XML choice matters more than it looks. XML carries no types. The vLLM parser for that format converts each parameter value using the JSON Schema when a schema is available and leaves it as a string when one is not. Loose or missing schema, and your integer 5 arrives as the string "5" and your boolean false arrives as "false". Argument type drift is not the model being sloppy. It is a lossy surface format doing exactly what it was told.

Moonshot shipped a public verifier for its own model because it could not trust the people serving it. That instinct is the whole lesson.

Your parser flag carries more risk than the weights

Both vLLM and SGLang select the conversion logic with a --tool-call-parser argument. Pick the wrong value and you get raw text where tool_calls should be. The parser names are family-specific, and the correct choice is not stable even inside a single vendor's catalogue.

"Qwen's own model cards make this explicit, though not on the function calling page. The Qwen3-Coder cards tell you to launch vLLM or SGLang with --enable-auto-tool-choice and --tool-call-parser qwen3_coder, while the guidance for the QwQ and Qwen3 series tells you to use Qwen-Agent's built-in parser and specifically not to add those server flags. One vendor, one release cycle, two opposite recommendations." One vendor, one release cycle, two opposite recommendations. If you standardised your serving configuration across a fleet, you standardised on something wrong for half of it.

Then the models drift within a family. A confirmed bad-case issue filed on the Qwen3.8 tracker concerns the previous generation. Under SGLang's qwen3_coder parser the reporter finds Qwen3.5-4B mostly produces valid JSON tool calls, Qwen3.5-9B is unstable and often emits XML instead and Qwen3.5-35B-A3B consistently fails to produce JSON-compatible calls. Same generation, same recommended settings, size-dependent output format. Note the parser is SGLang's here and the type-conversion behaviour described above is vLLM's, which is itself part of the point. Any evaluation you ran on the small model is void for the large one.

DeepSeek has the mirror-image bug. An SGLang issue reports V3.2 under the deepseekv32 parser intermittently emitting a raw function invocation snippet without the DSML markers that wrap it, which breaks client parsing. Intermittently, with no predictable trigger. A separate report against the current DeepSeek flagship describes calls arriving as plain text appended to the content field, with finish_reason set to stop and tool_calls null. The reporter counted roughly two occurrences in a nineteen-completion session. Treat that as one person's session, not a rate, and then note that a rate above zero on this failure is already an architectural problem for an unattended agent.

Streaming is a separate code path with its own defect history. Multiple open vLLM issues describe the Hermes parser returning raw text instead of parsed calls under streaming, or erroring while handling a streamed call. If you tested non-streaming and shipped streaming, you tested nothing.

Moonshot's K2 Vendor Verifier is the response to all of this. It fires roughly four thousand requests at a provider and compares against the official API, reporting how many responses finished with tool_calls, a schema accuracy figure defined as successful calls over triggered calls and an F1 score for trigger agreement with the reference. Half the dataset is published so you can reproduce it. Moonshot attributes much of the observed vendor variance to incorrect inference engine versions, which is a polite way of saying the weights were fine and the serving was not.

Five failure modes, and each needs a different fix

Separate these in your logging or you will chase the wrong one for days.

The call arrives as prose. finish_reason is stop, tool_calls is null and the invocation sits in the content string. Cause is almost always template or parser mismatch, occasionally model drift under long context. Fix is at the serving layer, and the detection is a regex over content for your own tool names on every response you thought was a plain answer.

The payload does not validate. Triggering a call and producing arguments that satisfy the schema are independent events, which is exactly why the vendor verifier reports them as separate metrics. Validate every call with a real JSON Schema validator before dispatch, never with a try-and-see json.loads.

The name does not exist. Hallucinated tool names are the cheapest failure to defend against and the one people skip. Look the name up in a whitelist, and never dispatch by string. The Berkeley Function Calling Leaderboard scores this class explicitly, including subsets where the required function is deliberately absent, which is the behaviour you care about most and test least.

Types drift. Covered above for XML surfaces, but it also appears when a schema uses anyOf or nested objects. DeepSeek's beta strict mode enforces schema adherence server-side at the cost of requiring every property of every object to be marked required, which is a real constraint on schema design rather than a free switch.

The result is rejected. When a tool returns an error or truncated output, some models loop, trying alternative commands rather than accepting the observation. "Users report DeepSeek models looping on tool errors rather than accepting the observation, and the same pattern shows up in agent harness trackers as repeated identical calls. DeepSeek's own harness ships a counter that injects escalating reminders after three, five and eight identical consecutive calls, which tells you both that the behaviour is real and that prompting is not powerless against it. I have found no published measurement ranking its cost in agent-style evaluation, so treat it as a failure mode to test for rather than a quantified one." There is also an open report of empty responses after tool results are fed back under streaming. Test the unhappy path deliberately: return an error, return an empty array, return a two-hundred-kilobyte blob, and watch what the next turn does.

What the model cards claim, and what they leave unsaid

The Kimi K2.6 card is precise about the things it can be precise about. One trillion total parameters with thirty-two billion active, a mixture of 384 experts selecting eight per token plus one shared expert across sixty-one layers, 256K context, a Modified MIT licence, recommended temperature of 1.0 in thinking mode and 0.6 in instant mode with top_p at 0.95, and a pinned transformers range. It claims long-horizon agentic execution and swarm orchestration at large scale. It cannot claim, and does not, that any given third-party deployment will parse its output correctly.

DeepSeek's documentation is similarly honest about scope. The API reference states a maximum of 128 functions per request, which is a limit on how many tools you may define and not a promise about parallel invocation. I have seen that number repeated on third-party sites as 128 parallel calls per turn. It is not the same claim. Strict mode is beta and requires a different base URL, with a documented subset of JSON Schema constructs supported.

Qwen3.8 splits its licence. The 27B dense model shipped under Apache 2.0 while the 2.4-trillion-parameter flagship carries a custom licence with revenue conditions. Kimi's Modified MIT adds an attribution obligation above thresholds of roughly a hundred million monthly active users or twenty million US dollars in monthly revenue. Read these before you architect, because the model you benchmark and the model you can legally ship at scale may not be the same one.

What no card states is the reliability figure you actually need, which is conditional on your engine version, your parser flag, your template file and your schema style. No vendor can state it. That is why they publish verifiers instead.

A harness you can build in an afternoon and rerun on every upgrade

I am describing a method here, not reporting results from it. Any numbers in this area that are not yours, produced against your stack this week, are decoration. Build the thing and generate your own.

Freeze six values first and record them alongside every result: model repository revision hash, inference engine version, parser flag, the hash of the chat template file actually loaded, sampling parameters and whether streaming is on. Five of those six change without you noticing.

Build a corpus of two hundred to four hundred prompts in five buckets. Should call exactly one tool. Should call two tools in one turn. Should call nothing, because the answer is in the prompt. Should ask for a missing required parameter rather than inventing one. Should recover after the tool returns an error. The last two buckets are where models separate and where public leaderboards are thinnest.

Instrument at the right layer. Log the raw generated text before parsing as well as the parsed object, because the interesting failures are invisible after the parser has swallowed them. For each response record finish_reason, whether tool_calls is populated, whether the name is in your whitelist, the result of full JSON Schema validation and whether any tool name appears in the content string.

Score four rates independently and never average them: trigger agreement against a reference, name validity, schema validity and error recovery. Run each configuration twice, once at temperature zero and once at the card's recommended temperature, then run the whole set again with streaming enabled. Reuse existing work where you can. The K2 Vendor Verifier script takes a samples file, a base URL and a key, and will point at any OpenAI-compatible endpoint. The Berkeley leaderboard's harness covers the model-side abstract syntax tree matching and its multi-turn subsets for missing functions and missing parameters.

Then wire the whole thing into CI and rerun it on every engine bump. An inference engine upgrade is a model change in this domain, and it arrives without a model card.

Constrained decoding fixes the shape and can suppress the call

The obvious answer to malformed arguments is grammar-constrained decoding. "vLLM integrates XGrammar and exposes structured output constraints through its structured_outputs parameter, which replaced the older guided_json and related guided_ fields when those were removed in v0.12.0.", and constrained generation gives structural correctness by construction with very low overhead. DeepSeek's strict mode is the hosted equivalent. Force the grammar and the payload validates, every time.

It is not free, though not in the way the framing suggests. A June 2026 paper by Fangzheng Li, Aimin Zhang and Chen Lv, Constraint Tax in Open-Weight LLMs, tests a different pairing: tool definitions together with a response_format JSON Schema on the reply itself. Under that pairing several open-weight models stop invoking tools altogether while still satisfying the schema, an effect the authors name tool suppression. They attribute it to the grammar itself. Compiling the JSON Schema into a token mask blocks the tokens a tool call would have to start with, so the call never forms. They call the explanation the Constraint Priority Inversion hypothesis, and their proposed remedy is a two-pass execution that separates the tool turn from the schema-constrained reply. So you can drive schema accuracy to a hundred percent and simultaneously collapse the trigger rate, and a single headline metric will show that as an improvement.

This is the concrete reason to keep trigger rate and schema validity as separate columns. A change that helps one and hurts the other is common, and it is invisible to anyone reporting a combined score. Measure both before and after you turn on constrained decoding, on the same frozen configuration.

The strongest case against doing any of this

The honest counter-argument is that this is a serving problem in a fast-moving ecosystem, and it is being fixed continuously. The SGLang DSML bug was triaged as high priority with a pull request attached. The Qwen format issue is labelled confirmed and assigned. vLLM and Moonshot debugged K2 compatibility together in public. Vendors now ship parsers alongside weights on day one, which was not true two years ago. Agent frameworks retry malformed calls automatically, and the retry usually succeeds. On the model side, tool use scores across these three families have climbed steadily on public leaderboards.

So if you are running a chat assistant with three tools and a human reading every response, most of this checklist is over-engineering. Pick the recommended parser, pin the engine version, validate arguments before dispatch and move on. That covers the majority of the risk for a fraction of the work.

The case flips when nobody is watching. An unattended agent that runs overnight, a tool that writes rather than reads, a schema where a string where an integer belonged silently coerces into a wrong record, a fleet where one node picked up a different engine build. There the failure is not a bad answer a user shrugs at. It is a wrong action nobody reviewed. And the argument that these bugs get fixed quickly cuts the other way for self-hosters, because a fix that lands upstream reaches you only when you upgrade, and the upgrade is itself the thing most likely to break your parser.

You own the parser now

A hosted API hides an entire stack behind one endpoint. When you take the weights, you take the stack. The template, the parser, the streaming path, the engine version and the schema translation all become yours, and none of them are covered by the benchmark that made you choose the model.

So the pre-build checklist is short. Know which dialect your model emits and which parser flag converts it. Pin the engine and the template file by hash, not by tag. Validate names against a whitelist and arguments against a real schema, before dispatch and after every upgrade. Test the unhappy path where the tool returns an error, because that is where the loops live. Keep trigger rate and schema validity in separate columns forever. Read the licence for the size you will actually deploy, not the size you benchmarked.

Moonshot shipped a public verifier for its own model because it could not trust the people serving it, and published half the test set so anyone could argue back with data. That instinct is the whole lesson. In this part of the stack the vendor cannot vouch for your deployment, the benchmark cannot vouch for your parser and the only number that means anything is one you generated yourself, this week, against the exact configuration you are about to run.

Tools referenced

vLLM, reviewed here: vLLM review.

Sources

DeepSeek API Docs: Tool Calls guide, including beta strict mode and supported JSON Schema constructs: https://api-docs.deepseek.com/guides/tool_calls/

DeepSeek API Reference: Create Chat Completion, tools and tool_choice parameters and the 128-function limit: https://api-docs.deepseek.com/api/create-chat-completion

Qwen documentation: Function Calling, Hermes-style tool use and the caveat that protocol adherence is not guaranteed: https://qwen.readthedocs.io/en/latest/framework/function_call.html

QwenLM/Qwen3.8 issue #125: Qwen3.5 9B and 35B-A3B emit XML-style tool calls under the coder parser (badcase-confirmed): https://github.com/QwenLM/Qwen3.8/issues/125

MoonshotAI/Kimi-K2: tool call guidance, special token format and the engine-dependent finish_reason warning: https://github.com/MoonshotAI/Kimi-K2/blob/main/docs/tool_call_guidance.md

moonshotai/Kimi-K2.6 model card: architecture, 256K context, Modified MIT licence and recommended sampling settings: https://huggingface.co/moonshotai/Kimi-K2.6

MoonshotAI/K2-Vendor-Verifier: schema accuracy and tool_call_f1 definitions, ~4,000 request test set, half published: https://github.com/MoonshotAI/K2-Vendor-Verifier

Li, Zhang and Lv, Constraint Tax in Open-Weight LLMs: Tool Calling Suppression Under Structured Output Constraints (arXiv, June 2026): https://arxiv.org/pdf/2606.25605

sgl-project/sglang issue #14695: DeepSeek-V3.2 occasional malformed tool call output, missing DSML markers under --tool-call-parser deepseekv32, labelled high priority: https://github.com/sgl-project/sglang/issues/14695