I trained a small, open-source decision model on my own machine to do one job: pick the next tool in a customer support conversation. Training took minutes, and the training data never left my network.
The model was Laya, which has roughly 420 million parameters. Those parameters are the internal values that training adjusts. Compared with a large general-purpose AI model, Laya is small. Yet after fine-tuning, it beat the cloud-hosted decision model Jev in one of my experiments.
In the other experiment, Jev won.
That difference matters. It shows why a model comparison needs more than a headline score. The information I supplied changed the results substantially. So did training the smaller model on examples of the exact decision I wanted it to make.
Then I tested something that accuracy comparisons often leave out: could I trust the model’s confidence enough to let it act on its own?
A system that picks the right tool most of the time can still make confident mistakes. If confidence controls whether an AI agent acts or hands a request to a person, those mistakes become an operational problem. My tests covered both sides: choosing the right action and deciding when that choice deserves human review.
Why I Used a Specialist Decision Model
A general-purpose model can handle many different tasks. A specialist has a narrower job. For this experiment, I didn’t need a model to write every support response or manage an entire conversation. I needed it to choose one action from a stable list of 30 tools.
That’s a useful distinction. Customer support software repeatedly makes decisions of the same kind. Given the conversation so far and the relevant procedures, which tool should run next?
Laya and Jev offer different ways to improve that decision.
Jev is a large, general-purpose decision model, but I couldn’t fine-tune it. My main lever was the state I passed into it. Here, “state” means the information available at the moment of the decision. I could supply a lean description or give it much more context.
Laya has open weights, so I could train it on my own examples. Its initial performance might be poor on a particular task. Fine-tuning can change that by teaching it the relationship between the input and the correct choice.
The Laya repository provides the project resources, and its Hugging Face model page provides access to the model.
The conditions that make fine-tuning useful
I see a strong case for a specialist when several conditions line up:
- The application makes the same kind of decision repeatedly.
- The available choices stay reasonably stable.
- Past examples include the correct answer.
- Existing mistakes show patterns that training might address.
My support task fit those conditions. Every decision involved the same tool list. Historical conversations supplied examples of the appropriate next action. That gave Laya something concrete to learn.
Local operation was another reason to test it. Running the model inside my own software gives me more control over data, cost, and latency. It also makes fully isolated deployment possible.
That doesn’t mean a small local model automatically becomes the best option. It means model size alone doesn’t settle the question. A compact model trained for one repeatable task can compete with a much larger model that hasn’t learned that task directly.
Where the decision model fits in an AI agent
A language model often chooses tools while handling a chat. A decision model can sit inside the surrounding application and make that choice separately.
Instead of producing a conversational answer, it assigns probabilities to the available actions. The application can use those probabilities to decide whether to proceed or escalate.
For example, a high-confidence choice could allow the language model to act. A lower-confidence choice could go to a larger model or a human. These decision checks can return in less than a tenth of a second, so they don’t necessarily add much delay.
The important word is “could.” Before using confidence as a permission signal, I need evidence that the number means what it appears to mean.
The Three Measures I Kept Separate
I evaluated accuracy, confidence, and calibration. They’re related, but each answers a different question. Treating them as interchangeable makes it easy to overestimate how much automation a model can safely handle.
Accuracy: how often did it choose correctly?
Accuracy measures the share of test cases where the model selected the correct tool.
If it gets 822 out of 1,000 decisions right, its accuracy is 82.2%. The calculation doesn’t care how confident the model felt.
A correct choice at 99% confidence counts exactly the same as a correct choice at 60%. That makes accuracy useful for comparing overall performance. It doesn’t tell me which individual decisions I should allow through without review.
Confidence: how sure was it about this choice?
The decision model gives each tool a probability. Sometimes one option dominates. It might receive 95%, with the remaining choices sharing the other 5%.
Other decisions are much less clear. One action might receive 40%, another 25%, and another 35%. The model still has a first choice, but it has little separation from the alternatives.
I can place a threshold on the winning probability. Above that threshold, the application can act automatically. Below it, the application can hand off the decision.
This turns confidence into a practical control. Rather than treating every prediction alike, the system distinguishes clearer decisions from uncertain ones.
Calibration: does that confidence match reality?
Calibration tests whether the confidence scores reflect actual success rates.
Suppose the model reports about 90% confidence on 100 cases. If it gets roughly 90 of those cases right, the confidence is well calibrated for that group. If it gets only 70 right, it’s overconfident.
A weather forecast offers a simple comparison. Across days with a 70% rain forecast, rain should occur on about seven out of ten days. A single dry day doesn’t disprove the forecast. The pattern across many days tells me whether the percentage is trustworthy.
The same principle applies to tool selection. A model can make an incorrect choice at 95% confidence. What matters is whether predictions at that confidence level succeed at approximately the expected rate across many cases.
- Accuracy tells me how often the model is right.
- Confidence tells me how sure it is about an individual choice.
- Calibration tells me whether I should believe that confidence.
Those three checks shaped the rest of the experiment. Improving the first number was only part of the job.
How I Built the Customer Support Test
I used the Action-Based Conversations Dataset, or ABCD. It contains around 10,000 back-and-forth conversations.
For this task, I stopped each selected conversation just before the support agent used a tool. The model then had to choose the next tool from the list of 30.
This kept the target clear. I wasn’t judging the quality of a written reply. Nor was I asking the model to complete the whole support workflow. Each example represented a specific decision point with a known correct action.
The input included the conversation so far, previous tool results, and a reference to the procedures the agent should follow. Across the prepared data, I had 8,883 examples, just under 9,000.
Every model faced the same test questions and the same choices. That consistency was essential. Otherwise, a score difference could reflect a different test rather than a different approach.
I split the data before training
The most important preparation step happened before I trained anything. I divided the examples into four separate groups, each with a different purpose.
| Data group | Examples | Purpose |
|---|---|---|
| Training | 5,900 | Teach Laya to choose the correct tool |
| Validation | 983 | Compare versions during training |
| Calibration | 1,000 | Adjust the confidence probabilities |
| Test | 1,000 | Measure performance on held-out examples |
The training set is the material Laya learns from. It reads an example, predicts a tool, and receives the correct answer. Training adjusts its internal values to make the correct action more likely.
The validation set acts like a practice exam. Laya doesn’t learn directly from those examples. Instead, I use them to check performance after a training pass and keep the version that performs best.
That matters because additional training doesn’t guarantee improvement. A later version can perform worse than an earlier one. Validation gives me a way to identify that and return to the better version.
The calibration set has a separate role. It helps adjust how certain the model sounds, rather than teaching it which tool to select.
Finally, the test set stays separate from those processes. It supplies the final assessment. All the reported accuracy comparisons use those same 1,000 held-out cases.
Why separation matters
Testing on examples the model has already learned can produce a misleading score. It might remember answers rather than handle unfamiliar cases well.
The four groups prevent me from using one pile of examples for every purpose. Training answers one question: can the model learn the task? Validation helps select a version. Calibration adjusts confidence. Testing measures the resulting system.
Coding agents make the practical work of fine-tuning easier. They can help prepare datasets and carry out the training process. Still, easier execution doesn’t remove the need for a sound evaluation setup.
A quick training run is useful only if the result tells me something meaningful about new decisions.
Two Experiments: Basic State and Rich State
I ran two experiments because training and context are different ways to improve performance.
The first tested a relatively lean input. The second supplied much more information about the task. Within each experiment, Laya and Jev received the corresponding state.
Experiment one: basic state
The basic state contained the conversation, the names of the 30 tools, and the procedures. It provided the essential decision material without extensive supporting explanation.
I initially fine-tuned Laya on that format. Jev received the same basic format at decision time.
This tested whether Laya could learn enough from historical examples to select the right action without a large amount of extra context in every request.
Experiment two: rich state
The richer state added several sources of guidance:
- A handbook explaining what each tool does.
- Worked examples from similar solved conversations.
- Relevant manual sections for the request type.
- Rationale explaining how to choose the next action.
- Information about tools already used in the conversation.
For Jev, this included ten similar chat examples with their appropriate tool choices. Each rich-state request reached about 7,000 tokens, the units models use to process text.
That’s a substantially larger input. The benefit is that the model doesn’t need to infer as much from short tool names. It receives explanations, examples, and procedural guidance at the point of decision.
I also fine-tuned Laya on this richer format. Comparing rich-state Jev with basic-state Laya alone wouldn’t answer how they perform with comparable information.
A gift-wrapping question showed the difference
One example involved a customer who wanted to buy boots and asked about the price of gift wrapping. The correct next action was the pricing tool.
The models chose different actions:
- Laya before fine-tuning: Validate Purchase.
- Fine-tuned Laya: Pricing.
- Jev with basic state: Boots.
- Jev with rich state: Pricing.
Jev’s basic-state choice came with about 60% confidence. Once I supplied the richer information, it chose the correct pricing action with greater confidence.
This example captures the distinction between the product mentioned in a conversation and the customer’s immediate need. The customer mentioned boots, but the next decision concerned a price.
Fine-tuning helped Laya make that distinction. More detailed context helped Jev reach the same answer.
One example can explain a failure, but it can’t establish overall performance. I needed the full 1,000-case test to see whether that pattern held more broadly.
What Happened When I Fine-Tuned Laya
Laya’s starting point was poor. Without task-specific training, it achieved only 7.5% accuracy.
It selected two tools for most of the test cases: Search Pricing and Validate Purchase. In my visualization, each dot represented a test decision. Correct choices appeared in teal, while incorrect choices appeared in red.
The initial pattern showed a model that wasn’t making useful distinctions across the 30 actions. It repeatedly favored a small part of the tool list.
Training changed that quickly.
I ran the fine-tuning on an RTX 5090. A single pass through the training data took under four minutes of GPU time. During that pass, Laya surpassed Jev’s basic-state result and then passed a simple classifier baseline.
After two further training passes, Laya reached 82.2% accuracy on the held-out test set.
That moved it from 75 correct decisions out of 1,000 to 822. The improvement came from teaching the small model the specific task, rather than replacing it with a larger model.
The simple classifier was a useful surprise
I also tested a traditional machine-learning classifier without a language model. It learned which words and patterns pointed to particular tools.
That classifier scored 77.3%. Jev with the basic state scored 67.3%.
I found that result particularly useful. A simpler approach trained on historical examples beat a much larger general-purpose decision model given a lean request.
The training difference matters here. The classifier could learn from the full training set. Jev couldn’t receive a comparable fine-tuning process. It had to make each decision using the information in the request.
So this isn’t evidence that traditional classification generally beats Jev. It shows that learning a narrow task from relevant examples can outweigh broader capability when the supplied context is limited.
It also gives a practical baseline. Without the classifier result, I might have attributed too much value to using a language-based model at all.
The Accuracy Results Changed With Better Context
Jev’s performance improved sharply once I supplied the rich state. Its accuracy rose from 67.3% to 87.4%.
That’s an increase of 20.1 percentage points, or 201 additional correct choices across the same 1,000 test cases. The model didn’t change. The information available to it changed.
Laya also improved with the richer format. After two training passes on that state, it achieved 86.1%.
| Approach | Input | Accuracy | Correct out of 1,000 |
|---|---|---|---|
| Laya without task-specific fine-tuning | Basic state | 7.5% | 75 |
| Simple classifier | Basic-state comparison | 77.3% | 773 |
| Fine-tuned Laya | Basic state | 82.2% | 822 |
| Jev | Basic state | 67.3% | 673 |
| Fine-tuned Laya | Rich state | 86.1% | 861 |
| Jev | Rich state | 87.4% | 874 |
With basic state, fine-tuned Laya beat Jev by 14.9 percentage points. With rich state, Jev led Laya by 1.3 percentage points, equivalent to 13 decisions on this test.
I thought further Laya training might narrow that gap. But that remained a possibility, not a measured result. The score I achieved was 86.1%.
The evidence supports two separate findings. Fine-tuning made Laya much better at this task. Supplying richer state made Jev much better too.
Neither result justifies a universal winner. They show two ways to improve a decision system: teach a model the task, or give it more guidance every time it decides.
A 90% Confidence Gate Exposed Another Problem
Accuracy still didn’t tell me how much work I could safely automate.
I arranged the same 1,000 test decisions by confidence. Lower-confidence choices sat on the left. Higher-confidence choices sat on the right, extending to 99.9%.
Then I added a gate at 90%. Choices above the gate could proceed automatically. Choices below it would go to a human.
Before calibration, Laya’s results at that gate were:
- 160 decisions sent for human review.
- 840 decisions allowed to proceed automatically.
- 97 incorrect decisions among those allowed through.
The model reported high confidence on too many wrong answers. A 90% threshold sounded cautious, but the underlying probabilities were overconfident.
This illustrates why a percentage alone isn’t a safety guarantee. The application can enforce the threshold perfectly and still make poor decisions if the confidence scale is misleading.
Raising the gate changes how many requests proceed. Calibration addresses a different issue: whether the scores provide a credible basis for placing that gate.
How Temperature Scaling Improved the Gate
I used temperature scaling to adjust Laya’s confidence.
Before the model converts its internal scores into probabilities, I divide those scores by a single number called the temperature. A temperature above one softens the probabilities, making the model less certain.
This doesn’t change the order of the tools. The winning action remains the winner. Temperature scaling changes how strongly the model favors it.
That distinction is important. I wasn’t correcting the wrong tool choices. I was making some of those choices less likely to pass the automatic-action gate.
I fitted the temperature on separate examples
The calibration group supplied 1,000 examples with known correct answers. I used that group to find the temperature that best matched confidence to actual performance.
For this experiment, the selected temperature was 1.63.
It wasn’t a universal setting for Laya. It was the value fitted for this model and task using the separate calibration data.
After applying it, the confidence scores shifted lower. More incorrect decisions fell below the existing 90% gate. Some correct decisions moved below it too.
That’s the trade-off. The system becomes more cautious, but a person now handles more requests that the model would have answered correctly.
The gate stayed fixed, but the workload changed
| Laya at the 90% gate | Before calibration | After calibration |
|---|---|---|
| Human review | 160 | 326 |
| Automatic decisions | 840 | 674 |
| Incorrect automatic decisions | 97 | 36 |
Calibration reduced incorrect automatic decisions from 97 to 36. Meanwhile, human review increased from 160 cases to 326.
The threshold remained at 90%. The predicted tools also stayed the same. Only the probabilities changed.
This is a different improvement from higher accuracy. The model still makes the same choices across the full test set, but the surrounding application becomes more selective about which choices it executes.
I also keep the denominator clear. The 36 mistakes occurred among 674 automatic decisions, not across 1,000 automatic decisions. That works out to about 5.3% of the decisions allowed through.
A useful automation report therefore needs both numbers: how many mistakes passed the gate and how many requests the gate handled automatically.
Jev and Calibrated Laya Had Different Workload Trade-Offs
I compared calibrated Laya with rich-state Jev at the same 90% gate. Jev provides probabilities through its API, as described in the Jev documentation.
In this test, Jev returned whole-number percentages. That made its confidence plot appear in grouped vertical bands, rather than the more continuous spread of Laya’s scores.
| Measure at the 90% gate | Calibrated Laya | Rich-state Jev |
|---|---|---|
| Human review | 326 | 282 |
| Automatic decisions | 674 | 718 |
| Incorrect automatic decisions | 36 | 39 |
Laya allowed slightly fewer mistakes through. Jev handled more cases automatically and sent 44 fewer requests to a person.
Those results don’t point to an obvious operational winner. One configuration reduced the count of automatic mistakes. The other reduced the review workload.
The choice depends on the business. How costly is an incorrect tool call? How much review can the support team handle? Those questions determine where the gate belongs.
Confidence calibration makes that discussion more meaningful. Without it, a threshold can give a false impression of control. With more credible probabilities, I can assess the balance between automation and review using actual results.
How I’d Evaluate a Repeated Decision Task
This experiment changed how I think about model selection for narrow workflows. I’d start with the decision itself, rather than assume the largest available model should make it.
The tool list, historical examples, and operating constraints all matter. So does the amount of context the application can supply consistently.
For a task like this, I’d keep the evaluation focused on a few concrete steps:
- Define one decision clearly. My target was the next tool, not the quality of the whole conversation.
- Separate the data by purpose. Training, validation, calibration, and testing answer different questions.
- Include a simple baseline. The classifier’s 77.3% result showed how much relevant examples could achieve without a language model.
- Test the input format. Jev’s improvement showed that a lean state can hide a model’s useful capabilities.
- Measure selective automation. Overall accuracy doesn’t reveal the mistakes that pass a confidence gate.
- Account for human workload. Fewer automatic errors can mean more reviews, including reviews of correct predictions.
I wouldn’t treat these support results as proof that Laya will beat Jev on another task. They establish what happened on this dataset, with these inputs and training runs.
Nor would I claim a measured cost or latency winner from the accuracy results. Local execution offers control, but an operational comparison needs its own measurements.
What I did establish is that a small local model can become highly capable at a repeated decision. Laya’s rich-state accuracy finished close to Jev’s, while keeping a route to local, fully isolated operation.
For this customer support task, the useful question wasn’t simply which model scored highest. It was which combination of training, context, confidence calibration, and review policy produced the right operating balance. That combination determines how much responsibility I can give the system.
