4B Small Model Closes In on o3 on the Swedish Medical Licensing Exam

In late July, a developer experiment based on the Swedish medical licensing exam produced a striking result: according to the experiment’s author, Qwen3.5-4B achieved 87% multiple-choice accuracy with reasoning enabled, while o3 had previously scored 88% on a smaller, partially overlapping dataset.

Looking only at the numbers, a 4B open-weight model appears to be within reach of o3.

But it is too early to jump to conclusions. The two models were not evaluated head-to-head on exactly the same data, prompts, and reasoning budgets, so 87% versus 88% cannot be directly framed as “a small model matching o3.” A more accurate assessment is: on vertical tasks with clear boundaries, objectively gradable answers, and stable knowledge structures, 4B models have entered a score range previously occupied only by flagship proprietary models.

Moreover, what developers should pay the most attention to here is not how miraculous fine-tuning is, but how quickly base models are improving—the new generation of 4B models has not even undergone domain-specific post-training, yet already significantly outperforms medical models that received SFT just a few months ago.

From 60% to 87% in Three Months

The experiment centered on MedQA-SWE. It consists of multiple-choice questions from the Swedish medical licensing exam, testing not everyday health knowledge but professional-level clinical knowledge, diagnostic judgment, and principles of treatment.

According to the data published by the author, the results were approximately as follows:

| Model | Training or Inference Method | Reported Accuracy | Caveats | |---|---|---:|---| | GPT-4 | Evaluated in 2024 | 84% | Historical result; experimental conditions may differ | | o3 | Evaluated in 2025 | 88% | Used a smaller dataset partially overlapping with the current one | | MedGemma-1.5-4B | SFT on questions from previous years | 60% | Reached the passing threshold on the final-year exam | | Gemma4-E4B | No domain-specific post-training | 77% | Result reported by the author | | Qwen3.5-4B | No domain-specific post-training | 77% | Result reported by the author | | Qwen3.5-4B | Reasoning enabled | 87% | Reasoning budget and stopping conditions significantly affect the result |

The original approach was conventional: use medical exam questions from earlier years for supervised fine-tuning, while reserving the final year’s questions for testing. After SFT, MedGemma-1.5-4B improved from its base performance to 60%, reaching the passing level set by the author.

This was already a reasonable example of a vertical model. With 4B parameters, domain data, and a temporal split, the goal was not to become an all-purpose doctor, but to cross the threshold of usability on a narrow task at relatively low cost.

What truly changed the assessment was the subsequent generation of models. According to the author, Gemma4-E4B and Qwen3.5-4B, released only about three months later, reached 77% without any post-training on MedQA-SWE. With reasoning mode enabled, Qwen3.5-4B climbed further to 87%.

In other words, the 60% that the previous approach had worked hard to achieve through SFT is now being surpassed directly by the zero-shot capabilities of newer base models.

This is also one of the easiest changes to underestimate in current small-model development: post-training certainly remains important, but the gains from base-model iteration can sometimes exceed those from an entire round of domain-specific SFT. If the wrong base model is selected, an engineering team may spend weeks cleaning data, tuning learning rates, and investigating catastrophic forgetting, only to be outperformed by a newly released model running out of the box three months later.

87% Is Close to o3, but It Is Not the Same Exam

These results are impressive—and easy to overstate.

First, o3’s 88% came from a smaller dataset that only partially overlaps with the current one. Question difficulty, year distribution, the proportion of repeated questions, prompt templates, and answer-parsing methods can all cause fluctuations of several percentage points. On a relatively small exam set, getting just two or three questions right or wrong may change the ranking.

Therefore, “only one percentage point behind” is a descriptive fact, not a rigorous conclusion about model capability. Demonstrating that a 4B model is equivalent to o3 on this task would require, at minimum:

1. Using exactly the same test questions and option order;
2. Fixing the system prompt, temperature, number of samples, and answer parser;
3. Recording the actual number of reasoning tokens consumed by each model;
4. Performing paired statistics on the same questions rather than comparing only overall accuracy;
5. Reporting confidence intervals and checking per-subject performance across medical specialties;
6. Ruling out question paraphrases, explanation leakage, and near-duplicates between the training and test sets.

A more robust replication setup could first be written as an experimental checklist rather than rushing to produce an aggregate score:

evaluation:
  split: final-year-held-out
  option_shuffle: true
  temperature: 0
  seeds: [11, 23, 47]
  answer_format: single_letter
  report:
    - accuracy
    - bootstrap_confidence_interval
    - subject_breakdown
    - invalid_answer_rate
    - reasoning_tokens
    - latency

contamination_checks:
  exact_match: true
  near_duplicate_detection: true
  explanation_overlap_audit: true

Shuffling the answer options is especially important. Medical multiple-choice models may learn positional biases, and unstable output formatting may also cause correct answers to be marked wrong. If an evaluation records only whether the final A, B, C, or D answer is correct—without tracking invalid outputs, reasoning length, or variance across random seeds—it can easily produce a result that looks impressive but is unsuitable for deployment.

The correct interpretation is therefore not that “4B is now fully equal to o3,” but rather: within a specific language, professional domain, and closed answer space, a 4B model can now approach the historical benchmark of a flagship model. That conclusion is less sensational, but more valuable for engineering decisions.

Long Reasoning Does Not Deliver Accuracy for Free

The key variable behind Qwen3.5-4B’s improvement from 77% to 87% was reasoning mode. This suggests that the model may already contain a substantial amount of medical knowledge, but ordinary decoding does not consistently convert that knowledge into the correct answer choice. Extending the reasoning process effectively gives the model more time to perform differential diagnosis and eliminate distractors.

However, the author also observed the other side of reasoning models: when the length limit was removed, performance increased slightly further, but some samples fell into repetitive loops. The model repeatedly agonized over output formatting or restated the instructions, eventually exhausting the context window without providing an answer.

This is not merely a user-experience issue; it is a production cost and reliability issue. A model with high average accuracy that “thinks” indefinitely on a small number of requests can create three types of failures simultaneously:

• A widening latency tail, with P95 and P99 response times spinning out of control;
• KV cache and compute being consumed by low-value repetitive tokens;
• The output parser never receiving a final option, forcing the application layer to treat the request as a failure.

The author stated that the “early exit” approach proposed in the S-GRPO work was helpful. Because the current public post does not provide sufficiently complete details about the intervention, it should not simply be interpreted as a universal fixed threshold. The engineering direction, however, is clear: reasoning models need more than a maximum-token limit; they also need an early-exit mechanism.

In real-world deployments, several safeguards can be combined:

• Require the model to stop immediately once it has stably produced a single answer choice;
• Detect repeated n-grams or loops in format self-checking;
• Set separate budgets for “analysis” and the “final answer”;
• Inject a forced-convergence prompt after the soft limit is reached;
• If no answer is produced by the hard limit, fall back to direct answering or a backup model;
• Track accuracy, average reasoning length, and timeout rate together rather than looking only at benchmark scores.

This reveals a fact often obscured by leaderboards: reasoning tokens are also a parameter. The same 4B weights, under budgets of 500 tokens and 8,000 tokens, effectively correspond to two products with different costs, latency, and reliability.

If the 87% score requires far more inference overhead than the 77% score, the deployment decision should not simply be whether to turn reasoning on or off. It should be a routing decision: answer routine questions directly, and reserve long reasoning for low-confidence or high-risk cases.

Small-Model Post-Training Is Shifting from “Injecting Knowledge” to “Correcting Behavior”

In the past, the most common approach to building vertical models was to feed domain materials and question banks into SFT data in the hope that the model would memorize more specialized knowledge. MedGemma-1.5-4B’s 60% score shows that this approach works; the new models reaching 77% without training shows that it may no longer be the most cost-effective route.

Once a base model already possesses a large amount of medical knowledge, post-training may be better directed toward four issues:

1. Enforcing a Fixed Output Protocol

The model should reliably output machine-parseable options, diagnostic codes, or structured fields, reducing cases where it “answers correctly but fails parsing.” In real systems, gains of this kind are usually more useful than one or two percentage points on a leaderboard.

2. Learning When to Stop

The goal is not to make the model always think longer, but to teach it to judge when additional reasoning is useful and when it should submit an answer. For a 4B model, suppressing unproductive chains of thought may be more cost-effective than continuing to add parameters.

3. Calibrating Uncertainty

What medical systems truly need is not a model that answers every question, but one that knows which questions should be escalated to a larger model or a human expert. An accuracy of 87% still means getting roughly one out of every eight questions wrong. If the error occurs in emergency management, the average score has no safety significance.

4. Adapting to Local Standards

The Swedish medical exam involves local clinical procedures, medication practices, and linguistic conventions. A general-purpose model may understand a disease without knowing the local guidelines. High-quality post-training data should prioritize these regional differences rather than repeatedly teaching basic medical definitions.

New models have therefore not made SFT obsolete. They have merely shifted SFT’s role from “teaching the model enough to pass” toward “turning a model that can already answer questions into a controllable system.”

The Value of 4B Is Not Just That It Is Cheap, but That It Can Serve as a Component

A 4B-parameter model’s weights alone typically occupy about 8 GB in FP16, and quantization can reduce this further to just a few gigabytes, although real deployments must still reserve memory for the KV cache, runtime, and concurrency. This means such a model can run on a single consumer GPU, workstation, or even some edge devices, without requiring every request to be sent to a remote flagship model.

Beyond medicine, the same pattern applies to legal clause classification, industrial fault multiple-choice questions, financial rule evaluation, and internal enterprise ticket routing: the clearer the task boundaries and the more structured the answers, the more likely a small model is to achieve sufficiently high business accuracy with fewer parameters.

A more realistic system architecture will not be “4B replaces every large model,” but a tiered arrangement:

• A 4B model handles high-frequency, standardized, and verifiable tasks;
• A retrieval system supplies the latest guidelines and internal knowledge;
• A large model handles cross-document reasoning and ambiguous questions;
• High-risk results are sent for human review.

This is easier to control in terms of cost than having a flagship model handle every request, and it also makes data isolation easier. In sensitive domains such as healthcare, local execution and auditability are valuable in themselves.

However, open weights do not necessarily mean fully open source, nor do they mean the model can be used directly in clinical practice. Model licenses, training-data transparency, bias assessments, and liability boundaries must still be reviewed separately. Medical multiple-choice questions measure knowledge recognition and exam reasoning—not patient consultation, communication, longitudinal decision-making, or medical-device-grade validation.

The Most Direct Lesson for Developers: Re-Evaluate the Base Model Before Deciding to Train

As of July 26, 2026, this experiment is not rigorous enough to constitute a fair head-to-head leaderboard, but it already sends a clear signal to developers: the capability threshold of small models continues to fall rapidly, and a training plan that made sense three months ago may no longer be optimal today.

Teams preparing to build vertical models would be better off reversing the usual order:

1. First, evaluate the latest batch of 4B–8B base models in zero-shot and reasoning modes;
2. Analyze whether errors actually come from missing knowledge, failed reasoning, or uncontrolled formatting;
3. Design SFT data only when stable knowledge or behavioral gaps remain;
4. Treat reasoning tokens, latency, and invalid outputs as core metrics;
5. Only then compare the total costs of local small models, remote large models, and hybrid routing.

In my view, the greatest significance of this experiment is not that “4B beat someone,” but that it demonstrates how specialized capabilities are shifting from exclusive features of expensive models into engineering components that can be compressed, trained, and deployed.

An 87% score is not proof of clinical usability and may not even survive strict replication unchanged. But when a 4B model can score near o3’s historical result on a Swedish-language medical exam, developers can no longer view “small models” through the lens of two years ago.

The next round of competition will not be only about reducing parameter counts, but about who can make small models reliably deliver the final answer with shorter reasoning, more stable stopping conditions, and less domain-specific data.

References

• Reddit: Open-weight 4B models approach o3-level medical question answering in Swedish: Scores published by the experiment’s author, differences between the datasets, and the phenomenon of long-reasoning loops.
• GitHub: MedQA-SWE MedGemma SFT implementation: Reproduction code for supervised fine-tuning MedGemma-1.5-4B on exam questions from previous years.
• Zhihu: Mathematical Reasoning and Scaling RL Practice with 4B Small Models: Additional examples showing how small models can narrow the gap with larger models through reinforcement learning and post-training.