USAF Open Source: 12G VRAM Fine-Tuning Qwen3-30B MoE

Can a 12GB consumer GPU fine-tune a 30B MoE model?

For nearly two years, almost no one dared to answer this question head-on. Until last week, a developer with the ID tsuyu122 threw his long-awaited project USAF (Unified Sparse Activation Fine-tuning) onto GitHub and open-sourced it directly under the Apache 2.0 license. His answer is straightforward: if your GPU can run inference, it should be able to run fine-tuning.

He himself used an AMD RX 6750 XT, with 12GB of VRAM, outside the CUDA ecosystem, and fine-tuned Qwen3-30B-A3B—an MoE model with 30B total parameters and 3B active. In the past, this setup would either rely on llama.cpp for inference or required an A100 for training. USAF offers a third route.

The old headache of MoE fine-tuning

Before diving into USAF, we need to clarify why fine-tuning MoE models has always been awkward.

Fine-tuning dense models via LoRA has matured—tools like Unsloth, PEFT, and TRL already make 7B–13B models straightforward on 24GB cards. But MoE models are a different beast, with counterintuitive challenges:

• Terrifying parameter count: Qwen3-30B-A3B only activates 3B, but you still have to load all 30B. During inference, you can get away with offloading or MoE-aware kernels, but during backpropagation the path complexity skyrockets.
• Gradients are sparse yet discrete: Only the experts selected by the router receive nonzero gradients; inactive experts' parameters don’t move throughout the batch. Deciding whether and how to allocate optimizer state is a delicate trick.
• Freezing strategy is hotly debated: The classic ST-MoE paper argued that freezing MoE layers while tuning only the non-MoE FFN achieves nearly full fine-tuning quality; others claim complete expert-layer freezing leads to performance collapse. There’s still no consensus.
• LoRA on MoE isn’t elegant: Unsloth recently introduced Split LoRA + custom Triton grouped-GEMM kernels, pushing Qwen3-30B-A3B’s 16-bit LoRA down to 63GB—an engineering marvel. But 63GB still means you need A100-level hardware.

In short, mainstream MoE fine-tuning has revolved around “adding adapters.” USAF reverses that idea—it adds no adapters, tuning the native weights directly but only a small portion.

The core idea of USAF: sparse native training without adapters

The author didn’t formally define USAF’s full name, but from its code and description, it can be summed up as: train a sparse subset of expert weights + router, freeze everything else.

The key differences:

1. Trains original weights, not LoRA’s low-rank projection. LoRA introduces extra A and B tensors and their optimizer states to maintain low-rank capability. For MoE, attaching LoRA to every expert adds huge VRAM overhead.
2. Applies internal sparsity to experts. It doesn’t “train only certain experts,” but instead updates only a subset of weights within each activated expert. Thus, optimizer state (Adam needs 2× parameter count in FP32) is allocated only for that subset.
3. Router is also trained. That’s important. Traditionally, freezing the router stabilizes routing distributions, but the author chose to fine-tune it along with the task, allowing expert assignment reshuffling.

From a memory perspective, the benefit is obvious. Taking Qwen3-30B-A3B as an example—3B active parameters—and assuming USAF only computes gradients and optimizer states for a sparse subset of 10–20%, optimizer memory usage drops from “30B full” or “LoRA 63GB” down to just a few GB. Combined with 4-bit or 8-bit quantized model loading, fitting the process into 12GB of VRAM isn’t magic.

Compared with Unsloth Split LoRA: which is more practical?

That was my first question when I saw this project. Unsloth’s MOE Triton kernel from early 2026 was jaw-dropping—fine-tuning gpt-oss-20b in only 12.8GB and compressing Qwen3-30B-A3B’s 16-bit LoRA to 63GB, 12–30× faster than Transformers v4. But Unsloth focuses on “faster and cheaper LoRA,” while USAF skips LoRA altogether.

Their trade-offs are clear:

| Dimension | Unsloth Split LoRA | USAF | |---|---|---| | Target Hardware | A100 / RTX PRO 6000 / B200 | Consumer 12GB GPUs | | Fine-tuning Object | LoRA adapter (rank=64) | Native sparse weight subset + router | | Ecosystem | CUDA + Triton | Cross-platform (author used ROCm) | | Maturity | Production-level | Personal project, early-stage | | Deployment Convenience | Adapter needs merging | Native weights directly usable |

For production teams, Unsloth is the safer bet—especially since, after deep cooperation with HuggingFace, it standardized torch._grouped_mm. But USAF offers what Unsloth doesn’t: it runs on AMD and fits in 12GB. For indie developers with a 6700 XT / 6750 XT / 7800 XT who want to tinker with MoE, that’s the leap from “impossible” to “worth a try.”

Some technical judgments

A few thoughts about this approach—open for debate.

1. Sparse native training is a promising direction.
LoRA became popular largely because it’s cheap and non-invasive. But MoE models are inherently sparse—only some experts activate, only some gradients are nonzero. Layering LoRA’s dense low-rank assumption on top masks that sparsity. USAF basically says MoE itself carries a PEFT-like nature; we should embrace that structure instead of forcing dense-model habits.

2. Training the router comes with risks.
Prior studies repeatedly reported that lightly freezing gates stabilizes training. USAF updates the router with the task—great for short-term alignment, but if data distribution is narrow, routing collapse is easy: all tokens get sent to a few fixed experts, others become dead weights. The author didn’t address this much in the README—users should monitor router load balance metrics.

3. Selecting the sparse subset is crucial.
The current implementation appears heuristic and simple. Future iterations could adopt something like MoE-Sieve (“hot expert identification + targeted fine-tuning,” from March 2026’s paper cutting costs by 70% with minimal drop). That could greatly enhance USAF’s upper bound performance. For now, that’s the area most worth improving.

4. AMD finally has a usable MoE fine-tuning route.
No need to elaborate—ROCm has long been sidelined in LLM training ecosystems. USAF shows consumer-grade AMD cards can actually do meaningful work here.

Who should give it a try

In short:

• Developers with a 12–24GB consumer GPU wanting to locally fine-tune Qwen3-30B-A3B or gpt-oss-20b
• Researchers curious about MoE internals and preferring unwrapped implementation code
• ROCm / AMD users, especially those discouraged by CUDA-only ecosystems
• Independent developers building vertical-domain small models (datasets under ~10k samples) and fine with slower training speeds

Not recommended for:

• Serious research requiring reproducible, paper-level numbers
• Production environment fine-tuning
• Projects depending on the latest torch._grouped_mm or flash-attn features

By the way

If you just want to use APIs like Qwen3, gpt-oss, DeepSeek, or Claude rather than running local fine-tuning, OpenAI Hub now aggregates all of them under one key, compatible with the OpenAI API format and accessible from China. It’s convenient for baseline comparison or constructing fine-tuning datasets. Fine-tune locally, and during evaluation, use Hub’s flagship models as judges—that’s a workflow I currently enjoy.

In closing

USAF isn’t perfect. It’s a personal project—docs incomplete, benchmarks missing, and an early-stage sparse strategy. But it makes one important claim—MoE’s sparsity should be actively leveraged by fine-tuning methods, not suppressed by LoRA’s density assumptions.

The author repeatedly emphasized, “I don’t plan to commercialize this; I just think it’s interesting and want to share.” This kind of spirit is increasingly rare in the 2026 open-source community. Regardless of how far it goes, it provides a clear, unconventional answer to “what MoE fine-tuning should look like.”

It deserves a star, a fork, and at least one test run.

References

• USAF GitHub Repository – Open-source MoE sparse fine-tuning solution by tsuyu122, under Apache 2.0
• Reddit r/MachineLearning Thread – Original Reddit post by the author and community feedback
• r/LocalLLaMA Discussion on MoE Fine-tuning Difficulty – Community observations about MoE fine-tuning barriers, providing context for USAF’s emergence