Monarch Arrives on AMD GPUs

PyTorch Brings Single-Controller Training to AMD Clusters

The PyTorch team recently revealed that the Monarch distributed programming framework can now run on AMD Instinct GPUs and the ROCm software stack. Starting from a single Python program, developers can create and manage cross-node processes, Actors, and GPU tasks without splitting complex training workflows into numerous independent jobs stitched together with launch scripts, environment variables, and external schedulers.

As of July 25, 2026, this port covers Monarch’s GPU runtime, memory and synchronization mechanisms, as well as GPU communication in multi-node environments. The teams involved have also validated training workloads on MI300-class clusters, showing that the effort has moved beyond merely getting the code to compile and is beginning to support real-world cluster scenarios.

On the surface, this may look like just another PyTorch component gaining ROCm support. What deserves more attention, however, is that AMD is gaining more than operator compatibility—it is gaining a control plane for next-generation training workflows.

A Single Controller Does Not Mean “Training in a Single Process”

Monarch’s signature single-controller model is often misunderstood as concentrating all computation in one Python process. The opposite is true: data and computation remain distributed across multiple machines, processes, and large numbers of GPUs. What is centralized is the task orchestration logic.

Traditional distributed training can be thought of as a group of actors receiving the same script. After each process starts, it reads environmental information such as rank, world size, and local rank, then follows different branches of the same code. In practice, control logic is scattered across every process. This is the classic SPMD model. Developers generally follow this approach when using torchrun with DistributedDataParallel, FSDP, or tensor parallelism.

Monarch is more like a director’s console. A single Python control program is responsible for declaring:

• Which hosts should run processes;
• Which GPUs form a logical mesh;
• Which Actor is responsible for training, sampling, evaluation, or data preparation;
• Which tasks can run asynchronously;
• How results, exceptions, and state are returned to the controller.

The actual computation is still performed by remote processes. The controller does not perform matrix multiplication in place of the GPUs, nor does it need to hold all training data. “Single controller” therefore describes the programming model, not a physical single point of computation.

The conceptual code below illustrates the difference between the two approaches. It is not a directly runnable example of the current Monarch API; consult the relevant project version for the exact interfaces:

# Traditional SPMD: every worker runs the same entry point and determines
# its behavior based on its rank
rank = distributed_rank()
model = build_model().to(local_gpu(rank))
train(model, shard=rank)

# Monarch-style control: the control program creates a mesh and remote Actors
mesh = create_process_mesh(hosts=cluster_hosts, gpus_per_host=8)
trainers = mesh.spawn(Trainer)
evaluators = mesh.spawn(Evaluator)

train_job = trainers.train_async(checkpoint)
eval_job = evaluators.evaluate_async(train_job.output)
await_all(train_job, eval_job)

The change may appear to be little more than replacing rank checks with object calls, but it represents a restructuring of the execution model. Process lifecycles, task dispatch, asynchronous results, and error propagation are incorporated into a single runtime instead of being left to users to stitch together with shell scripts, queueing systems, and distributed communication code.

Why Reinforcement Learning Workflows Need It More

If the task is fixed-scale pretraining, SPMD remains perfectly viable. Thousands of GPUs start at the same time, with every rank repeatedly executing the same training loop. This model is simple, mature, and has been validated by extensive infrastructure.

Problems emerge when training is no longer a straight line.

Today’s large-model post-training workflows often include policy-model training, inference sampling, reward computation, reference-model scoring, periodic evaluation, and checkpoint handling—all at the same time. Different stages also have different hardware requirements: training nodes need high-bandwidth interconnects, inference nodes prioritize throughput, reward models may occupy only a small number of GPUs, and evaluation tasks may dynamically join or leave.

In traditional systems, these stages are often split into multiple jobs: one scheduling system launches containers, message queues transfer samples, object storage exchanges checkpoints, and monitoring scripts determine whether a particular stage has failed. The result is a “shadow runtime” that grows outside the training code itself.

Monarch attempts to bring these operations back into the Python program. Actors encapsulate stateful remote execution units, process meshes describe structured groups of resources, and asynchronous calls allow training, sampling, and evaluation to proceed in parallel. For reinforcement learning, mixture-of-experts experiments, and workloads requiring dynamic resource combinations, this model is far more natural than having “every rank execute the same main function.”

This is also the practical value of the AMD port. It gives ROCm clusters an opportunity to run more complex, PyTorch-native workflows rather than merely demonstrating that a particular Transformer training script can run on an MI300.

Migrating from CUDA to ROCm Takes More Than Changing a Few Macros

Monarch’s previous GPU path contained clear assumptions about CUDA environments. Porting it first required converting CUDA-related components to HIP, but mechanically replacing APIs was only the first step.

The GPU runtime depends on details including device discovery, streams and events, asynchronous memory operations, error handling, and interprocess resource management. CUDA and ROCm may be conceptually similar, but that does not mean their behavior is identical at the boundaries. In an ordinary single-GPU program, these differences might appear only as a compilation error. In an asynchronous, cross-node runtime, they can turn into difficult-to-reproduce race conditions or stalled tasks.

The porting work primarily falls into three layers:

1. GPU runtime porting. Converting CUDA-specific implementations to HIP/ROCm paths while preventing hardware differences from leaking into the higher-level Actor interfaces.
2. Adjusting memory and synchronization semantics. Managing the relationships among device-memory allocation, streams, events, and asynchronous execution. Monarch allows the controller to manage large numbers of remote tasks simultaneously; if synchronization semantics are handled carelessly, errors can propagate across processes.
3. Integrating multi-node communication. Enabling high-performance GPU-to-GPU communication and RDMA paths on AMD GPU clusters, while preventing data from being routed through CPUs or slowed down by the control plane.

The third layer determines whether this port has production value. Successfully running eight GPUs in a single machine demonstrates only functional compatibility. Cross-node training must also contend with network topology, RDMA registration, collective communication, GPU-direct paths, and failure diagnosis. The teams’ experience on MI300-class clusters at least indicates that Monarch’s architecture can scale to multi-node ROCm environments.

However, the currently available information is insufficient to conclude that the ROCm version performs identically to the CUDA version. Public materials focus more on architectural portability, functionality, and developer experience, without providing sufficiently complete end-to-end throughput figures, communication-efficiency data, or large-scale scaling curves. For developers, the more reasonable assessment at this stage is that it is “ready for evaluation,” rather than that mature CUDA training stacks should immediately be replaced in their entirety.

Process Meshes Make Resource Topology a First-Class Object

Another key Monarch abstraction is the process mesh. It organizes a group of hosts, processes, and devices into an operable object, allowing the controller to create Actors, broadcast tasks, and collect results across either the entire mesh or a slice of it.

This shares a similar intuition with DeviceMesh in PyTorch distributed tensors: developers do not need to repeatedly calculate by hand which data-parallel group, tensor-parallel group, or pipeline stage a rank belongs to. Instead, they express topology through mesh dimensions. The difference is that Monarch is concerned not only with how tensors are partitioned, but also with how processes and stateful services are created, addressed, and controlled.

For example, a cluster can be logically divided into three parts:

• A large mesh handles policy-model training;
• Several smaller meshes generate samples in parallel;
• Independent Actors coordinate evaluation, logging, and checkpoints.

This abstraction is particularly well suited to heterogeneous workflows. Here, “heterogeneous” does not necessarily mean mixing AMD and NVIDIA GPUs in the same system. It can also mean having different nodes perform different roles and use different parallelization strategies. Monarch’s ROCm support demonstrates that its runtime is not exclusively tied to CUDA. However, seamless collaboration among hardware from different vendors within the same training job remains constrained by operators, communication libraries, and deployment environments. It should not be assumed that adding an AMD backend has already solved this problem.

For AMD, This Is a Critical Piece of the Software-Ecosystem Puzzle

Over the past several years, AMD has invested heavily in PyTorch operators, compilers, communication libraries, and support for mainstream models. But when developers select training hardware, they consider far more than whether a particular model can start successfully.

A mature GPU platform requires at least four layers of capabilities: low-level operators and compilation tools, high-performance collective communication, distributed training frameworks, and higher-level scheduling and observability. ROCm already covers many foundational capabilities in the first two layers, while Monarch helps fill the distributed orchestration layer.

It will not immediately reshape the GPU market. CUDA’s advantages still lie in ecosystem momentum, comprehensive tooling, and a vast body of production experience. In large-scale training in particular, performance problems are often hidden in a specific kernel, communication link, or topology. API compatibility alone is nowhere near enough.

Monarch’s support for AMD nevertheless sends a more important signal: PyTorch’s next generation of distributed abstractions is beginning to move away from dependence on a single hardware backend. If frameworks are designed from the outset with replaceable device runtimes and communication layers, AMD will not always have to wait until a capability has matured on CUDA for years before trying to catch up.

For cloud providers and large model teams, this also increases negotiating leverage. Meaningful hardware diversity is not achieved by replacing the cuda string in model code with hip; it requires the entire pipeline—including training, evaluation, reinforcement learning, and failure handling—to be portable.

It Can Simplify Distributed Programming, but It Will Not Eliminate Complexity

Monarch’s most compelling narrative is that a single Python program can control an entire GPU cluster. Developers should not, however, interpret this as meaning that “distributed systems are now simple.”

A single controller can reduce scattered control logic, proliferating rank-based branches, and glue code across jobs, but several practical issues still need to be addressed:

• High availability, state recovery, and task replay for the controller itself;
• Whether an Actor failure should trigger a local restart or terminate the entire training run;
• Whether large numbers of fine-grained remote calls place excessive pressure on the control plane;
• How imperative Python logic can be recorded, reproduced, and audited;
• How responsibilities should be divided between Monarch and Kubernetes, Slurm, or existing training platforms;
• How differences in functionality, performance, and diagnostic tooling between the ROCm and CUDA backends should be exposed.

In other words, Monarch shifts complexity from “how each worker coordinates itself” to “how the runtime reliably coordinates all workers.” This is generally the right direction because a framework can solve the problem once in a standardized way, rather than forcing every team to reinvent the wheel. Even so, the new runtime must still be validated against large-scale production workloads, long-running training jobs, and complex failure scenarios.

Who Should Try It at This Stage?

Monarch’s AMD port is likely to be most attractive to:

• Teams that have already deployed MI300-class clusters and want to reduce multi-job orchestration code;
• Teams building RLHF, RLAIF, or other closed loops spanning training, sampling, and evaluation;
• Framework developers who need to create remote services dynamically and are not satisfied with pure SPMD training loops;
• Infrastructure teams that want the same high-level distributed abstractions to support both CUDA and ROCm.

If a team performs only stable DDP or FSDP training, and its existing torchrun, Slurm, and monitoring systems are already reliable, the benefits of migration may not outweigh the cost of introducing a new runtime. At present, Monarch is better viewed as an infrastructure option for complex training systems than as a new default that every PyTorch project must adopt.

Our assessment is that the AMD port is not a “2× performance” update, but it may have greater long-term significance than an individual operator optimization. Operators determine how fast a single GPU runs; the control model determines whether developers can effectively organize hundreds or even thousands of GPUs. Monarch has now demonstrated that its single-controller architecture can extend beyond CUDA environments. The next question it must answer is whether it can run reliably over the long term on large-scale ROCm clusters and provide sufficiently convincing performance and failure-recovery data.

References

• meta-pytorch/monarch GitHub repository: Monarch’s official open-source project, covering the framework’s positioning, build process, process and Actor programming models, and backend-related information for CUDA, ROCm, and other platforms.