MUGEN

Motion Understanding & Generation

A Unified Framework for Efficient Motion Understanding and Generation

No codebook, one draw.

One continuous latent interface for both directions · HumanML3D & SnapMoGen

Paper Code Model Results

11.1GFLOPs

An order of magnitude less compute

Per generated motion, against 95.5 and 105.5 for the strongest published baselines.

Kslots

No quantization anywhere

A clip of any length becomes K continuous slots. Nothing in the pipeline looks up a codebook entry.

1draw

One shot, no refinement

K language-model steps, then one sample and one decoder pass. There is no denoising chain to run.

Generated by MUGEN from the text beneath each clip.

Method

A shared motion–language interface

Generation predicts the latent slots from text, and understanding reads the same slots back. Neither direction has a tokenizer or a decoder of its own.

01 Adaptive-length autoencoder Any-length clip in, K continuous slots out, and back to any frame count.
02 Depth-routed states A prompt-conditioned router gives each slot its own mixture over all layers.
03 Calibrated latent head One low-rank-plus-diagonal Gaussian over the whole latent set.
04 Joint training Both directions run on the same batches through the same backbone.

MUGEN pipeline: a MOT-seeded K-step rollout keeps all layer states, a router assigns each slot a depth mixture, and a calibrated low-rank head draws the latent set for the frozen ALAE decoder; the understanding branch maps an input motion through the frozen encoder and a projector back into the same language model. — Overview. Flames are trainable, snowflakes frozen. **Top:** a `<MOT>`-seeded K-step rollout keeps every layer state, the router picks a depth mixture per slot, and the calibrated head draws the whole latent set at once for the frozen decoder. **Bottom:** the frozen encoder turns an input motion into the same K slots, and a projector hands them to the same language model.

1 Adaptive-length autoencoder, no codebook

ALAE architecture: an encoder produces a temporal memory, K learned latent queries cross-attend to it to give K latent slots, and T sinusoidal phase queries cross-attend back to the slots to reconstruct the motion.

ALAE. Learned queries cross-attend to a temporal memory to give exactly K slots, and the decoder rebuilds the motion from those slots.

A clip of any length becomes exactly K slots

A convolutional backbone turns the input into a temporal memory, and K learned queries cross-attend to it instead of looking up codebook entries. An orthogonality penalty keeps the queries from collapsing onto each other.

The decoder asks for frame t by its relative phase in the clip rather than an absolute index, so one decoder covers every length while K stays fixed.

K slots, D = 512 4 encoder + 4 decoder layers Frozen after stage one

2 Depth-routed hidden states

Layer-selection preference of the two slots: k0 concentrates 54% of its routing mass on layer 10 with 2.9 effective layers, while k1 spreads over layers 5 to 11 with 6.3 effective layers.

Router. Layer preference of the two slots on HumanML3D. k₀ reads deep and narrow at layer 10, while k₁ scans layers 5–11.

Each slot reads from the depth it needs

Reading the final layer is a habit inherited from next-token prediction. At small budgets every slot carries a large part of the sequence, so slots that all read the last layer see the same kind of evidence.

MUGEN gives each slot its own distribution over the L layers, conditioned on the prompt. The logits are a bounded residual: a prompt-independent table plus a zero-initialized correction, each squashed separately. The last layer stays a special case, so nothing is given up.

FID 0.123 → 0.087 29% better at fixed budget Same top layer on only 49% of prompts

3 Calibrated low-rank factor head

One draw has to carry all the variance

A single-step sampler gets no second chance, so its one draw has to hold the variation the description allows. A deterministic head gives one motion per prompt, and per-dimension noise cannot express variation shared between slots.

MUGEN puts a low-rank-plus-diagonal Gaussian over the whole flattened latent set. The learned basis U spans across slots, so one sample carries correlation between slots as well as inside them. With the amplitudes at zero the family reduces to the diagonal model.

q(z̄ | y) = N( μ(y), U diag(a(y))² Uᵀ + diag(σ(y)²) )

The head is fitted by exact maximum likelihood with a decoupled anchor: one term trains the mean under a detached precision, the other fits the covariance to the detached residual. Without that split the likelihood explains mean error away as inflated variance. No KL term is needed, and σ(y) ends up genuinely text-dependent.

Text → Motion

Generated from the caption beneath it

Every clip comes from the HumanML3D test set, sampled once from the calibrated distribution at the reported operating point. We do not re-rank the samples or keep the best of several seeds.

Motion → Text

The same slots, read the other way

The frozen encoder maps a real motion to the same K slots. The dataset annotation under each caption is there for comparison and never reaches the model.

Evaluation

Two benchmarks, both directions

HumanML3D and SnapMoGen, each with its own splits and evaluator, under the official 20-replication protocol. Baseline numbers come from the original papers.

A Text-to-motion on HumanML3D

Method	R@1 ↑	R@2 ↑	R@3 ↑	FID ↓	MM-Dist ↓	Diversity	MM
Real motions	0.510	0.700	0.795	–	3.012	9.503	–
Generation only
MotionDiffuse	0.491	0.681	0.782	0.630	3.113	9.410	1.55
T2M-GPT	0.491	0.680	0.775	0.116	3.118	9.761	1.86
MMM	0.504	0.696	0.794	0.080	2.998	9.411	1.16
MoMask	0.521	0.713	0.807	0.045	2.958	9.620	1.24
BAMM	0.525	0.720	0.814	0.055	2.919	9.717	1.69
MoMask++ in-context	0.528	0.718	0.811	0.072	2.912	–	1.23
MoMask++ cross-attn	0.517	0.709	0.803	0.069	2.948	–	1.19
Generation & understanding
MotionGPT	0.492	0.681	0.778	0.232	3.096	9.528	2.01
MotionGPT3	0.553	0.747	0.837	0.208	2.725	9.700	1.02
UniMo	0.539	0.738	0.831	0.177	2.768	10.042	1.92
MUGEN (ours)	0.579	0.772	0.859	0.087	2.629	9.510	0.83

Table 1. MUGEN has the best retrieval precision and matching distance in the table, both past the real-motion reference, and the best FID among language-model generators. The masked-codebook pipelines keep the overall FID lead, but their best R@1 trails MUGEN by 0.051.

B Text-to-motion on SnapMoGen

Method	R@1 ↑	R@2 ↑	R@3 ↑	FID ↓	CLIP score ↑	Diversity	MM
Real motions	0.940	0.976	0.985	0.001	0.837	19.5–19.8	–
Generation only
MDM	0.503	0.653	0.727	57.78	0.481	–	13.41
T2M-GPT	0.618	0.773	0.812	32.63	0.573	–	9.17
StableMoFusion	0.679	0.823	0.888	27.80	0.605	–	9.06
MARDM	0.659	0.812	0.860	26.88	0.602	–	9.81
MoMask	0.777	0.888	0.927	17.40	0.664	–	8.18
MoMask++ in-context	0.805	0.904	0.938	15.56	0.684	–	6.56
MoMask++ cross-attn	0.802	0.905	0.938	15.06	0.685	–	7.26
Generation & understanding
MUGEN (ours)	0.815	0.916	0.947	21.05	0.698	19.79	6.09

Table 2. MUGEN is the only entry here that also does understanding, and it leads at every retrieval rank and on CLIP score with confidence intervals separated. FID is the one column where the residual codebooks stay ahead. An oracle decode traces that gap to the sampler, not the representation.

C Motion understanding on HumanML3D

Method	R@1 ↑	R@2 ↑	R@3 ↑	MM-Dist ↓	BLEU@1 ↑	BLEU@4 ↑	ROUGE ↑	CIDEr ↑	BERTScore ↑
Real motions	0.523	0.725	0.828	2.901	–	–	–	–	–
Understanding only
LaMP-M2T	0.547	–	0.831	2.808	47.8	13.04	37.1	28.9	32.7
Generation & understanding
TM2T	0.516	–	0.823	2.935	48.9	7.00	38.1	16.8	32.2
MotionGPT	0.543	–	0.827	2.821	48.2	12.50	37.4	29.2	32.4
MoTe	0.577	–	0.871	2.649	46.7	11.15	37.4	31.5	30.3
MotionGPT3	0.573	0.773	0.864	2.426	59.08	19.41	46.17	28.72	35.23
UniMo	–	–	–	–	63.10	19.74	48.8	46.69	54.26
MUGEN (ours)	0.586	0.784	0.873	2.536	60.75	21.98	47.37	50.41	43.37

Table 3. MUGEN trains no captioning-specific variant, yet it leads every reported retrieval rank plus BLEU@4 and CIDEr. UniMo, trained on curated chain-of-thought annotations, leads the remaining caption metrics and reports no retrieval.

D Ablations

HumanML3D	R@1 ↑	R@3 ↑	FID ↓	Div.	MM
Latent budget K
K = 1	0.574	0.859	0.109	9.50	0.67
K = 2 flagship	0.579	0.859	0.087	9.51	0.83
K = 4	0.577	0.862	0.091	9.46	0.70
K = 8	0.567	0.853	0.111	9.45	0.91
K = 16	0.566	0.852	0.143	9.46	0.74
Rollout interface at K = 2
Last layer only	0.574	0.860	0.123	–	1.05

Table 4. Retrieval is flat over K = 1–4 and drops after. Inference cost grows with K, so we take the smallest budget that holds quality. Depth routing helps more than any change of budget inside that range.

Training objective	FID ↓	R@1 ↑	CIDEr ↑	BERT ↑	Params
Generation only	0.107	0.578	–	–	221.9M
Understanding only	–	–	52.8	44.8	222.3M
Joint, no loss floor	0.100	0.574	40.7	39.2	222.3M
Joint, λ = 0.25	0.089	0.576	53.5	44.7	222.3M
Joint, λ = 1 flagship	0.087	0.579	50.4	43.4	222.3M
Joint, λ = 4	0.111	0.571	49.7	43.2	222.3M

Table 5. Dropping the captioning branch makes generation worse, not better (FID 0.107 vs. 0.087, disjoint confidence intervals): understanding regularizes the shared representation. The reverse cost is small, within 5% of a dedicated captioner's CIDEr for a 0.39M projector.

Where the router looks depends on the verb. k₀ stays on layer 10 for almost everything, while k₁ moves with the action: shallow for crawl, punch and kick, mid-depth for run, walk and jump, deep for pick, clap and wave. A router with fixed weights has no way to express that.

Per-verb layer preference heatmaps for the two slots, showing k0 fixed on layer 10 and k1 shifting with verb semantics. — Per-verb routing. Mean routing weight over GPT-2 layers for the 20 most frequent verbs in the HumanML3D test split.

Efficiency

K steps, one draw, one decoder pass

One NVIDIA L4, the same pinned prompt list, each model at the operating point its own paper reports.

Compute and latency do not rank the same way, so the table below reports both. MotionGPT3 needs 8.6× the compute but 14.3× the time, because a token-by-token rollout is latency-bound. MoMask++ needs 9.5× the compute but only 5.8× the time, because its 18 guided steps are big matrix multiplies that keep the GPU busy.

Method	Params (M)	GFLOPs / motion	Latency (ms)	Throughput (motion/s)
MoMask++	150.2	105.5	55	55.1
MotionGPT3	299.7	95.5	136	53.2
MUGEN (ours)	222.3	11.1	9	325.2

Table 6. Model size is not the explanation. MUGEN's parameter count and peak memory both sit between the two baselines'. The difference is 18 masked-decoding steps, or a token-at-a-time diffusion head, against K steps and one draw.

Citation

BibTeX

@misc{ye2026mugenunifiedframeworkefficient,
      title={MUGEN: A Unified Framework for Efficient Motion Understanding and Generation},
      author={Zhankai Ye and Yukai Jin and Bingyang Wei and Bofan Li and Yusen Wu and Fangyi Li and Shangqian Gao and Xin Liu},
      year={2026},
      eprint={2607.27581},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.27581},
}