131 hours of motion on an RTX 5070

XMimic trains a humanoid motion tracker entirely on the GPU: the simulator, the replay buffer, the networks and the reference motions all live in device memory, and whole training iterations are compiled into one program. That is what makes it fast, and it means the motion library has to fit on the card too.

With the 40 LAFAN1 clips this is a non-issue. BONES-SEED is another scale. After dropping the clips that cannot be done on a flat, empty floor (more on that below), 64,697 of its 67,036 clips remain: 131 hours of motion, 23.6 million frames at the 50 Hz control rate.

The GPU I have is an RTX 5070: a mid-range consumer card with 12 GB of memory. I wanted the whole set to train on it.

The trainer holds 4.7 GB of its own state and the compiled program needs another 3.7 GB of temporaries, which leaves 3.6 GB of the card for motion data. The library stored 1.35 kB for each frame, about 32 GB in all. Together that is 40.3 GB, more than the 32 GB of an RTX 5090 at the top of the same lineup, so the run belonged on a 48 GB workstation or datacenter card.

The rest of this post is how 32 GB of reference motion got into those 3.6 GB.

Most of those 1.35 kB were conveniences. For every frame the loader precomputed the root's linear and angular velocity, the joint velocities, and the position, rotation and velocities of every tracked body. All of it is a function of three things: where the root is, how it is rotated, and the 29 joint angles.

So the library now stores only those three, and the rest is computed at the moment a frame is indexed, with the same formulas the loader used:

• Velocities are central differences of the neighbouring frames, one-sided at the two ends of a clip.
• Body poses and velocities come from forward kinematics on the stored root and joints.

def vel(self, f):
    c = self.clip_id[f]
    fp = jnp.minimum(f + 1, self.clip_end[c])     # stay inside the clip
    fm = jnp.maximum(f - 1, self.clip_start[c])
    dt = (jnp.maximum(fp - fm, 1) * CONTROL_DT)[..., None]
    v = (self.root_p[fp] - self.root_p[fm]) / dt
    ...

The derived fields still index like arrays (motion.body_p[f], motion.qd[f]), so the reward and observation code did not change. Inside a compiled iteration the compiler merges the work that several fields share at the same frames.

This alone takes a frame from 1.35 kB to about 180 bytes, and the library from 32 GB to 4.26 GB.

4.26 GB is still about 0.7 GB more than the 3.6 GB the card has free. The remaining bytes went in four places:

• Root rotation as a float32 quaternion instead of a 3×3 matrix: 16 bytes instead of 36.
• Joint angles as int16 fixed point over ±3.2 rad: 58 bytes instead of 116, with a step of 9.8e-5 rad.
• Clip index not stored at all. Every frame already knows its one-second reset bin, and the bin knows its clip.
• Foot contacts as one byte per foot.

Q_MAX = 3.2  # rad: int16 fixed point over [-Q_MAX, Q_MAX], step 9.8e-5 rad

# when the library is built
q_i16 = np.round(np.clip(q, -Q_MAX, Q_MAX) * (32767 / Q_MAX)).astype(np.int16)

# when a frame is read
q = q_i16[f].astype(jnp.float32) * np.float32(Q_MAX / 32767)

That is 92 bytes a frame and 2.18 GB for the 131 hours. Compared against the float32 library on the same frames, the largest differences are:

A sixteenth of a millimetre in body position is far below anything a tracking reward can feel.

With everything precomputed the run needed 40.3 GB. Dropping the recomputable fields brought it to 12.7 GB, within reach of the card but still over, and the quaternion and the int16 angles got it under the line at 10.6 GB: 2.2 GB of motion next to the trainer's 8.4 GB, with 1.4 GB of the card to spare.

With that, XMimic trains on all 131 hours on the one RTX 5070, and I left it running overnight. What the policy learns from is the same as before: the reward and observation code read the same fields, to within the differences in the table above.

Clips that need furniture

A motion-capture set of this size is full of people sitting on chairs, climbing stairs and leaning on railings. The simulator has a flat, empty floor, so a robot asked to follow those clips can only fall. Clips whose description names a prop the body stands on, sits on or leans against are dropped; held or hit props stay.

A second check works from the reference motion alone. It flags a clip when tracked bodies sink into the floor, when the body floats, or when a nearly still pose has the pelvis outside the support of the limbs on the ground, which means sitting on or leaning against something that is not there. Push-ups, planks, crawling and get-ups pass. 64,697 of 67,036 clips survive.

Sampling where to start

Each simulated robot restarts from a random one-second bin of the library, weighted by how badly the policy does there. The stock categorical sampler builds an array of random numbers with one entry per robot and per bin. With roughly half a million bins that array was its own memory problem, so the bins are now drawn by inverse-CDF sampling, which needs one number per robot.

Loading it at all

Parsing tens of thousands of CSV files and resampling them to 50 Hz takes a while, so files are parsed in parallel processes and the resampled clips are cached next to the data.

• This post is about getting the training run onto the card. It makes no claim about how well the policy tracks all 131 hours.
• Deriving velocities and body poses on demand costs compute on every lookup. I have not measured how much it slows an iteration.
• Joint angles beyond ±3.2 rad would be clipped. The loader counts them and warns.
• The error figures compare the compact library with the float32 one, not with the original motion capture.
• "Fits" means 10.6 GB of the card's 12 GB, at the batch sizes of this run.