How can we make hot join deterministic with CharacterVirtual?

[Alan-FGR]

I’m trying to make hot joining deterministic in a multiplayer setup.

The client joins from an empty world, receives the current authoritative world snapshot, recreates bodies with explicit server-provided BodyIDs, and restores the rigid body state. For CharacterVirtual, I’m also passing the inner body ID authoritatively via mInnerBodyIDOverride.

Broadly: what is the expected/recommended way to hot join deterministically with Jolt?

A few specific questions:

1. For CharacterVirtual, is SaveState / RestoreState sufficient, or is there additional state that must be serialized manually?
2. Does the restore order matter between PhysicsSystem, CharacterVirtual, and CharacterVsCharacterCollision registration?
3. Is calling CharacterVirtual::ExtendedUpdate before PhysicsSystem::Update valid for this setup?

I’m not expecting Jolt to provide full game-world serialization; I’m trying to confirm the correct low-level requirements so the game layer can provide the rest.

Thank you in advance!

[jrouwe]

For CharacterVirtual, is SaveState / RestoreState sufficient, or is there additional state that must be serialized manually?

Everything you pass through CharacterVirtualSettings is not saved by SaveState so you must ensure that these settings are the same. If you modified one of these properties afterwards using e.g. CharacterVirtual::SetMaxStrength you need to ensure that you set them correctly on the remote. All properties that are changed by the simulation are saved in the SaveState function (position, rotation, contacts etc.).

Does the restore order matter between PhysicsSystem, CharacterVirtual, and CharacterVsCharacterCollision registration?

PhysicsSystem / CharacterVirtual order doesn't matter. CharacterVsCharacterCollision insertion order should not matter. The hit results are sorted.

Is calling CharacterVirtual::ExtendedUpdate before PhysicsSystem::Update valid for this setup?

The order in which you call CharacterVirtual::ExtendedUpdate / PhysicsSystem::Update is up to you. As long as you do this consistently, the sim should be deterministic.

Beware that there is currently 1 case that I'm aware of where determinism will break down for CharacterVirtual and that is when a character exceeds mMaxNumHits contacts. In this case the furthest hits will be discarded in a non-deterministic fashion (because the broadphase delivers hits in a non-deterministic order) and there is a chance of divergence. You can check if this happened through CharacterVirtual::GetMaxHitsExceeded (and the current solution is to make sure that mMaxNumHits is high enough so you won't hit it).

[Alan-FGR]

@jrouwe, thanks for the reply. I'm still having rare situations in which I'm experiencing indeterminism though that are very hard to repro.
In order to rule out some lines of investigation, could you please clarify whether enabling determinism simulation and disabling constraint warm starting and body pair contact caching should be enough to make the solver/integration stateless?

[jrouwe]

No there are other things like:

• Sleep timers (which you could bypass by disable sleeping)
• Some constraints cache axis to have a fallback in case the axis is degenerate next frame (cannot be bypassed)
• CharacterVirtual itself uses last frame's contacts to cancel movement on steep slopes (could be bypassed by calling RefreshContacts before updating)
  (and there may be more that I forgot)

What I would suggest is to define JPH_ENABLE_DETERMINISM_LOG in DeterminismLog.h. This will produce a huge detlog.txt log file. You can compare this file between clients to find the first moment where they deviate. You may need to sprinkle more JPH_DET_LOG statements throughout the code to narrow it down.

[Alan-FGR]

Thanks for the reply!

What I would suggest is to define JPH_ENABLE_DETERMINISM_LOG in DeterminismLog.h. This will produce a huge detlog.txt log file. You can compare this file between clients to find the first moment where they deviate. You may need to sprinkle more JPH_DET_LOG statements throughout the code to narrow it down.

This is exactly what I'm doing.

I managed to somehow workaround the issues I am having, and most of them seem to fundamentally stem from the fact the BP isn't deterministic. I've modified code as to use deterministic data for tie-breaks of BP queries and also am calling OptimizeBroadPhase() when replicating elsewhere to make sure that matches. Another source of troubles was the BodyManager::mActiveBodies list, which I'm now brute-force ordering for updates.

[jrouwe]

also am calling OptimizeBroadPhase() when replicating elsewhere to make sure that matches

This shouldn't be needed when you repeat the collision check against Body::GetWorldSpaceBounds as I specified in the documentation.

Another source of troubles was the BodyManager::mActiveBodies list, which I'm now brute-force ordering for updates.

Ah yes, that's true. I've added that to the docs now.