Scalable Group Choreography via Variational Phase Manifold Learning (Part 2)

In the previous part, we have explore the introduction about group dance scalability and the what is manifold. In this part, we introduct our main proposal Variational Phase Manifold Learning.

#Task Definition

Given an input music sequence $\mathbf{a} = \{a_1, a_t, ...,a_T\}$ with $t = \{1,..., T\}$ indicates the index of the music frames, our goal is to generate the group motion sequences of $N$ arbitrary dancers: $\mathbf{x} = \{x^1_1,..., x^1_T; ...;x^N_1,...,x^N_T\}$ where $x^n_t$ is the pose of $n$-th dancer at frame $t$. We use the 6D continuous rotation for every joint, along with 3D joint positions and velocities. Additionally, the corresponding 3D root translation vectors are concatenated into the pose representations to involve the trajectory of motion. Previous group dance methods, which generate the whole group at once, cannot deal with the increasing number of dancers and can only create group sequences up to a pre-defined number of dancers, due to the vast complexity of the architecture. In contrast, we aim to generate group dance with an unlimited number of dancers.

#Phase-conditioned Dance VAE

Our goal is to learn a continuous manifold such that the motion can be generated by sampling from this learned manifold. We assume that although different dancers within the same group may present visually distinctive movements, the properties of their motions, such as timing, periodicity, or temporal alignment are intrinsically similar. We aim to learn a generative phase representation for each group of dancer in order to synthesize their motion indefinitely. Our generative model is built upon the conditional Variational Autoencoder architecture, thanks to its diverse generation capability and fast sampling speed. However, instead of directly encoding the data into a Gaussian latent distribution as in common VAE approaches, we model the latent variational distribution by the phase parameters extracted from the latent motion curve, which we call variational phase manifold. The latent phase manifold is well-structured and can well describe key characteristics of motion (such as its timing, local periodicity, and transition), which benefits learning motion features.

The overview of our Phase-conditioned Dance VAE is illustrated in Figure 3. Specifically, the model contains three main networks: an encoder $\mathcal{E}$ to capture the approximate posterior distribution conditioned on both motion and music $q_\phi(\mathbf{z}|\mathbf{x},\mathbf{a})$, a prior network $\mathcal{P}$ to learn the conditional prior given only the music $p_\theta(\mathbf{z}|\mathbf{a})$ , and a decoder $\mathcal{D}$ to learn to reconstruct the data from the latent distribution $p_\theta(\mathbf{x}|\mathbf{z},\mathbf{a})$. The new motion is generated by sampling the frequency-domain parameters predicted by the prior network, which is then passed through the decoder network to reconstruct the motion in the original data space. Furthermore, we adopt Transformer-based architecture in each network to effectively capture long-range dependencies and holistic context of the dance sequence.

#Encoder

The encoder $\mathcal{E}$ is expected to take both the motion and music feature sequence as input, and produce a distribution over possible latent variables capturing the cross-modal relationship between them. To transform the joint input space into a learned phase manifold, we adopt the Transformer decoder architecture where the Cross-Attention mechanism is utilized to learn the relationship between the motion and the music. Accordingly, the output of the encoder is a batch of latent curves (i.e., the activation sequences per channel) that can particularly capture different spatial and temporal aspects of the motion sequence. However, instead of training the model to directly reconstruct the input motion from the extracted latent curves, we further enforce each channel of the latent space to have a periodic functional form (i.e., sinusoidal). This enables us to effectively learn a compact parameterization for each latent channel from a small set of parameters in the frequency domain.

#Generative Variational Phase Manifold

Here we focus on learning the periodicity and non-linear temporal alignment of the motion in the latent space. In particular, given the output latent curves from the encoder $\mathbf{L} = \mathcal{E}(x,a) \in \mathbb{R}^{D \times T}$ with $D$ is the number of desired phase channels to be extracted from the motion, we parameterize each latent curve in $\mathbf{L}$ using a sinusoidal function with amplitude ($\mathbf{A}$), frequency ($\mathbf{F}$), offset ($\mathbf{B}$) and phase shift ($\mathbf{S}$) parameters. To allow for variational phase manifold learning, we opt to predict two sets of parameters $\mathbf{\mu}_{\mathcal{E}} =\{\mathbf{\mu}^A; \mathbf{\mu}^F; \mathbf{\mu}^B; \mathbf{\mu}^S \}$ and $\mathbf{\sigma}_{\mathcal{E}} =\{\mathbf{\sigma}^A; \mathbf{\sigma}^F; \mathbf{\sigma}^B; \mathbf{\sigma}^S \}$, which corresponds to the mean and variance of $\mathbb{R}^{4D}$ dimensional Gaussian distribution:

To do so, we first apply differentiable Fast Fourier Transform (FFT) to each channel of the latent curve $\mathbf{L}$ and create the zero-indexed matrix of Fourier coefficients as $\mathbf{c}=FFT(\mathbf{L})$ with $\mathbf{c} \in \mathbb{C}^{D \times K+1}$, $K =\lfloor \frac{T}{2}\rfloor$. Correspondingly, we compute the per channel power spectrum $\mathbf{p} \in \mathbb{R}^{D \times K+1}$ as $\mathbf{p}_{i,j} = \frac{2}{N}|\mathbf{c}_{i,j}|^2$, where $i$ is the channel index and $j$ is the index for the frequency bands. Correspondingly, the distributional mean parameters of the periodic sinusoidal function are then calculated as follows:

where $\mathbf{f} = (0, \frac{1}{T},\dots,\frac{K}{T})$ is the frequencies vector. At the same time, the phase shift $\mathbf{S}$ is predicted using a fully-connected (FC) layer with two $\arctan$ activation as:

To predict the distributional variance of the phase amplitude and phase shift parameters $\{\mathbf{\sigma}^A, \mathbf{\sigma}^S\}$, We additionally apply a separate two-layer MLP network over each channel of the latent curves. The variational latent phase parameters are sampled by utilizing parameterization trick, i.e., $\mathbf{A}\sim\mathcal{N}(\mathbf{\mu}^A,\mathbf{\sigma}^A)$ and $\mathbf{S}\sim\mathcal{N}(\mathbf{\mu}^S,\mathbf{\sigma}^S)$. In our experiments, we find that sampling the phase frequency $\mathbf{F}$ and offset $\mathbf{B}$ often produce unstable and non-coherent group movements. This might be because the frequency amplitudes of the dancers within the same group are likely to associate with the rhythmic pattern of the musical beats while the offsets capture their alignment, thereby should be consistent with each other. Therefore, we treat those parameters as deterministic by constraining their variance to zero.

Finally, the sampled set of phase parameters $\mathbf{z} = \{\mathbf{A};\mathbf{F};\mathbf{B};\mathbf{S}\}$ are used to reconstruct a parametric latent space consisting of multiple periodic curves to represent each intrinsic property of the motion by:

where $\mathcal{T}$ is a known time window series obtained by evenly spacing the timesteps from $0$ to $T$. Intuitively this curve construction procedure can be viewed as a "quantization" layer to enforce the network to learn to represent the motion features in the frequency domain, which is useful in representing different aspects of human motion such as their timing and periodicity. In the last step, a decoder is utilized to reconstruct the original motion signals from the set of parametric latent curves.

#Decoder

To decode the latent space into the original motion space, previous works have to use a sinusoidal positional encoding sequence with duration $T$ as the proxy input to the sequence decoder. This is because their latent space is only formed by single latent vectors following a Gaussian distribution, which cannot span the time dimension. However, we observe that it usually results in unstable and inconsistent movements, as the proxy sequence is generic and usually contains less meaningful information for the decoder. Meanwhile, our method does not suffer from this problem as our latent space is built on multiple curves that can represent the motion information through time, thanks to the phase parameters. Subsequently, our decoder $\mathcal{D}$ is based on Transformer decoder architecture that takes the constructed parametric latent curve, as well as the music features as inputs, to reconstruct the corresponding dance motions. Here, we also utilize the cross-attention model where we consider the sequence of and music features as key and value along with the sampled latent curves as the query. The output of the decoder is a sequence of $T$ vectors in $\mathbb{R}^D$, which is then projected back to the original motion dimensions through a linear layer, to obtain the reconstructed outputs $\hat{\mathbf{x}}=p_\theta(\mathbf{x}|\mathbf{z},\mathbf{a})$. We additionally employ a global trajectory predictor to predict the global translation of the root joint based on the generated local motions, in order to avoid intersection problems between dancers.

#Prior Network.

Since the ground-truth motion is generally inaccessible at test time (i.e., we only have access to the music), we also need to learn a prior $\mathcal{P}$ to match the posterior distribution of motion from which the latent phase can be sampled. Specifically, We follow the manifold procedure to predict the Gaussian distribution conditioned on the music sequence $\mathbf{a}$, which is then used for sampling the latent phases:

where a Transformer encoder is used to encode the input conditioning music sequence and predict the corresponding $\mathbf{\mu}_{\mathcal{P}}$ and $\mathbf{\sigma}_{\mathcal{P}}$. We implement the prior network similarly to the encoder network, however, we use self-attention mechanism to capture the global music context. Learning the conditional prior is crucial for the conditional VAE to generalize to diverse types of music and motion. Intuitively speaking, each latent variable $\mathbf{z}$ is expected to represent possible dance motions $\mathbf{x}$ conforming to the music context $\mathbf{a}$. Therefore, the prior should be able to encode different latent distributions given different musics.

#Next

In the next part, we will explore the training procedure and experimental setups to validate the effectiveness of the proposed method.