Rebody is based on a full-body motion capture recording. The recording contains data for 17 joints of the human body (see names and hierarchy in the diagram to the right). Each joint is measured (reconstructed from an array of infrared video images) in its absolute three-dimensional position and rotation. Additonally, all joints are represented according to their hierarchy as relative positions and rotations as specified by the degrees of freedom of each joint.
The following steps describe the different stages of the data transposition in Rebody:
Point selection
(1) For Rebody only the absolute positions data (X, Y, Z) has been used. The coordinate system layout is such that if looking in the positive Y direction, then positive X is right and positive Z points up.
(2) In a first transformation, the joint positions have been expressed relative to the position of the pelvis, i.e. subtracting the global body movement (locomotion), effectively pinning it down at the pelvis. This makes the pelvis position the origin of the new coordinate system.
(3) Out of the 17 joints, 12 have been selected (head, lhumerus, lradius, lhand, rhumerus, rradius, rhand, thorax, ltibia, lfoot, rtibia, rfoot). They are marked in bold in the diagram to the right.
(4) The 3D positions are projected orthographically to the X/Z plane, i.e. the Y coordinate is omitted. Effectively, the 3D Z cordinate becomes a 2D Y coordinate.
(5) According to a parameter called circ, which can vary between 0 and 1, the 12 2D points are projected onto a circle with a fixed radius. If circ is zero, the coordinates are not translated. If it is one, all coordinates are translated to a position on the circle. In between, a linearly scaled translation is applied (i.e. for a circ value of 0.5, a transformed point lies midway between its original position and its projected position on the circle). This can be expressed with the following formula, where P is a 2D point and |P| is its distance from the origin:
Pt = P * (1 + (radius / |P| - 1) * circ)
Algorithmic drawing
(6) The 12 points are used to create an algorithmic drawing connecting some of the points with straight lines starting from 'rfoot'.
(7) The drawing is done following a pseudo-triangulation drawing method developed by Michael Schwab.
(7a) The three points closest to 'rfoot' are identified and noted in list L in the order of nearest first. Lines are drawn between 'rfoot' and each point repectively.
(7b) The three points closest to the first point in L (P1) are sought, providing they are not yet connected to P1. If they are not already in L, they are appended to L, again nearest first. Lines are drawn between P1 and each of those three points regardless of whether or not a point had been appended to L.
(7c) The next point in L (P2) is chosen and step (7b) is repeated until there is no further point in L. It may happen that not all 12 points will be used during this procedure.
Algorithmic sound production
(8) The algorithmic sound production is based on an analysis of the algorithmic drawing, estimating the following attributes for each frame:
(8a) instantaneous speed of each point
(8b) average speed of each point (parameter: averaging interval in frames)
(8c) relative position of each point in the (variable) bounding box of the drawing
(8d) appearance of a new line
(8e) disappearance of a line
(9) A source-filter sound model has been defined as a synthesis algorithm implemented in SuperCollider.
(9a) As source a resonant lowpass filtered Dirac pulse (a click) is generated for each point in the drawing. The bandwidth and the resonance frequency of the filter are variable. Each click is fed into two parallel filters:
(9b) a comb filter with variable delay and feedback, and
(9c) a 3-step octavation delay line followed by a lowpass filter connected in series with a 2-formant vocal filter realised as a parallel section of a bandpass and a resonant lowpass. The degree of actaviation can be varied by linearly mixing the 3 sections (two, four, and eight times the frame rate). The cutoff frequency of the lowpass filter allows variation in the brightness of the excitation signal (stress). The centre frequencies and bandwidths of the two formants are also variable.
(9d) The source click (9a) and the outputs of the two filters (9b and 9c) can be panned individually and sent directly to the model output and to two differently parametrised reverbs whose outputs are mixed with the model output. A 3x3 matrix allows control of all send levels independently.
(10) The result of the analysis is mapped to the synthesis parameters following the rules described here.