Here are some of the core technical elements that make Frequency Domain possible.

### Getting the frequency data

The core of Frequency Domain is the Fast Fourier Transform (FFT) data being used to generate the terrain. The FFT data is stored in an array, where each element represents a “bin”/frequency and the value of the element (ranging from 0 to 1) represents the amplitude of said frequency.

Unity makes it super easy to get FFT data from audio, either from a single audio source or from the main audio listener (what the player hears, all audio). Frequency Domain uses the single audio source variant of the function : **AudioSource.GetSpectrumData **

### Generating the mesh

The terrain is generated at runtime, in code. The Unity website has a great guide on how to create a mesh from scratch here. It basically comes down to 2 main steps: generate a list of vertices, then connect said vertices into triangles.

Here is the bit of code that handles it:

List<int> trianglesList = new List<int>();
List<Vector3> verticesList = new List<Vector3>();
Vector2[] uvArray;
List<Vector2> uvList = new List<Vector2>();
// initial line
for(int j = 0; j < verticesFrequencyDepthCount; j++)
{
verticesList.Add( new Vector3(0,0,j) );
uvList.Add( new Vector2(0,0) );
}
// populate the rest of the vertices, triangles
// use verticesFrequencyDepthCount to shift between frewuency collumns
for(int i = 1; i < verticesTimeDepthCount ; i++)
{
for(int j = 0; j < verticesFrequencyDepthCount; j += 2)
{
// bottom left triangle
verticesList.Add( new Vector3(i,0, j) );
int currentListIndex = verticesList.Count -1;
trianglesList.Add(currentListIndex);
trianglesList.Add(currentListIndex - verticesFrequencyDepthCount);
trianglesList.Add(currentListIndex - verticesFrequencyDepthCount + 1);
// fill triangles in between this and previous triangle below
if( j > 0) // is not at the edge
{
// bottom left triangle
trianglesList.Add(currentListIndex -1);
trianglesList.Add(currentListIndex - verticesFrequencyDepthCount -1);
trianglesList.Add(currentListIndex - verticesFrequencyDepthCount);
// top right triangle
trianglesList.Add(currentListIndex);
trianglesList.Add(currentListIndex - 1);
trianglesList.Add(currentListIndex - verticesFrequencyDepthCount);
}
// top right triangle
verticesList.Add( new Vector3(i,0, j + 1) );
currentListIndex++;
trianglesList.Add(currentListIndex);
trianglesList.Add(currentListIndex - 1);
trianglesList.Add(currentListIndex - verticesFrequencyDepthCount);
uvList.Add( new Vector2(0,0) );
uvList.Add( new Vector2(0,0) );
}
}
verticesArray = verticesList.ToArray();
trianglesArray = trianglesList.ToArray();
uvArray = uvList.ToArray();
Debug.Log(trianglesList.Count);
mesh.Clear();
mesh.MarkDynamic();
mesh.vertices = verticesArray;
mesh.uv = uvArray;
mesh.triangles = trianglesArray;
mesh.RecalculateNormals();

### Faking physics

My initial plan was to simply use Unity’s built-in physics engine, but it ended up being too slow to use on the entire mesh at once. I was tempted to use smaller sub-meshes instead, but it didn’t feel like a very elegant solution. I soon realized that I only really needed the height information of the player, and so using the vertices list as a heightmap would suffice.

Here’s the function that handles it:

public float getHeightFromPosition(float xPos, float zPos)
{
float height = 0;
float xScale = transform.localScale.x;
float zScale = transform.localScale.z;
//normalize postion to a unit scale
xPos = xPos / xScale;
zPos = zPos / zScale;
float xFloor = Mathf.Floor(xPos);
float xCeil = Mathf.Ceil(xPos);
float zFloor = Mathf.Floor(zPos);
float zCeil = Mathf.Ceil(zPos);
//make sure position is within mesh
if( xFloor <0 || xCeil >= (float)verticesTimeDepthCount )
return 0; //return height 0 is out of bounds
if( zFloor <0 || zCeil >= (float)verticesFrequencyDepthCount )
return 0;
// get height of 4 corners arround position
float TL = getHeightFromHeightMap( (int)xFloor, (int)zFloor );
float TR = getHeightFromHeightMap( (int)xFloor, (int)zCeil );
float BL = getHeightFromHeightMap( (int)xCeil, (int)zFloor );
float BR = getHeightFromHeightMap( (int)xCeil, (int)zCeil );
float xLeftLerp = Mathf.Lerp(BL, TL, (xCeil - xPos ) );
float xRightLerp = Mathf.Lerp(BR, TR, (xCeil - xPos) ) ;
height = Mathf.Lerp( xLeftLerp, xRightLerp, zPos - zFloor );
return height;
}

### Converting to a pseudo-log scale

The FFT data coming from Unity is linear, i.e. all the frequencies are equally spaced out from one another. Unfortunately, audio/music “notes” as we hear them are not linearly space out, they’re more on on a log scale. In order to represent the entire audio spectrum at maximum FFT resolution (8192 data points), I had to implement some sort of logarithmic scaling to the linear data I got from Unity. The following piece of code allows me to convert those 8192 data points into 100.

// get raw FFT data
audioSourceArray[0].GetSpectrumData(sampleArrayFreqBH, 0, FFTWindow.BlackmanHarris);//Rectangular);
// cleanup pseudolog array first
for(int i = 0; i < pseudoLogArray.Length; i++)
pseudoLogArray[i] = 0;
// doing the pseudo log scale
int decadeIndex = 0;
int fftSampleCounter = 0;
for(int i = 0; i < pseudoLogArray.Length; i++)
{
if( i != 0 && i%10 == 0)
decadeIndex++;
for(int j = 0; j < samplesPerDecadeArray[decadeIndex]; j++ )
{
pseudoLogArray[i] += sampleArrayFreqBH[fftSampleCounter];
fftSampleCounter++;
}
}