Development Platform: Unity 2020
Programming Platform: Visual Studio 2020
foreword
A typical audio visualization solution in the visualization-oriented solution design. It has a wide range of applications, especially in geological exploration, medical equipment, etc. There are more typical application cases. This article takes the realization of music audio visualization as a case, focusing on the design concept and implementation of audio visualization design.
Unity can provide API instructions
There are 13 types of API objects associated with audio (Audio) in Unity, as follows:
| AudioChorusFilter | AudioClip | AudioConfiguration | AudioEchoFilter | AudioHighPassFilter | AudioListenner | AudioLowPassFilter |
|---|---|---|---|---|---|---|
| Audio Chorus Filter | audio clip | audio configuration | audio echo filter | Audio Treble Filter | audio monitor | Audio Bass Filter |
| AudioRenderer | AudioReverbFilter | AudioReverbZone | AudioSettings | AudioSource | AudioDistortionFilter | |
| audio renderer | Audio Reverb Filter | Audio Reverberation Zone (Environment) | audio settings | audio player | Audio Distortion Filter |
The UnityEngine APIs related to audio are mainly AudioClip and AudioSource. The audio resources exist as AudioClip resource types in the Unity editor. AudioClip records readable attributes such as reverberation identification, number of channels, sampling frequency, data length (time, capacity) of the audio data. AudioSource acts as a player effect, converting audio data into sound that is recognized and played by computer hardware.
Principles of Audio Visualization
1) Obtain audio source data content
The visualization service object is audio (AudioClip), so the initial focus of visualization should be the data information stored inside the audio .
Refer to the Unity Chinese documentation - AudioClip can understand that variables samplescan be used to obtain audio data information. Randomly select a music resource and import it into AssetDatabase Debug.Log()and GUILayout.Label()identify and analyze the audio data through or . It can be roughly concluded that the following variables will be used in visualization development:
Channels: number of audio channelslength: total audio durationSamples: Audio sample size
During the sampling process, the compressed audio (AudioClip) cannot beSetData()obtained through the data . The solution to this resource is to set the resource type LoadType to Decompress On Load to solve the problem.
Difficulties :SamplesThe sample data type is intintegerSadly pointless for audio visualization. However, it is quite helpful for querying the data of the content of the audio segment.
Recall that the purpose of audio visualization is to dynamically reflect the change degree of data content in real time. Finding out how to obtain audio data information in real time becomes the next focus.
2) Obtain real-time data collection by the audio player
During the running of the project, AudioSource plays AudioClip through real-time analysis. In the Unity Chinese Documentation - AudioSource, the following methods are given to obtain data:
GetOutputData(): Get the current audio output dataGetSpectrumData(): Get the spectrum data block of the current audio source
The focus is on the spectral block data. Encyclopedia explanation is the abbreviation of frequency spectral density, which is the distribution curve of frequency. Complex oscillations are decomposed into harmonic oscillations with different amplitudes and frequencies, and the graphs of the amplitudes of these harmonic oscillations arranged by frequency are called spectrum. So GetSpectrumData()it is undoubtedly a more useful method for audio visualization at present.
About GetSpectrumData()the method description in the Unity Chinese documentation public void GetSpectrumData(float[] samples, int channel, FFTWindow window):
samples: The data that fills the audio samples (the length is required to be a power of 2).
Generally, the sample data length is recommended to be 64/ 128/256/512 /.../8192. Remember not to exceed its range.
It is required to be a power of 2. The reason: computer data is read in binary, which is necessary to ensure data readability and optimal reading efficiency.channel: Audio sampling channel.window: Belongs toFTTWindowthe spectrum analysis windowing type provided by the class. Its purpose is to control the audio signal and reduce signal leakage in the frequency band.
The more complex the window type, the higher the sound quality, but at the same time the lower the speed.
From highest to lowest: Rectangular > Triangle > Hamming > Hanning > Blackman > BlackmanHarris
More explanation : GetSpectrumData()It seems that the parameter design does not meet the normal understanding out, but its purpose is to store the sampled data information into the samplescreated float[]array. Perhaps the official should use it outto make it easier for developers to understand. It may also be that there is a reason why the internal layer code cannot be changed.
Audio Visualization Implementation
1) Spectrum data sampling
Depending on GetSpectrumData()the required parameter types, prepare the following parameters:
public AudioSource ASource;: Specifies the AudioSource player object to use.public float[] SamlpesArray = new float[64];: A container for collecting spectrum data.
Note: Think about the reasons for choosing 64 instead of 63. (See explanation in the Principles section).
public void SyncSpectrumData()
{
this.ASource.GetSpectrumData(SamplesArray, 0, FFTWindow.Rectangular);
}
This is the default to sample the audio data of only one track. Indexed according to the actual AudioClip track number. Take QQ Music - The Almight Voilet Thunder - HOYOMix as an example. The audio can be seen through the Unity Preview (preview window) has two audio channels. Or it can be known through channelsthe property . The sound channel index method is similar to an array, that is, sound channel 1 corresponds to signal position 0, sound channel 2 corresponds to signal position 1, and so on.
2) Sampling data observation
After completing the frame spectrum data collection task, first use Debug.Log($"{SamplesArray[0]} {SamplesArray[10]} )the check data collection situation. Confirmed by Debug, the data range is roughly between [0, 1] (the range here is an approximate range, not an exact range). After confirming the changes in dynamic data, you can conceive how to design obvious visualization solutions.
follow-up instructions
◼About spectral data sampling frequency
In general, select Update()methods are developed for frame data acquisition. But from the actual effect, it shows frequent slider change design. It is easy to cause loss of computer performance. Especially when the sampled data capacity is increased. Therefore, it is considered to FixedUpdate()obtain the data sampling result at a fixed time. That is, the sampled data information is acquired only at the same time interval, and in this case, the data change frequency in the data container is controlled, so that the overall image fluctuation frequency is reduced. It may further be appropriate to design, for example, to acquire data every second. In order to achieve the characteristics of low energy consumption of the computer in the overall process of visualization.
◼Effect display
◼Solution about data difference
The frame data acquired each time shows no warning and trend performance. Situations such as big and small suddenly happen from time to time. When using these data, it is easy to have a sense of frequent violation. So designing a set of transition transformations is a good choice. Before the next data sampling starts, transition from the previous data point to the data of this point. It is quite a good design to produce an animation-like effect, and the associated Lerp()interpolation, such as Math.Lerp()/ Vector3.Lerp()and other methods can be realized.
summary
For any data visualization, the first confirmation is the supportability of the data. If the data clearly cannot be visualized, the focus should be on visualization in other directions. As in this article, AudioClip's data cannot build a visual design. However, AudioSource can just complete the process of audio visualization design through real-time sampling of spectrum data when playing AudioClip. When it cannot be visualized, try to explore visualization possibilities from other objects associated with it.