|
The audio and visual forms are always treated separately although
they are often used together. It seems these two faculty of
our brains can only interact superficially.
In learning a language, we use our ears and mouth. The eyes may
be used to read the lips. But hardly anyone expect to ˇ§seeˇ¨
the sound in any way. Even the engineers equipped with an
oscilloscope or audio spectrograph equipment can't claim
to have much of an advantage. We are already accustomed to
the advise that ˇ§analyzing the molecular structure of candies
kills the satisfaction in eating itˇ¨. So there is a strong
inclination not to mix up our brain faculties.
In another area, namely entertainment, we try very hard to link
up the audio and visual forms. Musical performance is seldom
conducted without some sort of appeal to the eyes also. Lighting
systems, video walls, the crystal ball and swinging color
light bulbs in disco, or just the well dressed musicians in
concerts are now trying their best to immerse the audience
in a multimedia stimulation environment. And we all know too
well that musical fountains have the music and the fountain
composed separately. Here we recognize both the audio and
visual form as two related stimulation that can move us and
yet can only make them work loosely together.
When human language is used to teach musical skills, it is hopelessly
imprecise.
So there is a need to relate the audio and visual form in a more
coherent way.
The result is PHONATURE :
A means to show the NATURE of sound (PHON) in visual form.
Through this system, we can read the sigNATURE of sound.
PHONATURE
is a suite of rigorous mathematical transformations that reveal
the different aspects of sound in animated visual form in
response to audio input. Features of sound is naturally mapped
to features of shapes, color and movement of patterns. Different
aspects of sound are emphasized and hence more visible for
the several different forms of transformations which can mix-&-match
as needed.
In developing phonature products, Display Research Laboratory
has implemented this new theoretical framework in concrete
form with advanced signal processing technology and novel
computer and display architecture. For example, the dynamic
color transition effect makes the analog oscilloscope or similar
equipment pale by comparison. Using phonature products, people
can either immerse themselves in a holistic multimedia environment
or use it as an objective metric to evaluate and discuss a
musical performance. Or people can use them as tools for scientific
or learning purpose.
Getting to know
Basically, phonature is a set of mathematical transformations on sound
signal. As mathematical transformations, they are nonjudgmental
operations which only turn one kind of pattern into another
kind of pattern. As such, they introduce a minimum of artifacts.
On the other hand, care has been taken to use different means
to emphasize the features of musical sound or human voice
to make the features more conspicuous. It is like the high
heels that make ladies look more attractive.
All the interesting relationships between the audio and visual
patterns are derived from the properties of musical patterns,
characteristics of musical instruments, and interaction of
the sound sources.
In this position paper, we shall first describe the transformations.
Then follow with a number of observations about the possible
patterns that can arise and their interpretation. In order
to provide more information about the comprehensive potential
of this new field in limited space, we won't provide mathematical
proofs here. A proof or disproof can be left to the mathematicians
in conjunction with the musicians once a statement is made.
Interested readers can contact the inventor for draft steps
of the proofs. The less mathematically inclined can just try
out the equipment to satisfy themselves that it is indeed
having such properties as described.
|
Technology
