|
Phonature is a collection of technology related to biofeedback. It involves
processing of sound in real-time transformation into human recognizable patterns of
stimulation for other senses, in particular the visual sense. Applications include:
Music
/ Vocal
/ Speech training
/ Entertainment lighting
/ Research related to sound or music
What is Phonature ?
Phonature is a newly developed technology which reveals the nature of sound.
Based on the natural relationship between sound and mathematics, Phonature converts
sound into visual patterns according to the characteristics of the sound being
processed.
Any single sound or "phone" has its particular "signature".
The name Phonature combines these two elements together visually and lets people see
the signature of sound. While we may have different signatures for different
occasions, it is also possible for a sound to have different visual forms to suit
different occasions.
The core of Phonature technology is to tackle the following issues :
Biofeedback
The basic principle of biofeedback is to let us see the invisible states of our body,
particularly our brainwaves. Other bodily states such as skin temperature and heart
beat and about anything that is measurable can be used. Biofeedback is most often
used in relaxation training and attention deficiency disorders. So when you can see
from the equipment that your muscles are tense, you can be more consciously calm down
and improve your skill of mastery with the help of equipment.
Even without sophisticated equipment, you can conduct your first biofeedback
experiment with a small mirror. Go to the restroom with a mirror, bringing along a
small mirror of about 150mm diameter. Put the mirror behind your ear until you can
see your ear in the mirror in front of you. Then try to move your ear with its
surrounding muscles. You can see your achievement from the mirror. If you get it
right, you can move your ear without much effort. If you see your ear turning red, it
means you are using too much effort. Then you have to relax a bit before continuing.
If you still have difficulty, then try to move your jaws like chewing. You can see
your ear moving but it is not the right way. However, you can feel the tensions of
your face and your muscle movements as reference for your future practice. Many people
have already acquired this advanced body mastery in as short as one hour, broken down
into 10-15 minutes practice a day. You can do this too!! Then you know biofeedback is
not alchemy or pseudoscience.
Currently, biofeedback is almost exclusively referring to the measurement and display of
the nerve or brain signals as an aid for our improvement.
Sound
Sound is not visible, uncountable, and difficult to quantify by ourselves without
some kind of aids. Even if someone has good ears, vocabulary in describing what one
hears is very limited and not precise. Physically, sound is the rapid movement of
physical objects in the range of 20 to 20,000 times a second, bringing the
surrounding air into movement, which in turn transmits through air into our ears.
But what makes sense to us is the pattern such as its frequency component, their
periods of repetition and transition patterns over the longer time frame. We also
have different concerns for speech and music.
The learning of sound generating skills is difficult because the various qualities
of sound can only be controlled indirectly either by our vocal instrument or
manipulation of a musical instrument. It is further complicated because the
learning involves two steps, learning to produce sound and learning to hear the
difference between the intended sound and erroneous sound. Very often, the failure
to perform is due to the failure to hear correctly. As yet, we have no systematic
means to tell us how accurately we hear a certain sound. So it difficult for us to
improve our hearing.
Phonature uses various means to tell us what sound is being generated. It provides
a chance for us to check this against the sound intend to generate as well as
verifying it with what we hear. This is the badly needed aid for effective self
improvement or the common reference between a coach and a student.
Real-time
According to psychology experiments, if two sounds begin with a time difference of
less than about 0.05 second, we cannot tell which one begins first, and will
consider them simultaneous. Simultaneous feeling is a bit more relaxed for
excitation of different senses. The figures cannot be exact because the beginning
of natural sound is often soft or fuzzy without a clear starting point. Other forms
of senses also have uncertainties in determining its starting point. With caution in
our conclusions, this should not stop us in our scientific pursuit. What's more,
perfectionist approach is not necessary when we know that the speed of sound is just
about 300 meters per second. The time difference will vary by 0.05 second within a
small distance of about 15 meters.
When we can create a visual stimulation that changes in step with the incoming sound
within a time frame of 0.05 second, a coherent stimulation is created. For slow
music such that the duration of each note is much longer than 0.05 second, a longer
delay is acceptable. Though we can then see a delay, harmony between the audio and
visual stimulation is still maintained.
The sense of smell has even fuzzier boundary, and transmits in space more slowly.
Say for example, we want to generate smells corresponding to the mood of music, then
a longer delay is acceptable. It is also no point in changing the smell too rapidly.
Transformations
When we talk about sound or signal processing, the first thing that comes into mind
is frequency analysis. This is also the prevailing means for our auditory
perception system to make sense of sounds around us. To turn the time varying
physical phenomenon of air compression and rare fraction into its frequency parts is
a kind of transformation. About every engineers should have learned what we call
the Fourier Transform to turn a time varying signal into its frequency components.
The graphic equalizer of Hi-Fi equipment is a visual formal of the frequency
components based on the same principle.
Another main branch of signal processing is recognition, which can be considered a
special form of transform with sharp transition points. That means at certain
points, small change of input will result in sudden drastic change of output.
We set a boundary between one sound and another similar one to make classification
possible. So after the recognition step, subtle details of the sound are lost,
leaving behind a symbolic representation.
In as much as possible, phonature technology performs what we call continuous
transformations such that small changes of input will result in small change of
output. In the design of transformation, cares are also taken to preserve the
psychological metric of difference between signals. For examples, if two sounds
are similar in a certain aspect of interest, then after transformation, a
corresponding visual quality will also appear similar. The strict adherence to
this class of transformations creates a concerted harmony between the auditory and
visual forms without human effort. It enables spontaneity in performance while
giving the same result as those carefully programmed lighting effects of concerts
and shows. Phonature based lighting products becomes splendid amplifier of human
performance. During practice, the lighted echo of our sound generating performance
becomes a mirror and faithful critic for us to move towards better mastery and
excellence of our skills.
Human recognizable patterns
Human beings are very good in detecting patterns. For things to be human
recognizable, they have to match our pattern detecting mechanisms. For example,
we can read a graph or a chart by following the grid lines to get the readings from
the boundary coordinate scale. Fluctuation is less easy to interpret in this way.
On the other hand, a moving pointer that rotates or vibrates reveals much better its
quantitative variation intuitively. Likewise, we perceive color contrast and
relative position much better than absolute quantities. And there are texture
patterns and imagery that we can use to enrich the multimedia experience.
Phonature technology involves choice of easily recognizable patterns relevant to the
target aspects of sound to enable engaging interactive experience. Consistency is
stressed so that we can make sense of the patterns, whatever they are.
Visual
There is a saying that a picture is worth a thousand words. It reminds us of the
rich message that visual forms can create. In addition, engineers have over the
centuries developed elaborate forms of conveying quantifiable information with
visuals such as meter, thermometer, various scopes and recently direct digital
readouts. The important thing about visuals is that it provides a precise common
medium of discussion about abstract or intangible, invisible, uncountable things.
Phonature also depends heavily on visuals. But the choices of visual forms are
such as to allow fast grasping of the multiple qualities of sound, or the
interaction of multiple sounds. In addition to computer graphics, technologies for
fine control of various light sources and their special arrangements to create
aesthetic dynamic effects have been developed.
Application
Technology development and its application are very different things.
A technology can be very advanced and yet have no application, at least not at the
current moment. It all depends on how many people are being limited by the same
problem that it has solved. So each technology is posting a number of "what if"
questions to the world. What if this can be done ? Naturally, phonature has its own
"what if" questions.
What if you can see immediately the different components of sound that you hear or make ?
What if you can see clearly the development of a sound, such as a foreign word, from its
appearance to extinction with details of its transitional profiles ?What if you can see
the effect of mixing sounds ?
In all the above "what if's", the word "see" means "understand"
or "get it!"
As developer of the technology, we believe we have solved problems with profound implications
and wide applications such as in accelerated learning and entertainment. We can see the value
of the technology by observing how people tackle their current issue without the benefit of
this advancement in technology. Today, lighting engineers and stage designers of large
performances work long hours to program computerized lighting systems in order to give the
effect of coherent light and sound multimedia experience. Small scale lighting systems and
small shows without the big budget have to accept compromises. Spontaneous performance also
cannot enjoy the benefit of coherent simultaneous lighting accompaniment. And in vocal
training or speech learning, the lack of objective metric puts the teacher in the poor
positions of the judge, creating antagonism and resentment in students that affects progresses.
Students also cannot get relevant and immediate feedback to make informed adaptations and give
up in the early stage of learning. The arrival of Phonature will change these for the better.
Finally, the best application of technology is possible when developer works with knowledgeable
and enthusiastic users to apply the technology. We are fully aware of our situation and
is eager to work with experts who find new possibilities for our technology.
|
