Paper presented at the Third Annual National Meeting of the American Synesthesia Association in New York, Rockefeller University, 2-4 May 2003.

 This page contains the text, to see the slides of the presentation click here.

Combining artistic and scientific perspectives in the study of synesthesia

 Crétien van Campen & Clara Froger

Yesterday and today, I have heard fine talks by artists and researchers. I appreciate the way the organization has succeeded in organizing a meeting of artists and scientists from various disciplines. In this presentation I want to tell you about a Dutch collaboration of an artist and a researcher in the study of synesthesia.

The intention was to do a duo presentation by Clara Froger and me. Unfortunately Clara is hindered to come to NY. She apologizes and sends her best wishes to you all. At the exhibit of synesthetic art you can examine some of her works.

I am working as a health care researcher in the Netherlands. Besides my work I have studied synesthesia in science and art. Clara is an artist and color consultant. We developed the Netherlands Color Synesthesia Method that was presented by Carol Steen at the ASA conference in San Diego last year. An article will appear in the coming August issue of the American journal “Leonardo”.

Interest in synesthesia since 1800

Our backgrounds stretch beyond our personal lives and go back into European artistic and scientific traditions. As you all known the study of synesthesia has a long history in Europe. This figure gives an impression of the interest in synesthesia by artists and scientists in the last centuries. 

Parallel study of perceptual phenomena in art & science

Synesthesia is not the only perceptual phenomenon that has been studied by artists and scientists. It looks that artists before have explored almost every scientific subject of perception. Just to give you an impression of some phenomena that were studied by artists and scientists: the perception of linear perspective (Dürer), the perception of simultaneous contrast in color (Seurat), the perception of reversible patterns (Rubin), and the perception of symmetry and mathematical puzzles (Escher).

Differences in approach: art and science

In most cases artists discovered a phenomenon in their practice of sketching and painting. They explored the phenomenon and applied it in their art as they saw it. They used a first person perspective. Often, scientists rediscovered the phenomenon later, tried to isolate it under laboratory conditions and explained its underlying physiological and mental mechanisms. They used a third person perspective to establish objective mechanisms.

The elucidating article on first and third person perspectives in synesthesia research by Daniel Smilek and Mike Dixon provides a fruitful conceptual framework for our study. We consider our experiment as an illustration and elaboration of this distinction.

Combining perspectives +  picture slides

[Nick double bass]

We combined first and third person perspectives in a special way. One month ago we invited nine synesthetes to the BMBR gallery (Beauty Must Be Realized). All nine have colored-word synesthesia (most of them of the colored-grapheme type) and four persons hear music in color too.

We asked them to do two things:

[color chips]

1. They chose color chips that represented best their synesthetic color responses to specified stimuli.
[participants paint]
2. They painted their synesthetic responses to a selected number of the same stimuli.
[Cretien reads words]
The selected number of stimuli that were painted consisted of five spoken sounds: O E OE KOEL STROEP.
[Nick and Clara]
And six musical sounds: the D-chord played in major and minor key successively on double bass, guitar and trumpet.  

The selected color chips were used to analyze the consistency of the persons color-word and color-music relationships. Consistency tests are now widely used in scientific study of synesthesia. In addition to the traditional consistency test, the Netherlands Color Synesthesia method computes consistency scores for each of three dimensions of color synesthesia: i.e. hue, chroma and lightness.

So, we gathered two types of information for each participant: a first and a third person perspective of his or her colored-word and colored-music synesthesia. At the moment we are analyzing the results that we gathered only one month ago. We would like to present some of the first results and share our first interpretations with you. We like to hear your suggestions.

[Colors of the Dutch word KOEL (= cool)]
This slide shows two responses to the same stimulus: the spoken word KOEL, meaning cool in English (as you see some Dutch are bilingual; this person painted two interpretations of the sound)
On the left side you see the color chip that was selected for the word KOEL by this person in the test. An on the right side, you see a visual representation of the same word by the same person.

Comparison:
1. the painting shows more nuance (see e.g. the change of blackness and chroma in the O of the Dutch word KOEL)
2. the painting shows not only the color but also the form and space of the color (see the clouds around COOL)
On the other hand the color chip has direct computable information on the dimensions of the color. The code for this color chip [6030 R70B] indicates that it contains: hue = 70% blue 30% red; chroma= 30% of the pure color; lightness= 60% black 40% white.
Later, the person commented that the real color is more transparent than the selected color chip. The paintings capture the transparency and space of the color better than the color chip.

[Influence of context on colored letters I]
This slide shows that the method of painting is better suited to show the effects of surrounding letters on the color of a letter. This subject made five paintings of the spoken sounds. In Dutch the combination of O and E produces a different sound: OE. This person has apparently colored-grapheme synesthesia.

In the test she chose a light ochre yellow and a muddy yellow for the word KOEL. The painting shows that the O is black, her E is bright ochre yellow. When she hears the different sound of OE she sees the colors of O and E. But mentally, they mingle into a muddy green-yellow. The words KOEL and STROEP below show that more letters influence the colors of the OE. 

[Less influence of context on colors]
The next slide shows how individual synesthetes can differ. For this person the color of OE is green (on a yellow background and a shade of white on top). And it stays green irrespective of the surrounding letters, as you can see in the words KOEL and STROEP.

The green OE is made of the white O and a green-yellow E. Remarkable is that the colors mingle in the words KOEL and STROEP but not in the isolated sound OE.

Note: Most participants mixed the colors of O en E to represent the color of OE.

[Sounds of musical instruments]
In the musical session, a D major chord and a D minor chord were played on double bass, guitar and trumpet. This person has selected in this test the same color chips for major and minor chords; but for each instrument another color. The paintings show a nuance that the test did not show; namely the difference in sound between major and minor chord. As the sounds are higher they become lighter and more transparent, according to this synesthete. Note 1: Most participants used “earthly colors” for the double bass.
Note 2: Minor chords are painted redder/warmer than major chords by most participants.

[Comparing two types of information on colored-music synesthesia]
We can compare the painting of this person with his color synesthesia profile that is produced by the Netherlands Color Synesthesia method. The profile (at the right side) reveals a remarkable aspect of his color-music synesthesia. If one considers his synesthesia in three dimensions, it appears that it is built on two strong pillars and has one weak pillar. He has a strong sense for the lightness and the chroma of music, but a weaker sense for the hue of the music.
Looking back at the painting on the left, it is visible that he varies in lightness and chroma but not so much in hues. His hues seem to stem from the colors of the instruments: e.g. a brown double bass and a yellow trumpet.

[Improvisation]
An visual musical improvisation as an intermezzo . Halfway a day of hard and precise work we paused with a free improvisation on double bass by our musician Nick McGuire. All participants were allowed to paint whatever they wanted. Here is the painting I liked most. Strikingly, she is not a color-music synesthete.

[Discussion of results]
In this early stage of analyzing the results, we already see two differences between scientific testing and painting. The use of color chips restricts the study of synesthesia to color, while the use of painting shows the aspects of form, space of color too. (In fact one should be able to represent rhythm and movement too.) This implies, that using color chips means loss of information. Painting is a better approach to represent the real synesthetic experience. But paintings are more difficult to compare. Color chips have the advantage to be computable and give the opportunity to statistical analysis of the dimensions of color.
We will continue our analyses the coming months and hope to profit from your suggestions.

[Conclusion]
In conclusion, we want to make some statements on the collaboration of artists and scientific researchers.
1. Art and science seem to contribute in different ways to our knowledge of synesthesia. But we have seen in our historical review of synesthesia research and in a broader historical review of visual research that science and art have always been interacting.
2. Artists and researchers can both profit from critical interaction. The collaboration of Clara and me is an example. After many years of collaboration, I understand better now what a color experience in fact is. And Clara has seen the advantages of measurement and comparing individual synesthetes. We have profited  both ways from our expertise.
3. To return to the conceptual framework of first and third person perspectives. We recommend artists and researchers to change perspectives: between first and third person perspective. Not once but regular. Change from assessment and objective knowledge to personal records and back and so forth. Finally, we think that progress in synesthesia research is probably not a linear, but a cyclic process. Thank you for your attention.