Multimodal integration in the chicken

At any time we perceive information about the environment through our sensory organs. The majority of these signals are multimodal, meaning that they consist of more than one sensory modality. Thus, at the same time, they can be seen, heard or smelled and sometimes even touched. The multimodal nature of these signals greatly enhances the chance of detecting them (Stein and Meredith, 1993). Although such stimuli could be redundant because information from a single modality suffices, the information that is held by the two modalities can be additive. For example, humans are more accurate and rapid at identifying crossmodal compared with unimodal objects (Giard and Peronnet, 1999; Forster et al., 2002). Likewise, visual stimuli are perceived as brighter when accompanied by an acoustic cue (Stein et al., 1996) and visual displays help to detect laughter that is difficult to hear (Jordan and Abedipour, 2010).

Crossmodal information seems to be beneficial not only for humans, but also for animals. In monkeys, the simultaneous presentation of information from two modalities results in faster reactions and more correct responses (Lanz et al., 2013). Such benefits have also been reported in vertebrates beyond mammals. In nightingales, for example, the presence of a visual display helps young birds not only to learn a higher number of songs, but also to copy the songs from their tutors in a more accurate way (Hultsch et al., 1999). In another bird species, pied currawongs, the animals were found to come closer to a model that produced both acoustic and visual cues (Lombardo et al., 2008). And even fishes (Moller, 2002) and insects (Duistermars and Frye, 2010) were found to benefit from combining information from more than one modality.

The majority of the experiments on multimodal information processing have been performed in cats and owls, which use both the visual and auditory modalities to capture prey (Meredith and Stein, 1983; Luksch et al., 2000; Stanford and Stein, 2007; Winkowski and Knudsen, 2007; Zahar et al., 2009). Cats are able to detect the source of a crossmodal signal with a higher accuracy compared with unimodal signals (Stein et al., 1988, 1989; Gingras et al., 2009). In the well-studied barn owl, head movements toward the location of the stimuli are both faster and more accurate for audiovisual stimuli (Whitchurch and Takahashi, 2006). However, multimodal integration is not restricted to animals that are specialized to use auditory and visual components during hunting. Chickens forage on the ground and rely on their distance senses for escape responses. It has been shown that the presence of an auditory signal improves aversion learning of a colored pattern (Rowe, 2002). Likewise, imprinting is stronger when the stimuli consist of more than one modality (Van Kampen and Bolhuis, 1991, 1993). However, most of the demonstrated benefits of crossmodal stimuli can be attributed to learning and do not necessarily show multimodal integration. In addition, chickens use crossmodal signals during their courtship behaviour. During tidbitting, where the rooster shows the female where food can be found, the animals use visual and acoustic displays. Despite the multimodal nature of this behaviour, the animals spend the same amount of time searching for food, independent of whether the display was visual, acoustic or audiovisual (Smith and Evans, 2008). However, in this case, the unimodal stimuli were easily detectable and therefore the crossmodal stimuli might not have provided an additional benefit to the animals.

Taken together, it is still not fully understood how multimodal integration benefits generalistic birds such as chickens in a non-learning context. In order to answer this question, we performed behavioural experiments. We measured the minimal stimulus strength that the chickens were able to detect and to discriminate using a staircase paradigm. We hypothesize that if animals combine the information present in visual and acoustic stimuli they should be able to detect weaker stimuli when these are crossmodal. However, when they have to discriminate between two well-detectable stimuli, the additional modality is expected to have a smaller effect on the thresholds.

A total of 12 chickens (Gallus gallus domesticus, Linnaeus 1758) of both sexes were used. The animals were up to 8 months old. The birds were housed in groups of 6 animals and were placed on a 12 h light:12 h dark cycle. The housing measured 2×1 m and contained perches and sand baths. A corn mixture was provided ad libitum but was removed shortly before the experiments started. As the animals received a reward consisting of pellets or wheat within the experimental setup, they were highly motivated to participate despite the absence of long deprivation periods. Water was always available ad libitum.

All experiments were performed according to the principles regarding the care and use of animals adopted by the German Animal Welfare Law for the prevention of cruelty to animals and were approved by the Government of Upper Bavaria, Germany (55.2-1-54-2532-119-12).

Experiments were performed within a rectangular arena measuring 150×120×90 cm (L×W×H) illuminated by a light bulb (145 lx) (see Fig. 1). A TFT monitor measuring 33.8×27.0 cm (Acer V173, Taipeh, Republic of China) was placed in each corner. The speakers (NSW1-205-8A, AuraSound, Santa Ana, USA) were located under the monitors. The sound was amplified by two Yamaha amplifiers (AX396 Yamaha, Rellingen, Germany). The position of the animal was monitored using cameras above each monitor and one in the middle (Hama CM-330 MF, Monheim, Germany). Food was provided by custom-built feeders behind each monitor, which were controlled through a USB interface (VPCD4, Velleman, Gavere, Belgium). All programs were written in Python 2.7. Visual stimuli were programmed using the visionegg toolbox (Straw, 2008).

The visual stimulus consisted of a dark circle (diameter, 8.4 cm) on a grey background (luminance 93.5 cd m⁻²). We selected a dark stimulus because our pilot data showed that the detection thresholds, as well as the variance, were lower using black stimuli (threshold white, 24.12±3.20 cd m⁻²; threshold black, 15.12±2.61 cd m⁻²; pilot data based on 6 animals). In the detection staircase paradigm (see below), the initial luminance was 56.10 cd m⁻². The luminance increased in steps of 1.87 cd m⁻², becoming more similar to the background. In the discrimination paradigm (see below), a strong (dark grey) stimulus was shown on one monitor and a grey (slightly darker as background) stimulus was shown on the other. The stronger (S+ or darker) stimulus started at 28.05 cd m⁻² and faded to 46.75 cd m⁻² and the weaker (S− or lighter) stimulus at 64.45 cd m⁻² and raised to 46.75 cd m⁻² in steps of 0.935 cd m⁻².

The acoustic stimulus was a pure tone lasting 100 ms, which was turned on 5 times per second. The frequency was 4.5 kHz, which is one of the higher frequencies at which chickens still show good detection thresholds (Saunders and Salvi, 1993; Hill et al., 2014). In the detection staircase paradigm, the initial loudness was 68 dB. The loudness was decreased in steps of 3 dB after correct trials. In the discrimination paradigm, the louder stimulus was presented at 68 dB and decreased to 47 dB in steps of 1.5 dB. The weaker stimulus started at 26 dB and was increased to 47 dB.

The monitors were calibrated so that each step represented a difference of 0.935 cd m⁻² using a Spyder 3 elite colorimeter located directly in front of the monitor (Datacolor, Zürich, Switzerland). The loudspeakers were calibrated to 68 dB each day using a measuring amplifier (Type 2609, Brüel&Kjaer, Copenhagen, Denmark) and a microphone (Brüel&Kjaer) located 30 cm away from the loudspeaker.

Trials always started with the animal at the start position in the middle of the arena. There, the chicken faced one pair of monitors/speakers in two corners, which were positioned at 30 deg (see Fig. 1). The pairs were alternated so that the animal had to walk back and forth. Stimulus positions within each pair were chosen at random. The stimuli were presented on the monitors or by the speakers in front of the animal. The animal either had to detect a single stimulus (detection staircase paradigm: S+ is the stimulus, S− is no stimulus) or locate the strongest stimulus between a stimulus pair (discrimination staircase paradigm: S+ is the stronger stimulus, S− is the weaker stimulus).

The animal had to choose one of the stimuli by walking towards it. The trial ended when the chicken was detected in a small area of ∼15×20 cm in front of one of the monitors (see also Fig. 1). Stimuli were turned on for 10 s or until the trial was ended. As soon as the animal was detected in front of one of the monitors, the trial ended and the animal was rewarded or punished. The animal had to return to the start position in order to start the next trial. If the animal was not detected in front of one of the monitors before the stimuli turned off (after 10 s), no reward or punishment occurred and the trial was excluded. When no animal was detected in front of any of the monitors within 50 s after the stimulus turned off, the next trial started.

We used a method based on the staircase paradigm (Levitt, 1971). We modified the signal strength of the visual and acoustic stimuli to investigate which signal strength the animal could just detect or what difference the animal could discriminate. In order to reach this signal strength, the stimulus (S+) became weaker or the stimuli became more similar after the animals chose correctly. After the animals chose incorrectly the stimulus was made stronger or the two stimuli were made less similar. When the animal chose the correct stimulus, a few pellets of food were delivered. When the animal chose the incorrect stimulus the lights went off for 2 s. As a result, the animal reached a plateau around the stimulus strength it could just localize (the detection threshold) or around the difference in stimulus strength that the chicken could just discriminate (the discrimination threshold).

After the animals were acquainted with the setup, they learned to walk towards the stimulus. After the animals walked towards the presented stimulus, they were trained to return to the starting position after each trial.

Half of the chickens (N=6) started with visual stimuli and the other half (N=6) started with audiovisual stimuli. The first group (N=6) was trained initially with visual stimuli and then with the acoustic stimuli. As this proved to be difficult for some animals, the second group (N=6) was first trained with audiovisual stimuli and then with visual and acoustic stimuli alone.

The animals were trained using strong visual and acoustic signals. Training continued until they chose the correct stimulus (S+) in at least 80% of the trials in two consecutive sessions, for both visual and acoustic stimuli.

All animals reached criteria for the detection staircase paradigm. However, for the discrimination paradigm only 9 animals were tested because the other 3 animals failed to reach 80% correct answers.

After training was completed, the animals were tested with stimuli whose signal strength was modified based on the performance (see staircase paradigm above). The animals were first tested on the detection paradigm and subsequently on the discrimination paradigm. The birds had to overtly respond by walking towards one of the two monitors. When no choice was made, the trial was excluded and the programme continued after an intertrial period of 50 s. Sessions in which the animals made a choice in less than 80% of the trials were excluded from the analysis and repeated. The threshold was calculated as the average signal strength during the last 15 trials.

Every animal was tested three times with visual, acoustic and audiovisual stimuli on different days and the thresholds were averaged (therefore every animal was tested a minimum of 9 times for each paradigm; when criteria were not reached, tests were repeated).

We first measured the unimodal thresholds for the visual (in cd m⁻²) and acoustic stimuli (in dB). Next, the stimuli at the visual and the acoustic thresholds were combined to create the audiovisual stimulus. Therefore, the audiovisual stimulus was set at the threshold for both the visual and the acoustic stimuli. Visual and acoustic components were then modulated together, in similar steps as the visual and the acoustic stimuli.

The visual and acoustic thresholds were calculated. The audiovisual thresholds were then compared with the unimodal thresholds using a paired t-test. Data are presented as the mean, the s.e.m. and the 95% confidence interval (CI). In addition, Cohen's d is given as a measure of the effect size (reviewed in Nakagawa and Cuthill, 2007). An effect size of 0.2 to 0.3 is regarded as ‘small’, an effect around 0.5 as ‘medium’ and an effect size larger than 0.8 as ‘large’ (Cohen, 1988).

To investigate whether chickens combine the information present in multimodal signals, we tested their ability to detect and discriminate between unimodal (acoustic and visual) and crossmodal (audiovisual) stimuli using a staircase paradigm (see Fig. 1 for the experimental setup).

The animals were initially required to detect the location of a visual or an acoustic stimulus using a staircase paradigm (see Materials and methods for details). We found that chickens can detect weaker stimuli when the stimuli are crossmodal compared with unimodal stimuli (see Fig. 2). The signal strength of the audiovisual stimulus that the animals could detect was significantly lower than either the visual (t₁₁=5.19, P<0.001, 95% CI 3.57 to 8.81 cd m⁻² difference, Cohen's d=1.493, see Fig. 2A) or the acoustic stimulus (t₁₁=3.58, P=0.004, 95% CI 2.32 to 9.70 dB difference, Cohen's d=0.537, see Fig. 2C). Using a visual stimulus, the animals could detect a stimulus that showed a luminance difference of 14.29±1.45 cd m⁻² relative to the background (a difference of 14.3 cd m⁻², comparable to a 15.3% contrast). Using an acoustic stimulus, the animals could detect a stimulus of 27.95±3.12 dB. Using an audiovisual stimulus, this difference became 8.10±0.87 cd m⁻² (a difference of 8.1 cd m⁻² comparable to 8.7% contrast) for the visual part of the audiovisual stimulus and 21.94±3.33 dB for the acoustic part of the audiovisual stimulus.

Next, we tested whether crossmodal information also helped the animals to discriminate between two stimuli using a staircase paradigm (see Materials and methods for details). We found that the differences between two stimuli that the animals could discriminate was smaller for audiovisual stimuli (see Fig. 2). When tested with audiovisual stimuli, the discrimination threshold was significantly lower than when tested with visual stimuli (t₈=2.81, P=0.023, 95% CI 0.58 to 5.87 cd m⁻² difference, Cohen's d=0.908, see Fig. 2B). With unimodal visual stimuli, the animals could detect luminance differences of minimally 10.58±1.10 cd m⁻² (absolute luminance of 41.46 to 52.04 cd m⁻²). Using audiovisual stimuli, this difference was only 7.35±1.26 cd m⁻² (absolute luminance of 43.07 to 50.43 cd m⁻², comparable to contrasts of 46.1% and 53.9%, respectively). Please note that at the threshold, the absolute luminance of the two visual stimuli were well above the unimodal detection threshold of 14.29 cd m⁻² (see detection staircase paradigm).

In contrast to the comparison to visual thresholds, the crossmodal discrimination threshold was not significantly lower than the threshold measured with acoustic stimuli (t₈=1.13, P=0.291, 95% CI −2.31 to 6.22 dB difference, Cohen's d=0.305, see Fig. 2D). Under unimodal conditions, the minimum difference between the two acoustic stimuli that the animals could discriminate was found to be 10.76±1.55 dB (absolute loudness 41.62 and 52.38 dB) under unimodal conditions and 8.72±2.76 dB (absolute loudness 42.64 and 51.36 dB) under crossmodal conditions. Please note that at the threshold, the loudness of the two acoustic stimuli was well above the detection threshold of 27.95 dB (see detection staircase paradigm).

Chickens can combine the visual and the acoustic information present in the audiovisual stimuli to better detect weak near-threshold stimuli. In addition, audiovisual stimuli allowed the animal to differentiate between stimuli that were more similar compared with purely visual stimuli, but not compared with acoustic stimuli.

Both the visual and the acoustic detection thresholds were found to be comparable to previously published data. In our paradigm, the animals were able to detect an acoustic stimulus (4.5 kHz pure tone) if it was louder than 28.0 dB (range 6–42 dB). Our findings are comparable to other behavioural studies where the threshold was found to be around 20 dB (Saunders and Salvi, 1993; Hill et al., 2014). The visual stimulus that our animals could still detect had a difference to the background of 14.3 cd m⁻² (or 15.3% contrast), ranging from 8.4 to 24.1 cd m⁻². To the best of our knowledge, the detection threshold has not been measured with similar methods. However, contrast sensitivity has been measured using sinusoidal gratings (da Silva Souza et al., 2011). In these experiments, contrast sensitivity is often measured as a function of the number of cycles deg⁻¹ of a grating stimulus (Pelli and Bex, 2013). Our visual thresholds are in the same range as the contrast thresholds necessary to see a sinusoidal grating, which were found to be around 10–14 cd m⁻² (Schmid and Wildsoet, 1998; Gover et al., 2009; Jarvis et al., 2009).

It has been reported that the enhancement of responses by multimodal integration is strongest when the stimuli are weak – a phenomenon called inverse effectiveness (Meredith and Stein, 1986). We hypothesized that in our behavioural experiments the largest effects would also be found using weak near-threshold stimuli. When the task was to detect a single stimulus, thresholds were indeed significantly lower using audiovisual stimuli. In this case, the effect sizes were intermediate to high, indicating a large effect of the additional modality. However, when the task was to discriminate between two stimuli, the difference to both unimodal conditions did not become significant. The additional acoustic modality helped to discriminate between two visual stimuli, but not the other way around. The effect size was also smaller, which indicates that the additional modality had a weaker effect on the thresholds.

This discrepancy between the two different paradigms might be explained by the strength of the stimuli. In the detection staircase paradigm, the animals had to detect a single stimulus that progressively became weaker. Therefore, these thresholds represent the weakest stimuli that the animals could detect. However, in the discrimination staircase paradigm, the animals had to differentiate between two stimuli that progressively became more similar. Although the difference between the two stimuli was large at the start of a behavioural session, they became more and more similar when the animals correctly chose the stronger stimulus. Around the discrimination threshold, the two stimuli were both well above the detection threshold for visual and acoustic stimuli, and therefore, could be easily detected by the animal. Thus, the stimuli were not ‘weak’ anymore. As multimodal integration is strongest when stimuli of both modalities are weak, it might be that the effect of multimodal integration was less pronounced when the animal had to discriminate between two stimuli than when it had to detect a single stimulus.

It is sometimes argued that crossmodal stimuli hold redundant information. Even when the two modalities hold redundant information, it is possible that the crossmodal thresholds are lower. In our (detection) paradigm, the stimuli were around the threshold and therefore comparable to the background noise. Under such conditions, the additional modality in the crossmodal signal can lift the signal above the noise and therefore help to detect the signal. However, this can only be the case if the background for the two information channels is unrelated.

Indeed, we found that the detection thresholds were lower for crossmodal stimuli, indicating that the animals use the additional information present in such stimuli. It has been shown that crossmodal stimuli hold more information than double unimodal stimuli and therefore the effect is not simply due to the fact that an additional stimulus is presented (Alais and Burr, 2004; Gingras et al., 2009). Alternatively, the animals might use the visual and acoustic information differently, as was shown previously for owls. Owls do make more accurate and faster head movements towards crossmodal stimuli (Whitchurch and Takahashi, 2006). They seem to use the visual cues to localize the stimulus accurately, whereas the acoustic cue appears to be important for the response speed.

Our results show that the ability to discriminate between two signals was significantly better for audiovisual signals compared with purely visual signals. Although the threshold was also lower with audiovisual stimuli compared with acoustic stimuli, this effect was not significant. Therefore, the effect of an additional modality appears to be stronger for the discrimination of two visual stimuli than for the discrimination of two acoustic stimuli. However, as only nine of our animals participated sufficiently to be tested in the discrimination paradigm, it is also possible that the failure to reach significance is due to the small sample size. Nevertheless, our results showed that the effect size (a measurement independent of sample size) of the multimodal threshold was large compared with the visual thresholds, but small compared with the acoustic stimuli. Thus, the additional modality helped the animals to discriminate between two visual stimuli to a larger extent than it helped to discriminate between two acoustic stimuli.

It is possible that the animal focuses on the modality that holds more information; this would indicate that the visual and acoustic component of the stimuli ‘below threshold’ were not equally strong. Although we jointly made the visual and the acoustic stimuli weaker and stronger, we cannot determine whether the visual or acoustic stimulus one step below threshold was easier to discriminate. Thus, the animal could simply focus on the modality that it finds easier to discriminate. With crossmodal signals, behaviour can be guided by the perceived stronger modality (Ernst and Banks, 2002; Shams et al., 2005). When the visual stimulus was strong enough, the location of a sound was based on the position of the visual stimulus. However, when the visual stimulus was blurred, the source of the visual stimulus was determined by the sound (Alais and Burr, 2004).

Our results show that chickens can combine the visual and the acoustic information present in audiovisual stimuli to better detect weak stimuli. The signal strength of the stimulus that the animals were still able to detect was significantly lower for audiovisual stimuli than for visual or acoustic stimuli alone. In addition, audiovisual stimuli allowed the animal to differentiate between stimuli that were more similar compared with purely visual stimuli, but not compared with acoustic stimuli. Therefore, the benefits of multimodal integration are especially pronounced in the detection staircase paradigm when the task is to detect a weak, near-threshold stimulus. When the task is to discriminate between two stimuli, the magnitude of the effect is much lower.

We would like to thank Silke Kipper for valuable discussions and critical reading of the manuscript and Diego Sanjulián Alonso, Astrid Pledermann, Monika Gruber and Julia Claussen for their assistance during the data collection. We would like to thank Sjoerd J. de Vries for his help with the acquisition software and the two anonymous reviewers for their extensive and helpful feedback.