The roles of visual parallax and edge attraction in the foraging behaviour of the butterfly Papilio xuthus

Virtually all animals possessing eyes are faced with the issue of how to reconstruct the three-dimensional layout of their surroundings from the two-dimensional images formed on their retinas. Vertebrates like ourselves solve this problem using, amongst other things, stereopsis (i.e. comparing the images created in the two eyes) and accommodation (i.e. proprioception of the adjustment of the lens required to bring objects into focus). These approaches are generally not suitable for insects because their compound eyes typically exhibit little binocular overlap, poor spatial resolution and fixed-focus optics. Given these constraints, a more viable alternative is to use motion cues (van Breugel et al., 2014): when the animal is performing a translational movement, nearby objects will move more quickly on the retina than distant ones and hence depth can be estimated.

Many insect species measure translational optic flow to regulate their flight speed and/or distance to obstacles (e.g. Fry et al., 2009; Srinivasan et al., 1991). Indeed, the relationship between these variables means that smooth landings can be achieved using a very simple control scheme (Baird et al., 2013; Chahl et al., 2004; Franceschini et al., 2007). By assuming a roughly constant distance between their eyes and the ground, central-place foragers such as bees and ants can perform odometry by integrating motion parallax over time (Esch et al., 2001; Ronacher and Wehner, 1995). However, in this study we are concerned with the converse process to visual odometry: using visual motion to gauge the distance of objects, rather than to estimate one's own movement.

Honeybees can be trained to select artificial flowers of variable size at particular depths below a Perspex screen, but they fail if the objects offer no green contrast with the background, implying that their colour-blind motion vision channel is essential for this behaviour (Srinivasan et al., 1989). They can also recognise the shape of camouflaged objects (i.e. textured shapes against a similarly textured background) by detecting their edges using discontinuities in optic flow (Zhang et al., 1995) and will approach a camouflaged raised platform differently to a camouflaged hole, attempting to land on the outer edge of the former and pass through the latter (Lehrer and Srinivasan, 1993). There is evidence that bees engage in ‘active vision’, structuring their flight to optimally extract motion parallax information. Flying honeybees perform lateral ‘peering’ manoeuvres whilst stabilising their gaze when approaching a feeder (Boeddeker and Hemmi, 2010) and, similarly, bumblebees move in such a way as to maximise ‘pivoting parallax’ when performing learning flights around the nest (Riabinina et al., 2014).

Although these examples amply illustrate that motion parallax is a valuable cue for motor control and navigation, it is difficult to unambiguously demonstrate that insects use it to measure target distance without specifically manipulating parallax cues. This requires that the target is moved with respect to the changing viewpoint of the animal in a ‘closed-loop’ manner. Perhaps the earliest insect experiment of this kind was a pioneering study of locusts (Wallace, 1959). These animals perform a stereotyped peering behaviour (whilst standing otherwise still) before pouncing on prey. By manually moving the target with or against these peering movements, the locust could be induced to over- or under-shoot with its subsequent jump, respectively. This finding was later confirmed using more modern techniques (Sobel, 1990; reviewed in Kral and Poteser, 1997). Another closed-loop experiment demonstrated that fruit flies freely walking on a platform are more attracted to targets that give the impression via parallax cues of being closer (Schuster et al., 2002).

Obviously, performing experiments of this kind on flying insects is rather more challenging. In recent years, ‘virtual reality’ techniques for presenting closed-loop stimuli to 3D-tracked animals have emerged (Fry et al., 2008; Straw et al., 2010). To our knowledge, this method has not yet been used to investigate the role of visual motion in target approach and landing, so, in this study, we adopt this approach to ask whether butterflies use motion cues in addition to colour and brightness information when foraging. Specifically, we consider the Japanese yellow swallowtail Papilio xuthus, an animal noted for its elaborate retinal organisation (Arikawa, 2003). We train the butterflies to associate food reward with coloured disks; this paradigm was used a century ago to demonstrate colour vision in bees (von Frisch, 1914), but has been employed more recently to investigate phenomena such as colour constancy, colour contrast and polarisation discrimination in Papilio (reviewed in Kinoshita and Arikawa, 2014). Of particular relevance to the current study is the finding that intensity contrast is required for landing. Papilio can detect and approach, but not land on, coloured targets that offer only chromatic contrast with their equiluminant grey backgrounds (Koshitaka et al., 2011). This suggests that foraging behaviour is composed of several somewhat distinct processes that utilise different types of visual information, with visual edges playing a crucial role in the final stages of target approach.

We adapt the classical paradigm by using targets displayed on a monitor rather than paper ones. By tracking the animals' flight in real time, we manipulate the stimulus to generate motion cues corresponding to a virtual target at some position other than the monitor surface. A benefit of this approach is that it allows us to independently control different aspects of the motion stimulus, namely expansion and parallax. Three particular questions we address are whether the apparent distance of targets affects (1) their attractiveness, (2) the spatio-temporal structure of the descents the butterfly makes towards them, and (3) their likelihood of eliciting successful versus abortive approaches.

Animals were trained for 5 days to feed from a blue circle displayed in the centre of a monitor placed on the floor of the experimental cage, before being tested on an unrewarded target (Fig. 1A). We define three closed-loop conditions: ‘expansion only’ (EO), where the target remains in the centre of the monitor but changes its diameter to give the impression from the butterfly's viewpoint of a constant-size object either above or below the monitor; parallax only (PO), where the target's size on the monitor is constant but its position changes to create the illusion of depth; and the more realistic ‘expansion+parallax’ (EP), where both manipulations occur in tandem (Fig. 1B,C; supplementary material Movie 1). Additionally, we define the open-loop control condition ‘static’ (S) where the target is fixed, as in the training sessions. Fig. 2 shows a sample flight trajectory recorded in the S condition.

The closed-loop manipulation of the target's appearance has no obvious disruptive effect on gross flight behaviour; repeated-measures ANOVAs detect neither significant effects of closed-loop condition (i.e. S, EO, PO or EP) nor of the target's apparent depth on either mean horizontal flight speed (condition: P=0.64, d.f.=3,123; depth: P=0.28, d.f.=1,41; supplementary material Fig. S1A) or median altitude (condition: P=0.45, d.f.=3,123; depth: P=0.37, d.f.=1,41; supplementary material Fig. S1C). There are, however, pronounced differences between individuals in terms of these metrics (main effect of individual on flight speed: P<0.0001; altitude: P<0.0001; d.f.=41,531).

The animals tended to fly near the ceiling of the cage when no target was displayed on the monitor. In six 30 s control tests (in different individuals) with a blank screen, only one instance of flight below 15 cm altitude was recorded (median minimum altitude: 29.8 cm, supplementary material Fig. S2). Therefore, we consider descents to below 15 cm to represent attempts to approach the target. To quantify the attractiveness of the stimulus, we measured the time between its onset and the animal's first foray below this altitude (Fig. 3A). A significant main effect of apparent depth exists (P=0.0029, d.f.=1,41), such that ‘closer’ targets are approached sooner. This analysis includes the S condition (apparent depth=0 cm), but a similar result is obtained if we consider only those cases where the appearance of the target is manipulated (P=0.0064, d.f.=1,41). Continuing to exclude S, neither a main effect of closed-loop condition (P=0.71, d.f.=2,82) nor an interaction between closed-loop condition and apparent depth (P=0.38, d.f.=2,82) was found. However, performing separate ANOVAs for each closed-loop condition revealed significant relationships between apparent depth and time to approach in those involving parallax manipulation (PO: P=0.026; EP: P=0.025; d.f.=2,82), but not in EO (P=0.56).

To confirm that this effect could be attributed to the motion of the target from the butterfly's perspective, we repeated the experiment using a ‘yoked-loop’ set-up, where the animal was played a recording of the stimulus generated by a previous individual's closed-loop flight, i.e. a target whose size and/or position fluctuates with no relation to the current animal's behaviour. No main effect of ‘apparent depth’ (of course, the illusion was then broken) and time to approach existed (P=0.27, d.f.=1,22; Fig. 3B). More importantly, analysing both data sets together (excluding the S condition) revealed a significant interaction between paradigm (i.e. closed- versus yoked-loop) and apparent depth (P=0.013, d.f.=1,774).

To establish that flight behaviour was broadly similar between paradigms, we measured flight speed and altitude in the yoked-loop conditions, as we did previously for closed-loop conditions (supplementary material Fig. S1B,D). Because these experiments were performed on different individuals, and results from one individual in different trials cannot be considered statistically independent, we calculated means for each animal across all 12 EO, PO and EP trials. There was no difference in mean horizontal flight speed (P=0.54, d.f.=63, Mann–Whitney U-test), but a significantly higher median altitude was observed in the yoked-loop paradigm (P=0.039), which probably reflects the lower overall attractiveness of the yoked-loop targets (Fig. 3). The S condition was ostensibly identical in both paradigms, yet we observed a significant difference in time to approach (P=0.0007, d.f.=63, Mann–Whitney U-test). In addition to the aforementioned large individual differences, this may be caused by the differing experience of the previous trials in the session. The possibility of such a learning effect is a troublesome confounding factor that must be borne in mind when interpreting our results. However, as the order of trials within a session was randomised, it should introduce no systematic bias.

One might expect the animals to modulate the speed and/or steepness of their approach according to the perceived distance of the target, such that objects apparently further below the monitor would elicit more violent contacts with the surface. To assess whether this is the case, we identified the first descent in each trial that reached an altitude of less than 3 cm (Fig. 2B). This rather strict criterion was employed to exclude abortive approaches (or ‘fly-bys’; van Breugel and Dickinson, 2012) from our analysis. For each of these initial descents we measured the instantaneous downward velocity at 5 cm altitude; the downward velocity over the altitude interval 25 to 5 cm; the altitude of the animal at the temporal midpoint of this spatial range, to characterise deceleration; and the total horizontal trajectory length over that vertical interval, to characterise steepness (Fig. 4). Neither any main effects of apparent depth, nor any interactions between condition and apparent depth, were found for any of these metrics (supplementary material Table S1).

As Fig. 2B illustrates, not all descents resulted in successful landings. Fig. 5A shows the number of forays the animal made below 15 cm altitude prior to its first successful descent (i.e. reaching an altitude of less than 3 cm). A significant interaction between apparent depth and closed-loop condition was found (P=0.034, d.f.=2,82). As there was little difference in elicited behaviour between virtual targets 4, 8 or 12 cm below the surface, Fig. 5C pools the data according to the polarity of the apparent depth. In the EO condition, virtual targets above the surface (EO−) elicited significantly fewer abortive approaches than those below it (EO+) (P=0.014, d.f.=42, Wilcoxon signed-rank test), while the opposite pattern was observed in PO (P=0.0090). When both cues are present (EP), no significant difference was found (P=0.18). Furthermore, the number of abortive approaches in both EO− and PO+ was significantly lower than in S (P=0.018, 0.0086, respectively). This phenomenon was never observed in the yoked-loop paradigm (Fig. 5B,D; S versus PO+: P=0.11, d.f.=23), indicating that successful approaches are not facilitated by dynamic stimuli per se, although certain types of yoked-loop stimulus motion may hinder them: PO− elicited significantly more abortive approaches than PO+ (P=0.015, d.f.=23). Unlike the time to approach data, these results may therefore be attributed, at least partially, to the general nature of the target's movement rather than the illusion created from the animal's perspective by the closed-loop system.

Returning to the closed-loop paradigm, the absence of a significant difference in number of abortive approaches between EP− and EP+ suggests that the opposing effects of expansion and parallax may combine additively and thus cancel each other out. To further explore this possibility, we performed a separate experiment using ‘chimeric’ stimuli: E−P+, where the target gives expansion cues as though it were situated 4 cm above the monitor, but parallax cues as though it were 12 cm below; and E+P−, the converse case (Fig. 5E). The former target required significantly fewer attempts than the static control to approach successfully (P=0.024, d.f.=54, Wilcoxon signed-rank test), while the latter required more (P=0.00085, Fig. 5F).

Although it is known that Papilio require intensity contrast to land on targets (Koshitaka et al., 2011) and in the present study, we observe that they frequently land close to and extend their proboscis towards the perimeter rather than the centre of the target, it has not yet been established that they ‘aim for’ visual edges when landing. To confirm that this is the case, we presented trained individuals with a static asymmetrically blurred target (supplementary material Fig. S3). Landings were significantly biased towards the side with the steeper intensity gradient (Wilcoxon signed-rank test, P=0.021, d.f.=38).

Fig. 6 offers a dynamic systems analysis of approaches under various conditions, in terms of the elevation of the nearest edge of the target from the animal's viewpoint and the instantaneous gradient of its descent (Fig. 6A). Successful trajectories in the S condition include many segments where both parameters are rather well matched at around 60 deg (Fig. 6B). Note that, in general, as the animal's altitude decreases, the retinal position of the target edge will tend to become increasingly unstable, corresponding to a leftward drift on the diagram. Fig. 6D shows a similar analysis of abortive approaches reaching a minimum altitude between 5 and 10 cm. Here, the bimodal distribution of trajectory segments reveals two qualitatively different failure modes, of which one example of each type is shown in Fig. 6C: undershooting, i.e. descending too steeply (cyan) and overshooting, i.e. failing to descend sufficiently steeply (blue). In Fig. 6E,F we consider two closed-loop conditions where few abortive approaches were observed (Fig. 5): EO(−4) and PO(+12). In both cases, the distributions are shifted rightward; the nature of these closed-loop manipulations means that the target edge will tend to remain in a ventral position rather than drifting towards the horizon as the animal descends.

A serious issue with this analysis is that we presented the stimuli on an LCD monitor, so contrast was attenuated (and colours distorted) at shallow viewing angles. To quantify this effect, we measured the contrast between the target and background from various angles, and found that it was relatively constant up to at least 40 deg, but then drops off rapidly (supplementary material Fig. S4). Thus, an elevation of ∼45 deg may represent a ‘point of no return’ where the target becomes invisible and the approach cannot be recovered. This is reflected in the leftward directionality of almost all the arrows towards the extreme left of Fig. 6B,D.

Using a virtual reality system, we presented stimuli to free-flying Papilio that mimicked a (previously rewarded) target situated at some vertical displacement above or below the arena floor. The butterflies were slower to initiate approaches to targets the deeper they appeared to be, and this effect appears to be elicited primarily by parallax cues. We interpret this to mean that objects that move quickly across the retina (and thus are presumably close) are more attractive. The animal can perform this computation rather quickly; for targets of apparent depth −4 cm in the PO condition, the first quartile of time to approach is just 1.02 s from stimulus onset, which includes the time taken to descend to 15 cm altitude.

It is important to stress that this effect cannot be trivially explained by target size, as it occurs even when this parameter is (unrealistically) held constant, in the PO condition. Furthermore, if the butterfly were simply attracted to targets based on their proximity in the horizontal plane (either because closer targets subtend larger visual angles, or because the steeper viewing angle to the monitor maximises contrast) we would expect to see the opposite pattern (see Fig. 1B,C).

Contrary to our expectations, we find no clear evidence that the insects approach apparently shallower targets more slowly, with sharper deceleration, or at shallower angles; though of course, we cannot exclude the possibility that subtle effects on these kinematic metrics exist (see particularly Fig. 4B,C). However, we do find that the frequency of abortive approaches depends on the apparent depth of the target, and interestingly, the polarity of this effect is opposite for expansion and parallax cues. This effect is demonstrated particularly clearly using ‘chimeric’ stimuli, which give expansion and parallax cues corresponding to two different depths. The observed facilitation of successful approaches can be understood as the effect of the closed-loop system helping the animal to stabilise the edge of the target in a ventral retinal position.

On the one hand, it is rather remarkable that significantly fewer abortive descents are observed in the EO− and PO+ closed-loop conditions than the S control (Fig. 5C); why would a stationary target with which the animal has had at least 5 sessions' experience be ‘harder’ to land on than a jittery, unfamiliar, dynamic one? However, when one considers the nature of these closed-loop manipulations, perhaps this result is not so surprising. In EO−, the target expands as the butterfly approaches; and in PO+, it moves to track the animal's flight. In both cases, the animal cannot miss; as long as it descends towards some point on the monitor, the target will come to it (albeit briefly in the case of EO−, as it disappears when the butterfly reaches 4 cm altitude). Viewed in these terms, it is somewhat impressive that the animal is able to approach the target at all in the PO− and especially E+P− conditions, as these represent the opposite case where the butterfly repels rather than attracts the target. Furthermore, the static target is more challenging to land on than an equivalent physical object, as the nature of the experimental apparatus means that its visibility will be attenuated if it is approached from shallow angles (supplementary material Fig. S4).

At the very least, these findings strongly imply that target approach is not the result of an open-loop, feed-forward motor program, but rather that the insect tracks the target in a continuous fashion. Prior to initiating an approach, when the animal's altitude is >30 cm, it experiences little difference between any of the closed-loop conditions, so approaches are presumably aborted as a result of sensory input received once the manoeuvre is under way. This issue could be investigated more directly by having the target abruptly vanish at a certain point during the approach (for an analogous experiment on walking flies, see Strauss and Pichler, 1998).

We have based our analyses of the butterfly's flight control on the retinal position of the nearest edge of the target (Fig. 6). There are three reasons why we consider this to be a more salient feature than its centre. The first argument is logical: if one has the goal of contacting an object, then its edge – not its centre – defines the boundary between success and failure. Second, we found that when given a choice between sharp and gradual contrasts, butterflies display a clear bias towards landing on the former (supplementary material Fig. S3). The third line of evidence comes from the EO condition; as the position of the target is fixed in these trials, we would expect to see no difference in behaviour across apparent depths if the animal were targeting its centre. On the contrary, the frequency of abortive approaches is significantly higher in EO+ than EO− (Fig. 5C), which we attribute to the edge receding away from the animal during its descent. Note that the abortive approaches in EO+ cannot be explained simply by the target becoming too small to be clearly visible, as it still expands on the retina during the descent, albeit at a slower rate than a real object on the monitor surface would.

An obvious question is why Papilio do not use the apparent distance to the target to control their approach speed. One possible answer is that during training they have learnt the dimensions of the cage and thus ‘know’ how far they are from contact with the surface at all times. However, this account is difficult to reconcile with their increased attraction to apparently closer targets. We think a more plausible explanation is that the butterflies are physically robust enough that they simply have no need to land softly, at least over the range of flight velocities observed in this small arena. This may be a rather common feature of insect flight; when fruit flies land on a vertical post, over a third of landings are ‘crashes’ where the head or wing contacts the surface before the legs (van Breugel and Dickinson, 2012). This question could be clarified by investigating behaviours such as leg and proboscis extension, which are performed in anticipation of imminent contact. Unfortunately, our present camera set-up lacks the necessary resolution for such an analysis.

Given this apparent disregard for the perceived distance of the target, together with the evidence for closed-loop edge tracking discussed above, it seems unlikely that Papilio truly plan their trajectories in three-dimensional space. Rather, our results are more consistent with them simply adjusting the steepness of their descent in order to stabilise the position of the target on the retina. The tendency of flying insects to fixate and land upon visual edges is well documented (e.g. van Breugel and Dickinson, 2012; Maimon et al., 2008). However, most studies of this kind have been concerned with the problem of stabilising vertical contrasts in the yaw dimension (but see Lehrer and Srinivasan, 1993). Our study provides evidence that a similar targeting process occurs towards edges on ventrally positioned horizontal surfaces. Like male blowflies pursuing potential mates by holding their retinal position and size constant (Boeddeker et al., 2003) or human baseball players catching fly balls by moving to keep their retinal trajectory linear (McBeath et al., 1995), this may represent a computational short-cut to reduce a complex but underconstrained three-dimensional problem to a simpler one in two-dimensional visual space.

If we consider only the EO condition (where parallax is not manipulated), there is no significant relationship between the attractiveness of the target and its apparent depth. However, it would be hasty to conclude that object expansion is not used to estimate distance; we quantify attractiveness by measuring the time taken to initiate an approach, but by definition, little expansion can be observed by the animal until it starts approaching the target. Having said this, it is reasonable to suppose that in general, expansion is a less-reliable cue than parallax in terms of signal-to-noise ratio. This is because the magnitude of the optic flow generated by an object at a certain distance is proportional to the sine of the angle between the object and the direction of translation, and probably explains why peering locusts and mantids sweep their heads laterally rather than back and forth.

At first blush, foraging for flowers or other static, highly visible food sources seems to be a somewhat trivial task: as soon as a target is detected, approach it. Having acute vision to extend the range at which targets can be detected is presumably advantageous, but introduces a complication: some means of favouring closer targets is required if foraging efficiency is to be maintained. While retinal size could be used as a proxy for proximity, this may be unreliable as a result of the effects of orientation and partial occlusion. Worse still, such a system could be fooled by large distant objects that shared the same visual properties (e.g. colour, polarisation, brightness, etc.) as target flowers. However, as long as the targets are approximately stationary, motion cues provide a way to unambiguously estimate their distance. Despite finding themselves in a restricted, artificial situation where the target can never be more than a metre away, the butterflies in this study do appear to be performing a computation of this kind, suggesting that it represents a fundamental foraging strategy. It should be stressed, however, that without performing experiments using physical targets at various distances, it remains somewhat unclear how well our behavioural metrics such as time to approach correspond to real-world distance perception, or indeed whether Papilio can truly be said to possess such a capability.

From this and previous behavioural studies on Papilio xuthus, a detailed picture is emerging of the sensorimotor processes involved in foraging. First, the butterfly must acquire a target, which it presumably does based on spectral (and possibly polarisation) information; Papilio can discriminate colours of small targets observed at distances where they would be imaged by a single ommatidium (Takeuchi et al., 2006). Then, it gauges the distance using motion cues, and based on this information, decides whether to approach. Finally, once the target is close, it aims to land on an edge, requiring a contrast in intensity (Koshitaka et al., 2011). Far from being trivial, foraging flight in fact relies on the delicate interplay of a number of visual modalities and visuomotor control schemes.

Laboratory-reared spring-form adult Japanese yellow swallowtail butterflies (Papilio xuthus Linnaeus) were used for all experiments. The laboratory culture was derived from eggs laid by females caught in Kanagawa, Japan. Larvae were fed fresh citrus leaves under a 10 h:14 h light:dark cycle. Pupae were stored at 4°C for at least 3 months before being allowed to emerge at room temperature. Between training or test sessions, animals were kept in polystyrene boxes at room temperature, with no more than three individuals to a box, all of the same sex. They were not fed other than during training or testing.

The experimental arena was a cuboid cage measuring 70×50×50 cm (length, width, height), consisting of a frame of aluminium tubes covered with plastic netting with 9 mm square holes (Fig. 1A). It was lit with six overhead xenon lamps, and care was taken to light the arena evenly at approximately 4000 lx at floor level. A 24″ (61 cm) LCD monitor (Benq XL2420T) was placed flat on the arena floor, with a transparent acrylic sheet put directly on its surface to protect it. We define the centre of the monitor surface as the origin of our co-ordinate system, meaning that the ceiling's height was approximately 44.5 cm. The animals were tracked using two orthogonally oriented cameras (AVT Prosilica GE680, monochrome), which were synchronised by triggering them programmatically using a USB interface board (PhidgetInterfaceKit 8/8/8). The cameras each recorded 60 frames s⁻¹ with a 2.0 ms exposure at a resolution of 640×480 pixels (cropped to 500×480 for the camera oriented along the x-axis).

During training, a 100% brightness blue circle of diameter 5 cm was displayed in the centre of the monitor, with several drops of 6% sucrose solution placed on the acrylic sheet over it. One box of butterflies at a time were released into the arena and allowed to feed ad libitum. Any individuals not spontaneously feeding were caught and had their proboscis manually extended onto the sucrose drops. This procedure was repeated for 5 days, starting 24–48 h post-emergence.

For testing, a single animal was released at a time. As before, it was allowed to feed from the target, but would be quickly shooed away in order to prevent satiety and thus maintain high foraging motivation. After three successful landings, the monitor was blanked, the sucrose solution wiped off, and the testing phase begun. Any animal failing to land would not be tested, but rather given a further training session with manual feeding. If failure continued for five attempted test sessions, the animal was discarded.

In the main experiment, each testing session consisted of fourteen 30 s trials presented in a random order: three closed-loop conditions (EO, PO and EP) at four apparent depths each [−4 (i.e. above the monitor), 4, 8 and 12 cm], plus two control (S) tests. If on any particular trial the animal failed to descend below 5 cm altitude, that trial was repeated. If this happened more than five times in the whole session, or if the animal stopped flying, the test was abandoned and a further training session given. A similar procedure was followed for the chimeric experiment, but with just five trials per session (S, E−P+, E+P−, and two other chimeric conditions not discussed in this paper).

The cameras were calibrated by placing vertical posts in each corner of the monitor, giving eight points (top and bottom, ×4) in monitor-centric 3D space from which the projection matrices of the cameras could be fitted using a least-squares method.

The closed-loop system was controlled by a custom-written Java program. Immediately prior to the start of each trial, each camera captured 15 frames from which the pixel-wise median was taken to effectively erase the flying animal and obtain a background image of the empty arena. Once the trial was under way, each incoming frame was subtracted from the background to identify the dark region corresponding to the butterfly. This difference image was thresholded, dilated and the largest connected region found. The centroid of this was taken as the animal's position in the image plane; note that this could be ∼2 cm from the head in the worst case. Together with the projection matrix, this defined a ray in 3D space along which the animal must have lain. The point at which the rays of the two cameras come closest to intersecting was calculated; if they did not pass within 2.5 cm of each other, a tracking error was deemed to have occurred and the animal's position was unknown, otherwise the 3D position was recorded and made available to the stimulus presentation process.

The monitor was set to a resolution of 1024×768 pixels and a refresh rate of 60 Hz, to match the frequency of image acquisition. This is lower than the flicker fusion frequencies of Papilio xuthus photoreceptors, which are around 100 Hz (Nakagawa and Eguchi, 1994). However, filming the monitor at 1000 frames s⁻¹ revealed negligible flicker.

Every refresh cycle, the display was updated based on the last logged 3D position (x,y,z) of the butterfly. In all closed-loop conditions, the ‘true’ 3D position of the centre of the virtual target was (0,0,−d), where d was the apparent depth (Fig. 1B,C).

If the virtual target was above the monitor (d<0), then its appearance was undefined when the animal was below its altitude (z<−d), so no target was displayed. For this reason, virtual targets >4 cm above the monitor were not used, because the target would disappear long before the animal reached the surface, making comparisons between conditions difficult. In the case of chimeric stimuli (Fig. 5E), one value of d was used for Eqns 1 and 2, and a different one for Eqn 3.

In the yoked-loop paradigm, everything was done identically, except that instead of using the last logged 3D position of the animal, the corresponding time-point of a previously recorded closed-loop trajectory was used as the basis for generating the stimulus. Each yoked-loop animal was paired with a different closed-loop counterpart in a one-to-one fashion.

To quantify the latency of the closed-loop system, an LED was placed in the arena and one of the cameras was repositioned such that it imaged the monitor surface. The program was modified to display an indication on the monitor as soon the value of a pixel corresponding to the LED exceeded a threshold. Whilst the system was tracking a butterfly, the LED was lit. In the resulting recording, the delay between the LED being activated and the pattern appearing on the monitor was rather repeatably found to be three frames, corresponding to a latency of approximately 50 ms.

To test edge attraction, a special static stimulus was prepared, based on an 8-cm-diameter blue circle. This was convolved with a 1D Gaussian blur in the x-dimension, where the standard deviation varied linearly according to the position on x-axis (supplementary material Fig. S3). The resulting image was then blurred again with a uniform 1D Gaussian in the y-dimension. This stimulus was designed so that the centre of mass remained at (0,0). For testing, the stimulus was presented twice, in mirror-image orientations about the y-axis, and the animal's trajectory was tracked in real-time as before. In each trial the landing location was taken as the (x,y) co-ordinates of the lowest point of the first descent that reached <2 cm altitude within a 9 cm radius of the centre of the monitor.

The trajectories obtained from real-time tracking are rather jittery; in particular, an oscillation in the z-dimension due to the wing-stroke cycle is often apparent. Therefore, we smoothed the data by resampling it at 20 ms intervals using Gaussian averaging with s.d.=30 ms. Note that this is for analysis purposes only; the closed-loop stimulus presentation used the raw signal as averaging would increase latency.

When calculating time-to-approach, in the rare case that the butterfly was at an altitude <25 cm at the onset of the stimulus, the trial was excluded. Otherwise, we simply recorded the time at which its altitude first drops below 15 cm. For the analysis of descents, we identified the first manoeuvre in each trial where the butterfly moved from an altitude >30 to <3 cm, without climbing more than 2 cm, i.e. its altitude could never exceed the lowest point reached so far in the descent +2 cm. If no descents satisfying these criteria were found, that trial was excluded from the analysis.

A ‘successful’ approach was defined as one reaching a minimum altitude <3 cm. We term these manoeuvres ‘approaches’ rather than ‘landings’, because the butterflies frequently made only fleeting contact with the surface, particularly (but not exclusively) in those closed-loop conditions where the target disappears in the final stages of the descent. To ensure consistency across closed-loop conditions, the position of the animal in the horizontal plane was disregarded. Thus, in rare cases, a trajectory where the animal approached some part of the monitor distant from the target could be deemed successful. ‘Abortive’ approaches were those reaching a minimum altitude <15 cm but >3 cm, unless otherwise specified.

When performing ANOVAs, virtual target depth was treated as an interval type independent variable, whereas closed-loop condition was a categorical variable with four categories: EO, PO, EP and S (though S was often excluded).

We thank the anonymous reviewers for their very helpful comments.