Neural and molecular mechanisms underlying female mate choice decisions in vertebrates

Few decisions in life are more consequential for individual fitness than choosing a mate. Mate choice (see Glossary) is the result of selectively mating with only some individuals of the opposite sex, whose members compete for fertilization opportunities. Because of its evolutionary significance, the ultimate causes and consequences of mate choice have been studied in depth (Andersson and Simmons, 2006; Jennions and Petrie, 1997; Widemo and Sæther, 1999). Variability in trait expression in one sex is often mirrored by correlated variability in preferences of the opposite sex. This results in non-random mating patterns (Edward, 2014) and functions as a primary driver of sexual selection (see Glossary; Andersson, 1994). As a consequence, variation in the phenotypes of the chosen sex are positively correlated with differences in evolutionary fitness (Halliday, 1983). In fact, individuals prevented from exercising mate choice (e.g. in forced pairings) experience reduced reproductive success and lower offspring viability (Gowaty et al., 2003; Iyengar and Eisner, 1999; Lancaster et al., 2009).

In species that exercise mate choice, a display producer expresses traits that attract mates, and these traits evolve in tandem with the mechanisms that allow the chooser to discriminate amongst such traits displayed by potential mates. For example, in the guppy Poecilia reticulata, a poeciliid fish, males display their tails to attract female mates, and females prefer longer tails to shorter ones (Bischoff et al., 1985). Hence, both tail length in males and the corresponding discriminatory mechanisms in females must correlate for the selection of longer-tailed males as mates. As we discuss below, females are often the ‘choosier’ sex. Given the critical importance of female mate choice in individual fitness, sexual selection and evolution, it is surprising how few studies have examined the underlying neural and molecular mechanisms, although this is beginning to change. In this Review, we summarize what is currently known about the neural and molecular mechanisms of female mate choice and how these mechanisms fit within a broader understanding of vertebrate social behavior (see Glossary). Given the paucity of studies exploring the mechanistic basis of female mate choice in invertebrates, here we discuss only vertebrates. We focus on neurochemical signals and brain areas that have previously been identified as evolutionarily conserved across vertebrates and have known roles in the regulation of social behaviors outside of mate choice, including (but not limited to) aggression, parental care and sexual behavior (O'Connell and Hofmann, 2011a,b; Weitekamp and Hofmann, 2017). Finally, we highlight future avenues of fruitful research opportunities.

Mate preference and mate choice (see Glossary), two distinct forms of behavior, are often inherently related (Lynch et al., 2005; Rosenthal, 2017). This is especially true in species that display long-term associative partnerships prior to reproduction (DeAngelis and Rhodes, 2016; Donaldson et al., 2010), where preference for a particular mate precedes the choice to mate. A relationship between mate preference and mate choice is also observed in non-monogamous species, where a preference to associate with individuals displaying a particular phenotype is expressed prior to mating (Cummings et al., 2007; Desjardins et al., 2010; Wong and Cummings, 2014). In many species, the formation of a particular mate preference may be more critical to fitness than the moment of choice when mating occurs. Importantly, the research we discuss in this Review encompasses studies addressing both the mechanisms of female mate choice and those underlying female preference, as they are intertwined.

Females, by definition, produce a limited number of larger and more metabolically expensive gametes compared with males, which tends to limit their reproductive opportunities, whereas males are more often limited by the number of females that are available for mating. Hence, females increase their fitness by mating with optimal male phenotypes and investing in offspring survival. The resulting skew of a small proportion of males obtaining a larger proportion of available mating opportunities leads to a greater chance of selection from female choice than from male choice. In many species, females are therefore the choosier sex and males are more likely to show a greater response to the pressures of sexual selection compared with females. Of course, there are many examples where males display mate choice behaviors – such as in fishes, anurans, reptiles, birds and mammals (Liao and Lu, 2009; Preston et al., 2005; Shine et al., 2004; Werner and Lotem, 2003) – and males may even be the choosier sex in some instances. However, males generally choose mates based on non-heritable characteristics, such as reproductive state (Edward and Chapman, 2011). The evolutionary consequences of male mate choice are muted in such cases, because sexual selection occurs only if the preferred phenotype increases fitness in the chosen sex. Therefore, in this Review, we primarily focus on mechanisms found within the female brain that underlie the decision to mate with distinct and specific male phenotypes.

The ultimate consequences of female mate choice are an increase in individual fitness and the corresponding evolutionary persistence of a particular male trait (Edward, 2014; Lancaster et al., 2009). Although these ultimate consequences are well understood, the specific proximate mechanisms that underlie the processes of ‘how’ female mate choice occurs remain enigmatic. In many species, mate choice is fundamentally a social process that relies on the integration of (often multimodal) sensory information – that signals sex, species and the quality of potential mates – with internal physiological conditions, such as reproductive status and available energy reserves. The brain's decision-making circuit must evaluate this information, possibly in conjunction with social signals from other conspecifics (see Glossary), before expression of a mate preference or choice. Importantly, neither choice nor preference are necessarily static, as both can vary with season, environmental condition, physical condition, reproductive state, hormone levels, and previous experiences and/or affiliations. The recognition that the neural and molecular processes in the female brain are a key substrate of sexual selection, in combination with conceptual and technological advances, has resulted in a growing interest in studying the neural substrates underlying this behavior.

Although different vertebrate species display a diversity of social behaviors, many of which have evolved independently, these behavioral outputs often share common underlying neural and molecular substrates (O'Connell and Hofmann, 2011b; Weitekamp and Hofmann, 2017; Young et al., 2019). An evolutionarily ancient social decision-making network (SDMN; see Glossary) located in the forebrain and midbrain of all vertebrates evaluates the salience and valence of a social stimulus by integrating sensory information about the (social) environment with an individual's own condition and prior experience, eventually resulting in a behavioral choice (O'Connell and Hofmann, 2011a,b, 2012). Importantly, deeply conserved signaling pathways – such as steroid hormones, neuropeptides and biogenic amines – are involved in regulating SDMN function in the context of aggression, parental care, pair bonding and sexual behavior (reviewed in Weitekamp and Hofmann, 2017). This framework is well suited for gaining an integrative understanding of the neural circuits and signaling molecules underlying any social behavior across diverse species. In the following paragraphs, we will place what we currently know about the neural and molecular underpinnings of female mate choice and related behaviors within this framework (Fig. 1). Because of its tight functional integration with the SDMN, we include here also discussion of the role of the neocortex, and its non-mammalian homologs, in mate choice. The neocortex was originally not included in the SDMN because the putative neocortical homologs in other vertebrate lineages were not well known at the time (O'Connell and Hofmann, 2011a,b), but recent research has provided important new insights in this regard (Briscoe et al., 2018; Ito and Yamamoto, 2009; Karten, 2015).

Females prefer to mate with males based on phenotypic traits that are perceived as attractive. Body coloration, odor cues and song production are all examples of traits that different taxa use to assess potential mates. However, before they make a decision, females first need to recognize and discriminate conspecifics from heterospecifics (see Glossary), as well as males from other females. Several studies in diverse species have uncovered potential neural and molecular mechanisms involved in preference formation, sex discrimination and species recognition (see Glossary). Brain regions such as the nucleus accumbens, amygdala, preoptic area and cortical areas have all been implicated in the discriminatory process of mate selection. The role of an individual's genotype, as well as genomic responses, have also been the subject of mate choice studies. Although this field of research has already generated important insights, a broader theoretical framework for how this process occurs across vertebrates remains to be developed.

In monogamous species, the formation of a mate preference is often closely linked to mate choice and is therefore critical for individual fitness and its evolutionary consequences. This process has been extensively studied in prairie voles (Microtus ochrogaster; Johnson and Young, 2015; Young et al., 2011). Depending on various ecological and demographic factors, monogamy (see Glossary) can be advantageous, as it eliminates the need to find additional mating partners once a pair bond is formed (Emlen and Oring, 1977). Further, vigorous territory defense and biparental care can increase offspring survival. The initial act of mating is critical for the formation of a pair bond, where specific males and females express a strong preference for each other, while avoiding other conspecifics (Young, 2003; Young and Wang, 2004). The nonapeptides arginine vasopressin (AVP) and oxytocin (OXT), along with the dopaminergic system, play a critical role in this process (Johnson and Young, 2015; Lim and Young, 2004; Walum et al., 2012; Young and Wang, 2004). Specifically, the act of mating activates the ventral tegmental area, resulting in increased dopamine activity in the nucleus accumbens and prefrontal cortex, which synchronize with the medial amygdala and lateral septum, areas rich in neuropeptide receptors, to associate social learning with encoded reward. This orchestrated activity of dopaminergic and peptidergic brain areas reinforces the act of mating to a conditioned partner preference, thus forming enduring pair bonds (Young and Wang, 2004). While this process has been most extensively studied in voles, several studies in birds and fish have provided evidence that similar mechanisms regulate pair bonds across vertebrates (Day et al., 2019; Kelly and Goodson, 2014; Klatt and Goodson, 2013; Nowicki et al., 2020; Oldfield and Hofmann, 2011).

Species recognition is a critical aspect of reproduction, as mating with a heterospecific individual in most cases squanders reproductive effort and diminishes fitness (Burdfield-Steel and Shuker, 2011). The process of species recognition during mate choice has been studied in several vertebrate species, including fish and birds, (Caspers et al., 2009; Couldridge and Alexander, 2002; Tokarz, 1995; Uy et al., 2009), yet the cognitive architecture of this discriminatory process remains unclear (Phelps et al., 2006). The ability to discriminate conspecifics from heterospecifics is thus a critical task during mate selection. This process was investigated by Hoke et al. (2008), who examined the induction of the immediate early gene (IEG; see Glossary) egr-1, a marker of neural activity, in the túngara frog (Physalaemus pustulosus), a model system in research on sexual selection (see Glossary). When the authors exposed males and females of this species to calls from either conspecifics or a closely related species, the IEG response in the superior olivary nucleus, an auditory region in the lower brainstem, did not differ by sex. However, there were sex differences in IEG expression in the laminar nucleus of the torus semicircularis, a midbrain auditory region. Specifically, egr-1 expression increased in males following exposure to both conspecific and heterospecific calls. Conversely, in females, this region was only activated in response to conspecific calls. This sex difference in selectivity for conspecifics over heterospecifics may be a reflection of their higher investment in reproduction, which means that missed mating opportunities are more costly for females. Patterns of neural activity in the laminar nucleus of the torus semicircularis mirrored behavioral responses, inducing calling in males and phonotaxis (see Glossary) in females (Hoke et al., 2008). These results suggest that sex differences in mate selectivity are mirrored by selectivity in midbrain regions, which may act as decision-making areas in relaying auditory cues to forebrain processing areas (Wilczynski and Ryan, 2010).

Like species recognition, sex discrimination is paramount in mate selection, as same-sex mating squanders reproductive effort. Individuals in search of reproductive opportunities must be able to recognize and express a preference for members of the opposing sex. In female Syrian hamsters (Mesocricetus auratus), lesions of the medial preoptic area result in no differences in lordosis (see Glossary; a precopulatory motivational behavior) or vaginal scent marking compared with controls, but do eliminate the normal preference for male compared with female odors. However, these hamsters retain the ability to discriminate between male and female scent markings (Martinez and Petrulis, 2013). These results suggest that although the medial preoptic area may not be necessary in sex discrimination, it is critical in regulating female preferences for males. Although the ability to recognize the opposite sex is obviously important, opposite sex preference is also necessary for successful reproduction. However, the processes in the brain through which this occurs remain largely unknown.

In many species, kin recognition (see Glossary) has been suggested to minimize inbreeding (Tang-Martinez, 2001). Specifically, genes belonging to the major histocompatibility complex (MHC), which encodes proteins that identify foreign substances within the body, have been implicated in kin recognition and inbreeding avoidance (Grob et al., 1998), immune competence (Kamiya et al., 2014; Sommer, 2005) and genetic compatibility (Penn, 2002). In fact, individuals of numerous species can recognize even unfamiliar kin based on their MHC profiles (Gerlach and Lysiak, 2006). In this way, MHC genes likely play an important role in mate choice and sexual selection, although how variation in MHC alleles affects the underlying sensory and decision-making mechanisms in the brain remains largely unknown (Santos et al., 2018).

Although an individual's genotype may bias its mating decisions, variation in neural gene expression profiles can also be associated with mating behavior. In fact, researchers are increasingly applying behavioral genomics approaches to examine the extent to which the neural and molecular mechanisms underlying social behavior are evolutionarily conserved (e.g. Rittschof et al., 2014; Young et al., 2019). To date, only one transcriptomics study (see Glossary) has compared the neural gene expression profiles associated with female mate choice across both populations and sex. Keagy and colleagues (J. Keagy, J. W. Boughman, H.A.H., submitted) examined how gravid females of three different populations of stickleback fish (Gasterosteus aculeatus) respond to nesting males from their own or a different population. As expected, females prefer males from their own population, and both male and female trait complexes (principal components of behavioral and/or morphological traits) vary across populations. Interestingly, although population explains most of the variation in gene expression, the authors identified several gene co-expression modules that vary depending on whether focal females had viewed males from their own or a different population. Individual candidate genes that were previously associated with female mate choice behavior and social decision-making more generally (for review, see Weitekamp and Hofmann, 2017) were also investigated. Remarkably, neuroligin-3b and neuroserpin are differentially expressed according to treatment (i.e. they show increased expression in females exposed to males of their own population), which is consistent with the findings of Cummings et al. (2007) – see below. In summary, the study by Keagy et al. (J. Keagy, J. W. Boughman, H.A.H., submitted) was the first to show that, across species, the activity of specific gene co-expression modules is consistently associated with a female's preference. The extent to which these molecular pathways associated with mating decisions are similar across diverse species, possibly revealing an evolutionarily ancient decision-making system, remains to be seen.

Of course, any association between gene expression changes in response to a social stimulus and the resulting behavioral response does not establish the direction of any causal relationship between genes and behavior. In fact, given the fast-paced social lives of many animals, it is possible that the molecular pathways uncovered by behavioral transcriptomic studies in relation to a variety of social behaviors may mainly serve to prepare the individual for similar situations in the future.

Behavioral ecologists have provided ample evidence that, in many species, females prefer to associate (and often mate) with males that are perceived as more ‘attractive’ (Andersson and Simmons, 2006). However, few studies have attempted to uncover the neural basis of this preference. In one example, during estrous, female mice (Mus musculus) prefer intact versus castrated males, and show increased neural activity in the preoptic area (as measured by IEG induction) during lordosis (see Glossary) following exposure to intact males. Lesions to either the preoptic area or medial amygdala abolish this preference, although females treated in this way are still able to discriminate between intact and castrated males (DiBenedictis et al., 2012; Sakuma, 2008). Moreover, it has also been shown that the vomeronasal organ and accessory olfactory bulb are important for social odor discrimination and sexual behavior in rodents (Bressler and Baum, 1996; Kondo et al., 2003). These results suggest that the preoptic area and medial amygdala are not involved in olfactory discrimination per se, but play an important role in coordinating adaptive behavioral responses to associate with attractive males prior to mating.

Several studies in songbirds have utilized IEGs to identify patterns of neural activity following experimental manipulations of the social and ecological environment that females experience prior to mating. In European Starlings, Sturnus vulgaris, for example, female preference for longer songs (considered to be an indicator of male quality and, thus, attractiveness) can be modulated by recent social experience (exposure to long versus short songs) (Sockman et al., 2002), and also by current ecological conditions (Sockman and Ball, 2009). Females with recent experience listening to long songs display an increased IEG response in the caudomedial mesopallium (CMM, putatively homologous to auditory cortical structures in the mammalian brain) when later exposed to long songs; previous experience with shortened day length (simulating the onset of the breeding season) amplifies this result (Sockman and Ball, 2009; Sockman et al., 2002).

Poeciliid fishes are a well-studied group in the context of understanding the neural basis of mate choice. Poeciliids are livebearers and exhibit a diversity of mating systems (see Glossary), with males often displaying alternative reproductive tactics (see Glossary; Lynch et al., 2012), rendering this family well suited to exploring the mechanisms of mate choice. In a pioneering behavioral genomics study, Cummings et al. (2007) exposed female northern swordtails (Xiphophorus nigrensis) to different male reproductive strategies and an all-female control. Females prefer to mate and associate with attractive courting males, and actively avoid smaller coercive males. Using a brain-specific cDNA microarray, the authors identified 306 differently expressed genes (Cummings et al., 2007) across treatments. Interestingly, a greater number of genes are upregulated when females were exposed to smaller coercive males. This genomic response may relate to the behavioral strategy of actively avoiding small males. Conversely, a greater number of genes show a reduction in expression when females are exposed to large (courting) males, and these genes are more highly expressed during exposure to all-female groups, suggesting that these two conditions (attractive large males versus sexually not-salient females) diametrically affect expression of the same genes (Cummings et al., 2007). Moreover, female preference is associated with the increased expression of genes related to synaptic plasticity (e.g. neuroserpin a and neuroligin-3) independent of social affiliation, whereas intrasexual affiliation increases expression of genes typically related to social bonding (e.g. OXT and AVP) (Ramsey et al., 2012). Interestingly, a follow-up study showed that inhibiting synaptic plasticity reduces female preference behaviors (Ramsey et al., 2014).

How do distinct mating contexts dynamically regulate brain gene expression profiles when females are exposed to attractive versus non-attractive males? The mating system of northern swordtails, where males either are courting or mate coercively, was compared with that of the mosquitofish Gambusia affinis, where males do not court and instead only pursue a coercive mating strategy. Following exposure to males, genes in the brains of females displayed opposite patterns of expression in response to these contrasting mating dynamics (Lynch et al., 2012). Expression of genes underlying synaptic plasticity (e.g. neuroserpin and neuroligan-3) was positively correlated with females' preference for attractive large males in swordtails, whereas in mosquitofish the situation was reversed. In another study, Wang et al. (2014) found that this pattern is reversed in mosquitofish following exposure to heterospecific courting swordtail males, suggesting that the relationship between expression of synaptic plasticity genes and mating behavior is dependent on the mating system and the mate choice environment. Finally, using in situ hybridization, Wong and Cummings (2014) showed that female preference is positively correlated with gene expression of neuroligin-3 in the telencephalic areas Dm (a putative homolog of the mammalian basolateral amygdala) and Dl (homologous to the hippocampus), the ventral medial hypothalamus, as well as the preoptic area and the ventral telencephalic area Vv (homologous to the lateral septum). Interestingly, expression of tyrosine hydroxylase, an enzyme that catalyzes the rate-limiting step in the synthesis of catecholamines (and the expression of which is often used to assess dopamine activity) was not dependent on choice contexts (Wong and Cummings, 2014). Given the role of dopaminergic signaling in reward reinforcement, this result might be surprising, although the experimental design may simply not have allowed sufficient time for tyrosine hydroxylase expression to change in response to a female's preference to mate with one male over another. Conversely, it is also possible that courtship only induces release and not necessarily synthesis of dopamine. Taken together, these studies suggest that the neural architecture underlying cognitive functions important for mate discrimination may be altered by the differential expression of synaptic plasticity genes in a dynamic social environment where females must continually discriminate amongst potential mates while also avoiding coercive males.

In a recent analysis exploring the changes in gene expression associated with mate preferences in another poeciliid species, the guppy (Poecilia reticulata), Bloch et al. (2018) exposed females to colorful and drab males for 10 min and subsequently measured transcriptomic changes in the optic tectum (which integrates visual information) and the telencephalon (where many of the SDMN nodes reside). Some females were known to show a preference for ‘attractive’ colorful males, and these females showed a different pattern of differentially expressed genes in the telencephalon than females that did not have a preference regarding male coloration. However, all females showed similar changes in gene expression in the optic tectum in response to colorful and drab males, suggesting that all females are able to discern the differences in male appearance, but some lack the ability to integrate that information appropriately within the telencephalon. In females that prefer colorful males, male coloration causes a differential genomic response at the sensory processing and decision-making levels. This study represents one of the most comprehensive examples for identifying unique transcriptional responses underlying mate preference formation. It is likely that brain regions sensitive to sensory information relay those inputs to decision-making areas that orchestrate the appropriate behavioral response (Bischof and Rollenhagen, 1999; Fisher et al., 2006; Hoke et al., 2008), a hypothesis that requires more research. In summary, species recognition, sex discrimination and discrimination of attractive versus unattractive mates are all important aspects of mate selection in choosy females. Although several brain areas, neurotransmitters, genes and genomic responses have been identified (Fig. 1), the integrative study of these pathways across species will provide further insight into the neural and molecular regulation of mate discrimination and selection.

Mate preferences are often highly dynamic, depending on age and reproductive state, as well as social and ecological factors. In many species, males and females acquire mate preferences early in life through exposure to particular traits of the opposite sex. Similarly, familiarity (see Glossary) with an individual can bias mate choice even in adulthood. Finally, seasonal and other ecological factors regulate reproductive state, which in turn influences selectiveness, receptivity and choice.

Early-life experiences – such as rearing environment, social interactions, social learning and parent–offspring bonds – can affect mate-choice behaviors and predispose individuals to specific mate preferences. One of the best-studied examples of early learning is sexual imprinting (see Glossary), which predicts mating displays and mating preferences in adulthood and has been described in teleost fishes (Delclos et al., 2020; Verzijden and ten Cate, 2007), songbirds (Ten Cate and Vos, 1999) and mammals (Kendrick et al., 1998), including humans (Bereczkei et al., 2004). Although the mechanistic basis of sexual imprinting has been addressed primarily in birds and, more recently, in poeciliid fish, it likely has important consequences for sexual selection and evolution in a wide range of species (Owens et al., 1999; Yang et al., 2019).

In a recent behavioral transcriptomics study, Delclos et al. (2020) raised females of the sheepshead swordtail fish (Xiphophorus birchmanni) with adult groups of either conspecifics or those of a sister species, the Highland swordtail X. malinche. The authors then tested the preference of these females for olfactory cues of either species, followed by whole-brain transcriptome profiling. The results showed that females preferred the odors of males to which they had been exposed in early life. Moreover, the authors discovered specific gene co-expression modules associated with rearing environment and odor preference, suggesting that specific molecular pathways might underlie sexual imprinting. Although this study provides an important foundation for future research in swordtails and other fishes, the specific neural circuits underlying the developmental acquisition of mate preferences have been studied in much more detail in songbirds.

In songbirds, juveniles of both sexes can become sexually imprinted on their father's song in a two-stage process that includes the acquisition of the song memory during a critical period early in life as well as a stabilization phase during the first sexual experience (Bischof and Rollenhagen, 1999). In several elegant studies in male zebra finches (Taeniopygia guttata), Bischof and coworkers implicated higher-order auditory projection areas putatively homologous to the mammalian auditory association cortex [the hyperpallium apicale (HA), the caudomedial nidopallium (CMN) and the caudomedial mesopallium (CMM), according to the avian nomenclature as revised by (Jarvis et al., 2005)] in both the acquisition and stabilization phases (Bischof and Rollenhagen, 1999; Lieshoff et al., 2004; Sadananda and Bischof, 2004). The extent to which this occurs in females, which do not sing, but memorize their father's song and become imprinted on it, has scarcely been investigated.

Where are these preferences formed and stored within the female brain? When zebra finch females are re-exposed to their fathers' song during the stabilization period, neural activity increases in the CMM, but not the CMN or hippocampus (Terpstra et al., 2006). In conjunction with the results in males discussed above, this finding suggests that the CMM may be an important brain area in the consolidation of learned songs and formation of preference, independent of the ability to produce the song. Subsequent research by Woolley and Doupe (2008) demonstrated that the activity of the CMM is most pronounced in response to song that is directed at the female, whereas the CMN responds the most to songs the female was exposed to previously, suggesting that these auditory association regions integrate discrete information independently and likely work in concert to coordinate mate preferences. We can conclude that the current social environment likely primes the brain to respond to previously formed preferences as a result of social exposure.

In another study exploring how development affects song preferences as adults, Chen et al. (2017) reared females either with both parents present or without the father present. Using the IEG egr-1 as a marker of neural activity, these authors demonstrated that CMN activity is dependent on developmental exposure and song stimulus. Females reared with their father present show increased egr-1 activity in response to courtship song compared to non-courtship song. Females reared in the absence of male song show no difference in egr-1 expression following the normally preferred courtship song compared with non-preferred non-courtship song. Finally, egr-1 activity in the CMM is not dependent on rearing environment, and is higher in response to courtship song versus non-courtship song (Chen et al., 2017). Hauber et al. (2013) went beyond these auditory association regions by examining how variable song stimuli and social rearing environment interact to modulate neural activity in the field L complex, which is the primary auditory forebrain area activated by hearing natural sounds and which receives input from both the CMN and the CMM. Specifically, these authors reared female zebra finches in one of three conditions: with both parents present, with only the mother present (and the father absent) or with Bengalese finches (Lonchura striata domestica) as foster parents. Once they had reached adulthood, these females were then exposed to song playbacks from either zebra finch, Bengalese finch, including their own (foster) father's song, or a Parson's finch (Poephila cincta), and neuronal activity was recorded in the field L complex. In both the control and father-absent groups, field L complex neurons were more active in response to conspecific songs, and no differences were found in cross-fostered females following exposure to conspecific or their foster father's song. In cross-fostered females, neuronal firing was higher following exposure to the foster species' song compared with the song of the Parson's finch (Hauber et al., 2013). These findings underscore the importance of early-life social experience in the context of species recognition and sexual imprinting.

Day et al. (2019) explored the role of dopamine receptors following the formation of a song preference in adult female zebra finches. These authors exposed paired and unpaired females to either a known or a novel song. Not surprisingly, paired females preferred their partner's song to that of a stranger, whereas unpaired females showed no preference. Then, using a series of antagonist and agonist treatments at both the D1 and D2 dopamine receptors, the authors provided evidence that the D2 receptor is both necessary and sufficient for the maintenance of this preference (Day et al., 2019). They suggest that the dopaminergic reward system is likely activated during pair-bond formation, making the preference for familiar song rewarding and maintaining the social bond.

Taken together, these data highlight the dynamic interactive nature of both current context and previous experience on neural activity patterns in auditory processing regions in the context of female mate choice.

Clearly, previous experiences and/or familiarity with potential mates can strongly influence female mating behavior. Additional factors such as the current availability and quality of potential mates (Sockman and Ball, 2009), as well as familiarity between potential mates, can bias female mate choice (Kidd et al., 2013b). In fact, this is also the case in humans, where social familiarity with potential mates is an important prerequisite for partner affiliation and, ultimately, romantic love.

The phenomenon of love appears to be universal across human cultures (Jankowiak and Fischer, 1992), and is thought to be an evolutionary elaboration of the mammalian neural mechanism of mate choice (Fisher et al., 2006). Therefore, many of the neural signatures underlying familiarity in other vertebrates are likely also involved in the orchestration of human love. During the perception of romantic love, several brain regions operate synchronously, including those involved in sensory perception and emotional centers. Dopamine plays an important role in this process, as it may rewire neural circuits to encode sensory stimuli from loved ones in a way that is more potently rewarding, specifically through dopamine release in dopamine-rich brain areas (Lim and Young, 2004). In a functional magnetic resonance imaging (fMRI) study of people who self-reported as intensely in love, exposure to their beloved was followed by activation of dopamine-rich areas associated with mammalian reward and motivation, such as the ventral tegmentum area and right caudate gyrus (Fisher et al., 2005). Another study suggests that the neural substrates encoding sexual preference in humans include phylogenetically ancient and evolutionarily highly conserved subcortical brain structures, including the anterior and preoptic area of the hypothalamus, the anterior and mediodorsal thalamus, the septal area and the perirhinal parahippocampus, including the dentate gyrus, and excluding more derived regions of the neocortex (Poeppl et al., 2016). Although ethologists have often classified attachment and sexual affinity together with sex drive or motivation, several fMRI studies have provided evidence that the neural circuits and brain networks promoting reproductive motivation are distinct from those underlying the formation of romantic love (Arnow et al., 2002; Fisher et al., 2005, 2006; Gibson, 2015). Thus, to understand the neural underpinnings of mate choice, we need to dissociate courtship, attraction and choice from sexual motivation. Although these fMRI data from human studies provide compelling evidence that distinct mechanisms regulate each independently, corroborating evidence from non-human study systems is lacking.

Familiarity with an opposite-sex individual can also affect mate preferences in non-monogamous species. An elegant study by Okuyama et al. (2014) used Japanese rice fish (Oryzia latipes), also known as medaka, to explore how social familiarity can affect female mate choice. First, the authors showed that females can identify and recognize potential mates and that familiarization enhances female receptivity. They then identified two mutant lines with defective mating behaviors, in which females did not display enhancement of receptivity following mate exposure. Focusing on gonadotropin-releasing hormone (GnRH) and examining patterns of neural migration, they identified that these mutant lines showed abnormal development of terminal nerve (TN) GnRH3 neurons, thus demonstrating that normal GnRH3 peptides are required for female preference of familiar males. Familiarization facilitates TN-GnRH3 neuron activity, as firing rates are correlated with female receptivity (Okuyama et al., 2014).

In another study on medaka fish, Yokoi et al. (2020) generated lines carrying mutations in oxytocin (OXT) and oxytocin receptor (OTR) genes. Results indicate that the OXT/OTR pathway is critical for the formation of female preference for familiar males. In males, which prefer to mate with unfamiliar females, mutant lines display a loss of unrestricted promiscuous mating. The mutant lines display a series of transcriptional changes related to metabolism which differ by sex; these changes may explain the sex differences in behavior following mutagenesis, where mutant males display a loss of the normal preference to mate with unfamiliar females, whereas mutant females lose their preference for familiar males (Yokoi et al., 2020). These studies represent some of the most comprehensive attempts to explain the mechanistic neural basis of mate choice. Although the results highlight specific neural and molecular components in the maintenance of mate preference, future studies in a variety of model systems require more spatial resolution, possibly informed by the SDMN, to determine the extent to which the mechanistic basis of mate choice may be shared across vertebrate taxa.

Female familiarity and male–male interactions can also affect female mate-choice behaviors. In the highly social Burton's mouthbrooder, Astatotilapia burtoni, an African cichlid fish, females observe males as they competitively interact, and they integrate information based on the outcomes of these observed interactions, which may then alter the females' reproductive behaviors. In one study (Desjardins et al., 2010), A. burtoni females were first familiarized with specific males. Subsequently, females were allowed to watch an aggressive interaction between familiar and unfamiliar males. Neural activity patterns in nodes of the SDMN, including the lateral septum, preoptic area and ventromedial hypothalamus, were highly dependent on whether familiar males won or lost the fight. Following observation of familiar males losing a fight, the lateral septum (a region associated with anxiety and social recognition) was activated, whereas observing familiar males winning a fight activated the preoptic area and ventromedial hypothalamus, reproductive centers within the brain (Desjardins et al., 2010). These findings demonstrate how social interactions and group dynamics influence female reproductive behavior and highlight a need for future work exploring how social information is processed within the brain and subsequently influences female reproductive decisions.

Another important factor affecting mating behavior is female reproductive state. As physiological attributes vary with maturity and seasonality, and within the reproductive cycle, the motivation to find and select a mate also varies (Hunt et al., 2005; Lynch et al., 2005; Moore and Moore, 2001). Circulating steroid hormones are a major factor contributing to reproductive state, as they play an important role in the regulation of reproductive physiology (Adkins-Regan, 1998). In humans, variation within the menstrual cycle can affect female mate preferences (Puts, 2005). In female grey tree frogs (Hyla versicolor), exogenous administration of progesterone and prostaglandin increases the frequency of phonotaxis; however, treatment does not affect female discriminatory abilities (Gordon and Gerhardt, 2009). Non-reproductive A. burtoni females normally prefer to associate with small subordinate males as they are less aggressive than larger dominant males, but on the day of spawning they switch their preference and mate with larger dominant males (Kidd et al., 2013b). Remarkably, non-reproductive females treated with prostaglandin F2α (PGF2) dramatically reverse their normal preference (Kidd et al., 2013a). Building on this observation, Juntti et al. (2016) used gene editing to show that PGF2 signaling is a necessary factor for normal female reproductive behavior in this species. Furthermore, mRNA levels of the PGF2 receptor (Ptgfr) increase in the preoptic area around the time of mating. These results underscore the importance of the preoptic area in female sexual behavior and provide strong support for a causal role of the PGF2 pathway in regulating female mating preferences. The synthesis and release of neuropeptides also influences female reproductive state. For example, in the female grey tree frog, intracerebroventricular injections of AVP increase the speed of phonotaxis (allowing females to more quickly direct attention towards the acoustic signals of males), and blockade of AVP inhibits phonotaxis (Boyd, 2019). In summary, ancient signaling pathways – steroid hormones, neuropeptides, prostaglandins and other hormones – that regulate female reproductive physiology across vertebrates appear to also affect mating preferences, although much remains to be discovered about the neural circuits involved (Fig. 1).

A substantial body of work has addressed the neural and molecular substrates of social behavior across vertebrates. This work has identified a suite of brain regions, gene regulatory pathways, neurotransmitters and hormones that regulate social behaviors such as parental care, aggression, pair bonding and sexual behavior. However, how these brain systems function in mate choice remains understudied. Future studies exploring mate choice across diverse social systems, incorporating phylogenetic comparative methods (see Glossary) and utilizing new genetic and genomic techniques will substantially expand our understanding of how mate choice is mediated within the brain and the extent to which it is evolutionarily conserved across taxa.

Importantly, although mechanisms of cognition and choice are often shared across the sexes, there may also be certain differences, as the physiology of males and females can be very different (DeAngelis and Rhodes, 2016; Dulac and Kimchi, 2007; Goodson, 2005; O'Connell et al., 2013). Additionally, males and females often rely on different sensory inputs to assess mate quality. For example, during sexual imprinting in birds, females memorize their father's song and, as adults, prefer songs sung by potential mates that are similar to those of their fathers. Conversely, male birds evaluate potential mates through visual and/or olfactory cues, as females do not sing. Although different brain areas process these distinct sensory modalities, these cues may be relayed to the same association centers, which may act as a decision-making brain area in the processing of sensory information for the facilitation of mate choice. It will be fascinating to see future studies test the hypothesis that conserved brain areas act as decision-making centers both within a species (where sexes rely on distinct sensory inputs) and across taxa, to uncover the degree of evolutionary conservation of this process. If there are highly conserved decision-making areas present across taxa, we should see that although different groups rely on different sensory inputs, shared decision-making areas may act similarly in their orchestration of mate choice. These brain areas potentially include the highly conserved preoptic area, and future studies exploring this region in mate selection could provide further insight.

Diverse social systems across taxa with variable social dynamics also provide promising systems with which to investigate the neural mechanisms of mate choice. One example is the cichlid fish A. burtoni, a species that displays multiple phenotypes within a sex, where dominant reproductive males can be either blue or yellow in color (Dijkstra et al., 2017). Another is when alternative reproductive tactics are present, as are in ruffs (Philomachus pugnax), a lekking bird species where both courting and satellite males are present (Lank et al., 1995). These species provide ideal study systems for investigating female mate choice. In these species, females must make mating decisions after assessing different options in dynamic social environments.

With recent advances in genomics and data processing, and corresponding reductions in cost, phylogenetic comparative approaches can yield strong inferences similar to those from experimental approaches. One recent example utilizing a comparative framework was in exploration of a conserved transcriptomic signature underlying monogamy. Young et al. (2019) compared transcriptomes of paired monogamous and non-monogamous species across diverse vertebrate lineages and found substantial evidence to support the hypothesis that conserved and ancient gene modules have been recruited repeatedly in evolutionarily independent transitions to monogamy. This suggests that there may be other universal molecular mechanisms underlying similarly fundamental social behaviors in vertebrates and beyond (O'Connell and Hofmann, 2011a,b; Toth and Robinson, 2007). How mate choice decisions rely on similarly conserved transcriptomic profiles across distantly related taxa remains unknown and provides an exciting avenue of future comparative research.

Finally, recent technological advances in neuroscience provide many exciting opportunities to clarify the relationship between brain and behavior. For example, in a recent study by Kohl et al. (2018), virus-mediated retrograde trans-synaptic tracing, fiber photometry and calcium imaging (see Glossary) were used to elegantly detail the relationship between galanin circuit architecture and parental behavior in the mouse (Mus musculus) (Kohl et al., 2018). In another example, Kingsbury et al. (2019) used calcium imaging to simultaneously record neural activity in two interacting mice, illustrating that brain activity is correlated between individuals interacting in real time. These experiments represent remarkable examples of how new technologies can uncover not only the neural architecture of specific neuronal circuits, but also how discrete components functionally regulate social behavior. Although there are limitations in their current application to non-traditional model systems, some of these technologies are becoming feasible in diverse species. A comparative approach, exploring a diversity of organisms with unique behavioral strategies, will paint a clearer picture of the evolution and function of neural substrates involved in social decision-making (Dulac et al., 2014; Pollen and Hofmann, 2008; Roland and O'Connell, 2015; Yartsev, 2017).

In this Review, we have discussed a series of neurochemicals – including neurotransmitters, nonapeptides and other neurohormones – and genes associated with female mate choice across vertebrate taxa. Fig. 1 summarizes which taxa these have been studied in, and where in the brain they have been explored (when this information is available). From pair-bonding mammals, we know that activation of the ventral tegmental area and subsequent dopamine activity in the nucleus accumbens is critical for the formation of female mate preferences, and involves the complementary synthesis and release of both AVP and OXT. These neurochemical pathways have also been identified as substrates of female mate choice and/or preference in birds (Day et al., 2019) and amphibians (Boyd, 2019). Additionally, in the swordtail X. nigrensis, genes related to synaptic plasticity are expressed in core nodes of the SDMN (such as the lateral septum, medial amygdala and preoptic area) during mate selection (Cummings et al., 2007), a finding that has been corroborated in other teleosts using transcriptomic approaches (Bloch et al. 2018). However, the extent to which these processes are conserved across vertebrates remains unclear.

Importantly, dynamic social environments modulate how the brain integrates external information to display appropriate adaptive mate-choice behavior. Factors such as early life experiences, social familiarity, the current social context and reproductive state all influence how and when females make mate-choice decisions. Different species experience distinct social environments and may rely on differing sensory modalities in mate choice. Although diverse species likely differ in how the brain responds to these social and environmental dynamics, we can expect certain brain areas to function in an evolutionarily conserved role as decision-making areas in the integration of information and facilitation of female mate choice. The preoptic area and medial amygdala are two such regions. These have been identified in mammals as important for preference, but not necessary for mate discrimination (DiBenedictis et al., 2012; Martinez and Petrulis, 2013; Sakuma, 2008). Other nodes of the SDMN, though clearly important in a variety of social behaviors, have been largely ignored in the context of mate choice and thus provide promising future areas to explore.

Female mate choice is fundamentally a social behavior. Even though it has long been recognized as a powerful driver of sexual selection (Andersson and Simmons, 2006; Emlen and Oring, 1977), it is astonishing how little is known about the neural and molecular processes by which the female brain recognizes and selects for male traits. This is in stark contrast to the proximate mechanisms underlying sexually selected traits displayed by males, which have received much attention, in part because they often are very conspicuous (Andersson and Simmons, 2006). Even though our current understanding of mate choice mechanisms is woefully inadequate, the evidence we have discussed here suggests that the evolutionarily conserved brain regions and neurochemical pathways that regulate social decision-making across vertebrates also play a critical role in the recognition and selection of suitable mates. Given novel methodological advances that facilitate research across time scales and levels of biological organization, even in non-traditional model systems, we can look forward to exciting new insights into mate choice mechanisms and how they evolved.

We thank Chelsea Weitekamp and Eva Fischer for helpful comments on earlier versions of the manuscript, and members of the Hofmann lab for discussion.