Social status differences regulate the serotonergic system of a cichlid fish, Astatotilapia burtoni

The monoamine neurotransmitter serotonin (5-hydroxytryptamine, 5-HT), acts throughout the brain to modulate social behaviors, including territorial aggression (Lepage et al., 2005; Winberg and Nilsson, 1993) and reproduction-related behaviors (Deemyad et al., 2013; Dominguez and Hull, 2010; Verma et al., 1989). Studies have shown that serotonin inhibits aggression in mammals, reptiles and fish (Dahlbom et al., 2012; Elofsson et al., 2000; Lepage et al., 2005; Munro, 1986; Perreault et al., 2003; Poletto et al., 2011; Winberg et al., 1992). Accordingly, in a social dominance context in which subordinate–dominant relationships are formed, serotonin turnover generally tends to be higher in subordinates, compared with dominant conspecifics (Nelson and Chiavegatto, 2001; Nelson and Trainor, 2007).

In all vertebrate species studied, the main population of serotonin-producing neurons is restricted to the hindbrain raphe nucleus. Such neuroanatomical conservation suggests that functional elements of the 5-HT system are shared across species because of common ancestry. However, how this homology relates to serotonin's role in aggression or basic homeostatic processes is unknown. Despite concerted efforts to characterize serotonin's multiple roles, the identities of relevant receptors and target areas responsible for its capacity to modify behavioral and neuroendocrine responses are still poorly understood. Furthermore, plasticity in the serotonergic system, including changes in 5-HT cell size or number in relation to phenotypic change, remains largely unexplored (Lorenzi and Grober, 2012).

To understand the role of serotonin in the maintenance of dominance hierarchies, we used the African cichlid fish Astatotilapia burtoni (Günther 1894), in which social status and mating opportunities depend largely on male aggression. Dominant males are brightly colored and aggressively defend territories used for shelter and spawning. In contrast, subordinate males do not have territories, are drably colored, school with females and rarely perform dominant behaviors (Fernald, 1977). In addition to body coloration and behavior, neural and physiological features of the brain–pituitary–gonadal (BPG) axis depend on social status. For example, compared with subordinate males, dominant males have larger gonadotropin-releasing hormone (GnRH1) neurons (Davis and Fernald, 1990) that synthesize greater amounts of GnRH1 mRNA, have higher levels of circulating 11-ketotestosterone (11-KT) (Soma et al., 1996) and have larger testes that produce more sperm (Fraley and Fernald, 1982; Kustan et al., 2012).

In mammals, the classification of 5-HT neurons into distinct subpopulations has led to important insights into their functions. Serotonergic neurons in dorsal and median raphe subnuclei can be distinguished based on their neuroanatomical location, cell morphology, projection regions and functional properties (Abrams et al., 2004; Gaspar and Lillesaar, 2012). Importantly, 5-HT cells in these subnuclei differ in electrophysiological properties (Beck et al., 2004; Kirby et al., 2003), selective hormone sensitivity and patterns of excitatory response to social stimuli (Abrams et al., 2004), and these characteristics vary among species. For example, after social defeat in hamsters, 5-HT neurons in the dorsal raphe, but not median raphe, show increased cFos expression, an immediate early gene commonly used to detect neuronal activation (Cooper et al., 2009). A similar neuroanatomical distinction between dorsal and medial 5-HT raphe neurons was described for the three-spined stickleback (Gasterosteus aculeatus) (Ekström and Van Veen, 1984). More recently, the mapping of 5-HT projection areas from the raphe in zebrafish (Lillesaar et al., 2009) provided further evidence in support of this classification of subpopulations. Based on numerous functional differences reported among subpopulations of raphe 5-HT neurons in mammals, we tested whether social status was associated with differences in 5-HT cell size and number in dorsal (Rd) and medial (Rm) subregions of the raphe in A. burtoni.

To understand the actions of serotonin, its distribution and that of its receptors must be known. Therefore, we mapped the mRNA distribution of 5-HT receptors in the A. burtoni brain and compared relative abundance between subordinate and dominant fish. In mammals, seven classes of serotonin receptors (5-HTRs) have been described; of these, six are G-protein coupled and one has a 5-HT-gated ion channel (Roth, 2006). Serotonin receptor subtypes belonging to classes 1 and 2 are among the best characterized and several lines of evidence suggest they are important for the behavioral effects of serotonin. Class 1 5-HTRs are autoreceptors expressed in 5-HT neurons to regulate 5-HT-mediated cell firing and 5-HT release, as well as in non-5-HT neurons throughout the entire brain (Roth, 2006).

Serotonin transmission is also important for the proper display of sexual behaviors in rodents, particularly in the preoptic area (POA), anterior hypothalamus (AH) and ventromedial hypothalamus (VMH). For example, 5-HT injected directly into the POA of rats inhibits ejaculation in males (Verma et al., 1989). Similarly, injection of a 5-HTR_1A agonist into either the medial preoptic area (mPOA) or the VMH inhibits lordosis in females (Verma et al., 1989). In addition, other studies show that systemic activation of 5-HTR₂ receptors with subtype-specific agonists decreases the frequency of courtship vocalizations in male electric fish Apteronotus leptorhynchus (Smith and Combs, 2008).

Here, we microdissected targeted brain areas and measured 5-HT with high performance liquid chromatography (HPLC) to discover whether social status differences in serotonin turnover are restricted to the raphe, or elsewhere in the POA and ventral tuberal nucleus (VTn), a putative partial homolog of the mammalian AH (Goodson, 2005). Previously, Winberg et al. found that in midbrain and hindbrain, subordinate A. burtoni males had a higher ratio of the serotonin catabolite 5-hydroxyindoleacetic acid (5-HIAA) to 5-HT, compared with dominant males, which was indicative of high 5-HT turnover and activity in subordinates (Winberg et al., 1997). The 5-HIAA/5-HT ratio is commonly used as a proxy for serotonin turnover because an increased 5-HIAA level indicates that a large amount of 5-HT was recently released. Thus, a relatively high 5-HIAA/5-HT ratio can be interpreted as high 5-HT demand in the brain and this information may go undetected if 5-HT or 5-HIAA levels are measured alone.

We examined the POA because of its essential role in the control of the reproductive axis and evidence from other species that implicates serotonergic signaling in the control of sexual behavior (Hull et al., 1993; Verma et al., 1989). We analyzed the VTn because of its putative partial homology to the mammalian AH (Goodson, 2005), which has been implicated in controlling aggression. In Syrian golden hamsters, arginine vasopressin (AVP) neurons in the AH are proposed to mediate aggression because AVP microinjections increase offensive aggression. These AVP effects, however, can be blocked by pretreatment with fluoxetine, which increases 5-HT levels (Ferris et al., 1997). Furthermore, 5-HT varicosities throughout the AVP population as well as putative synaptic contacts between 5-HT terminals and AVP neurons were described through immunohistochemical studies in this species (Delville et al., 2000; Ferris et al., 1997). Through studies based on electrical stimulation (Kruk et al., 1984), lesioning (Kruk, 1991) and mapping of immediate early gene expression following aggressive encounters (Davis and Marler, 2004; Hasen and Gammie, 2005; Kollack-Walker and Newman, 1995) the AH is claimed to play a fundamental role in rodent aggression [but see Lin et al. (Lin et al., 2011) for the exception of mice]. We predicted that if serotonin is important for inhibiting behaviors associated with dominance, then serotonin turnover in the POA and VTn would be greater in subordinate A. burtoni males than in dominant males.

We identified three populations of 5-HT-ir cells in the A. burtoni male brain: dorsal and ventral parts of the periventricular pretectal nucleus (PPd, PPv), nucleus of the paraventricular organ (PVO) and raphe (Fig. 1) (for nomenclature details, see Discussion; supplementary material Table S1). Negative controls had no staining [primary antibody omitted or pre-adsorbed with 5-HT bovine serum albumin (BSA) conjugate; supplementary material Fig. S1]. In the hindbrain, the largest population of serotonin perikarya was in the raphe (Fig. 1C). In the most anterior part of the raphe population, 5-HT-ir neurons in the dorsal raphe surrounded the medial longitudinal fasciculi (MLF) tracts beneath the fourth ventricle; these cells were elongated and densely overlapped (Fig. 1C). In more posterior sections, as 5-HT-ir cells in the medial raphe increase in abundance, a subset of cells align on both sides of the midline (Fig. 1C). We found that on average males had 187.3±11.5 (mean ± s.e.m.) serotonin cells in the raphe (N=10 fish).

In dominant and subordinate males, the distribution of raphe 5-HT neurons along the antero-posterior (AP) axis exhibited a single peak of highest density tapering to fewer cells (Fig. 2). We classified 5-HT-ir cells as belonging to either Rd or Rm raphe following Ekström and Van Veen (Ekström and Van Veen, 1984). We asked whether 5-HT cell number and/or size vary by social status. Dominant males had higher gonadosomatic index (GSI) values [GSI=(testes mass/body mass)×100] compared with subordinate males [N=5 per group, P=0.008, Mann–Whitney U-test (MWU)]. There were no significant differences between social groups in average cell size in the Rd (P=0.751, MWU), Rm (P=0.530, MWU) or entire raphe (P=0.859, MWU), or in average number of 5-HT-ir cells in the Rd (P=0.841, MWU), Rm (P=0.095, MWU) or entire raphe (P=0.151, MWU).

We compared the distribution of 5-HT cells along the AP axis between social groups in anterior, middle and posterior locations for post hoc tests. We found a significant interaction between social status and region (Fig. 2B) (two-way ANOVA, P=0.016), indicating the effect of status on the number of 5-HT-ir cells varies as a function of position along the AP axis. Pairwise comparisons with Bonferroni correction confirmed that the largest difference between social groups was in the middle raphe region (subordinate versus dominant in middle region, P=0.002), whereas for anterior and posterior raphe regions there were no differences between social groups (anterior, P=0.122; posterior, P=0.330).

To identify whether a particular 5-HT subpopulation could account for the difference in the number of cells between social groups in the middle raphe region, we analyzed Rd and Rm data separately. Only Rd cells in the middle raphe region showed a difference by social status (Fig. 2C) (Bonferroni-corrected subordinate versus dominant in anterior, P=0.156; middle, P=0.015; posterior, P=0.889). In contrast, there were no differences between social groups for Rm cells alone (Fig. 2D) (Bonferroni-corrected subordinate versus dominant in anterior, P=0.976; middle, P=0.338; posterior, P=0.584). Interestingly, when all males were analyzed as a single group, the total number of 5-HT-ir cells in the middle raphe region was positively correlated with GSI (Pearson r=0.72, P=0.018) (Fig. 3A) but not with standard length (SL) or body mass (M_b) (SL, r=0.39, P=0.268; M_b, Pearson r=0.48, P=0.157) (Fig. 3B,C). There were no significant correlations between GSI and Rd or Rm cell numbers (Rd, Pearson r=0.53, P=0.115; Rm, Pearson r=0.28, P=0.434).

The 5-HIAA to 5-HT ratio (i.e. 5-HT turnover) in microdissected brain regions was higher in subordinate males in the POA (P=0.041, MWU) and raphe (P=0.020, MWU), but not the VTn (P=0.548, MWU) (Fig. 4). Levels of 5-HT and 5-HIAA were comparable to those in other studies for fish of comparable size to A. burtoni (Clotfelter et al., 2010; Larson et al., 2003) and did not differ between social groups in any of the brain regions (POA 5-HT P=0.356, 5-HIAA P=0.6; VTn 5-HT P=0.436, 5-HIAA P=0.106; raphe 5-HT P>0.99, 5-HIAA P=0.18; all MWU) (supplementary material Fig. S2).

GSIs and 11-KT levels were higher in dominants compared with subordinates (GSI P=0.001, t-test; 11-KT P=0.0008, MWU). We chose to measure 11-KT because in teleosts it is the more behaviorally important of the androgens (Hirschenhauser et al., 2004; Kime, 1993). When all male data were pooled, GSI was negatively correlated with 5-HIAA/5-HT levels in the POA (Spearman r=−0.52, P=0.046) and raphe (Spearman r=−0.48, P=0.044), but not the VTn (Spearman r=0.07, P=0.8) (Fig. 5), and GSI did not correlate with either 5-HT or 5-HIAA in any brain nuclei (all P>0.05) (supplementary material Table S2). Interestingly however, when groups were analyzed separately, the relationship between GSI and raphe 5-HIAA/5-HT was strengthened and became even more significant for subordinates (Pearson r=−0.83, P=0.005), while for the dominant group this relationship was lost altogether (Pearson r=0.25, P=0.52) (supplementary material Table S3). There were no significant correlations between 11-KT levels and 5-HT, 5-HIAA or 5-HIAA/5-HT in any brain nuclei (supplementary material Table S2).

For animals used to measure 5-HTR mRNA, GSI and 11-KT levels were higher in the dominant group, but only the 11-KT difference was statistically significant (GSI P=0.522; 11-KT P=0.014, all MWU).

The 5-HTR_1A receptor, classified as an autoreceptor, is typically present on the surface of 5-HT cell bodies, axons, dendrites and terminals in projection areas (Norton et al., 2008). In A. burtoni, brain region explained 87% of the variation in expression (two-way ANOVA, P<0.0001) while neither status nor the brain region × status interaction was significant. In the telencephalon, subordinate males had higher levels of this mRNA compared with dominants (P=0.041, MWU) (Fig. 6). Interestingly, there was a significant negative relationship between 11-KT and expression levels of this receptor in the telencephalon (Pearson, r=−0.704, P=0.011), but not in the midbrain or hindbrain. In both subordinate and dominant groups, mRNA levels of this receptor were highest in the hindbrain compared with the midbrain and telencephalon (P<0.001 for all pairwise comparisons between brain regions within both social groups). To control for the effects of body length and M_b, which were also correlated with htr1a expression in the telencephalon (SL, Pearson r=−0.671, P=0.017; M_b, Pearson r=−0.715, P=0.009), we repeated correlation tests using expression values corrected for SL (htr1a_corr=relative expression/SL). After applying this correction, the strength and significance of correlations between relative htr1a mRNA levels and SL and M_b were lost (SL, Pearson r=−0.332, P=0.164; M_b, Pearson r=0.141, P=0.553). In contrast, the correlation between htr1a and 11-KT remained (Pearson r=−0.746, P=0.005), demonstrating that this correlation did not depend on differences in body length or mass.

There was a significant status × region interaction effect (two-way ANOVA, P=0.006), indicating that the influence of social status on the expression of the htr2a receptor varied among brain regions with main effects for region (P=0.007) and status (P=0.01). In both the telencephalon and midbrain, subordinate males had higher expression than dominants, and for the telencephalon this difference was significant (P=0.009, MWU) (Fig. 6). In subordinates, highest expression among regions was found in the telencephalon but this difference was only significant between the telencephalon and hindbrain (P<0.001). In contrast, in dominant males, no statistically significant differences in expression levels across regions were found.

Brain region explained 54% (two-way ANOVA, P<0.0001), 23% (two-way ANOVA, P=0.049) and 68% (two-way ANOVA, P<0.0001) of the expression variance for receptors htr1d, htr3b and htr6, respectively, but status had no significant effect. htr1d expression showed a trend for being higher in the telencephalon (Fig. 6); and htr6 was highest in the telencephalon (versus midbrain, both social groups P<0.001; versus hindbrain, subordinate P<0.001 and dominant P<0.01).

The behavior of fish used for HPLC assays was as expected for stable subordinate and dominant males. All dominant males (N=11) displayed only dominant behaviors, and for nine out of the 10 subordinate males tested, subordinate behavior (i.e. fleeing) comprised an average of 97% of total behaviors performed (one subordinate male remained motionless for 30 min before the experiment was ended). We tested for correlations between 5-HT, 5-HIAA and 5-HIAA/5-HT levels and each behavior. There were no significant correlations with 5-HT or 5-HIAA. A summary of results for 5-HIAA/5-HT correlations is listed in supplementary material Table S4. Significant correlations were only present in the dominant group, as follows: 5-HIAA/5-HT from POA was negatively correlated with number of lateral displays (Spearman r=−0.857, P=0.024, N=7) and positively correlated with courtship leads (Spearman r=0.847, P=0.025, N=7); 5-HIAA/5-HT from VTn was negatively correlated with number of quivers (Spearman r=−0.85, P=0.004, N=9).

In the A. burtoni male brain we identified three populations of serotonin cells: in PPd and PPv, the PVO and raphe. After 5 weeks in a stable social status, subordinate males had higher serotonin turnover in the raphe, but fewer 5-HT-ir cells, and higher serotonin turnover in the POA, but not in the VTn as compared with dominant males. We also found that in the telencephalon, but not the midbrain or hindbrain, subordinates had higher expression of mRNAs for 5-HTR subtypes 1A and 2A.

The patterns of 5-HT-ir neuron populations we observed agreed with prior reports in other fishes with the exception that the PVO population did not extend posterior and laterally into the nucleus of the lateral recess (NRL) or ventrally into the nucleus of the posterior recess (NRP) as described in some species (reviewed in Lillesaar, 2011). We propose this difference is due, in part, to nomenclature variations for these nuclei between species and even within a single species (Lillesaar, 2011).

A summary of nomenclature variations is listed in supplementary material Table S1 and briefly explained here. First, the cells that we refer to as belonging to the PVO have been localized to the anterior paraventricular nucleus (PVOa) in zebrafish (Kaslin and Panula, 2001), which in turn corresponds to the PVO according to Rink and Wullimann (Rink and Wullimann, 2001). Most recently, Lillesaar (Lillesaar, 2011) proposed a revision of this 5-HT-ir population to designate its location as part of the posterior tuberculum (PT). Second, the more caudal populations, which are described as contained within the NRL and NRP (e.g. Lorenzi and Grober, 2012), can also be found under the alternative names dorsal (Hd) and caudal (Hc) zones of the periventricular hypothalamus, respectively. It is important to note that a consistent methodology for delineating Hd and Hc nuclei borders is lacking, so comparisons are not in perfect correspondence. Furthermore, the part of the NRL that has been reported to harbor 5-HT-ir cells has also been called the intermediate paraventricular organ (PVOi) (Elipot et al., 2013; Kaslin and Panula, 2001). Similarly, the NRP has been called the posterior paraventricular organ (PVOp) (Elipot et al., 2013; Kaslin and Panula, 2001). Third, some authors refer to the PVO as the collective of the posterior periventricular nucleus (NPPv), NRL and NRP [(e.g. Lorenzi and Grober, 2012), bluebanded goby; (Fryer et al., 1985), goldfish; (Batten et al., 1993), sea bass]. In light of these nomenclature issues, in supplementary material Fig. S3 we show representative photomicrographs of coronal sections from two males to illustrate that the location of immunoreactivity, to the best of our knowledge, is within the PVO and does not extend to the NRL or NRP.

In earlier work, Winberg et al. (Winberg et al., 1997) showed that after 3 days of being housed in subordinate/dominant pairs, subordinate males had higher hindbrain 5-HIAA/5-HT ratios compared with dominant males. Here, we show this difference persists after 5 weeks of maintaining a stable social status and originates from 5-HT cells in the raphe. We tested whether high 5-HT turnover in subordinates was also associated with 5-HT cells that were relatively larger than those of dominants. First, we reasoned that because increased 5-HT turnover requires a high demand for tryptophan hydroxylase synthesis, cells may increase in size to meet this demand. Second, increased 5-HT turnover could lead to an increase in dendritic branching or other structural changes that can require an overall increase in cell metabolism and, consequently, cell growth. Third, in A. burtoni, neuropeptide-producing cells have been shown to vary by social status (Davis and Fernald, 1990; Francis et al., 1993), and although serotonin cells are not neuropeptide-producing cells per se, we hypothesized social status correlates with cell size differences because, in adult rats, changes in stress hormone levels can produce 5-HT cell area increases in dorsal, but not median, raphe of up to ~80% in only 3 days (Azmitia et al., 1993).

In rodent models, 5-HT cells in dorsal and median raphe nuclei have unique responses in paradigms involving social aggression and defeat. For example, in mice after social defeat, firing of 5-HT cells in the dorsal raphe is inhibited by neighboring GABAergic neurons and this inhibition is a necessary step in the acquisition of social avoidance that develops after repeated bouts of social defeat (Challis et al., 2013). In contrast, in hamsters even within the dorsal raphe 5-HT population there can be regional differences in excitatory responses to social defeat: animals that lost an aggressive encounter had a greater number of 5-HT cells in the rostral part of the dorsal raphe that were positive for c-Fos protein, a marker of neuronal activity, compared with winners and controls (Cooper et al., 2008; Cooper et al., 2009).

We found subordinate males had fewer 5-HT-ir cells, specifically in the dorsal raphe, compared with dominants, but there was no difference in average cell size between social groups. This topographical distinction suggests there may be functional differences between dorsal and medial raphe 5-HT cells in A. burtoni. Our interpretation is that our results exemplify the striking difference in staining intensity between depleted and non-depleted 5-HT cells (e.g. Fickbohm et al., 2001). Because high 5-HT turnover is an indirect readout of a high demand on 5-HT release elsewhere in the brain, subordinates would be expected to have less intense staining than dominants. Accordingly, the staining intensity in 5-HT cells with relatively high 5-HT depletion levels may have been so faint that they were not considered 5-HT positive, because our criteria required even staining throughout the cell and well-defined borders. Therefore, the lower average number of dorsal 5-HT cells reported for subordinates was likely an underestimate, because it resulted from the fact that cells with extreme 5-HT depletion were not counted.

A similar trend in 5-HT cell number differences linked to social status has been reported for the hermaphroditic goby Lythrypnus dalli. In this species, when the dominant male is removed, the alpha female changes sex and eventually replaces the removed male (Lorenzi and Grober, 2012; Lorenzi et al., 2009). By 2 h after the onset of the sex change, transitioning fish showed a trend for greater numbers of 5-HT cells in the raphe, compared with stable and later stage (>1 day) transitioning fish (Lorenzi and Grober, 2012). These results are compatible with the notion that 5-HT depletion is highest in subordinate animals and is replenished in higher social statuses. Discovering the timing of when these cells switch from low to replenished during social status transitions would provide insight as to whether they facilitate, or are a consequence of, social ascent or descent.

We found that subordinates had higher expression of htr1a and htr2a in the telencephalon as well as higher 5-HT turnover in the POA, compared with dominants. These results are interesting because both receptors are implicated in the control of aggression (Clotfelter et al., 2007) and courtship (Smith and Combs, 2008) in fishes, but to our knowledge, this is the first study that shows differential htr1a and htr2a expression by social status in a particular brain region. In addition to its release at the synapse, 5-HT can also be present at extrasynaptic sites as a result of volume diffusion (Bunin and Wightman, 1998; Descarries and Mechawar, 2000). Our qPCR data suggest the expression of these receptors in the olfactory bulb, telencephalon and/or anterior POA may be key for mediating social interactions within dominance hierarchies. We assume it is possible that some telencephalon samples contained anterior POA because GnRH1 mRNA has been detected by qPCR in some telencephalon samples (that were not a part of this study) derived from this dissection protocol (J.L.L., unpublished data). Furthermore, these receptors may be involved in the maintenance of high 5-HT turnover in the POA of subordinates, which could have direct consequences on the control of sexual behavior and reproductive physiology, as seen in rodents (e.g. Hull et al., 2004).

In rodents, serotonin's main effects in the POA are to inhibit different aspects of sex-related behaviors (Uphouse, 2010). For example, measurement of 5-HT levels in microdialysates from the POA of behaving male rats presented with an opportunity to mate with females showed that successful copulations were associated with low 5-HT levels. Further, failed attempts to copulate occurred in animals that had relatively higher 5-HT amounts, suggesting 5-HT in the POA impairs copulatory behavior. In fishes, the POA also controls sexual behaviors and in the green sunfish, Lepomis cyanellus, direct electrical stimulation of the POA causes sperm release in males (Demski et al., 1975), suggesting a conserved role for the POA in ejaculation control (Hull et al., 2004). Future studies are needed, however, to test whether in fishes activation of 5-HTRs has similar effects on sexual behaviors to those reported for rodents. Next, we will discuss specific cases for 5-HT effects on GnRH1 and arginine vasotocin (AVT) populations, neuropeptides that are also known to vary with social status and are known targets for direct modulation by 5-HT (Wada et al., 2006; Ferris and Delville, 1994).

Manipulation of AVT levels in fish can affect courtship behaviors and induce social avoidance (Salek et al., 2002; Silva et al., 2013; Thompson and Walton, 2004). In the golden hamster, AVP facilitated offensive aggression in males, which can be blocked by pretreatment with a selective serotonin re-uptake inhibitor (SSRI) (Delville et al., 1996; Ferris and Delville, 1994) or 5-HTR_1A agonist (Ferris et al., 1999). A direct interaction between serotonergic and vasopressin neurons in the anterior hypothalamus of hamsters was described (Ferris et al., 1997) and the possibility of a similar case in the weakly electric fish Gymnotus omarorum was recently speculated (Zubizarreta et al., 2012). Using in situ hybridization, Greenwood et al. demonstrated that in subordinate A. burtoni, AVT expression is higher in parvocellular neurons compared with dominant males (Greenwood et al., 2008). Accordingly, 5-HT may be involved in regulating this AVT difference. Future studies however, will require information on AVT projection mapping and functional evidence to substantiate a causal relationship between AVT release in projection areas and specific behaviors, as well as determining whether and how 5-HT might control AVT neurons. Interestingly, however, in situ hybridization data for the htr1a homolog in zebrafish provides some support for the feasibility of such contacts because its expression is restricted to the anterior, but not posterior, parvocellular subnucleus of the POA (Norton et al., 2008). Accordingly, if this pattern were conserved in A. burtoni, we would predict serotonin has defined target regions within the anterior parvocellular subnucleus, which might include, in addition to AVT, GnRH1 and vasoactive intestinal peptide cells.

The production of GnRH1 in the anterior POA and its delivery to the pituitary is essential for reproduction in all vertebrates (Fernald, 2012). In A. burtoni, GnRH1 neurons increase in size and in GnRH1 production during social ascent (Francis et al., 1993; White et al., 2002), but exactly what initiates these changes is not understood. Activation of serotonin receptors in immortalized GnRH cells in tissue culture affects many cellular properties. Specifically, administration of a 5-HTR_1A agonist has a strong inhibitory effect on firing rate and release of GnRH1, and a 5-HTR_2C agonist produces a delay in GnRH1 release but does not influence firing rate (Wada et al., 2006). Phylogenetic analysis of vertebrate 5-HTR sequences shows that in fishes only htr2a and htr2b are present, and it appears that htr2c was either lost or the gene duplication from which it originated never occurred in fishes (Kroeze and Roth, 2006). Ultimately, if GnRH1 neurons are a source of increased expression of htr1a and htr2a receptors in subordinates compared with dominants, this would suggest that in subordinates, serotonin has an inhibitory effect on the control of GnRH1 neuron firing, as well as its synthesis and release. Further studies are still needed to determine whether these receptors are in fact differentially expressed in GnRH1 neurons between social groups. Altogether, these studies highlight the potential for interactions between AVT, GnRH1 and 5-HT systems, and future studies shall test hypotheses relating to the identity of the specific 5-HTRs and POA cell types that contribute to the control of aggressive behavior.

Here, we have expanded on previous work on the role of serotonin in the dominance hierarchy of A. burtoni in several ways. First, we characterized 5-HT-ir neurons and receptor distributions in A. burtoni and compared somata size and number of 5-HT raphe neurons between social statuses. In the telencephalon, we identified differential mRNA expression of htr1a and htr2a receptors, with highest expression in subordinates compared with dominants. Second, using a targeted approach aimed at discrete nuclei, we found that higher 5-HIAA/5-HT ratios in subordinates are localized to the main site of serotonin production (the raphe) as well as the POA, a nucleus essential for the control of reproductive physiology and sexual behaviors. In contrast, the VTn, a proposed partial homolog to the mammalian anterior hypothalamus, implicated in the control of aggression in some species, did not show any social status differences in 5-HT turnover. These results suggest that differences in serotonin signaling between animals of different social status is mediated, at least in part, by 5-HTR_1A and 5-HTR_2A receptors in the telencephalon, and that serotonergic transmission in the POA, but not the VTn, contributes to facilitating, and perhaps even maintaining, the physiological and behavioral changes that are characteristic of social descent.

We used sexually mature, adult male fish, descendants of wild-caught A. burtoni from Lake Tanganyika, Africa, maintained in aquaria under conditions that mimic their native habitat (pH 8.0, 24–26°C, 12 h light:12 h dark cycle). Aquaria contained gravel substrate and terracotta pots to facilitate territory establishment. All protocols were approved by the Stanford Animal Care and Use Committee.

To ensure that we sampled animals that had maintained a stable social status, body coloration and behavior of focal animals was monitored 3 times a week by observing each tank for 5 min over a 5 week period prior to sample collection; during this time we did not observe any biting and there were no visible signs of bodily harm. Dominant behaviors included chase, lateral display, border fight, quiver, lead, shelter entry and bite. Fleeing was scored as a subordinate behavior (Fernald, 1977). Animals were killed by withdrawal of blood from the caudal vein followed immediately by rapid cervical decapitation. Blood was centrifuged at 8000 r.p.m. for 10 min; plasma supernatant was collected, frozen and stored at −80°C. SL, M_b and testes mass were recorded and GSI [=(testes mass/body mass)×100] was calculated. Brains were removed and stored for relevant protocols as described below.

Stable dominant (SL=5.88±0.12 cm, M_b=5.43±0.12 g, GSI=0.71±0.03; N=5) and subordinate males (SL=5.56±0.26 cm, M_b=4.19±0.60 g, GSI=0.42±0.04; N=5) were killed between 09:00 h and 10:00 h before regular feeding time. Whole brains were removed, fixed in 4% paraformaldehyde overnight at 4°C and cryoprotected in 30% sucrose solution at 4°C the following day. Cryoprotected brains were sectioned coronally at 20 μm in a single series (Micron HM550) onto charged slides (Superfrost Plus, VWR, Radnor, PA, USA).

To label cells expressing 5-HT, sections were rehydrated in phosphate-buffered saline (PBS) and incubated in blocking solution [0.2% bovine serum albumin (BSA) and 0.3% Triton X-100 in PBS] supplemented with 10% normal goat serum (NGS) for 1 h at room temperature. Incubation with the primary antibody (1:40,000, polyclonal 5-HT rabbit antibody, cat. no. 20080, lot no. 541317, Immunostar) in blocking solution was at 4°C overnight followed by a 1 h incubation with secondary antibody (1:400, goat biotinylated anti-rabbit antibody, cat. no. BA-1000, lot no. W0524, Vector) in blocking solution containing 10% NGS. For stain visualization, sections were treated with hydrogen peroxide for 15 min, washed in PBS and incubated for 1.5 h with ABC reagent (Vectastain ABC kit, cat. no. PK4000, Vector) followed by immunoreactive staining with the ImmPACT DAB (3′-3′-diaminobenzidine) substrate kit (cat. no. SK4105, Vector) according to the manufacturer's instructions for 4–12 min. To aid with neuroanatomical landmarks, sections were immediately counterstained with Nuclear Fast Red (cat. no. H3403, lot no. L0613, Vector), dehydrated in a series of increasing alcohol concentrations, cleared in xylene and coverslipped with Cytoseal 60 (cat. no. 23-244257, Thermo Scientific, USA). A separate set of male brains sectioned at 20 μm in two alternative series were used as pre-adsorption and primary antibody omission controls. For pre-adsorption, lyophilized 5-HT BSA conjugate (cat. no. 20081, lot no. 841001, Immunostar) was reconstituted with 500 μl of blocking solution supplemented with 10% NGS. Then, to 1 ml of diluted primary antibody (1:40,000), 200 μl of reconstituted 5-HT BSA conjugate was added for a final conjugate concentration of 20 μg ml−1 and stored at 4°C for 20 h prior to treating tissue. For the second series the primary antibody was omitted. In both groups, the secondary antibody staining was carried out with the standard protocol and used as a positive control.

Images were captured and analyzed with Spot Imaging Software (Spot Diagnostic Instruments Inc., Sterling Heights, MI, USA). We defined the region of interest (ROI) for each subject to be contained between the first and last sections that had at least one 5-HT-ir cell. A neuron was counted as 5-HT-ir if staining was even throughout the entire cell and its borders could be reliably identified. All positively stained neurons were counted on every other section within the ROI. Positive cell counts for each section were corrected using the Abercrombie (Abercrombie, 1946) formula: N_i=n_i(t/t+d), where N_i is the number of labelled cells in section i, n_i is the raw count in section i, t is the section thickness (20 μm) and d is the mean diameter of the cell nuclei measured in sections orthogonal (i.e. sagittal) to the plane of sections used for quantification (i.e. coronal). We compared the average diameter of 5-HT-ir cell nuclei between two males that had body sizes representative of the smallest and largest males in our sample groups, based on 20 cells (randomly chosen) per animal. Because there was no difference between these two fish (P=0.86, t-test), we used the grand average for all corrections. Cells were assigned as belonging to either dorsal or medial raphe, following Ekström and Van Veen (Ekström and Van Veen, 1984).

For each subject, ROI images were taken at ×20 magnification and zoomed to ×40 in Spot. The perimeter of cells was traced manually using a Wacom tablet (Wacom Technology Corp., Vancouver, WA, USA). Soma area estimates were then calculated for 20 cells chosen at random within the region of highest 5-HT-ir cell density for each animal. To control for the unequal number of sections counted across animals, we used the average number of cells per section and mean soma area for each animal to calculate group means. All counting and cell size measurements were performed by a single experimenter blinded to the social status of each subject. Boundaries between anterior, middle and posterior regions (i.e. matched sections 1–6, 7–12 and 13–20, respectively) were based on the distribution of pooled data across all sections. Boundaries were marked between consecutive sections that had the largest difference in average cell number.

To measure baseline serotonin activity in subordinate and dominant males, we first established seven tanks with stable social conditions as follows: focal males were placed either into community aquaria containing smaller males so they would ascend in social status and become dominant (three tanks) or in community aquaria containing larger males so they would become subordinate (four tanks). Each tank had four terracotta pots as territories and housed nine males and five females. Animals spent 5 weeks in their respective community tanks before being moved individually to a new tank, where they spent 2 days before being killed. We chose 5 weeks to ensure full suppression of behavior and the reproductive axis in subordinates (see Francis et al., 1993; Maruska and Fernald, 2010b; White et al., 2002).

In the new tank, a central compartment with a single pot shelter was flanked on each side by transparent barriers containing smaller males and females to allow visual, but not physical, interaction (see Burmeister et al., 2005; Maruska and Fernald, 2010b). We confirmed social status of focal males in this new tank with 30 min recordings starting at light onset, on the day before and day on which fish were killed. Focal males were killed 30 min after light onset. This protocol ensured that each animal was killed at the same time of day under similar housing conditions. In total, 10 subordinate (SL=6.22±0.06 cm, M_b=5.9±0.2 g, GSI=0.44±0.05) and 11 dominant (SL=6.45±0.08 cm, M_b=6.48±0.15 g, GSI=0.68±0.04) were collected.

An observer blind to the experimental conditions scored videos corresponding to the day of killing with a custom-written software package in MATLAB (Wu et al., 2009). The observer was trained to identify behaviors with videos that were not a part of this study until intra-observer reliability across three consecutive scoring sessions was >90%. We tested correlations between 5-HT, 5-HIAA and 5-HIAA/5-HT levels and each behavior. One subordinate male, which was included in analyses for monoamine data, was excluded from correlational analyses on behavior because it remained motionless throughout the recording.

Brains were removed, immediately frozen in mounting medium on dry ice and kept at −80°C until sectioning (coronal, 300 μm). Slides were placed on a frozen stage (BFS-30MP, Physitemp, Clifton, NJ, USA) and visualized under a dissecting microscope. Brain nuclei were microdissected by a single experimenter over a 2 day period with a modified 23 gauge needle (inner diameter ~390 μm) as described elsewhere (Maruska et al., 2013) to isolate the VTn, POA and raphe nucleus. Microdissected samples were placed into 200 μl of perchloric acid on dry ice and centrifuged at 13,200 rpm at 4°C for 10 min. The supernatant (containing monoamines) was removed and stored at −80°C for HPLC analysis and the remaining pellet was dissolved in 100 μl 0.2 mol l⁻¹ sodium hydroxide for protein concentration assays. Protein concentration was quantified using a Micro-Volume UV-Vis Spectrophotometer (Nanodrop 2000c, Thermo Scientific, USA).

For each sample, 5-HT and 5-HIAA were measured in duplicate using HPLC (SRI International, Menlo Park, CA, USA). Samples (20 μl) were measured (Model 5014B Coulometric Analytical Cell) with electrochemical detection (ESA Model 582 Solvent Delivery System, ESA Model 542 Autosampler, ESA CoulChem III) under the following conditions: ESA MDTM Mobile Phase, flow rate at 0.6 ml min⁻¹, Thermal Organizer set to 35°C. Data were normalized to the amount of protein in each sample and expressed as pg per μg of protein.

Stable dominant (N=7) and subordinate males (N=10) were decapitated, and skull and tissue were carefully removed so that approximately three-quarters of the top brain surface was visible. The brain was then divided into three parts, hereinafter referred to as the telencephalon, midbrain and hindbrain, which were immediately frozen and kept at −80°C until RNA extraction. Using small scissors, first the telencephalon (with olfactory bulbs attached) was removed. The remaining brain was then halved coronally with a razor blade aimed slightly posterior to the middle of the optic tectum (Fig. 6A). Performing this cut while the brain was partially encased minimized variability in dissections between samples. We divided the brain this way to restrict all raphe 5-HT neurons to hindbrain samples, and PPv/d and PVO subpopulations to midbrain samples.

RNA was isolated from homogenized brain tissue (RNeasy Micro Plus Kit, Qiagen, Valencia, CA, USA). RNA concentrations were measured with a Nanodrop 2000c spectrophotometer. For each sample, 1 μg of RNA was used for cDNA synthesis using the iScript cDNA synthesis kit (Bio-Rad Laboratories, Hercules, CA, USA). cDNA samples were diluted 1:20 before qPCR assay.

We measured mRNA levels of five serotonin receptors (htr1a, htr1d, htr2a, htr3b and htr6) and reference genes gapdh and rpl32. Primer pairs were commercially synthesized (Integrated DNA Technologies, San Diego, CA, USA) and amplicons verified by sequencing (Sequetech, Mountain View, CA, USA); accession numbers and primer details are listed in supplementary material Tables S5 and S6. To control for inter-plate effects, each plate contained subordinate and dominant samples as well as a positive control template to verify proper reaction preparation.

PCR Miner software (Zhao and Fernald, 2005) was used to calculate reaction efficiencies (E_gene) and cycle thresholds (CT_gene) from the fluorescence readings of individual wells during the reaction. Gene efficiencies averaged across 12 plates were (means ± s.e.m.): gapdh 0.93±0.01, rpl32 0.91±0.02, htr1a 0.95±0.01, htr1d 0.98±0.01, htr2a 0.99±0.01, htr3b 0.97±0.02 and htr6 0.97±0.01. Each primer pair produced a single melting curve peak in the presence of cDNA template, and showed no amplification when water was used as a template in the reaction mix, or when reverse transcriptase was omitted from the cDNA synthesis reaction. The relative amount (R_o) of mRNA in each sample was calculated with the formula R_o=1/(1+E)_CT. R_o values were then normalized by the geometric mean of the Ro of reference genes as: R_o for the gene of interest over the geometric mean of the Ro of reference genes. For each brain region, we compared subordinate and dominant reference gene R_o values and found no differences (P>0.05, MWU), which confirmed these genes were suitable reference genes.

Comparisons between dominant and subordinate group means were performed using two-tailed t-tests if variance and distribution assumptions for parametric tests were satisfied. Tests for unequal variances between groups were performed with an F-test and distributions were assessed with the D'Agostino–Pearson normality test (omnibus K2). When necessary, data transformations were performed to achieve a normal distribution to satisfy parametric test assumptions. If the normality assumption was not met after transformation, or if the number of data points in either group was less than seven, comparisons were performed using non-parametric MWU. To examine relationships between data sets, Pearson correlations were performed for those with a normal distribution, and Spearman tests for non-normal data. For cell count and qPCR data, two-way ANOVA were used to test for main effects and interactions between status and AP region, and status and brain region, respectively, followed by Bonferroni tests for multiple comparisons. In all tests, P<0.05 was considered significant.

The 5-HT and 5-HIAA data from each group were analyzed separately for outliers in each brain nuclei: values that were beyond 2 s.d. from the group mean were excluded from further analyses. From a total of 113 data points, six values originating from one subordinate and two dominant fish, were excluded. For group comparisons, because we were interested in determining differences between stable statuses, we excluded fish with GSI values below or above 0.5 from dominant and subordinate groups, respectively. Because of small samples sizes for HPLC data and the non-independence of data across brain regions, the P-values from t-tests or MWU tests for 5-HT, 5-HIAA and 5-HIAA/5-HT were not adjusted for multiple comparisons. For correlational tests between 5-HIAA/5-HT and GSI, the two social groups were combined and GSI-based exclusions were not applied.

Plasma 11-KT levels were measured as described and validated for this species elsewhere (Maruska and Fernald, 2010a) using the Enzyme ImmunoAssay (EIA) kit (Cayman Chemical, Ann Arbor, MI, USA). We followed the same protocol except we used a 4 μl sample of plasma reconstituted in a 1:32 dilution. Steroid concentrations were determined based on the standard curve (y-intercept=0.917, slope=−0.841, R²=0.997). Samples were assayed in duplicate on a single plate and the mean coefficient of variance for replicate concentration values was 10%. Altogether, 11-KT assays were performed for all samples used for HPLC (subordinate, N=10; dominant, N=11); all dominants (N=7) and a representative subset of subordinates (N=7, out of 10) used for qPCR.

We thank Olivia Chords for behavior scoring; Annie Smart for technical assistance; Dr Jacqueline Vazquez-DeRose at SRI for managing and processing HPLC samples; and Caroline Hu and two anonymous reviewers for comments on the manuscript.