GIANI – open-source software for automated analysis of 3D microscopy images

The ability to routinely acquire multi-dimensional datasets with modern microscopy techniques is transforming, among other fields, cell biology, developmental biology and cancer research. There has long been an acceptance that two-dimensional (2D) cell cultures might not accurately recreate behaviours found in complex three-dimensional (3D) in vivo environments (Duval et al., 2017). Commonly used 3D culture formats include, but are not limited to, populations of single cells in organotypic matrices, spheroid models, tissue sections or whole embryos and organisms.

However, the development of software for the quantitative analysis of such data has not kept pace with imaging advances, and there is now a pressing need for automated solutions (Meijering et al., 2016). Manual annotation of such data is not feasible in a reasonable time frame. Commercial packages, such as Imaris (Bitplane) and Vison4D (Arivis) provide excellent visualisation functionality and are also equipped with analysis tools. However, licences for such software are expensive and they also rely on proprietary file formats. Furthermore, the closed-source nature of such software prevents detailed interrogation of specific calculations and processes.

There are numerous excellent, freely available bioimage analysis tools in the open source domain. However, their support for 3D analysis is often limited – for example, several CellProfiler (Lamprecht et al., 2007) modules are not yet compatible with 3D images. In cases where 3D datasets are supported, analyses can be challenging to execute for novice users, potentially requiring a complex combination of commands from multiple plugins [e.g. FIJI (Schindelin et al., 2012) and Icy (de Chaumont et al., 2012)]. This can lead to undesirable compromises being made, such as 2D slices from a 3D volume being analysed individually, blinding the analysis to information in adjacent slices. Alternatively, 3D data might be compressed into 2D via projection, which, consequently, artificially reduces distances between objects and can lead to spurious results. A number of very useful open-source MATLAB (MathWorks, Cambridge, UK)-based tools have also been implemented, most notably MINS, which has been effectively used to analyse embryo datasets (Lou et al., 2014) and LOBSTER (Tosi et al., 2020). However, these require the purchase of a MATLAB licence.

We have therefore developed General Image Analysis of Nuclei-based Images (GIANI; https://djpbarry.github.io/Giani), a generally applicable open source tool, implemented as a plugin for the widely-used image analysis platform FIJI. With an emphasis on detection and segmentation of cells in 3D microscopy images, GIANI has been implemented specifically with batch-processing in mind. Although an understanding of fundamental concepts of bioimage analysis is beneficial, GIANI's user interface has been implemented in a wizard format to facilitate use by non-specialists (Fig. 1) and is fully documented (https://github.com/djpbarry/Giani/wiki). Analysis protocols may be reproduced by loading a single parameter file.

The utility of GIANI is illustrated here using three examples. In the first, we generated a series of simulated datasets to evaluate the accuracy of segmentations produced by GIANI using known ‘ground truths’. In the second proof-of-concept, we use a series of mouse preimplantation embryo datasets, and demonstrate the ability of GIANI to detect variations in morphology and protein expression in different experimental conditions. Finally, to show that GIANI can also be used on much larger datasets, we present segmentations of nuclei from light-sheet microscopy images of the flour beetle embryo. It should be noted that GIANI can be used to analyse 3D images from a range of cellular and developmental contexts, and we anticipate that it will be a useful tool to automate quantification of a wide variety of complex imaging data.

GIANI was written using Java 8 as a plug-in for FIJI (Schindelin et al., 2012), making extensive use of the underlying FIJI and ImageJ (Schneider et al., 2012) libraries. A number of other open-source projects were leveraged. Reading of image data is facilitated by interfacing to Bio-Formats (Linkert et al., 2010), making GIANI compatible with a wide range of file formats (https://docs.openmicroscopy.org/bio-formats/6.7.0/supported-formats.html). Detection of nuclear blobs makes use of either TrackMate's spot detector (Tinevez et al., 2017) or FeatureJ (https://imagescience.org/meijering/software/featurej). Segmentation of cells and nuclei takes advantage of the marker-controlled watershed functionality in MorphoLibJ (Legland et al., 2016) and 3D Image Suite (Ollion et al., 2013). The browsing of results is based upon the 3D region of interest (ROI) manager from 3D Image Suite. Complete source code, documentation and test data are available online (https://djpbarry.github.io/Giani).

The design philosophy behind GIANI is inspired by CellProfiler. It is assumed that the user wishes to detect ‘primary objects’ of some sort (typically cell nuclei), followed by the subsequent segmentation of ‘secondary objects’ (typically cells) and then wishes to measure either the morphology of, or the expression of a fluorescent signal within, these objects. The principal difference in the case of GIANI is that, in order to facilitate ease of use, the order of steps in the pipeline is fixed, although some flexibility is present where necessary (Fig. 2).

GIANI can accept as input any images that are readable by Bio-Formats (Linkert et al., 2010), which, at the time of writing, can read almost 160 different formats. It is expected that this data will be 3D in nature, containing any number of different channels. As a minimum requirement, in order to facilitate accurate segmentation, these channels must contain some form of marker for both nuclei and cells. The nuclear marker must be a volume marker, whereas the cell marker can be either membrane localised or volumetric in nature, although the former is preferred for more accurate segmentation. In the example pipeline illustrated in Fig. 2, a nuclear volume marker (DAPI) and cell membrane marker (E-cadherin) are used.

The nuclear channel is first subjected to one of two forms of blob detection to estimate the centre of each nucleus (Fig. 2B). The first option is Laplacian of Gaussian (LoG) blob detection, which involves the application of an LoG filter to the image and then identifying local extrema (Fig. 2B, left panels). This is implemented in GIANI by using the spot detector of TrackMate (Tinevez et al., 2017), which is based on LoG detection. The advantage of this approach is that it is relatively fast and generic. The disadvantage is it assumes the objects to be detected are approximately Gaussian in nature, so will likely perform poorly for irregularly shaped nuclei.

For this reason, a second option is given for nuclear centre detection, based on the eigenvalues Hessian matrix of the image (Fig. 2B, right panels) – this is similar to the approach used for nuclear detection in MINS (Lou et al., 2014). The Hessian is a square matrix of second-order partial derivatives describing the local curvature in the image. For a detailed explanation on how Hessian eigenvalues can be used to detect blobs in images, see Cautun et al. (2012). This is implemented in GIANI by using FeatureJ, a sub-component of the ImageScience plugin (https://imagescience.org/meijering/software/featurej). This approach has the advantage of being capable of detecting ‘blobs’ of any shape, such as irregularly shaped nuclei. The main disadvantage is that it is much more computationally demanding than the simpler LoG detector, so will take longer to run.

Prior to full segmentation of nuclei, Gaussian filtering is employed to smooth any noise that may have been present in the input image (Fig. 2C). To both enhance the nuclei and homogenise the image background, the output of the Gaussian filtering operation is input into a top-hat filter (Fig. 2D), implemented using MorphoLibJ (Legland et al., 2016). Top-hat filtering can be computationally expensive for large datasets, so GIANI includes an option to downsize datasets by a specified factor prior to this step – the dataset is then restored to its original size post-filtering.

Nuclei are fully segmented using a marker-controlled watershed approach (Fig. 2E). GIANI uses MorphoLibJ to achieve this, with the previously detected nuclear centres serving as seeds and the Euclidean Distance Transform of the nuclear centre mask, calculated using the 3D ImageJ Suite (Ollion et al., 2013), serving as the image to be flooded. In addition, the top-hat filtered nuclei channel is thresholded (using one of FIJI's in-built thresholding algorithms, specified by the user) to act as a mask, restricting the overall extent of the segmentation.

It can be seen that as λ tends to infinity, d tends towards the Euclidean distance.

To first validate the performance of GIANI, we generated a series of 189 ‘simulated embryos’ exhibiting different levels of signal-to-noise ratio and cell density (Fig. 3A). Simulated data has the significant advantage of having a known ‘ground truth’ (Fig. 3B). That is, because the images are generated artificially, we know what the ‘correct’ segmentation should look like. This permits us to compare the segmentation results generated with any piece of analysis software with the known ‘true’ values.

Cell count errors generally increased with increased cell density and decreasing signal-to-noise ratio (Fig. 3C,F). However, this error was equal to 1 or less in 82.0% of embryos analysed using GIANI's basic nuclear detection and 98.9% when using advanced detection (Fig. 3I). GIANI detected the correct number of cells in 93.7% of cases when using advanced detection. Nuclear localisation errors also generally increased with increased cell density and decreasing signal-to-noise ratio (Fig. 3D,G). The interquartile range (IQR) of nuclear localisation errors was 654–788 nm using basic nuclear detection and 639–674 nm for advanced nuclear detection (Fig. 3J). The cell centroid localisation error was similar for both basic and advanced nuclear detection, with IQRs of 1.87–2.61 μm and 1.78–2.56 μm for basic and advanced detection, respectively (Fig. 3E,H,K).

To benchmark GIANI against other available software, we analysed the same simulated embryos using Imaris (Fig. S1). We found that while Imaris produced a lower nuclear localisation error (IQR was 423–519 nm; Fig. S1F), it miscounted the number of cells in each embryo (38.1% of embryos) far more often than GIANI did (6.3%; Fig. S1E). We also performed a limited evaluation of CellProfiler on a subset of the same simulated data, which resulted in errors larger than those produced by GIANI (Fig. S2) – the nuclear localisation error was 776–786 nm and cell localisation error was 52.3–55.0 μm.

We subsequently applied GIANI to the analysis of two populations of mouse preimplantation embryos at the morula stage (3 to 3.25 days post fertilisation), one control (n=18) and one treated with a small-molecule inhibitor (Gerri et al., 2020) (n=20). At this stage, two distinct cell populations are discernible – inner and outer cells. In subsequent cell divisions, a blastocyst is formed, whereby the inner cells give rise to an inner cell mass (ICM), and the outer cells become the trophectoderm (TE), a polarized epithelium that will form fetal components of the placenta (Cockburn and Rossant, 2010).

At the morula stage, inner and outer cells display different polarisation states, which influence their cell fate acquisition. The outer cells acquire an apical domain, enriched with the atypical protein kinase C (aPKC) (Plusa et al., 2005). In the polar outer cells, aPKC prevents activation of downstream Hippo pathway kinases, large tumour suppressor kinases 1 and 2 (LATS1/2) (Hirate et al., 2013). Consequently, in outer cells, YAP1 accumulates in the nucleus, where it promotes the expression of GATA3 (Ralston et al., 2010). In contrast, in the apolar inner cells, the activation of the Hippo pathway results in YAP1 cytoplasmic retention, thus maintaining the inner cells in an unspecified state (Cockburn et al., 2013; Frum et al., 2018; Hirate et al., 2013).

We then applied GIANI to detect variations in morphology and protein expression in mouse preimplantation embryos. We began with a morphological analysis of cells in control embryos. No significant difference was found between median nuclear volumes in inner and outer cell populations (Fig. 4C; P=0.861, Wilcoxon rank sum test). A difference in median cell volume between the two populations was also identified, but it was not statistically significant (Fig. 4D; P=0.069). However, a comparison of cell-to-nucleus volume ratios revealed a significant difference between median values, with outer cells having a proportionately greater cytoplasmic volume than inner cells (Fig. 4E; P<0.001). We also confirmed previous analysis of YAP1 translocation (Nishioka et al., 2009), illustrating that nuclear localisation is significantly higher in outer versus inner cells (Fig. 4F; P<0.001). Similarly, GATA3 expression was shown to be higher in outer compared to inner cells (Fig. 4G; P<0.0001).

To further demonstrate the utility of GIANI, we sought to analyse GATA3 and YAP1 expression after treating mouse embryos with a small-molecular inhibitor against aPKC, the upstream regulator of YAP1 and GATA3. The aPKC inhibitor, CRT0276121, has previously been confirmed to specifically inhibit aPKC in various biological and cellular contexts (Aguilar-Aragon et al., 2018; Kjær et al., 2013; Rodriguez et al., 2017). Specifically, in mouse preimplantation embryos, aPKC inhibition has been recently shown to efficiently abrogate YAP1 and GATA3 expression in outer cells (Gerri et al., 2020).

Expression of GATA3 in the nucleus (normalised to DAPI intensity to correct for diminished signal intensity with increasing sample depth) was found to be similarly low in the inner cells of both control and treated embryos (Fig. 4G; P=0.841). However, GATA3 expression in outer cells was found to be significantly lower in the treated embryos relative to the control group (P<0.002). Normalised expression of GATA3 was also found to be significantly higher in outer cells of control embryos relative to their inner cells (P<0.001), while the same was not true of treated embryos (P=0.171). In addition, differences in distribution are evident between control and treated embryos, with two distinct populations evident in both inner and outer cells in control populations (Fig. 4G).

Although no statistically significant difference in nuclear YAP1 expression (normalised to cytoplasmic expression) between control and treated cells were observed when comparing inner cells (Fig. 4H; P=0.609), a large difference was observed in outer cells (P<0.0001). Moreover, the nuclear/cytoplasmic YAP1 expression ratio was still higher in the outer cells of treated embryos versus inner (P<0.0001). Altogether, this demonstrates that GIANI allows for the automated quantification of expression differences between cells following perturbation.

To illustrate that GIANI can be run successfully on later stage mouse embryos, we analysed two examples of blastocysts (mouse embryos just prior to implantation, ∼4 days post fertilisation) and compared with results from Imaris. A qualitative assessment (in the absence of ground truth segmentations) shows that the results produced by GIANI are comparable to those obtained using Imaris (Fig. S3).

To test GIANI's ability to handle more complex datasets, we generated a large simulated volume consisting of ∼2000 nuclei (available to download from https://dx.doi.org/10.5281/zenodo.5270244). We compared the segmentations generated by GIANI with those generated by Imaris (Fig. 5). A qualitative assessment shows that although both software successfully segment the vast majority of cells, errors are apparent, particularly in cases where nuclei are highly clustered (Fig. 5A). We attempted to quantify these segmentation errors using a number of different metrics.

We first compared the distribution of localisation errors produced by GIANI and Imaris (Fig. 5B). Although in both cases nuclei were localised to within ∼1 μm or less of their true locations most of the time, the localisation errors produced by Imaris were, overall, marginally lower. But, Imaris had a greater tendency to produce outliers (localisation errors above ∼2 μm).

We next looked for any link between localisation error and nuclear density. To do so, we examined the relationship between the localisation error for each successfully detected nucleus and the distance between that nucleus and its nearest neighbour (calculated as the distance between nuclear centroids; Fig. 5C). Although the relationship between these two variables is highly non-linear, it is apparent that, for both GIANI and Imaris, a trend exists – localisation error increases as the distance between nuclei decreases.

Finally, we compared the ability of GIANI and Imaris to successfully detect the simulated nuclei and investigated whether nuclear density influenced the likelihood of successful nuclear detection (Fig. 5D). We found that although both GIANI and Imaris successfully detected the vast majority of nuclei, the sensitivity (percentage of nuclei detected) was slightly higher for GIANI (93.3% vs 89.0%). There was also a clear relationship between inter-nuclear distance and likelihood of successful detection. Of the 85 nuclei that both Imaris and GIANI failed to detect, 95% were 10 μm or less from their nearest neighbour. Given that the longest dimension of the simulated nuclei is ∼10 μm, two nuclei that are less than 10 μm apart at their centres are likely to significantly overlap. This suggests that a nuclear density of less than approximately μm⁻³, where x is the longest dimension of the nuclei, is necessary for reliable segmentation using GIANI.

To further emphasise the capacity of GIANI to analyse large volumes of data, we tested it on a Tribolium castaneum embryo dataset derived from light-sheet microscopy (available to download from https://dx.doi.org/10.5281/zenodo.5270323), again benchmarking against Imaris (Fig. 6). We obtained this data from the Cell Tracking Challenge (Ulman et al., 2017) and, unfortunately, the ground truth data available is very limited (annotations for only three cells is provided). However, the intention here is to demonstrate that GIANI is capable of segmenting large numbers of nuclei in such data (∼5600 were detected) and the results are comparable to those obtained using market-leading commercial software (Imaris).

The accurate quantitative analysis of a multicellular image dataset, such as an embryo, is dependent on the correct identification and segmentation of nuclei. While GIANI presently employs generic blob detectors for this purpose, which is similar to other previously-described methods (Lou et al., 2014), other approaches could be incorporated (Blin et al., 2019; Caicedo et al., 2019; Kostrykin et al., 2019; Weigert et al., 2020; Yang et al., 2020). In particular, the ability to load pre-trained neural networks, by leveraging DeepImageJ (Gómez-de-Mariscal et al., 2021), for example, or machine-learning classifiers trained with, for example, Weka (Arganda-Carreras et al., 2017) or Ilastik (Berg et al., 2019), may be added in a future release. However, it is already possible for Java developers to customise and extend GIANI – a template for doing so is provided online (https://github.com/djpbarry/Giani/wiki).

One of the challenges associated with the imaging of 3D samples is that slices deeper within the sample will be more prone to scattering effects and therefor exhibit lower intensity. We modelled this process in the generation of our simulated embryos – the simulation process took into consideration depth of slices within a sample when modelling fluorescence intensity. Finding a single global thresholding strategy that does not under-segment the dimmer cells while also not over-segmenting the brighter cells is challenging.

There are strategies that could be employed to mitigate against these factors. For example, some form of adaptive thresholding, whereby the intensity threshold changes according to z-location, could be used. However, while an optimal adaptive thresholding strategy could be found for a given dataset (such as the simulated data used in this study), implementing a universally-applicable strategy would be difficult. One of the principal design aims of GIANI was simplicity of use, which does not allow for the incorporation of a variety of case-specific segmentation approaches.

Given the ongoing interest in studying cells in ‘native’ 3D extracellular environments (Yamada and Sixt, 2019), future extension to the capabilities of GIANI will include the ability to analyse time-lapse data, as there is currently a lack of open source tools for the quantification of 3D cell migration (Masuzzo et al., 2016). The incorporation of additional functionality from TrackMate [and MaMuT (Wolff et al., 2018) or Mastodon] will be explored to facilitate 3D cell tracking.

Future development will also include the replacement of the 3D ROI Manager interface for visualizing results with a new, custom-built interface. At present, segmented objects are saved solely as FIJI ROI files, but it is intended that support for other formats (such as .stl, .ply, .obj and .x3d) will be added to allow the import of objects into a variety of different software.

More generally, with a view to improving and optimising performance, further use of ImgLib2 (Pietzsch et al., 2012) will be incorporated in future releases. Taking advantage of GPU acceleration is also an aim, most likely by exploiting CLIJ (Haase et al., 2020). At present, the analysis of a single simulated embryo used in this study (approx. 2.1 GB) takes between approximately 30 and 60 min, depending on the number of CPUs available (Fig. S4).

We have used GIANI to quantitatively analyse mouse embryos in 3D. This analysis has revealed differences in morphology and protein expression between different experimental conditions. Analysis of simulated ground truth data was used to confirm the validity of these results. Further development of GIANI is planned, with the specific aim of improving segmentations in noisy and dense fields of view, common in 3D images of cells. Extension to timelapse analysis is also planned. GIANI is freely available on GitHub (https://github.com/djpbarry/Giani) and we anticipate that it will be a useful resource for the community to perform routine, automated quantification of complex imaging data.

Simulated data sets were generated using an extension of a previously described method (Rajasekaran et al., 2016). Simulated nuclei (ellipsoidal with axis dimensions of ∼10.0×7.5×7.5 μm) were positioned as previously described (Rajasekaran et al., 2016), then cell membranes were approximated using a Euclidean Distance Map constructed around nuclei. The simulated images were then convolved with a Gaussian point-spread function, sub-sampled and noise added from a gamma distribution. The complete code for simulated image generation is available on GitHub (https://github.com/djpbarry/Embryo-Generator).

Four- to eight-week-old (C57BL6×CBA) F1 female mice were super-ovulated using injection of 5 IU of pregnant mare serum gonadotrophin (PMSG; Sigma-Aldrich). At 48 h after PMSG injection, 5 IU of human chorionic gonadotrophin (HCG; Sigma-Aldrich) was administered. Superovulated females were set up for mating with eight-week-old or older (C57BL6×CBA) F1 males. Mice were maintained on a 12 h light–12 h dark cycle. Mouse zygotes were isolated in EmbryoMax FHM mouse embryo medium (Sigma-Aldrich; MR-122-D) under mineral oil (Origio; ART-4008-5P) and cumulus cells were removed with hyaluronidase (Sigma-Aldrich; H4272). All animal research was performed in compliance with the UK Home Office Licence Number 70/8560.

Mouse embryos were cultured in drops of pre-equilibrated Global medium (LifeGlobal; LGGG-20) supplemented with 5 mg/ml protein supplement (LifeGlobal; LGPS-605) and overlaid with mineral oil (Origio; ART-4008-5P). Preimplantation embryos were incubated at 37°C and 5.5% CO₂ and cultured up to the day of analysis.

Inhibitor experiment was performed as previously described (Gerri et al., 2020). Briefly, the aPKC inhibitor CRT0276121 (Cancer Research Technology) was dissolved in DMSO to 10 mM stock concentration and diluted to the optimal concentration of 8 μM in pre-equilibrated embryo culture medium. Mouse embryos were incubated in pre-equilibrated medium with 8 μM of CRT0276121 from four-cell to morula stage (3 to 3.25 days post-fertilisation). Control mouse embryos were developed in pre-equilibrated medium where volume-matched DMSO was added.

Embryos were fixed with freshly prepared 4% paraformaldehyde in PBS that was pre-chilled at 4°C. Embryo fixation was performed for 20 min at room temperature (RT) and then the embryos were transferred through three washes of 1× PBS with 0.1% Tween-20 to remove residual paraformaldehyde. Embryos were permeabilized with 1× PBS with 0.5% Triton X-100 and then blocked in blocking solution (3% BSA in 1× PBS with 0.2% Triton X-100) for 2 h at RT on a rotating shaker. Then, embryos were incubated with primary antibodies (listed in Table S1) diluted in blocking solution overnight at 4°C on rotating shaker.

The following day, embryos were washed in 1× PBS with 0.2% Triton X-100 for 20 min at RT on a rotating shaker and then incubated with secondary antibodies diluted in blocking solution for 1 h at RT on a rotating shaker in the dark. Next, embryos were washed in 1× PBS with 0.2% Triton X-100 for 20 min at RT on rotating shaker. Finally, embryos were placed in 1× PBS with 0.1% Tween-20 with Vectashield with DAPI mounting medium (Vector Lab; H-1200) (1:30 dilution). Embryos were placed on μ-Slide 8-well dishes (Ibidi; 80826) for confocal imaging.

All images were acquired on a Leica SP5 laser scanning confocal microscope using a Leica 1.3 NA 63× HCX PL APO CS glycerol objective and a voxel size of approximately 0.1×0.1×1.0 μm in x, y and z, respectively.

All statistical analyses in this study were performed using RStudio (https://www.rstudio.com). The tests performed in this paper are two-sample Wilcoxon tests (also known as Mann–Whitney) (Mann and Whitney, 1947); no assumptions are made regarding the nature of data distribution. All box plots show the median and inter-quartile range, with the whiskers extending 1.5 times the interquartile range from the 25th and 75th percentiles. Unless otherwise stated, each dot in dot plots represents a single cell.

Thanks to Kurt Anderson for his constructive comments on the manuscript.

The peer review history is available online at https://journals.biologists.com/jcs/article-lookup/doi/10.1242/jcs.259511.