Table of contents
Organ Detection and Random Forest
2018 Extracting abdominal planes by anatomical landmark detection and aligning them with fetal organ models.
Summary
Locating abdominal cross-sections and measuring the corresponding abdominal circumference as part of a typical fetal screening program is discussed.
To this end, a fully automated pipeline is designed, starting with random forest-based anatomical landmark detection .
A feature-trained model of fetal torso shape, including internal organs, and abdominal cross-sectional planes was encoded into the model, which was then transformed into patient space using landmark localization.
In a free-form warping step, the model is personalized to the image, using a probability map of the torso generated by a convolutional neural network as an additional feature image. After adaptation, the ventral plane and the contour of the ventral torso on this plane are obtained directly. This allows measurement of abdominal girth as well as drawing of a plane for visual assessment. The method has been trained on 126 samples and evaluated on 42 abdominal 3D ultrasound datasets. In the evaluation set, the average plane offset error was 5.8 mm, and the average relative perimeter error was 4.9 %.
Figure 1: Overview of the processing pipeline. The "Spine Detection", "Landmark Detection", "Torso Probability Map" and "Organ Model Adaptation" algorithms automatically generated a 3D segmentation of the fetal torso and major organs, from which the position, orientation and contour of the AC plane were estimated.
Figure 2. Left: Abdominal plane and circumference (red) of the clinical specialist. Important structures in this plane are the gastric vein and the internal umbilical vein. Right: Plan and profile results of the automated procedure.
Target
First, data navigation can be facilitated by automatically identifying fetal anatomy and displaying anatomically relevant views.
Second, automated contouring of anatomical structures enables automated biometry.
Finally, biometric measurements and other image processing results can be used to support diagnosis.
Fetal screening includes a variety of fetal growth measures, but also includes detection of fetal abnormalities. Most examinations begin at 18-22 weeks gestational age (GA) with specific recommended standard measurements. Typically, a set of anatomically defined planes of the fetal head, thorax, and abdomen are obtained and examined, with the choice of an appropriate two-dimensional plane varying widely between observers.
Second trimester screening of the abdomen, specifically the standard abdominal plane (ac plane). Abdominal circumference (AC) is a standard measurement for estimating fetal size and growth, where performance varies significantly between centers and individuals.
Target:
From a 3D US scan
i) Automatic extraction of AC planes
ii) Estimation of abdominal circumference
For the AC plane, we follow the definition given in the 21st interleaving item. 1 According to this definition, the presence of the stomach needs to be given as well as an appropriate part of the umbilical vein, compare Fig. 2.
Anatomical landmark localization based on random forests,
Fetal Chest Probability Map Based on Convolutional Neural Networks
Model-Based Segmentation of Fetal Chest and Internal Organs.
method
An overview of the processing pipeline used is shown in Figure 1, and a typical result is shown in Figure 2. Several sources of information guide the fetal organ model in its personalization to a given 3D US image. It provides estimates of AC plane position, plane orientation, and AC profile. This pipeline is fully automated.
Fetal feature point detection
The first step in estimating the AC plane and AC measurements is the detection of fetal organs. They are used as landmarks to orient fetal organ models.
Organ detection was performed using a learning algorithm based on a random forest approach . 5–7 Here, the datasets used for training and testing consist of volumes in which the orientation, position, and size of the fetus are highly variable, making organ detection difficult when the size of the training database is limited. To eliminate this size and position variation and let the learning algorithm focus on anatomical variation, the processed volume is automatically aligned in an arbitrary common coordinate system that is centered and oriented relative to the fetal spine and abdomen , as shown in Figure 3.
Figure 3. Definition of the fetal-based coordinate system
volume alignment
The coordinate system used for volume alignment is defined as follows.
Origin and z-axis definition from spine detection
Since the spine is a unique and echogenic elongated structure , it is a good reference for anatomical alignment.
Spine detection is done by combining two methods.
First, apply a morphological filter that detects elongated structures.
It got a fairly accurate detection map, but also produced many false positives (red overlay in Figure 4 (left)).
A cone detector using a convolutional neural network is independently applied to 2D slices extracted perpendicular to the original z-axis of the volume.
Such slicing produces a large amount of data with similar characteristics, which is suitable for deep learning methods.
The output of the network is a downsampled probability map with values closer to where the spine might be. A 3D cone detector was built by superimposing all obtained 2D probability maps as a volume. The heatmap of this output is coarser than the output of the morphological filter, but more stably located around the vertebral bodies (Fig. 4 (center)).
Finally, the neural network output is refined by selecting the intersection of the vertebral body detector with the morphological filter response, and rejecting the filter response outside the spine, resulting in a more robust spine binary mask (Fig. 4 (right)).
The origin of the fetal-based coordinate system was defined as the binary mask point closest to its centroid .
The ends of the spine binary mask are then used to define the z-axis, a vertical axis tangent to the spine (see Figure 3).
Figure 4 Spine detection. Left: Morphological filtering, Center: DL method with 3 consecutive convolutional layers with ReLU activation, two dense equivalent blocks and a final 1×1 convolution, Right: Combination of both methods.
The x-axis defines the detection from the abdomen
A set of planes is extracted orthogonally to the detected spine. The abdomen was detected on these planes by the Hankel transform, a variant of the Hough transform, used to detect circular shapes. The stack of detected circles is an estimate of the abdominal mask, and their centers are used to estimate the sagittal plane of the fetus, from which the x-axis is defined.
Head / toe orientation
It is determined in the sagittal plane by a classifier similar to AlexNet using a convolutional neural network. 8
zoom
Scaling employs a scaling factor based on an estimated genetic algorithm to reduce size variability in the database.
Organ Detection and Random Forest
Once these volumes are aligned in a fetal-based coordinate system, a random forest learning algorithm is used to detect the location of fetal organs. The principle is to learn the relative orientation of a given point in a volume to the target organ landmark. In the description below, this relative direction is called a voting vector.
Training: The algorithm is trained on a dataset consisting of manual landmark annotations provided by experts .
It computes some image-based features, mostly derived from local gradients, such as locally normalized gradients and distances to gradients.
A segmentation criterion is defined for all nodes of the forest tree.
Its purpose is to find two subsets of training points such that the sum of the entropy of the two subsets is minimized, and entropy is defined as the variance of the voting vector for each landmark.
In order to obtain the segmentation criterion, a large number of random features are tested on each node, and the features that provide the optimal subset separation and corresponding segmentation threshold are selected and stored in the node.
Multiple stopping criteria are defined for all leaves:
(i) reached a given tree depth
(ii) the variance within the subset is below a given threshold
(iii) The subset is too small.
The average of the voting vectors is stored in each leaf. This will be the vote vector for each point classified in this leaf.
Testing : This step describes the actual landmark localization process, which is constrained to volumetric regions located within the abdomen, estimated from the abdomen detections obtained during volume alignment.
For a given input volume, a random set of test points is chosen and propagated through the tree using the chosen splitting condition until they reach a leaf.
Then, each test point provides a voting vector.
All voting vectors are converted into milestone predictions.
To provide a single prediction, all predictions are combined via Gaussian estimation, resulting in a localization probability map for each landmark.
The obtained landmark locations of the heart, stomach, umbilical vein , and bladder were then used to position the fetal organ models .
Fetal Torso Probability Map
In order to increase the capture range of deformable segmentation and reduce the damaging effect of irrelevant edges (such as placenta) in the image, the output of convolutional neural network is incorporated in the model-based segmentation method.
The network is trained to estimate the probability that each voxel corresponds to either the fetal torso or the background (i.e. fetal head, limbs or maternal tissue).
To this end, this paper adopts the Fovea Fully Convolutional Network (FovFCN). 9 This particular architecture allows the network to take a "global" view of the image without significantly increasing resource requirements, as ordinary fully convolutional networks process the entire image at once.
This is achieved by simultaneously considering equidistant central patches of different sizes and resolution levels. Compared to the multi-resolution methods proposed in 10 our network continuously integrates information from arbitrary networks
( Voxel is a volume element (Volume Pixel), which is the smallest unit of digital data in three-dimensional space segmentation. Voxel is used in fields such as three-dimensional imaging, scientific data and medical imaging. Conceptually similar to the smallest unit of two-dimensional space - pixel )
Figure 5. Schematic diagram of a three-layer FovFCN. To segment a patch of a given size (here: 74×74×74 voxels, red box), three input patches of different sizes are cropped from the input image. The larger "context" patch is then rescaled to a lower resolution. For each level, features are extracted using the cross-entropy loss function (convolutional layers with ReLU activations). The feature maps of the rescaled levels are upsampled to the original resolution and then added to integrate features from all levels.
number of solutions, not just two. The method is shown in Figure 5. First, features are extracted at each resolution level. Then, the coarse-resolution feature maps are up-sampled using average pooling and additively concatenated with finer layers. The network is trained using the cross-entropy between the output layer and known labels as the loss function
fetal organ model
Although only the gastric and umbilical veins are directly relevant to the definition of the ac plane, it is also useful for locating other prominent structures that the sonographer uses for localization. Therefore, a triangular surface mesh of 5 components (fetal trunk, heart, stomach, umbilical vein, and bladder) was used, which were interconnected to model the approximate anatomical relationship between them. An illustration of the interconnection model is shown in Figure 1. The segmentation of model-based combined models relies on trained triangle-specific features that search for target points in images, mainly gradients with additional gray-value attribute filters. For the torso, the feature function is also defined on the torso probability map in Section 2.2. Warp the model to minimize the distance to maximize the eigenresponse. This is achieved by optimizing an external (=feature) and an internal (=shape) energy formulation. 11 The weighting between external and internal energies determines the magnitude that is allowed to be derived from the average shape of the structure. For this particular model, there is a 3-step adaptation procedure:
1. The model is initialized by helping with the landmarks in 2.1. The torso triangle then just finds the target point based on the probability map from 2.2 and moves accordingly. All other structures follow passively, guided by the connectors. This results in an approximate positioning of the entire 5-component model and a good fit to the external body contour (=trunk) of the fetus.
2. The torso mesh is frozen in place, and the other four structures are rigidly deformed toward the target point in the ultrasound image, guided by the connectors. This results in a better position of the four internal organs.
3. All 5 structures are looking for valid target points in the ultrasound image (image gradient) to get close to the current position. The deformation is very flexible but guided by the adjacent structure through the connector. This results in a precise adaptation to the ultrasound anatomy, with accuracy depending on the visibility of the structure and the correct positioning in steps 1 and 2. See for more details on model-based segmentation. 9, 11
Based on the adapted triangular surface mesh, the AC plane was determined in the intergrowth-21 project, i.e. containing the appropriate portion of the gastric and umbilical veins.
result
The experiments in this paper are based on the 166-2 3D transducer obtained from the Philips EPIQ 7 ultrasound system. The imaged field of view covers the abdomen and part of the thorax of the fetus. Typically, the bladder, stomach, (internal) umbilical vein, and heart are included in the image. The gestational age range is 18-32 weeks. The image matrix (x,y,z) contains (256···512)×(263···510)×(143···256) voxels according to the gestational age and the size of the fetus, and the voxel size is (0.1 ···0.5)×(0.06···0.4)×(0.18···0.8) mm3. The images are divided into 126 training (i.e. model generation, random forest landmark locator training and neural network training) and 42 test cases with the aim of obtaining the same age distribution. For evaluation, ground truth annotations from a panel of clinical experts are available. These annotations include the location and orientation of the ac plane and the contour of the abdominal girth.
Signature detection
Quantitative results show that the average distance between the detected spine center and the manually annotated spine is 12 mm [±6.3], and its relative angle is 12◦ [±7]. Assessing the effect of the x-axis definition from the abdomen detected in the transverse plane, the spatial distribution of the variance annotation heart and stomach is calculated in two reference coordinate systems: first with the x-axis directly from the detection of the spine curvature and second with the x-axis detected from the abdomen, As stated by sec. 2.1. Regarding the heart, the variance of the first coordinate system is 13.9 mm and that of the second coordinate system is 5.6 mm. For the stomach, it increased from 10.4 mm to 4.3 mm, indicating that the x-axis definition is more accurate using abdominal detection. The success rate for head/toe classification was 86%.
All steps described in Volume Alignment and Landmark Description Procedure. 5 A major conclusion is that the volume alignment process greatly reduces spatial localization and size variability in the US volume database, enabling the detection of internal organs using a random forest learning algorithm. As previously described, experiments were performed on a dataset of 126 ultrasound volumes, which were divided into 4-fold. To test each fold, the algorithm will be trained on the remaining 3 folds.
Compare detected organ locations with manual annotations. Using the fully automated pipeline, the mean values for the umbilical vein, heart, stomach, and bladder were {10.2; 11.1; 11.1; 15.4} mm, respectively, close to {10.0; 10.8; 11.0; 15.3} for the same organs aligned using manually annotated spine initialization volumes mm. These results suggest that automatic spine and abdomen detection for determining the fetal coordinate system is satisfactory
torso segmentation
A thorough evaluation is performed on the torso segmentation step. 9 found that including torso probability maps estimated using FovFCNs significantly improves the accuracy of model-based segmentation, in particular enhancing robustness to initialization errors. Quantitatively, a mean surface distance of 2.24 mm and a dice score of 0.87 was reported.
AC Plane Estimation
Four exemplary results of AC plane estimation are shown in FIG. 6 . For quantitative evaluation, plane offset error, plane angle error, AC length error, and error in contour point positions were calculated (see Table 1 for exemplary cases and Table 2 for quantitative results). The plane offset error is measured along the plane normal of the ground truth plane at the center of the ground truth contour. In addition, the clinical usability of the contours was assessed by clinical experts with a success rate of 62% ("no correction required").
Figure 6. Demonstration results for AC plane estimation. Left: Expert AC profile Middle: Segmentation result of combined torso model, Right: Torso probability map. In (A), a good result with an extremely small AC length error of only 0.1% is given. (B) shows the best result of AC plane selection, the umbilical vein and stomach are clearly visible, and the length error is only 1.8%. In (C), the AC plane is well chosen, and the larger error in the AC length is due to the shape of the abdomen.
In (D), the image is noisy and difficult, resulting in suboptimal AC planes, however, the AC length error is only 2.7% due to both clinician and automated methods relying on shape priors.
in conclusion
This paper presents a new fully automated method for the localization and measurement of standard AC planes based on manually acquired 3D volumes, which can be used for second-trimester screening. The process employs random forest-based landmark localization, neural network-based torso region detection, and model-based organ segmentation. Although the gestational age of the test data ranged widely from 18-32 weeks, the average offset error obtained on the test set was 5.8 mm and the average relative circumference error was 4.9%.
In this way, the relative error in AC length was smaller than the reported inter-observer variability of 8.8%, and even the intra-observer variability of 5.6%. 4 In 62% of cases, the automatically estimated AC plane was judged to be suitable for clinical use by the clinical observer without modification. For the majority of the remaining cases, clinically appropriate results can be obtained with only minor changes in planar geometry. Due to the large inter-observer variability, not only when measuring abdominal circumference in a given 2D ultrasound image, but also when finding the correct AC plane in the volume, the advantages of the proposed solution may not be in the clinical workflow Full automation, but a reduction in the time required to achieve appropriate results.
The proposed method is evaluated with only a limited number of cases. Further work on larger numbers is needed to ensure that the method remains robust and accurate across a variety of acquisition settings and trimesters.
REFERENCES
1. A. T. Papageorghiou, E. O. Ohuma, D. G. Altman, T. Todros, L. C. Ismail, A. Lambert, Y. A. Jaffer,
E. Bertino, M. G. Gravett, M. Purwar, J. A. Noble, R. Pang, C. G. Victora, F. C. Barros, M. Carvalho,
L. J. Salomon, Z. A. Bhuttaf, S. H. Kennedy, and J. Villar, “International standards for fetal growth based
on serial ultrasound measurements: the Fetal Growth Longitudinal Study of the INTERGROWTH-21st
Project,” The Lancet 384, pp. 869–879, 2014.
2. A. Coomarasamy, N. M. Fisk, H. Gee, and S. C. Robson, “The investigation and management of the small–
for–gestational–age fetus,” Royal College of Obstetricians and Gynaecologists, Guideline No. 31 , 2002.
3. M. B. Landon, M. C. Mintz, and S. G. Gabbe, “Sonographic evaluation of fetal abdominal growth: predictor
of the large-for-gestational-age infant in pregnancies complicated by diabetes mellitus,” American journal
of obstetrics and gynecology 160(1), pp. 115–121, 1989.
4. I. Sarris, C. Ioannou, P. Chamberlain, E. Ohuma, F. Roseman, L. Hoch, D. Altman, and A. Papageorghiou,
“Intra-and interobserver variability in fetal ultrasound measurements,” Ultrasound in Obstetrics & Gynecology 39(3), pp. 266–273, 2012.