1 Introduction

Breast cancer is a fatal disease that infects both women and men, especially women that largely die in that disease worldwide [1]. There are several ways to image breast cancer, such as mammogram, computed tomography (CT), ultrasound, magnetic resonance imaging (MRI), and thermogram [1, 2]. Image segmentation is an important step in image processing and is employed indispensable in several fields such as computer vision, pattern recognition, agriculture, robotic vision, medical images, and cryptography [3]. The segmentation of an image refers to extracting the homogenous regions with similar features such as texture, color, contrast, brightness, form, and size based on several methods including feature-based clustering, region-based, threshold-based, and edge-based. Thresholding is the most common segmentation method due to its simplicity, ease of use, speed, accuracy, and limited storage space required [4].

Thresholding is classified into two classes: bi-level, and multilevel. In the former, a single threshold value needs to be identified to segment the background and foreground from an image, while the latter class is applied to find more than two similar regions in the image using pixel intensities called a histogram [5]. Thresholding has been formulated as an optimization problem using either nonparametric or parametric techniques [6]. In the parametric approach, the probability density function is employed to compute some parameters for each region to find the optimal threshold values. Meanwhile, the nonparametric approach seeks to maximize one of the functions such as fuzzy entropy [7], Kapur’s entropy (maximizing class entropy) [6], and Otsu function (maximizing between-variance) [8]. Unfortunately, by those approaches, finding the optimal threshold values for multi-level thresholding is so hard and computationally expensive time, especially with increasing threshold levels. Therefore, the need for a strong modern alternative was so necessary. Due to the significant success achieved by the metaheuristic algorithms in several fields [5], the researchers pay attention to them to overcome the multilevel image segmentation problem. Several studies in the literature have employed metaheuristic algorithms for tackling this problem, some of which will be discussed in some of the following paragraphs.

Akay et al. [9] developed a hybrid metaheuristic algorithm by integrating teaching–learning-based artificial bee colony (TLABC) with levy flight to present a new strong variant, namely MTLABC, for finding multilevel threshold values of plant disease. This algorithm has been compared with four other optimizers using five performance metrics to show its effectiveness. The experimental findings affirmed its superiority compared to the others. Furthermore, in [10], the equilibrium optimizer has been modified by Laplace distribution-based random walk and opposition-based learning (OBL) to propose a new strong variant called opposition-based laplacian equilibrium optimizer (OB-L-EO) having stronger exploration and exploitation operators, respectively. Afterward, this variant has been applied to determine the optimal threshold values for multilevel thresholding image segmentation problems and could come true superior results for various employed metrics.

A new metaheuristic optimization algorithm based on hybridizing three optimization algorithms within three stages: primary stage, booster stage, and final stage was proposed for tackling ISP based on multilevel thresholding technique [11]. The three algorithms employed within those three various stages were artificial bee colony optimization (ABC), particle swarm optimization (PSO), and ant colony optimization (ACO). The experimental findings affirmed that this hybrid algorithm is better than all the others in terms of SSIM, PSNR, and Wilcoxon rank-sum test. The marine predators algorithm enhanced by opposition-based learning to improve its exploration operator and convergence speed has been developed to tackle multilevel thresholding image segmentation [12]. This variant was termed MPA-OBL and could be superior to all compared algorithms for tackling this problem. The cuckoo search algorithm (CS) improved using chaotic maps to initialize population for reaching better diversity of solutions, in addition to improving both step size factor and the complexity of algorithm was developed for tackling the multilevel thresholding image segmentation [13]. This variant was abbreviated ICS and its outcomes were better than those of the competing algorithms. There are several other metaheuristic algorithms proposed recently for tackling this problem, some of them are the water cycle algorithm [14], the growth optimizer [15], and the opposition-based Runge Kutta optimizer [16].

Unfortunately, after reviewing some metaheuristic algorithms employed in the literature for tackling this problem, it is observed that each one of them suffers from at least one of the following problems: falling into local optima which prevent from reaching the optimal threshold levels, low convergence speed, and lack of the population diversity. Therefore, in this paper, a new metaheuristic algorithm known as the jellyfish search optimization algorithm (JSO) is hybridized with an effective method to propose a new strong variant, namely HJSO, having better exploration and exploitation capabilities. Afterward, this variant is employed for tackling the breast cancer images, where 12 breast cancer images are employed to validate its performance using Otsu’s method as an objective function. In addition, HJSO is extensively compared with several well-established metaheuristic algorithms using various statistical analyses and the Wilcoxon rank-sum test to show its effectiveness. The experimental findings show the superiority of HJSO compared to the others. The main contributions of this study are as follows:

  1. 1.

    Presenting a hybrid algorithm, namely HJSO, based on effectively integrating the JSO with an effective improvement strategy to segment breast cancer images under multilevel thresholding technique.

  2. 2.

    Validating the performance of this algorithm using various breast cancer images and conducting an extensive comparison with several rival algorithms to explore its effectiveness.

  3. 3.

    The experimental findings reveals the HJSO’s superiority in terms of several performance metrics

The next sections of this research are arranged as that:

  • Section 2 explains the Otsu method.

  • Section 3 describes the proposed work.

  • Section 4 presents findings and discussion.

  • Section 5 shows some conclusions and future work.

2 Otsu Method

This method is a variance-based technique and proposed in [8] to search the optimal threshold values which separate the heterogeneous out of an image by maximizing the between-class variance, or equivalently, minimizing the intra-class intensity variance. To extract \(m\) threshold values, [\({t}_{0},{t}_{1},{t}_{2},\dots \dots .., {t}_{m}]\) for an image with \(m+1\) homogenous regions, the following fitness function has to be applied:

$$F\left({t}_{0},{t}_{1},{t}_{2},\dots \dots .., {t}_{m}\right)={{\sigma }_{0}}^{2}+{{\sigma }_{1}}^{2}+{{\sigma }_{2}}^{2}+\dots \dots ..+ {{\sigma }_{m}}^{2}$$
(1)
$$where:$$
$${{\sigma }_{0}}^{2}={\omega }_{0}{({\mu }_{0}-{\mu }_{T})}^{2}, {\omega }_{0}={\sum }_{i=0}^{{t}_{1}-1}{p}_{i}, {\mu }_{0}={\sum }_{i=0}^{{t}_{1}-1}\frac{i{p}_{i}}{{\omega }_{0}}$$
(2)
$${{\sigma }_{1}}^{2}={\omega }_{1}{({\mu }_{1}-{\mu }_{T})}^{2}, {\omega }_{1}={\sum }_{i={t}_{1}}^{{t}_{2}-1}{p}_{i}, {\mu }_{1}={\sum }_{i={t}_{1}}^{{t}_{2}-1}\frac{i{p}_{i}}{{\omega }_{1}}$$
(3)
$${{\sigma }_{2}}^{2}={\omega }_{2}{({\mu }_{2}-{\mu }_{T})}^{2}, {\omega }_{2}={\sum }_{i={t}_{2}}^{{t}_{3}-1}{p}_{i}, {\mu }_{2}={\sum }_{i={t}_{2}}^{{t}_{3}-1}\frac{i{p}_{i}}{{\omega }_{2}}$$
(4)
$${{\sigma }_{m}}^{2}={\omega }_{m}{({\mu }_{m}-{\mu }_{T})}^{2}, {\omega }_{m}={\sum }_{i={t}_{m}}^{L-1}{p}_{i}, {\mu }_{m}={\sum }_{i={t}_{m}}^{L-1}\frac{i{p}_{i}}{{\omega }_{m}},$$
(5)

where \({{\sigma }_{0}}^{2}, {{\sigma }_{1}}^{2}, {{ \sigma }_{2}}^{2},\dots \dots .., and\, {{\sigma }_{m}}^{2}\) indicates variances of various similar classes; \({\omega }_{0}, {\omega }_{1}, {\omega }_{2},\dots \dots ..,{\omega }_{m}\) refer to the class probabilities; \({\mu }_{0}, {\mu }_{1}, {\mu }_{2},\dots \dots ..,{\mu }_{m}\) indicates the class means; \(L\) is the maximum grey level; and \({\mu }_{T}\) is computed using the following formula:

$${\mu }_{T}={\sum }_{i=0}^{L-1}i{p}_{i}$$
(6)

3 Standard Algorithm: Artificial Jellyfish Search Optimizer

A new optimization algorithm known as artificial jellyfish search optimizer (JSO) has been recently presented for tackling optimization problems [17]. This algorithm has been based on following the ocean current, or movements in the swarm as the behaviors of jellyfish for finding food in the ocean.

3.1 Initialization

At the outset, most metaheuristic algorithms have randomly distributed a number of solutions within the search space of the problem to generate the initialized positions which are updated by the optimization process for reaching better positions. However, the authors of JSO have found that the distribution of the initialized positions using the chaotic maps is more coverage and accurate. According to [17], the logistic chaotic map is the best way to initialize the solutions before starting the optimization process and modeled mathematically using the following equation:

$${\overrightarrow{{X}^{{^{\prime}}{^{\prime}}}}}_{i+1}=\eta {\overrightarrow{X}}_{i}\left(1-{X}_{i}\right), { 0\le \overrightarrow{{X}^{{^{\prime}}{^{\prime}}}}}_{0}\le 1$$
(7)

where \({\overrightarrow{{X}^{{^{\prime}}{^{\prime}}}}}_{0}\) is an initial vector randomly assigned at the range of 0 and 1 and employed for generating the next logistic chaotic vector, and \({\overrightarrow{{X}^{{^{\prime}}{^{\prime}}}}}_{i}\) is a vector including the logistic chaotic values employed for generating the initial position of the \({i}^{th}\) jellyfish. \(\eta\) is a fixed-value of 4 as recommended in [17]. After generating the logistic vectors, the initial position for the \({i}^{th}\) jellyfish will be generated using the following formula:

$${\overrightarrow{X}}_{i}={\overrightarrow{X}}_{L}+{\overrightarrow{{X}^{{^{\prime}}{^{\prime}}}}}_{i}.\left({\overrightarrow{X}}_{U}-{\overrightarrow{X}}_{L}\right)$$
(8)

\({\overrightarrow{X}}_{L}\) and \({\overrightarrow{X}}_{U}\) include the lower and upper boundaries of each dimension in an optimization problem, respectively, and \(.\) is the entry-wise multiplication operator.

3.2 Ocean Current

This section describes the mathematical model of the ocean current followed by the jellyfish for searching for food. This model is described as follows:

$${\overrightarrow{{X}^{^{\prime}}}}_{i}\left(t+1\right)={\overrightarrow{X}}_{i}\left(t\right)+\overrightarrow{r}.({\overrightarrow{X}}^{*}-\beta *{r}_{1}*\mu )$$
(9)

where t indicates the current iteration, \(\overrightarrow{r}\) is a vector including random numbers between 0 and 1, and \(\beta >0\) stands for the distribution coefficient and is set to \(3\) in the cited paper. \({\overrightarrow{{X}^{^{\prime}}}}_{i}\) is a vector to contain the updated position of the \({i}^{th}\) jellyfish. \({\overrightarrow{X}}^{*}\) is the best-so-far solution, \(\mu\) is the current population mean and \({r}_{1}\) is a number generated randomly in the range of 0 and 1.

3.3 Movements Inside the Swarm

This behavior is based on two motions: passive and active. The former indicates the motion of the jellyfish around their locations as described in Eq. (10), while the latter considers the motion of the jellyfish in the best direction to food and is mathematically described in Eq. (11).

$${\overrightarrow{{X}^{^{\prime}}}}_{i}\left(t+1\right)={\overrightarrow{X}}_{i}\left(t\right)+{r}_{3}*\gamma *\left({U}_{b}-{L}_{b}\right),$$
(10)

where \({r}_{3}\) is a number generated randomly between 0 and 1, and \(\gamma >0\) is the motion length around the current location.

$${\overrightarrow{{X}^{^{\prime}}}}_{i}\left(t+1\right)={\overrightarrow{X}}_{i}\left(t\right)+\overrightarrow{r}*\overrightarrow{D},$$
(11)

where \(\overrightarrow{r}\) is a vector assigned randomly between 0 and 1. \(\overrightarrow{D}\) is computed as follows:

$$\vec{D} = \left\{ {\begin{array}{*{20}c} {\vec{X}_{i} \left( t \right) - \vec{X}_{j} \left( t \right),} & {{\text{iff}}\left( {\vec{X}_{i} } \right) < f\left( {\vec{X}_{j} } \right)} \\ {\vec{X}_{j} \left( t \right) - \vec{X}_{i} \left( t \right),} & {{\text{otherwise}}} \\ \end{array} } \right.,$$
(12)

where \(f\) is the fitness function, and \({\overrightarrow{X}}_{j}\) is a jellyfish randomly-selected from the current solutions. Exchanging between those motions: active and passive, and ocean current is achieved by the time control mechanism with a predefined value \({c}_{0}\) (see Fig. 1). The mathematical model of this mechanism is as that:

$$c\left(t\right)=\left(1-\frac{t}{{t}_{max}}\right)*\left(2*r-1\right),$$
(13)

where \({t}_{max}\) is the maximum iteration, and \(r\) is a number created randomly at the range of 0 and 1.

Fig. 1
figure 1

Flowchart of the time control mechanism

Full size image

4 Proposed Algorithm

This section describes the steps employed for developing the proposed algorithm for overcoming the multilevel thresholding image segmentation for color breast cancer images; those steps are initialization, evaluation, improvement method, and finally proposed algorithm called hybrid JSO.

4.1 Initialization

At the outset, N solutions will be randomly distributed within the search space of the problem, where each solution will have d variables (threshold level). In color images, the intensities of three components: Red, Green, and Blue are employed to build those images. For each component of those, N solutions will be randomly initialized within the lower bound \({\overrightarrow{X}}_{L}\) of 0 and the upper bound \({\overrightarrow{X}}_{U}\) of 255 (maximum intensity) using the following equation:

$${\overrightarrow{X}}_{i}={\overrightarrow{X}}_{L}+\overrightarrow{r}.\left({\overrightarrow{X}}_{U}-{\overrightarrow{X}}_{L}\right),$$
(14)

where \(\overrightarrow{r}\) is a vector assigned randomly at the range of 0 and 1. Afterward, each initialized solution of those will be evaluated and compared with the solutions belonging to the same component, and then the best solutions for each component will be determined to be incorporated in the next iterations for generating better solutions.

4.2 Improvement Method

Actually, the classical jellyfish search algorithm suffers from falling into local optima during searching for a better solution, so our first updating equation is designed to deal with this problem by giving the algorithm a new capability to search around the current position using two different step sizes: the first one tries to take the solution in the right direction of the best solution obtained even now, while the other seeks to take it to the positions of a solution picked randomly from the population in the hope of averting local minima problem. Generally, this updating formula is described below:

$${\overrightarrow{{X}^{^{\prime}}}}_{i}={\overrightarrow{X}}_{i}+{r}_{1}.\left({\overrightarrow{X}}^{*}-{\overrightarrow{X}}_{i}\right)+{r}_{2}.\left({\overrightarrow{X}}_{a}-{\overrightarrow{X}}_{b}\right)$$
(15)

where \({r}_{1}\) and \({r}_{2}\) are two values selected randomly between 0 and 1. \({\overrightarrow{X}}_{a}\) and \({\overrightarrow{X}}_{b}\) are two solutions selected randomly from the current population.

In addition, as the second attempt, the near-optimal solution might nearby the best solution obtained even now, therefore, the optimization process needs to focus on searching around this solution in the hope of finding the near-optimal solution in fewer function evaluations. Our improvement for this point is based on two-folds. The first fold is based on searching around the best-so-far solution using two different step sizes: the first one is based on searching around the best solution using two solutions selected randomly from the population, while the second one mutates the best solution obtained so-far within the search boundaries of the problem according to a specific predefined probability. The mathematical model of the first fold is modeled below:

$${\overrightarrow{{X}^{^{\prime}}}}_{i}={\overrightarrow{X}}^{*}+\left(r\left(1-{r}_{3}\right)+{r}_{3}\right).\left({\overrightarrow{X}}_{a}-{\overrightarrow{X}}_{b}\right) +{\overrightarrow{r}}_{5}.\left({\overrightarrow{X}}_{U}-{\overrightarrow{X}}_{L}\right).\overrightarrow{U},$$
(16)

where \(r\) and \({r}_{3}\) are two randomly selected numbers between 0 and 1. \({\overrightarrow{r}}_{5}\) is a vector assigned randomly between 0 and 1. \(\overrightarrow{U}\) is a vector containing 0 and 1 values which are randomly generated according to the following formula:

$$\vec{U} = \left\{ {\begin{array}{*{20}c} 0 & {r_{4} > \gamma } \\ 1 & {otherwise} \\ \end{array} } \right.,$$
(17)

where \({r}_{4}\) is a random number between 0 and 1; \(\gamma\) is a predefined probability to determine the percentage of 1 in this vector. In the second fold, the current solution will be updated using the following equation as a new attempt to generate various steps helping in covering the regions around the best-so-far solution possible.

$${\overrightarrow{{X}^{^{\prime}}}}_{i}={\overrightarrow{X}}^{*}+{r}_{6}.\left({r}_{4}.{\overrightarrow{X}}^{*}-{\overrightarrow{X}}_{c}\right),$$
(18)

where \({r}_{6}\) and \({r}_{4}\) are two values selected randomly between 0 and 1. Exchanging between Eq. (15), (16), and (17) is randomly achieved as shown in the following equation:

$${\overrightarrow{{X}^{^{\prime}}}}_{i}=\left\{\begin{array}{l}Applying\, eq.\, \left(15\right) {r}_{7}<\alpha \\ Applying\, eq.\,\left(16\right) { r}_{7}<\delta \\ Applying\, eq.\,\left(18\right) otherwise\end{array}\right.,$$
(19)

where \(\alpha\) and \(\delta\) are two predefined probabilities, such that \(\alpha <\delta\). Finally, the steps of integrating the classical JSO with the improvement method for segmenting breast cancer images under multilevel thresholding are described in Algorithm 1. In a more sense, this algorithm starts with distributing N solutions within the lower and upper bounds of the optimization problem, as defined beforehand in the initialization section. Those initial solutions are evaluated using the Otsu-based objective function, and the solution with the highest fitness value is set as the best-so-far solution \(\overrightarrow{{X}^{*}}\), as described in Line 2 within Algorithm 1. In Lines 4–25, the HJSO’s optimization process is fired to improve the quality of the initial solutions for finding better segmented images. This process first updates the current solutions using the updating behaviors of the classical JSO to generate new solutions called the updated solutions, which are evaluated and further improved using the improvement method. This optimization process is continued until the termination condition is satisfied. In this study, the termination condition is achieved after \({t}_{max}\) iterations.

figure a

5 Results and Discussion

5.1 Test Images

Our experiments are based on segmenting 12 color images for breast cancer with threshold levels of 10, 15, 20, 25, 30, 35, and 40. Those images are taken from the visualLab [18] to validate the performance of HJSO. The experimental findings of HJSO are compared to those of several rival optimization algorithms like the classical JSO [17], sine–cosine algorithm (SCA) [19], improved marine predators algorithm (IMPA) [4], marine predators algorithm (MPA) [4], improved salp swarm algorithm (ISSA) [20], equilibrium optimization (EO) [21], cuckoo search algorithm (CSMC) [22], and WOA [23]. As aforementioned that the color images are compounded of three components: Red, Green, and blue, the histogram of each component of those in addition to the original image for some test images are depicted in Fig. 2. It is worth mentioning that the test images in our study are renamed as img1, img2, img3, img4, and so on.

Fig. 2
figure 2

Depiction of the original image and the histogram of its components (Red, Green, Blue)

Full size image

Regarding the parameters of the rival algorithm, they are set according to the published papers except for \({t}_{max}\) and N which is set to 50 and 30 for all algorithms to come true an equitable comparison. However, the proposed algorithm has three parameters that have to be estimated accurately to maximize its performance; those parameters are \(\gamma\), \(\alpha\), and \(\delta\). Therefore, extensive experiments with various values for each parameter are conducted and the obtained outcomes are depicted in Fig. 3. Inspecting this figure appears that the best values for those parameters respectively are 0.04, 0.1, and 0.8. All algorithms are implemented using MATLAB R2019a on the same device.

Fig. 3
figure 3

Parameters tuning of HJSO

Full size image

5.2 Performance Evaluation Criteria

The performance of the proposed algorithm will be evaluated using six performance metrics: standard deviation (SD), fitness values for each component (F-value), CPU time, Peak signal-to-noise ratio (PSNR), Structural similarity index (SSIM), and Features similarity index (FSIM), and their outcomes will be compared with those produced by the rival algorithms mentioned before.

5.2.1 Standard Deviation (SD)

This metric is employed to see the deviation of the outcomes obtained through 30 independent executions and the algorithm with less SD is classified as the best because its outcomes within the different runs are so converged. The mathematical equation of SD is as that:

$$SD = \sqrt {\frac{1}{{n - 1}}\sum\nolimits_{{i = 1}}^{n} {\left( {f_{i} - \bar{f}} \right)^{2} } }$$
(20)

n is the independent runs, \({f}_{i}\) refers the fitness value generated in the \({i}^{th}\) run using Eq. (1), and \(\overline{\mathrm{f} }\) is estimated using the following formula:

$$\overline{f }=\frac{\sum_{i=1}^{n}{f}_{i} }{n}$$
(21)

5.2.2 PSNR

This metric abbreviated PSNR [24] is employed to measure the quality of the segmented image compared to the original one by computing the ratio of the error between them according to the following formula:

$$PSNR=10\left(\frac{{255}^{2}}{MSE}\right)$$
(22)

MSE (mean squared error) is computed according to:

$$MSE=\frac{{\sum }_{i=1}^{M}{\sum }_{j=1}^{N}\left|A\left(i,j\right)-S(i,j)\right|}{M*N}$$
(23)

where \(A\left(i,j\right)\) and \(S(i,j)\) are the intensity level of the segmented and original image within the row, \({i}^{th}\) and column \({j}^{th}\), respectively. M and N indicate the number of rows and columns in the image, respectively.

5.2.3 SSIM

Unlike PSNR which doesn’t take into consideration the image structure, SSIM is employed to take into account the brightness, similarity, and contrast distortion between the segmented and source images. SSIM is mathematically estimated using the following formula [24]:

$$SSIM(O, S)=\frac{\left(2{\mu }_{o}{\mu }_{s}+a\right)\left(2{\sigma }_{os}+b\right)}{\left({{\mu }_{o}}^{2}+{{\mu }_{s}}^{2}+a\right)\left({{\sigma }_{o}}^{2}+{{\sigma }_{s}}^{2}+b\right)}$$
(24)

where \({\mu }_{o}\) and \({\mu }_{s}\) are the average intensities of the source and segmented images, respectively. \({\sigma }_{o},\) \({\sigma }_{s}\) are the SD of the two images, respectively. \({\sigma }_{os}\) refers to the covariance between the original and segmented images, and \(b\), and \(a\) are two-fixed values of 0.003 and 0.001, respectively. This metric need to be maximized to enhance the segmented image quality

5.2.4 FSIM

FSIM [25] is another metric utilized to estimate the feature similarity between the segmented and source images. The mathematical model of this metric is found in [25].

5.3 Comparison Under the Fitness Value of the Green Level

In this section, the proposed algorithm will be compared with the other algorithms under the fitness values for the Green component. Each algorithm is executed 30 independent times for each threshold level on each test image, and the average of the fitness value under each threshold level is computed and presented in Figs.4a and b. Inspecting those figures clarifies the superiority of HJSO (Proposed algorithm) for all threshold levels separately. In addition, the average of the fitness values of all threshold levels is computed and depicted in Fig. 4c which affirms that HJSO occupies the first rank with an amount of 3639.57, JSO is the second-best one with a value of 3639.25, and SCA is the worst one. It is concluded that HJSO has a strong performance for segmenting the Green component of the color images of breast cancer under various threshold levels.

Fig. 4
figure 4

Comparison of the fitness values for the Green component among various algorithms

Full size image

5.4 Comparison Under the Fitness Value of the Blue Level

Herein, the Blue component for each test image will be segmented using various algorithms, and the fitness values estimated by those algorithms on each test image according to various threshold levels will be computed and exposed in Fig. 5a and b. From those figures, it is concluded that HJSO is better than all for all threshold levels. Furthermore, the average fitness values on all threshold levels are presented in Fig. 5c which shows the superiority of the proposed algorithm with an average fitness value of 2026.06 as the first rank, while SCA comes as the worst one with an amount of 2015.13. for this level, the proposed algorithm approved its efficiency for reaching the threshold values which segment the red component of various test images under all observed threshold levels.

Fig. 5
figure 5

Comparison of the fitness values for the Blue component among various algorithms

Full size image

5.5 Comparison Under the Fitness Value of the Red Level

Regarding the Red component, Fig. 5a–c are presented to show the performance of HJSO on each threshold level and totally on all threshold levels. Broadly speaking, each algorithm is running 30 independent runs, and the average of the fitness values for each threshold level on all test images is presented in Fig. 6a and b. According to those figures, HJSO occupies the first rank for all threshold levels separately, while SCA is the worst. Additionally, for each algorithm, Fig. 6c is presented to compute the average of the fitness values under all test images and threshold levels. From this figure, it is observed that HJSO is the best one with a value of 9407.25, JSO is the second-best one with a value of 9406.81, and SCA has the worst performance. From the previous analysis, the superiority of HJSO for three components (R, G, and B) is so clear which makes it a strong alternative to all the existing techniques for tackling the image segmentation problem of breast cancer images based on the multilevel thresholding technique.

Fig. 6
figure 6

Comparison of the fitness values for the Red component among various algorithms

Full size image

5.6 Comparison Under FSIM

In this section, another performance metric will be used to evaluate the quality of the segmented images by the proposed algorithm and the others rival. Broadly speaking, the best solution obtained by each algorithm in each independent run on each threshold level for each test image will be employed to build the segmented image which is compared with the source image using FSIM, and the average FSIM for each threshold level on all test image, and all threshold levels on all test images for each algorithm is presented in Fig. 7. From this figure, it is concluded that HJSO is the best, and SCA is the worst one. Based on this analysis, it is concluded that the quality of the segmented image produced by HJSO is better than those of the other algorithms.

Fig. 7
figure 7

Comparison of the average FSIM values among various algorithms

Full size image

5.7 Comparison Under PSNR

PSNR is another performance metric used to compute the error ratio between the segmented and original images. Therefore, the segmented images of various algorithms are compared with the source ones by the PSNR, and the average outcome on each threshold level for all test images is presented in Fig. 8a and b. Those figures appear the outperformance of the proposed algorithm in terms of PSNR, the weakness of the other algorithms for this metric, especially SCA which is the worst one for all threshold levels. Moreover, the average of all threshold levels on all test images which is produced by each algorithm is presented in Fig. 8c. From this figure, it is clear that HJSO is the best with a value of 34.78, and JSO is the second best one with a value of 34.4; meanwhile, SCA is the worst with an amount of 30.016.

Fig. 8
figure 8

Comparison of the average PSNR values among various algorithms

Full size image

5.8 Comparison Under SSIM

Finally, SSIM is employed to further measure the segmented images quality and the average of the obtained outcomes within 30 independent times on each threshold level for all test images, and on all threshold levels for all test images for each algorithm is presented in Fig. 9. It is obvious from this figure that HJSO is better for all threshold levels, where, from Fig. 9c, HJSO comes in the first rank with an amount of 0.9406, and JSO is the second-best one with an amount of 0.9373; meanwhile, SCA is the worst one with a value of 0.8924.

Fig. 9
figure 9

Comparison of the average SSIM values among various algorithms

Full size image

Consequently, HJSO is a robust alternative for segmenting the color images of breast cancer since it could reach segmented images with better quality.

5.9 Comparison Under Wilcoxon Rank-Sum Test

This test [26] is employed to measure the difference between the fitness values for each color component (Green, Blue, and Red) obtained by HJSO with those of the rival optimizers on all test images employed in our experiments. This statistical test produces the p-value of the two-sided Wilcoxon rank-sum test. Afterward, this value is compared with a significant level than 5%, and the Null hypothesis which says that there is no difference between the paired data is accepted if this value is less than the significant level; otherwise, the alternative hypothesis is accepted. The outcomes produced by applying this test on the fitness values for each component (Red, Green, and Blue) of HJSO and each rival algorithm are presented in Tables 1, 2 and 3. Inspecting these tables show that the p-value is less than 5% for most test cases, and this notifies that the outcomes of HJSO are significantly-different from those of the rival optimizers.

Table 1 Comparison on Blue component under Wilcoxon rank-sum test
Full size table
Table 2 Comparison on Green component under Wilcoxon rank-sum test
Full size table
Table 3 Comparison on Red component under Wilcoxon rank-sum test
Full size table

5.10 Comparison Under Boxplot

Figure 10 draws the fitness values for the various components (R, G, and B) of the test image: img1 on all threshold levels using the boxplot which analyzes the outcomes using a five-number summary: the minimum, the maximum, the sample median, and the first and third quartiles. Inspecting this figure shows the outperformance of HJSO for the five-number summary on all color components' overall threshold levels. Finally, it is concluded from all previous analyses that HJSO is a strong alternative to segment the breast cancer images instead of the existing techniques.

Fig. 10
figure 10figure 10

Comparison among algorithms using Boxplot of the fitness values of img1

Full size image

5.10.1 Comparison Under CPU Time and Standard Deviation

In this section, the performance of HJSO in terms of computational cost and stability is shown. In Fig. 11, the average of the SD on all threshold levels for all test images is shown. According to the results introduced in this figure, it is concluded that HJSO has better stability with a value of 13.037, while SCA is the one with the lowest stability. Regarding the CPU time that is displayed in Fig. 12, HJSO is almost competitive with the others, where it could occupy the sixth rank after EO, WOA, IMPA, MPA, and SCA, respectively.

Fig. 11
figure 11

Comparison in term of average SD

Full size image
Fig. 12
figure 12

Comparison in term of CPU time

Full size image

6 Conclusions and Future Work

This paper presents a new image segmentation algorithm based on the multilevel thresholding for breast cancer images. This algorithm is based on integrating the artificial jellyfish search algorithm with an effective improvement method to enhance its searchability for averting stuck into local minima, in addition to accelerating the convergence speed in the right direction of the near-optimal solution. This proposed algorithm called the hybrid jellyfish search algorithm (HJSO) is validated using 12 breast cancer images and compared with nine algorithms using several performance metrics to observe its effectiveness. The experimental findings show the effectiveness of HJSO for reaching better-segmented images. Our future work involves applying this algorithm for segmenting the brain MRI images, in addition to using HJSO with the convolutional neural network for brain tumor detection.