Electronic Journal of Differential Equations, Vol. 2020 (2020), No. 71, pp. 1–17. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu ASYMPTOTIC BEHAVIOR OF STOCHASTIC THREE-SPECIES PREDATOR-PREY SYSTEMS WITH WHITE AND LÉVY NOISE YIHAN ZHAO, YUANPEI XIA, ZHICHUN YANG Abstract. In this article, we propose a three-species prey-predator system with Holling II functional response and stochastic perturbations involving white noise and Lévy noise. Firstly, we study the existence and uniqueness of a global positive solution and stochastic ultimate boundedness. Then, we obtain sufficient conditions for stability, extinction, strongly persistence in the mean and stochastic permanence in the sense of probability for the stochastic system. The results show that both white noise and Lévy noise may change the asymptotic properties of the population system. Finally, some examples that chaotic dynamics can be influenced by stochastic noises. 1. Introduction The predator-prey models have attracted great attention because of their rich and complicated dynamical behaviors, in which functional response plays an im- portant role to determine dynamical behaviors such as stability, oscillation, bifurca- tion and even chaos (see [4]-[18]). In the past few decades, food chain models with Holling-type functional response have been widely studied by many researchers; see [3, 4, 5, 7, 18]. For instance, the famous Hastings and Powell’s model depicted a three-species food chain with the Holling II functional response [3] dx1(t) = x1(t)[1− x1(t)− a1x2(t) 1 + b1x1(t) ], dx2(t) = x2(t)[−r2 + a1x1(t) 1 + b1x1(t) − a2x3(t) 1 + b2x2(t) ], dx3(t) = x3(t)[−r3 + a2x2(t) 1 + b2x2(t) ], (1.1) where xi are the population densities of prey, middle predator and top predator [3, 7] respectively. A more general three-special predator-prey model with the Holling II 2010 Mathematics Subject Classification. 34D20, 92D25, 60H10. Key words and phrases. Predator-prey system; Lévy noise; white noise; extinction; stochastic permanence. c©2020 Texas State University. Submitted February 6, 2020. Published July 6, 2020. 1 2 Y. ZHAO, Y. XIA, Z. YANG EJDE-2020/71 functional response has the form dx1(t) dt = x1(t)[r1 − a11x1(t)− a12x2(t) 1 + b1x1(t) ], dx2(t) dt = x2(t)[−r2 + a21x1(t) 1 + b1x1(t) − a22x2(t)− a23x3(t) 1 + b2x2(t) ], dx3(t) dt = x3(t)[−r3 + a32x2(t) 1 + b2x2(t) − a33x3(t)], (1.2) where xi(t), i = 1, 2, 3 denote the population densities of prey, meso-predator and super-predator at time t respectively, r1 is intrinsic growth rate and ri > 0 (i = 2, 3) are the death rates, aii > 0 (i = 1, 2, 3) represent the intraspecies competition coefficients, a12 > 0 and a23 > 0 stand for the capture rates, a21 > 0 and a32 > 0 represent the efficiency of food conversion, and 1/bi (i = 1, 2) denote the half- saturation constant of meso-predator and super-predator respectively. As we known, in the real world, ecosystems are unavoidably subject to stochastic perturbations because of random fluctuation of the birth rates, death rates, carrying capacity and so on. In recent years, white noise driven by Brownian motion has been taken into consideration in the process of modeling [11], and the study of dynamical behaviors for stochastic population systems with white noise has become fascinating [6, 8, 10, 13]. Except for white noise, ecosystems may suffer sudden environmental perturbations such as earthquakes, hurricanes, floods and so on, which may cause jumps of population number and great influences for dynamical properties of the systems. So it is reasonable to introduce Lévy noise described by Lévy random processes into the systems. Stochastic population systems with Lévy noise have been extensively studied by some scholars in the last few years [17]. In most of ecosystems, the functional responses are linear, but linear ones have some limitations and may be unable to accurately describe various natural phenomena [12]. In fact, nonlinear function response has much richer dynamical behaviors than linear function response, and lots of critical properties are demonstrated only via nonlinear function response [3]. Therefore, it is interesting to further study the population systems with nonlinear functional response and stochastic perturbations such as white noise and Lévy noise. Motivated by the above discussions, we take white noise and colored Lévy noise into the model (1.2), and formulate the following hybrid stochastic three-species predator-prey system with the Holling II functional response dx1(t) = x1(t)[r1 − a11x1(t)− a12x2(t) 1 + b1x1(t) ]dt + σ1x1(t)dB1(t) + x1(t−) ∫ Y γ1(u)Ñ(dt, du), dx2(t) = x2(t)[−r2 + a21x1(t) 1 + b1x1(t) − a22x2(t)− a23x3(t) 1 + b2x2(t) ]dt + σ2x2(t)dB2(t) + x2(t−) ∫ Y γ2(u)Ñ(dt, du), dx3(t) = x3(t)[−r3 + a32x2(t) 1 + b2x2(t) − a33x3(t)]dt + σ3x3(t)dB3(t) + x3(t−) ∫ Y γ3(u)Ñ(dt, du), (1.3) EJDE-2020/71 ASYMPTOTIC BEHAVIOR OF STOCHASTIC SYSTEMS 3 where xi(t −) is the left limit of xi(t), Bi(t) is the standard Brownian motion defined on a complete probability space (Ω,F , {Ft}t≥0, P ) with a filtration {Ft}t≥0 satisfy- ing the usual conditions, σ2 i is the intensity of white noise, N is a Poisson counting measure with characteristic measure λ on a measurable subset Y of (0,+∞) with λ(Y) < +∞, Ñ(dt, du) = N(dt, du)−λ(du)dt is the compensated random measure, γi(u) > −1(u ∈ Y) are bounded functions (i = 1, 2, 3), and the meaning of other parameters are same with model (1.2). We shall investigate the dynamical behaviors such as well-posedness, bounded- ness, stability, extinction and persistence for the above stochastic system. The main contributions of this paper are listed as follows. Firstly, we formulate a three-species predator-prey system with the Holling II functional response and hybrid stochastic perturbations involving white noise and Lévy noise. Secondly, we discuss the extinc- tion and persistence in the mean and in the stochastic trajectory path. Lastly, we show that both white noise and Lévy noise have significant impacts on dynamical properties of the system. The remaining part of this paper is organized as follows. In Section 2, we give some preliminary results on system(1.3). In Section 3, we analyze the asymptotic behaviors of system(1.3). In Section 4, the theoretical results are illustrated by some examples. 2. Preliminaries Throughout this paper, we denote R3 + = {x = (x1, x2, x3)T ∈ R3 : xi > 0, i = 1, 2, 3} with the norm |x| = √ x21 + x22 + x23, and assume Bi(t)(i = 1, 2, 3) and N are independent. For convenience, we define the following notations βi = σ2 i 2 − ∫ Y ln(1 + γi(u))λ(du), i = 1, 2, 3; Qi(t) = ∫ t 0 ∫ Y ln(1 + γi(u))Ñ(ds, du), i = 1, 2, 3; f(t) = 1 t ∫ t 0 f(s)ds, f∗ = lim sup t→∞ f(t), f∗ = lim inf t→∞ f(t). To obtain the main results, we introduce the following assumptions. (A1) There is a positive constant c such that ∫ Y[ln(1 + γi(u))]2λ(du) < c, i = 1, 2, 3; (A2) For any t ≥ 0, supt≥0 ∫ t 0 ∫ Y e s−t[γi(u) − ln(1 + γi(u))]λ(du)ds < ∞, i = 1, 2, 3; (A3) B = min{r1 − β1,−r2 − β2,−r3 − β3} > 0. Definition 2.1 ([8]). The solutions x(t) of system (1.3) are called stochastically ultimately bounded if for each ε ∈ (0, 1), there is a positive constant H := H(ε) such that x(t) with any initial value x(0) ∈ R3 + has the property that lim sup t→∞ P (|x(t)| > H) < ε. Definition 2.2 ([1]). The system (1.3) is said to be stochastically permanent if for any ε ∈ (0, 1), there exist constants δ1 = δ1(ε) > 0 and δ2 = δ2(ε) > 0 such that lim inf t→∞ P{|x(t)| ≥ δ1} ≥ 1− ε, lim inf t→∞ P{|x(t)| ≤ δ2} ≥ 1− ε. 4 Y. ZHAO, Y. XIA, Z. YANG EJDE-2020/71 Definition 2.3 ([20]). Let x(t) = (x1(t), x2(t), x3(t))T ∈ R3 + be a solution to system (1.3), then for i = 1, 2, 3, (1) the population xi(t) becomes extinct if limt→∞ xi(t) = 0 a.s.; (2) the population xi(t) becomes strongly persistent in the mean if lim inft→∞ 1 t ∫ t 0 xi(s)ds > 0 a.s.; (3) the population xi(t) is said to be stable in the mean if limt→∞ 1 t ∫ t 0 xi(s)ds = c > 0 a.s. From the above definitions we can find that the stability in the mean must be strongly persistence in the mean, stochastic permanence implies stochastically ultimate boundedness, and stochastically ultimate boundedness means the solution will be ultimately bounded with large probability, stochastic permanence is the strongest property, indicating the eternal existence of the population. To ensure that the system (1.3) has biological significance, we give well-posedness for the solution of system (1.3). Lemma 2.4. For any given initial value x(0) ∈ R3 +, the system (1.3) has a unique global solution x(t) ∈ R3 + for all t ≥ 0 almost surely. Proof. First, we prove that (1.3) has a unique positive local solution. For t ≥ 0, we consider the system du1(t) = (r1 − β1 − a11eu1(t) − a12e u2(t) 1 + b1eu1(t) )dt+ σ1dB1(t) + ∫ Y ln(1 + γ1(u))Ñ(dt, du), du2(t) = (−r2 − β2 + a21e u1(t) 1 + b1eu1(t) − a22eu2(t) − a23e u3(t) 1 + b2eu2(t) )dt+ σ2dB2(t) + ∫ Y ln(1 + γ2(u))Ñ(dt, du), du3(t) = (−r3 − β3 + a32e u2(t) 1 + b2eu2(t) − a33eu3(t))dt+ σ3dB3(t) + ∫ Y ln(1 + γ3(u))Ñ(dt, du), (2.1) with initial value (u1(0), u2(0), u3(0))T = (lnx1(0), lnx2(0), lnx3(0))T. Clearly, (2.1) satisfies local Lipschitz condition, there is a unique local solution (u1(t), u2(t), u3(t))T on [0, τe), where τe is the explosion time. By Itô’s formula, (x1(t), x2(t), x3(t))T = (eu1(t), eu2(t), eu3(t))T is the unique positive local solution to the system (1.3) with initial value xi(0) > 0. Then, we will use the comparison theorem to prove x(t) is global, i.e., τe = +∞. Considering the following stochastic EJDE-2020/71 ASYMPTOTIC BEHAVIOR OF STOCHASTIC SYSTEMS 5 system dy1(t) = y1(t)[r1 − a11y1(t)]dt+ σ1y1(t)dB1(t) + y1(t−) ∫ Y γ1(u)Ñ(dt, du), dy2(t) = y2(t)[−r2 + a21 b1 − a22y2(t)]dt+ σ2y2(t)dB2(t) + y2(t−) ∫ Y γ2(u)Ñ(dt, du), dy3(t) = y3(t)[−r3 + a32 b2 − a33y3(t)]dt+ σ3y3(t)dB3(t) + y3(t−) ∫ Y γ3(u)Ñ(dt, du), (2.2) with initial value yi(0) = xi(0) > 0, i = 1, 2, 3. By the comparison theorem for stochastic differential equation, we obtain for t ∈ [0, τe), xi(t) ≤ yi(t), a.s., i = 1, 2, 3. According to [1, Theorem 2.1], the system (2.2) has a unique global solution y1(t), y2(t) and y3(t) for t ≥ 0. Hence we have τe = +∞. � The following lemma gives ultimate boundedness for the system (1.3). Lemma 2.5. For any initial value x(0) ∈ R3 + and p > 0, there is a constant K such that the solution x(t) of system (1.3) satisfies lim supt→∞E|x(t)|p ≤ K, and is stochastically ultimately bounded. Proof. Define a Lyapunov function V (x) = xp1 + xp2 + xp3, p > 0. Applying the generalized Itô’s formula, we obtain E(etV (x)) = V (x(0)) + E ∫ t 0 es[V (x(s)) + LV (x(s))]ds, where LV (x) =− a11pxp+1 1 + xp1(pr1 + p(p− 1) 2 σ2 1 + ∫ Y [(1 + γ1(u))p − 1]λ(du))− a12px p 1x2 1 + b1x1 − a22pxp+1 2 + xp2(−pr2 + p(p− 1) 2 σ2 2 + ∫ Y [(1 + γ2(u))p − 1]λ(du)) + a21px p 2x1 1 + b1x1 − a23px p 2x3 1 + b2x2 − a33pxp+1 3 + xp3(−pr3 + p(p− 1) 2 σ2 3 + ∫ Y [(1 + γ3(u))p − 1]λ(du)) + a32px p 3x2 1 + b2x2 . From aij > 0, we can deduce that there exists a constant K(p) > 0 such that V (x) + LV (x) ≤− a11pxp+1 1 + xp1(1 + pr1 + p(p− 1) 2 σ2 1 + ∫ Y [(1 + γ1(u))p − 1]λ(du))− a22pxp+1 2 + xp2(1− pr2 + a21p b1 + p(p− 1) 2 σ2 2 6 Y. ZHAO, Y. XIA, Z. YANG EJDE-2020/71 + ∫ Y [(1 + γ2(u))p − 1]λ(du))− a33pxp+1 3 + xp3(1− pr3 + a32p b2 + p(p− 1) 2 σ2 3 + ∫ Y [(1 + γ3(u))p − 1]λ(du)) ≤ K(p). Hence, E(etV (x1(t), x2(t), x3(t))) ≤ V (x1(0), x2(0), x3(0)) +K(p)(et − 1). Then lim sup t→∞ E(xp1(t) + xp2(t) + xp3(t)) ≤ K(p). Since n(1− p 2 )∧0|x|p ≤ ∑n i=1 x p i ≤ n(1− p 2 )∨0|x|p, for all p > 0, x ∈ Rn+, we can find a constant K = K(p) 3(1− p 2 )∧0 > 0, this yields that lim supt→∞E|x(t)|p ≤ K. And combining with Chebyshev inequality, we can derive that the solution of (1.3) is stochastically ultimately bounded. The proof is complete. � The following lemma gives the pathwise estimation of system state. Lemma 2.6. Let (A2) hold, for any initial value x(0) ∈ R3 +, the solution x(t) of (1.3) has the property that lim supt→∞ ln xi(t) t ≤ 0 a.s., i = 1, 2, 3. Proof. Using the same method as in [1, Lemma 4.4] with (A2), we obtain that the solution (y1(t), y2(t), y3(t)) of (2.2) satisfies lim supt→∞ ln yi(t) ln t ≤ 1 a.s., i = 1, 2, 3. Combining this and the limit limt→∞ ln t t = 0, we have lim supt→∞ ln yi(t) t ≤ 0 a.s., i = 1, 2, 3. Then by the inequality xi(t) ≤ yi(t), t ≥ 0, i = 1, 2, 3, we can gain the desired result. � Lastly, we also introduce the following basic lemma given in [9]. Lemma 2.7. Let (A1) hold and Z(t) ∈ C(Ω× [0,+∞), R+). (1) If there exist two positive constants T and λ0 such that for all t ≥ T , lnZ(t) ≤ λt− λ0 ∫ t 0 Z(s)ds+ n∑ i=1 σiBi(t) + n∑ i=1 λiQi(t), where λ,σi,λi are constants,then Z ∗ ≤ λ λ0 quada.s., ifλ ≥ 0; lim t→∞ Z(t) = 0 a.s., if λ < 0. (2) If there exist there positive constants T , λ and λ0 such that for all t ≥ T , lnZ(t) ≥ λt− λ0 ∫ t 0 Z(s)ds+ n∑ i=1 σiBi(t) + n∑ i=1 λiQi(t), then Z∗ ≥ λ λ0 a.s. EJDE-2020/71 ASYMPTOTIC BEHAVIOR OF STOCHASTIC SYSTEMS 7 3. Asymptotic behavior of system (1.3) In this section, we shall investigate the asymptotic behaviors such as extinction, persistence and stability for system (1.3). Firstly, we give main results on extinction, strongly persistence in the mean and stability in the mean. Theorem 3.1. Let (A1) and (A2) hold. We have the following statements for system (1.3). (i) If r1 − β1 < 0 and −ri − βi < 0, i = 2, 3, then all populations become extinct. (ii) If r1 − β1 > 0, −r2 − β2 + a21 b1 < 0 and −r3 − β3 < 0, then the populations x2(t), x3(t) become extinct and x1(t) is stable in the mean, namely, lim t→∞ x1(t) = r1 − β1 a11 a.s. (iii) If −r3 − β3 + a32 b2 < 0, then population x3(t) becomes extinct. Moreover, if r1 − β1 > max{0, a12 −r2−β2+ a21 b1 a22 } and −r2 − β2 > 0, then the populations x1(t), x2(t) are strongly persistent in the mean, that is, r1 − β1 − a12 −r2−β2+ a21 b1 a22 a11 ≤ x1(t)∗ ≤ x1(t) ∗ ≤ r1 − β1 a11 a.s., −r2 − β2 a22 ≤ x2(t)∗ ≤ x2(t) ∗ ≤ −r2 − β2 + a21 a22 a.s. (iv) If−r3 − β3 > 0, then the population variable x3(t) satisfies −r3 − β3 a33 ≤ x3(t)∗ ≤ x3(t) ∗ ≤ −r3 − β3 + a32 a33 a.s. Furthermore, if r1 − β1 > max { 0, a12 −r2−β2+ a21 b1 a22 } and −r2 − β2 > max { 0, a23 −r3−β3+ a32 b2 a33 } , then r1 − β1 − a12 −r2−β2+ a21 b1 a22 a11 ≤ x1(t)∗ ≤ x1(t) ∗ ≤ r1 − β1 a11 a.s., −r2 − β2 − a23 −r3−β3+ a32 b2 a33 a22 ≤ x2(t)∗ ≤ x2(t) ∗ ≤ −r2 − β2 + a21 a22 a.s. That is, all populations are strongly persistent in the mean. Proof. Applying generalized Itô’s formula to lnx1(t) leads to d lnx1(t) = (r1 − β1 − a11x1(t)− a12x2(t) 1 + b1x1(t) )dt+ σ1dB1(t) + ∫ Y ln(1 + γ1(u))Ñ(dt, du). Integrating from 0 to t and then dividing it by t yields ln(x1(t)/x1(0)) t = r1 − β1 − a11x1(t)− a12 x2(t) 1 + b1x1(t) + σ1B1(t) t + Q1(t) t . (3.1) 8 Y. ZHAO, Y. XIA, Z. YANG EJDE-2020/71 Similarly, ln(x2(t)/x2(0)) t = −r2 − β2 + a21 x1(t) 1 + b1x1(t) − a22x2(t)− a23 x3(t) 1 + b2x2(t) + σ2B2(t) t + Q2(t) t , (3.2) ln(x3(t)/x3(0)) t = −r3 − β3 + a32 x2(t) 1 + b2x2(t) − a33x3(t) + σ3B3(t) t + Q3(t) t . (3.3) Firstly, we shall prove the conclusion in case (i). By (3.1), ln(x1(t)/x1(0)) t ≤ r1 − β1 − a11x1(t) + σ1B1(t) t + Q1(t) t . Note that r1 − β1 < 0, hence by case (1) in Lemma 2.7, lim t→∞ x1(t) = 0 a.s. Thus we have that | x1(t) 1 + b1x1(t) | ≤ |x1(t)| < ε, for sufficiently large t, where 0 < ε < β2+r2 a21 . Then for(3.2), we obtain ln(x2(t)/x2(0)) t ≤ −r2 − β2 + a21ε− a22x2(t) + σ2B2(t) t + Q2(t) t . Note that −r2 − β2 < 0 and 0 < ε < β2+r2 a21 , hence by case (1) in Lemma 2.7, lim t→∞ x2(t) = 0 a.s. Similarly, applying this to (3.3), we have lim t→∞ x3(t) = 0 a.s. Secondly, we will give the proof of case (ii). From (3.2), ln(x2(t)/x2(0)) t ≤ −r2 − β2 + a21 b1 − a22x2(t) + σ2B2(t) t + Q2(t) t . Since −r2 − β2 + a21 b1 < 0, by case (1) in Lemma 2.7, lim t→∞ x2(t) = 0 a.s. The proof of limt→∞ x3(t) = 0 a.s. is the same with that in (i), hence the details are omitted. For (3.1), we obtain ln(x1(t)/x1(0)) t ≤ r1 − β1 − a11x1(t) + σ1B1(t) t + Q1(t) t . By case (1) in Lemma 2.7, we deduce that x1(t) ∗ ≤ r1 − β1 a11 a.s. EJDE-2020/71 ASYMPTOTIC BEHAVIOR OF STOCHASTIC SYSTEMS 9 From limt→∞ x2(t) = 0, we have | x2(t) 1+b1x1(t) | ≤ |x2(t)| < ε, for sufficiently large t, where 0 < ε < r1−β1 a12 . Then for (3.1), we obtain ln(x1(t)/x1(0)) t ≥ r1 − β1 − a12ε− a11x1(t) + σ1B1(t) t + Q1(t) t . From case (2) in Lemma 2.7, we deduce that x1(t)∗ ≥ r1 − β1 − a12ε a11 a.s. In view of the arbitrariness of ε, we obtain lim t→∞ x1(t) = r1 − β1 a11 a.s. Thirdly, we shall prove the conclusion in case (iii). By (3.3), ln(x3(t)/x3(0)) t ≤ −r3 − β3 + a32 b2 − a33x3(t) + σ3B3(t) t + Q3(t) t . Note that −r3 − β3 + a32 b2 < 0, hence by case (1) in Lemma 2.7, lim t→∞ x3(t) = 0 a.s. According to (3.2), ln(x2(t)/x2(0)) t ≤ −r2 − β2 + a21 b1 − a22x2(t) + σ2B2(t) t + Q2(t) t . By case (1) in Lemma 2.7, we deduce that x2(t) ∗ ≤ −r2 − β2 + a21 b1 a22 a.s. From limt→∞ x3(t) = 0, we have that | x3(t) 1+b2x2(t) | ≤ |x3(t)| < ε, for sufficiently large t, where 0 < ε < −r2−β2 a23 . Then for (3.2), we obtain ln(x2(t)/x2(0)) t ≥ −r2 − β2 − a22x2(t)− a23ε+ σ2B2(t) t + Q2(t) t . Using case (2) in Lemma 2.7, we deduce that x2(t)∗ ≥ −r2 − β2 − a23ε a22 a.s. Therefore, in view of the arbitrariness of ε, we obtain −r2 − β2 a22 ≤ x2(t)∗ ≤ x2(t) ∗ ≤ −r2 − β2 + a21 b1 a22 a.s. (3.4) Through (3.1), ln(x1(t)/x1(0)) t ≤ r1 − β1 − a11x1(t) + σ1B1(t) t + Q1(t) t . It follows from case (1) in Lemma 2.7 that x1(t) ∗ ≤ r1 − β1 a11 a.s. Combining inequality (3.4) and Lemma 2.6, from (3.1) we deduce that a11x1(t)∗ ≥ lim inf t→∞ { r1 − β1 − ln(x1(t)/x1(0)) t − a12x2(t) + σ1B1(t) t + Q1(t) t } 10 Y. ZHAO, Y. XIA, Z. YANG EJDE-2020/71 ≥ r1 − β1 − lim sup t→∞ lnx1(t) t − a12x2(t) ∗ ≥ r1 − β1 − a12 −r2 − β2 + a21 b1 a22 . So, r1 − β1 − a12 −r2−β2+ a21 b1 a22 a11 ≤ x1(t)∗ ≤ x1(t) ∗ ≤ r1 − β1 a11 a.s. Finally, we shall prove case (iv). From (3.3), ln(x3(t)/x3(0)) t ≤ −r3 − β3 + a32 b2 − a33x3(t) + σ3B3(t) t + Q3(t) t . By case (1) in Lemma 2.7, we deduce that x3(t) ∗ ≤ −r3 − β3 + a32 b2 a33 a.s. Again from (3.3), ln(x3(t)/x3(0)) t ≥ −r3 − β3 − a33x3(t) + σ3B3(t) t + Q3(t) t . Using case (2) in Lemma 2.7, we have x3(t)∗ ≥ −r3 − β3 a33 a.s. Therefore, −r3 − β3 a33 ≤ x3(t)∗ ≤ x3(t) ∗ ≤ −r3 − β3 + a32 b2 a33 a.s. (3.5) Through (3.2), ln(x2(t)/x2(0)) t ≤ −r2 − β2 + a21 b1 − a22x2(t) + σ2B2(t) t + Q2(t) t . According to case (1) in Lemma 2.7, we have x2(t) ∗ ≤ −r2 − β2 + a21 b1 a22 a.s. Combining inequality (3.5), (3.2) and Lemma 2.6, we can deduce that a22x2(t)∗ ≥ lim inf t→∞ { − r2 − β2 − ln(x2(t)/x2(0)) t − a23x3(t) + σ2B2(t) t + Q2(t) t } ≥ −r2 − β2 − lim sup t→∞ lnx2(t) t − a23x3(t) ∗ ≥ −r2 − β2 − a23 −r3 − β3 + a32 b2 a33 . Therefore, −r2 − β2 − a23 −r3−β3+ a32 b2 a33 a22 ≤ x2(t)∗ ≤ x2(t) ∗ ≤ −r2 − β2 + a21 b1 a22 a.s. The estimation for the ultimate infimum and ultimate supremum of x1(t) is similar with one in case (iii), hence it is omitted. � EJDE-2020/71 ASYMPTOTIC BEHAVIOR OF STOCHASTIC SYSTEMS 11 Remark 3.2. When a22 = a33 = 0, we easily check the conclusion in the case (i) and case (ii) of Theorem 3.1 still holds. This means that the conclusion can be applied to the model (1.1) with stochastic effects and see the case from Example 4.1 later. Furthermore, we will give a condition weaker than the one given in the above case (iv) to discuss the stochastic permanence in the sense of probability for the stochastic system (1.3). Theorem 3.3. If (A3) holds, then system (1.3) is stochastically permanent. Proof. We define a Lyapunov function V (x(t)) := 1 x1(t)+x2(t)+x3(t) , where x(t) = (x1(t), x2(t), x3(t))T is any positive solution of (1.3). By generalized Itô’s formula, we obtain dV = { − V 2(x)[x1(r1 − a11x1 − a12 1 + b1x1 x2) + x2(−r2 + a21 1 + b1x1 x1 − a22x2 − a23 1 + b2x2 x3) + x3(−r3 + a32 1 + b2x2 x2 − a33x3)] + V 3(x)( 3∑ i=1 σixi) 2 + ∫ Y ( 1∑3 i=1 xi(1 + γi) − V (x))λ(du) } dt− V 2(x) 3∑ i=1 σixidBi(t) + ∫ Y ( 1∑3 i=1 xi(1 + γi) − V )Ñ(dt, du). Note that lim θ→0+ {maxi=1,2,3 σ 2 i 2 θ + ∫ Y [ 1 θmini=1,2,3(1 + γi(u))θ − 1 θ ]λ(du) } = ∫ Y ln 1 mini=1,2,3(1 + γi(u)) λ(du) = − ∫ Y ln[ min i=1,2,3 (1 + γi(u))]λ(du). By (A3), we can find a sufficiently small θ > 0 such that min r1 − maxσ2 1 2 (1 + θ)− ∫ Y [ 1 θmin(1 + γ1(u))θ − 1 θ ]λ(du) > 0, −max i=2,3 ri − maxi=2,3 σ 2 i 2 (1 + θ)− ∫ Y [ 1 θmini=2,3(1 + γi(u))θ − 1 θ ]λ(du) > 0. Then there is a small positive η such that min r1 − maxσ2 1 2 (1 + θ)− ∫ Y [ 1 θmin(1 + γ1(u))θ − 1 θ ]λ(du) > η θ , −max i=2,3 ri − maxi=2,3 σ 2 i 2 (1 + θ)− ∫ Y [ 1 θmini=2,3(1 + γi(u))θ − 1 θ ]λ(du) > η θ . (3.6) 12 Y. ZHAO, Y. XIA, Z. YANG EJDE-2020/71 We define another Lyapunov function, U(x) = eηtV θ(x). Then dU(x) = eηt { F (V (x))dt− θV θ−1(x)V 2(x) 3∑ i=1 σixidBi(t) + ∫ Y [( 1∑3 i=1 xi(1 + γi) )θ − V θ(x)]Ñ(dt, du) } , (3.7) where F (V (x)) =ηV θ(x)− θV θ−1(x)V 2(x)[x1(r1 − a11x1 − a12 1 + b1x1 x2) + x2(−r2 + a21 1 + b1x1 x1 − a22x2 − a23 1 + b2x2 x3) + x3(−r3 + a32 1 + b2x2 x2 − a33x3)] + θV θ−1(x)V 3(x) ( 3∑ i=1 σixi )2 + θ(θ − 1) 2 V θ−2(x)V 4(x)( 3∑ i=1 σixi) 2 + ∫ Y [( 1∑3 i=1 xi(1 + γi) )θ − V θ(x)]λ(du). By (A3), we see that −ri ≥ B + σ2 i 2 − ∫ Y ln(1 + γi(u))λ(du) (i = 2, 3) and r1 ≥ B + σ2 1 2 − ∫ Y ln(1 + γ1(u))λ(du). Thus, we can find constants θ and η satisfying (3.6) such that Bθ − θ2 2 max i=1,2,3 σ2 i − ∫ Y [ 1 mini=1,2,3(1 + γi(u))θ − 1 + θ ln min i=1,2,3 (1 + γi(u)) ] λ(du) > η > 0. (3.8) Accordingly, F (V (x)) ≤ ηV θ(x)− θV θ−1(x)V (x) 3∑ i=1 xi(B − ∫ Y ln(1 + γi(u))λ(du))V (x) − θV θ−1(x)V (x) 3∑ i=1 xi σ2 i 2 V (x) + θV θ−1(x)V 2(x) 3∑ i=1 aiix 2 i + θV θ−1(x)V 2(x)(|a12 − a21|x1x2 + |a23 − a32|x2x3) + θV θ−1(x)V (x) ( 3∑ i=1 σixi )2 V 2(x) + θ(θ − 1) 2 V θ−2(x)V 2(x) ( 3∑ i=1 σixi )2 V 2(x) + ∫ Y [( 1∑3 i=1 xi(1 + γi) )θ − V θ]λ(du) := O(V θ(x))V θ(x) +G(V (x)), (3.9) EJDE-2020/71 ASYMPTOTIC BEHAVIOR OF STOCHASTIC SYSTEMS 13 where limV→+∞ G(V (x)) V θ(x) = 0. Since 0 ≤ V 2(x) ∑3 i=1 aiix 2 i ≤ maxi=1,2,3 aii, 0 ≤ V 2(x)(x1x2 + x2x3) ≤ 1 2 , we obtain O(V θ(x)) =η − θ 3∑ i=1 xi(B − ∫ Y ln(1 + γi(u))λ(du))V (x)− θ 3∑ i=1 xi σ2 i 2 V (x) + θ(θ + 1) 2 ( 3∑ i=1 σixi) 2V 2(x) + ∫ Y [( x1 + x2 + x3∑3 i=1 xi(1 + γi) )θ − 1]λ(du). In view of Jensen’s inequality and (3.8), we deduce that O(V θ) ≤ η −Bθ + θ ∫ Y 3∑ i=1 xiln(1 + γi(u))V λ(du) + θ2 2 ( 3∑ i=1 σixi )2 V 2(x) + ∫ Y [( x1 + x2 + x3∑3 i=1 xi(1 + γi) )θ − 1 ] λ(du) ≤ η −Bθ + θ2 2 max i=1,2,3 σ2 i + ∫ Y ∞∑ n=2 θn n! (ln x1 + x2 + x3∑3 i=1 xi(1 + γi) )nλ(du) ≤ η −Bθ + θ2 2 max i=1,2,3 σ2 i + ∫ Y [ 1 mini=1,2,3(1 + γi(u))θ − 1 + θ ln min i=1,2,3 (1 + γi(u)) ] λ(du) < 0. (3.10) From (3.7), (3.9) and (3.10), there exists H(θ) > 0 such that E[eηtV θ(x(t))]− V θx((0)) ≤ E ∫ t 0 eηsH(θ)ds = H(θ) η (eηt − 1). So we have lim sup t→∞ E(V θ(x(t))) ≤ H(θ) η . In light of 1 |x(t)|θ ≤ 2 θ 2 V θ(t), we obtain lim sup t→∞ E( 1 |x(t)|θ ) ≤ 2θ/2 H(θ) η . Based on Chebyshev’s inequality, for any ε > 0, there exists H = √ 2 2 ( ηε H(θ) ) 1/θ > 0 such that lim sup t→∞ P{|x(t)| < H} = lim sup t→∞ P { 1 |x(t)| > 1 H } ≤ Hθ lim sup t→∞ E ( 1 |x(t)|θ ) ≤ ε. Therefore, lim inf t→∞ P{|x(t)| ≥ H} ≥ 1− ε. Combining this and Lemma 2.5, it follows that (1.3) is stochastically permanent. � Remark 3.4. According to Theorems 3.1 and 3.3, the dynamical behavior of sys- tem (1.3) may be changed by stochastic perturbations. In fact, when the determinis- tic system (1.2) is persistent, the species in the stochastic system (1.3) always trend to extinction if we take large enough white noise parameters σ2 i or large enough Lévy noise parameters γi(·) such that r1−β1,−ri−βi < 0, i = 2, 3. Whereas, when the species in the deterministic system (1.2) becomes extinct, the stochastic system 14 Y. ZHAO, Y. XIA, Z. YANG EJDE-2020/71 (1.3) will become persistent by handling Lévy noise satisfying B > 0. However, our results may be unable to handle white noises to change the extinction for the de- terministic system (1.2) into the persistence for the stochastic system (1.3) because σi ≥ 0. That is, there should be different effects on dynamics of (1.3) between white noises and Lévy noise. 4. Examples and conclusions In this section, we shall give some numerical examples to illustrate our theoretical results, and show the effects of white noise and Lévy noise to dynamical properties of the system. Example 4.1. Consider the following stochastic system based on the Hastings and Powell’s model (1.1) dx1(t) = x1(t)[(1− x1(t)− a1x2(t) 1 + b1x1(t) )dt] + σ1x1dB1(t) + x1(t−) ∫ Y γ1(u)Ñ(dt, du), dx2(t) = x2(t)[(−r2 + a1x1(t) 1 + b1x1(t) − a2x3(t) 1 + b2x2(t) )dt] + σ2x2dB2(t) + x2(t−) ∫ Y γ2(u)Ñ(dt, du), dx3(t) = x3(t)[(−r3 + a2x2(t) 1 + b2x2(t) )dt] + σ3x3dB3(t) + x3(t−) ∫ Y γ3(u)Ñ(dt, du), which is a special example of system (1.3) with r1 = a11 = 1, a22 = a33 = 0, a12 = a21 = a1, a23 = a32 = a2. Take a1 = 5, a2 = 0.1, b1 = 3, b2 = 2, r2 = 0.4, r3 = 0.01. According to [7], the deterministic Hastings and Powell’s model exhibits chaotic dynamics in long-term behavior (i.e., σi = γi(·) = 0, i = 1, 2, 3). According to Theorem 3.1 and Remark 3.2, we shall show that the chaotic behaviors can be eliminated under certain stochastic perturbations by choosing different values of σi, γi and λ(Y) = 1. Case I. Let γi(u) = 0, i = 1, 2, 3, σ1 = 2, σ2 = 1, σ3 = 0.5, then r1 − β1 = −1, −r2 − β2 = −0.9, −r3 − β3 = −0.135. From the case (i) in Theorem 3.1, we have all populations become extinct. Case II. Let γi(u) = 0, i = 1, 2, 3, σ1 = 1, σ2 = 2, σ3 = 0.5, then r1 − β1 = 0.5, −r2 − β2 + a21 b1 = −0.7333, −r3 − β3 = −0.135. By the case (ii) in Theorem 3.1, the populations x2(t), x3(t) become extinct, x1(t) is stable in the mean and limt→∞ x1(t) = 0.5 a.s. Case III. Let σi = 0, i = 1, 2, 3, γ1(u) = −0.8, γ2(u) = −0.4, γ3(u) = −0.3, then r1 − β1 = −0.6094, −r2 − β2 = −0.9108, −r3 − β3 = −0.3667. It follows from the case (i) in Theorem 3.1 that all populations go to extinction. Case IV. Let σi = 0, i = 1, 2, 3, γ1(u) = 0.2, γ2(u) = −0.8, γ3(u) = −0.6, then r1 − β1 = 1.1823, −r2 − β2 + a21 b1 = −0.3428, −r3 − β3 = −0.9263. From case (ii) EJDE-2020/71 ASYMPTOTIC BEHAVIOR OF STOCHASTIC SYSTEMS 15 in Theorem 3.1, we see that the populations x2(t), x3(t) become extinct, x1(t) is stable in the mean and limt→∞ x1(t) = 1.1823 a.s. The above cases illustrate also that the chaotic dynamics can be suppressed by either white noises or Lévy noises. Example 4.2. Consider the following stochastic system with white noises or Lévy noises dx1(t) = x1(t)[0.8− 0.4x1(t)− 0.3x2(t) 1 + 3x1(t) ]dt + σ1x1(t)dB1(t) + x1(t−) ∫ Y γ1(u)Ñ(dt, du), dx2(t) = x2(t)[−0.5 + 0.2x1(t) 1 + 3x1(t) − 0.4x2(t)− 0.2x3(t) 1 + 2x2(t) ]dt + σ2x2(t)dB2(t) + x2(t−) ∫ Y γ2(u)Ñ(dt, du), dx3(t) = x3(t)[−0.3 + 0.1x2(t) 1 + 2x2(t) − 0.4x3(t)]dt + σ3x3(t)dB3(t) + x3(t−) ∫ Y γ3(u)Ñ(dt, du). (4.1) In the following, we take different values of white noise and Lévy noise to show that the system (4.1) has different dynamical behaviors. Case I. Let σi = 0.2, γi(u) = 0, i = 1, 2, 3, then r1 − β1 = 0.78, −r2 − β2 + a21 b1 = −0.4533, −r3 − β3 = −0.32. From the second statement of Theorem 3.1, it follows that the meso-predator and super-predator become extinct while the prey is stable in the mean, and limt→∞ x1(t) = 1.95 a.s. Case II. Let σi = 0, i = 1, 2, 3, γ1(u) = −0.6, γ2(u) = −0.4, γ3(u) = −0.2, then r1 − β1 = −0.1163, −r2 − β2 = −1.0108, −r3 − β3 = −0.5231. The first statement of Theorem 3.1 exhibit that all populations go to extinction. Case III. Let γ1(u) = 0.3, γ2(u) = 0.8, γ3(u) = 0.4, then r1 − β1 = 1.0624, −r2 − β2 = 0.0878, −r3 − β3 = 0.0365, r1 − β1 − a12 −r2−β2+ a21 b1 a22 = 0.9466, −r2 − β2 − a23 −r3−β3+ a32 b2 a33 = 0.0446. The fourth statement of Theorem 3.1 show that all populations are strongly persistent in the mean. Case IV. Let σi = 0.2, i = 1, 2, 3, γ1(u) = −0.6, γ2(u) = −0.4, γ3(u) = −0.2, then r1 − β1 = −0.1363, −r2 − β2 = −1.0308, −r3 − β3 = −0.5431. From the first statement of Theorem 3.1, we can see that all populations go to extinction. Case V. Let σi = 0.2, i = 1, 2, 3, γ1(u) = 0.6, γ2(u) = −0.4, γ3(u) = −0.2, then r1 − β1 = 1.25, −r2 − β2 + a21 b1 = −0.9642, −r3 − β3 = −0.5431. The second statement of Theorem 3.1 tells us that the populations x2(t) and x3(t) become extinct and the population x1(t) is stable in the mean, and limt→∞ x1(t) = 3.125 a.s. Case VI. Let σi = 0.2, i = 1, 2, 3, γ1(u) = 0.3, γ2(u) = 0.8, γ3(u) = −0.2, then r1 − β1 = 1.0424, −r2 − β2 = 0.0678, −r3 − β3 + a32 b2 = −0.4931, r1 − β1 − a12 −r2−β2+ a21 b1 a22 = 0.9416. From the third statement of Theorem 3.1, it follows that the population x3(t) becomes extinct and the populations x1(t) and x2(t) are strongly persistent in the mean. 16 Y. ZHAO, Y. XIA, Z. YANG EJDE-2020/71 Case VII. Let σi = 0.2, i = 1, 2, 3, γ1(u) = 0.3, γ2(u) = 0.8, γ3(u) = 0.4, then r1−β1 = 1.0424, −r2−β2 = 0.0678, −r3−β3 = 0.0165, r1−β1−a12 −r2−β2+ a21 b1 a22 = 0.9416, −r2 − β2 − a23 −r3−β3+ a32 b2 a33 = 0.0346. The fourth statement of Theorem 3.1 exhibit that all populations are strongly persistent in the mean. Case VIII. Let σi = 0.2, i = 1, 2, 3, γ1(u) = −0.5, γ2(u) = 0.7, γ3(u) = 0.4, then r1 − β1 = 0.0869, −r2 − β2 = 0.0106, −r3 − β3 = 0.0165. From Theorem 3.3 we can see that the system (4.1) is stochastically permanent. From the above cases, we can switch dynamical behaviors between the extinction and the permanence by handling the parameters of Lévy noises for the stochastic system. We can also handle the parameters of white noises to change permanence into extinction, but our results are invalid to switch the dynamical behaviors from extinction to permanence by utilizing white noises. This may be because there are have different impacts on dynamical properties of the system between white noises and Lévy noises. Conclusion. This paper formulated a Holling-II type three-species prey-predator system with white noise and Lévy noise. First of all, we showed that the system admits a unique global positive solution, and discuss stochastic ultimate bounded- ness of the solution. Next we obtained sufficient conditions for extinction, strongly persistence in the mean and stability in the mean of the population and stochastic permanence of the system. Finally, our theoretical analysis reveals that dynamical behaviors of the system are closely related to stochastic noises. That is, under sto- chastic perturbations the extinct species can become persistent and the persistent species can go to extinction, and there are different effects on dynamical properties between white noises and Lévy noises for the stochastic system. In addition, we found an interesting result that the chaotic dynamics can be supressed by stochastic noises for the Hastings and Powell’s model. However, we didn’t further investigate how to generate chaos by white noises and Lévy noises for the stochastic system. This leaves some interesting works to develop this direction in future. Acknowledgements This work is partially supported by National Natural Science Foundation of China (NSFC) under Grant No. 11971081, by the Jilin Scientific and Technological Development Program under Grant No. 20170101143JC, by the Fundamental and Frontier Research Project of Chongqing under Grant No. cstc2018jcyjAX0144, and by the Program of Chongqing Graduate Research and Innovation Project under Grant CYS19290. The authors would like to thank Professor Zhaosheng Feng for valuable suggestions and comments for improving the quality of this article. References [1] J. Bao, X. Mao, G. Yin, C. Yuan; Competitive Lotka-Volterra population dynamics with jumps, Nonlinear Anal, 74 (2011), 6601-6616. [2] J. Bao, C. Yuan; Stochastic population dynamics driven by Lévy noise, J. Math. Anal. Appl., 391 (2012), 363-375. [3] A. Hastings, T. Powell; Chaos in a three-species food chain, Ecology, 72 (1991), 896-903. [4] C. S. Holling; The functional response of predator to prey density and its role in mimicry and population regulation, Mem. Entomol. Soc. Can, 45 (1965), 1-60. [5] C. Huang, H. Zhang, J. Cao, H. Hu; Stability and Hopf bifurcation of a delayed prey-predator model with disease in the predator, Internat. J. Bifur. Chaos, 29(2019), 1950091-16. EJDE-2020/71 ASYMPTOTIC BEHAVIOR OF STOCHASTIC SYSTEMS 17 [6] C. Ji, D. Jiang, N. Shi; Analysis of a predator-prey model with modified Leslie-Gower and Holling-type II schemes with stochastic perturbation, J. Math. Anal. Appl., 359 (2009), 482- 498. [7] A. Klebanoff, A. Hastings; Chaos in three species food chains, J. Math. Biol., 32 (1994), 427-451. [8] X. Li, A. Gray, D. Jiang, X. Mao; Sufficient and necessary conditions of stochastic per- manence and extinction for stochastic logistic populations under regime switching, J. Math. Anal. Appl., 376 (2011), 11-28. [9] M. Liu, K. Wang; Stochastic Lotka-Volterra systems with Lévy noise, J. Math. Anal. Appl., 410 (2014), 750-763. [10] X. Mao, G. Marion, E. Renshaw; Environmental Brownian noise suppresses explosions in population dynamics, Stochastic Processes and Their Applications, 97 (2002), 95-110. [11] R. M. May; Stability and Complexity in Model Ecosystems, Princeton University Press, Princeton, 1975. [12] J. D. Murray; Mathematical Biology I: An Introduction, 3rd edn. Springer, New York, 2001. [13] Y. Takeuchi, N. H. Du, N. T. Hieu, K. Sato; Evolution of predator-prey systems described by a Lotka-Volterra equation under random environment, J. Math. Anal. Appl., 323 (2006), 938-957. [14] B.J. Wang, H.J. Gao; Neutral stochastic partial functional integro-differential equations driven by G-Brownian motion, Electron. J. Differential Equations, 2019 no. 120 (2019), 1-15. [15] S. Wang, L. Wang, T. Wei; Well-posedness and asymptotic behaviors for a predator-prey system with Lévy noise, Methodol Comput Appl Probab, 19 (2017), 715-725. [16] J. Wu, Stability of a three-species stochastic delay predator-prey system with Lévy noise, Phys. A, 502 (2018), 492-505. [17] R. Wu, X. Zou, K. Wang; Dynamics of logistic systems driven by Lévy noise under regime switching, Electronic Journal of Differential Equations, 2014 no. 76 (2014), 1-16. [18] C. Xiang, J. Huang, S. Ruan, D. Xiao; Bifurcation analysis in a host-generalist parasitoid model with Holling II functional response, J. Differential Equations, 268 (2020), 4618-4662. [19] L. Zhang, L. Zou; Bifurcations and control in a discrete predator-prey model with strong Allee effect, Internat. J. Bifur. Chaos, 28 (2018), 1850062-15. [20] X. Zhang, W. Li, M. Liu, K. Wang; Dynamics of a stochastic Holling II one-predator two-prey system with jumps, Phys. A, 421 (2015), 571-582. Yihan Zhao School of Mathematics, Chongqing Normal University, Chongqing 401331, China Email address: zhaoyihan13@126.com Yuanpei Xia School of Mathematics, Chongqing Normal University, Chongqing 401331, China Email address: xiayuanppei@163.com Zhichun Yang (corresponding author) School of Mathematics, Chongqing Normal University, Chongqing 401331, China Email address: yangzhch@126.com 1. Introduction 2. Preliminaries 3. Asymptotic behavior of system (1.3) 4. Examples and conclusions Conclusion Acknowledgements References