Electronic Journal of Differential Equations, Vol. 2022 (2022), No. 37, pp. 1–15. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu DYNAMICAL BEHAVIOR IN A REACTION-DIFFUSION SYSTEM WITH PREY-TAXIS YINGWEI SONG, TIE ZHANG, JINPENG LI Abstract. In this article, we study a diffusive predator-prey system with prey-taxis under homogeneous Neumann boundary conditions. We establish the existence and boundedness of nonnegative global solutions. Through com- parison with the system without prey-taxis, we find that the positive constant equilibrium remains stable for positive prey-taxis, while negative prey-taxis makes it unstable. 1. Introduction The interaction of predator and prey is one of the most fundamental relationships in complex biological systems. For some predator-prey systems, under certain cir- cumstances, predators have to look for food, share food or compete for food among other predators. In some predator-prey models, the carrying capacity of predator is proportional to the densities of prey [2, 6, 7, 8, 14, 15, 18, 19, 21, 23, 24, 25]. The predator’s movement to find prey is decided by the pheromone released by the prey in certain extent. Prey-taxis is the spatiotemporal variations of preda- tors in response to prey gradient, and predator-prey systems with prey-taxis have captured considerable attention in various forms [3, 4, 9, 11, 12, 16, 17, 26]. In this article, we consider a spatial model with prey-taxis the form ∂u ∂t = d1∆u+ ug(u)− p(u)v, x ∈ Ω, t > 0, ∂v ∂t = d2∆v − χ∇ · (α(v)v∇u) + σv(1− v u ), x ∈ Ω, t > 0, ∂u ∂ν = ∂v ∂ν = 0, x ∈ ∂Ω, t > 0, u(x, 0) = u0(x) > 0, v(x, 0) = v0(x) ≥ 0, 6≡ 0, x ∈ Ω, (1.1) where the habitat of both species Ω is a bounded domain in Rn(n ≥ 1) with the smooth boundary ∂Ω, ν is the outer normal vector and homogeneous Neumann boundary conditions (no flux boundary condition) is imposed on both u and v, so the system is closed. Here u(x, t) and v(x, t) denote the densities of prey and predator at place x and time t respectively, d1 and d2 are the dispersal rates of prey and predator, σ stands for the intrinsic growth rate of the predator, and the carrying capacity of the predator is proportional to the densities of prey [14], the 2020 Mathematics Subject Classification. 35B32, 35K57, 37L10. Key words and phrases. Reaction; diffusion; global solution; stability. ©2022. This work is licensed under a CC BY 4.0 license. Submitted August 26, 2021. Published May 13, 2022. 1 2 Y. SONG, T. ZHANG, J. LI EJDE-2022/37 function ug(u) refers to the net growth rate of the prey without predators, and g(u) is per capita growth rate satisfying the following condition: (A1) g ∈ C1([0,+∞)), there exists k > 0, such that g(u) is positive for 0 < u < k, and g(u) is negative for u > k and g(k) = 0. As pointed in [14], the following four classical types of the functional response p(u) are typical and useful: p(x) = x (Leslie-Gower type), p(x) = x x+ a (Holling-Tanner type), p(x) = x2 (x+ a)(x+ b) (Sigmodial type), p(x) = 1− e−ax (Ivlev functional response). Note that the above four types of functional response p(u) satisfy the following hypotheses: (A2) p ∈ C1([0,+∞)), p(0) = 0, p(u) > 0 for u > 0 and p′(u) > 0 for u ≥ 0. Moreover, there exists a positive constant P > 0 such that p′(u) ≤ P for all u > 0. There are quite a few qualitative analyzes on predator-prey system (1.1), e.g., [8] for Leslie-Gower functional response and [6] for Holling-Tanner functional response. The term χ∇ · (α(v)v∇u) is the sensitivity of predator to prey, which quantifies the tendency of predator to move toward the direction of the increasing prey gra- dient. χ is the prey tactic coefficient, χ ≥ 0 measures the intensity of the directed motion of predator. As pointed in [27], α(v) can be taken as α(v) = { 1− v N , 0 ≤ v ≤ N, 0, v > N, (1.2) where N represents the maximum carrying capacity of predators in a unit volume. If the number of predator exceeds the volume N , the trend of direct motion of predator will approach 0. When the response function in the predator equation is typical Holling Type I and Holling Type II, Lee [16] considered the traveling wave solutions, and investi- gated the pattern formation under homogeneous Neumann boundary conditions in a bounded interval [17]. Chakraborty et al [3] showed that the mode of functional response function plays an important role in resolving spatial patterns through nu- merical simulation. The global existence of nonnegative solutions and the stability of steady-state solutions were presented in [9, 26]. For the constant prey tactic, the global existence of the nonnegative solution was discussed in [11], where two di- mensions for any χ > 0 were considered. The diffusive predator-prey system (1.1) without prey-taxis (i.e., χ=0) has been extensively studied in [5, 8, 14]. In this paper, we focus on the dynamical behavior change under homogeneous Neumann boundary conditions from χ = 0 to χ > 0. The rest of the paper is structured as follows. In Section 2, we show the global existence of non-negative solution in system (1.1). In Section 3, the sta- bility/instability of positive constant steady state is investigated for different χ. EJDE-2022/37 DYNAMICAL BEHAVIOR IN A REACTION-DIFFUSION SYSTEM 3 2. Existence of global solutions We start with the existence of classical solutions of system (1.1) when χ = 0, see e.g. [14]. Lemma 2.1. Suppose that p, g satisfy (A1) and (A2), σ > 0, d1, d2 > 0 and χ = 0 in (1.1). Let u0(x) > 0, v0(x) ≥ 0, then 0 < u(x, t) ≤ m1, 0 < v(x, t) ≤ m2, (2.1) where m1 = max{‖u0‖∞, k}, m2 = max{‖v0‖∞,m1}. (2.2) In the following part, we shall prove that for χ > 0, the system (1.1) with prey- taxis still permit a global classical solution. It is known that k is the carrying capacity of prey in (1.1), and N indicates the maximal number of predators that can fill a unit volume. Based on the meaning of m2 and N , the condition N > m2 proposed in [20] always holds in our paper. Note that v = N is not a differentiable point of α(v). To obtain classical solu- tions, by the example in [28], we extend α(v) as follows: α(v) =  > 1, v < 0, 1− v N , 0 ≤ v ≤ N, < 0, v > N. (2.3) Then α(v) is a smooth extension of α(v). Now we narrow our attention to the existence of classical global solutions of the system ∂u ∂t = d1∆u+ ug(u)− p(u)v, x ∈ Ω, t > 0, ∂v ∂t = d2∆v − χ∇ · (α(v)v∇u) + σv(1− v u ), x ∈ Ω, t > 0, ∂u ∂ν = ∂v ∂ν = 0, x ∈ ∂Ω, t > 0, u(x, 0) = u0(x) > 0, v(x, 0) = v0(x) ≥ 0, 6≡ 0 x ∈ Ω. (2.4) It turns out that the existence of classical global solutions to system (2.4) implies the existence of classical global solutions to system (1.1). This is because α(v) = α(v) when 0 ≤ v ≤ N and we will show that v ∈ [0, N ] later, while α(v) = 0 when v > N and Lemma 2.1 gives the desirable results. We define X = { ω ∈W 1,p(Ω) : ∂ω ∂ν = 0, x ∈ ∂Ω } . Lemma 2.2. (1) Assume u0, v0 ∈ W 1,p(Ω), where p > n, and χ > 0. Sup- pose (A1) and (A2) hold. Then there exists a maximal existence time Tmax, such that system (2.4) has a unique nonnegative solution u, v ∈ C([0, Tmax);W 1,p(Ω)) ∩C1((0, Tmax), C1(Ω)), where Tmax depends on ini- tial data (u0, v0) ∈ X2, and u,v satisfy u(x, t) > 0, v(x, t) ≥ 0, x ∈ Ω, 0 ≤ t < Tmax. (2.5) (2) If for every T > 0 there exists a constant containing M0(T ) such that ‖(u(t), v(t))‖∞ ≤M0(T ), 0 < t < min{T, Tmax}, (2.6) 4 Y. SONG, T. ZHANG, J. LI EJDE-2022/37 where M0(T ) is a constant depending on T and ‖(u(t), v(t))‖1,p, then Tmax = +∞. Proof. Let ζ = (u, v). We can rewrite equalities in (2.4) in the form ζt = ∇ · (a(ζ)∇ζ) + Φ(ζ), x ∈ Ω, t > 0, ∂ζ ∂ν = 0, x ∈ ∂Ω, t > 0, ζ(·, 0) = (u0, v0), x ∈ Ω, (2.7) where a(ζ) = ( d1 0 −χα(v)v d2 ) , Φ(ζ) = ( ug(u)− p(u)v σv(1− v u ) ) . By [1, Theorem 14.4 and 14.6], the local existence of solutions in (2.4) is obtained. Moreover, the diffusion matrix a(ζ) in (2.7) is lower-triangular, the result in (2) follows from [1], so we have Tmax =∞. � Theorem 2.3. Assume that χ > 0 and (A1) and (A2) hold. Then the solution u(x, t) of system (1.1) satisfies 0 < u(x, t) ≤ max{‖u0‖∞, k} = m1, lim t→∞ sup u(x, t) ≤ k. (2.8) Proof. From the first equation in (1.1), it holds ∂u ∂t − d1∆u = ug(u)− p(u)v ≤ ug(u), x ∈ Ω, t > 0, ∂u ∂ν = 0, x ∈ ∂Ω, t > 0. u(x, 0) = u0(x), x ∈ Ω. (2.9) Let u∗(t) be the solution of the ODE problem du∗(t) dt = u∗(t)g(u∗(t)), t > 0, u∗(0) = ||u0||∞. (2.10) Then hypothesis (A1) gives u∗(t) ≤ m1 = max{‖u0‖∞, k}. Furthermore u∗(t) is a super-solution of the PDE problem ∂U ∂t − d1∆U = Ug(U), x ∈ Ω, t > 0, ∂U ∂ν = 0, x ∈ ∂Ω, t > 0. U(x, 0) = u0(x), x ∈ Ω. (2.11) Therefore, 0 < U(x, t) ≤ u∗(t), for all (x, t) ∈ Ω× (0,∞). (2.12) By the comparison principle, we have 0 < u(x, t) < U(x, t) ≤ u∗(t) ≤ m1, (x, t) ∈ Ω× (0,∞). (2.13) Since g(u) < 0 for all u > k by hypothesis (A1), we have that lim t→+∞ sup u∗(x, t) ≤ lim t→+∞ u∗(t) = k, which along with (2.13) gives (2.8). � EJDE-2022/37 DYNAMICAL BEHAVIOR IN A REACTION-DIFFUSION SYSTEM 5 Theorem 2.4. Assume that 0 ≤ v0(x) ≤ N , then the solution (u(x, t), v(x, t)) of (2.4) for all (x, t) ∈ Ω× (0, T ) satisfies 0 ≤ v(x, t) ≤ N . Proof. We define Lv = vt − d2∆v + χ∇ · (α(v)v∇u)− σv ( 1− v u ) . (2.14) Note that 0 ≤ v0, so v = 0 is a lower solution of the equation. Moreover, LN = −σN ( 1− N u ) . (2.15) Because N > m2, choosing sufficiently large N gives LN ≥ 0. (2.16) Thus v = N is an upper solution of the v equation by (2.16). The comparison principle [22] gives 0 ≤ v(x, t) ≤ N. (2.17) � 3. Effect of prey-taxis on the dynamics In this section, we investigate the effect of prey-taxis on the dynamics. From The- orems 2.3 and 2.4, and Sobolev embedding Theorem, the solution (u(x, t), v(x, t)) of (2.4) becomes classical solution. In the following, we consider the local stability and global stability of (u∗, v∗) in Y = { ω ∈ C2(Ω) : ∂ω ∂ν = 0, x ∈ ∂Ω } . 3.1. Global stability of positive constant steady state. First we study the global stability of positive constant solution (u∗, v∗) in (1.1). For this purpose, we impose the following additional hypothesis as pointed in [14]: (A3) there exists some constant p̂ > 0, such that −p̂ ≤ d du ( p(u) u ) ≤ 0 for u > 0. (A4) g′(u) ≤ −ĝ, where ĝ > 0 is a positive constant. It is easy to find that system (1.1) has a unique positive equilibrium (u∗, v∗), where u∗ = g(u∗) p(u∗) = v∗. Theorem 3.1. Assume that (A1)–(A4) hold and k 2 < u∗ ≤ ĝ p̂ , then the positive equilibrium (u∗, v∗) of (1.1) is globally asymptotically stable if χ2 < d1d2σ u∗p(u∗)v∗ . Proof. It follows from Theorems 2.3 and 2.4 that Y := {(u, v) ∈ R2|0 < u ≤ k, 0 ≤ v ≤ N} is positive invariant for (1.1). Let (u(x, t), v(x, t)) be a positive solution of (1.1). Define a Lyapunov function E(t) = ∫ Ω (∫ u u∗ ξ − u∗ ξp(ξ) dξ +A ∫ v v∗ η − v∗ η dη ) dx for some positive constant A, which will be chosen later. Then Ė = ∫ Ω u− u∗ up(u) ∂u ∂t dx+ ∫ Ω A v − v∗ v ∂v ∂t dx = ∫ Ω u− u∗ up(u) [d1∆u+ ug(u)− p(u)v]dx 6 Y. SONG, T. ZHANG, J. LI EJDE-2022/37 + ∫ Ω A v − v∗ v [ d2∆v + σv ( 1− v u ) − χ∇ · (α(v)v∇u) ] dx := E1(t) + E2(t), where E1(t) = ∫ Ω [u− u∗ up(u) d1∆u+A v − v∗ v (d2∆v − χ∇(α(v)v∇u)) ] dx = − ∫ Ω [ d1 u2p2(u) (up(u)− (u− u∗)(p(u) + upu(u))) |∇u|2 ] dx − ∫ Ω [ Ad2 v∗ v2 |∇v|2 +Aχv∗ ∫ Ω α(v) v |∇u||∇v| ] dx, and E2(t) = ∫ Ω [u− u∗ up(u) (ug(u)− p(u)v) +A v − v∗ v σv(1− v u ) ] dx = ∫ Ω [u− u∗ up(u) f1(u, v) +A v − v∗ v f2(u, v) ] dx. According to (2.8), we can find a large T such that u(x, t) ≤ k + ε in [T,∞) × Ω for any positive constant ε with ε ≤ 2u∗ −K. Rewrite E1(t) as E1(t) = − ∫ Ω ZTBZ, where Z = ( ∇u ∇v ) , B = ( d1 u2p2(u) (up(u)− (u− u∗)(p(u) + upu(u))) Aχv∗α(v) 2v Aχv∗α(v) 2v Ad2 v∗ v2 ) . Since d du (p(u) u ) = upu(u)−p(u) u2 < 0 for u > 0 from (A3), thus upu(u) < p(u) for u > 0. Using this fact and the assumption K 2 < u∗, we have up(u)− (u− u∗)(p(u) + upu(u)) = −u2pu(u) + u∗(p(u) + upu(u)) ≥ −u2pu(u) + u∗[2upu(u)] = upu(u)[2u∗ − u] ≥ upu(u)[2u∗ −K − ε] ≥ 0 for t ≥ T , which implies the result. It is clear that trace(B) > 0, and the determinant of B is detB = Ad1d2v ∗ u2p2(u)v2 (up(u)− (u− u∗)(p(u) + upu(u)))− A2χ2(v∗)2α2(v) 4v2 . (3.1) Thus det(B) > 0 is equivalent to d1d2[up(u)− (u− u∗)(p(u) + upu(u))] ≥ u2p2(u)Aχ2v∗α(v). (3.2) Because 0 < u ≤ u∗, 0 ≤ v ≤ N , a sufficient condition for (3.2) to hold is d1d2 > u∗p(u∗)Aχ2v∗. (3.3) Therefore, if (3.3) holds, then E1(t) = − ∫ Ω ZTBZ ≤ 0. (3.4) EJDE-2022/37 DYNAMICAL BEHAVIOR IN A REACTION-DIFFUSION SYSTEM 7 Moreover, E2(t) = ∫ Ω [ u− u∗ p(u) ( g(u)− g(u∗) ) ]dx − ∫ Ω {u− u∗ p(u) [(p(u) u v − p(u) u v∗ ) + (p(u) u v∗ − p(u∗) u∗ v∗ )]} dx + ∫ Ω [ Aσ(v − v∗) ( − v u + v∗ u − v∗ u + v∗ u∗ )] dx = ∫ Ω [ 1 p(u) ( gu(ξ)− v∗ d du (p(u) u ) |u=η ) (u− u∗)2 ] dx + ∫ Ω [ (u− u∗)(v − v∗) u ( − 1 +Aσ v∗ u∗ ) + (v − v∗)2 u (−σA) ] dx for some ξ and η. If we choose A = 1 σ , then −1 +Aσ v ∗ u∗ = 0. Thus E2(t) ≤ 0 since gu(ξ)− v∗ d du (p(u) u ) |u=η ≤ −ĝ + p̂v∗ = −ĝ + p̂u∗ ≤ 0. (3.5) from the hypotheses (A1), (A3), and (A4). Thus Ė ≤ 0 for all t ≥ 0, which implies the desired result since the equality holds if and only if when (u, v) = (u∗, v∗). That is lim t→+∞ ‖u(x, t)− u∗‖Y = 0, lim t→+∞ ‖v(x, t)− v∗‖Y = 0. � Remark 3.2. If d du (p(u) u ) ≡ 0, then (3.5) is always satisfied since gu(ξ) < 0, and the same result holds when k 2 < u∗. It points out that the predator-prey models with Leslie-Gower functional response [13] satisfy the condition d du (p(u) u ) ≡ 0. 3.2. Effect of prey-taxis on the stability of (u∗, v∗). The linearization of (1.1) at e∗ = (u∗, v∗) can be expressed as Ψt = L(χ)Ψ := G∆Ψ + JΨ with domain Y , where Ψ(x) = ( φ(x) ψ(x) ) ∈ Y, G = ( d1 0 −χα(v∗)v∗ d2 ) , J = ( M −p(u∗) σ −σ ) and M = g(u∗) + u∗gu(u∗)− pu(u∗)v∗. tr(G) = d1 + d2 > 0, det(G) = d1d2 > 0, (3.6) tr(J) = M − σ, det(J) = σ(p(u∗)−M). (3.7) Then the eigenvalue λ of L(χ)Ψ = λΨ can be obtained from the Fourier decompo- sition of the matrix Li(χ), where Li(χ) = ( −d1µi +M −p(u∗) C + χα(v∗)v∗µi −d2µi − σ ) , (i = 0, 1, 2, . . . ) and µi is the eigenvalue of operator −∆ under Neumann boundary conditions, which implies that eigenvalues 0 = µ0 < µ1 ≤ µ2 ≤ . . . and limi→∞ µi =∞. tr(Li) = −(d1 + d2)µi +M − σ = − tr(G)µi + tr(J), (3.8) 8 Y. SONG, T. ZHANG, J. LI EJDE-2022/37 det(Li) = d1d2µ 2 i + (d1σ − d2M + p(u∗)χα(v∗)v∗)µi + (p(u∗)−M)σ = det(G)µ2 i + F (J,G)µi + det(J), (3.9) where F (J,G) = −(d2A+ d1D +Bχα(v∗)v∗) = −(d2M + d1σ − p(u∗)χα(v∗)v∗). The expression of det(Li) lead to its minimum minµ∈R+ det(Li): min µ∈R+ det(Li) = 4d1d2(p(u∗)−M)σ − (d1σ − d2M + p(u∗)χα(v∗)v∗)2 4d1d2 , (3.10) at µ = µ∗ = −d1σ − d2M + p(u∗)χα(v∗)v∗ 2d1d2 = d2M − d1σ − p(u∗)χα(v∗)v∗ 2d1d2 . (3.11) After direct calculations, we obtain the stability/instability of (u∗, v∗) for χ = 0. Theorem 3.3. Let d1, d2 > 0 and Ω is a bounded domain with smooth boundary. Assume σ > M > 0 and p(u∗) > M . (1) If σ ≥ d2 d1 M for any d1 > 0, d2 > 0, (u∗, v∗) is locally asymptotically stable in Y . (2) If max{M,σ1} < σ < min{d2Md1 , σ2}, (u∗, v∗) is locally asymptotically stable in Y . (3) If M < σ < σ1 or σ2 < σ < d2M d1 , e∗ = (u∗, v∗) is unstable in Y , where σ1 = d2 d1 (√ p(u∗)− √ p(u∗)−M )2 and σ2 = d2 d1 (√ p(u∗)+ √ p(u∗)−M )2 . Next, we study the effect that the prey-taxis χ has on the stability of (u∗, v∗) for different parameter ranges. Theorem 3.4. Let d1, d2 > 0 and Ω be a bounded domain with smooth bound- ary. Assume σ > M > 0 and p(u∗) > M . If σ ≥ d2 d1 M , then (u∗, v∗) is locally asymptotically stable for any χ > 0. Proof. From (3.7), tr(J) < 0 as σ > M . From (3.8), it is noticed that tr(Li) = −(d1 + d2)µi + tr(J) < 0 for i = 0, 1, 2, 3, . . . . From (3.9), det(L0) = det(J) > 0 as p(u∗) > M . According to (3.9), by σ ≥ d2 d1 M and χ > 0, we have F (J,G) > 0, which gives rise to det(Li) = det(G)µ2 i + F (J,G)µi + det(J) > 0, i = 0, 1, 2, 3, . . . . Thus each eigenvalue of det(Li) has a negative real part, and (u∗, v∗) is locally asymptotically stable for any χ > 0 from [10]. � Moreover, the case of σ < d2M/d1 can be stated as follows. Theorem 3.5. Let σ1 and σ2 be the smaller and larger roots of minµ∈R+ det(Li) = 0. Assume d1, d2 > 0 and p(u∗) > M . If max{M,σ1} < σ < min {d2M d1 , σ2 } , then e∗ = (u∗, v∗) is locally asymptotically stable for system (1.1) for any χ > 0. Proof. According to (3.7), by σ > M , we have tr(J) < 0, which gives rise to tr(Li) = −(d1 + d2)µi + tr(J) < 0, i = 0, 1, 2, 3, . . . . If d2M − d1σ − p(u∗)χα(v∗)v∗ > 0, i.e. χ < d2M−d1σ p(u∗)α(v∗)v∗ , then 4d1d2(p(u∗)−M)σ − (d1σ − d2M)2 4d1d2 > 0 EJDE-2022/37 DYNAMICAL BEHAVIOR IN A REACTION-DIFFUSION SYSTEM 9 for σ1 < σ < σ2. Thus we get det(Li) > 0 for i = 0, 1, 2, 3, . . . , which implies that the constant positive equilibrium solution (u∗, v∗) is locally asymptotically stable. If d2M − d1σ − p(u∗)χα(v∗)v∗ ≤ 0, then det(L0) = det(J) > 0 from (3.9). It is easy to find that det(Li) > 0 for i = 0, 1, 2, 3, . . . , thus (u∗, v∗) is still locally asymptotically stable. � Similarly, we have the following stability change for different χ > 0. For conve- nience, we denote D1 := d2M − d1σ − 2 √ d1d2(p(u∗)−M)σ p(u∗)α(v∗)v∗ , D2 := d2M − d1σ + 2 √ d1d2(p(u∗)−M)σ p(u∗)α(v∗)v∗ Theorem 3.6. Let σ1 and σ2 be the smaller and larger roots of minµ∈R+ det(Li) = 0. Assume d1, d2 > 0 and p(u∗) > M . If M < σ < σ1 or σ2 < σ < d2M d1 , then (1) e∗ = (u∗, v∗) is locally asymptotically stable for χ > D1; (2) e∗ = (u∗, v∗) is unstable for χ ≤ D1. Comparing Theorems 3.6 and 3.3, the positive prey tactic coefficient χ makes e∗ = (u∗, v∗) from being unstable to being stable. Furthermore, we can find the negative prey tactic χ < 0 also has significant influence on the stability/instability of (u∗, v∗). Theorem 3.7. Assume 0 < M < min{p(u∗), σd1d2 }. Then (u∗, v∗) is unstable for (1.1) if χ ≤ D1 < 0, (3.12) or D2 ≤ χ < 0. (3.13) Proof. It is easy to find that (3.10) is smaller than zero when (3.12) or (3.13) holds. � Remark 3.8. Note that e∗ = (u∗, v∗) is globally asymptotically stable when p(u∗) > M > 0 and σ ≥ d2 d1 M for χ = 0 from Theorem 3.3. While the neg- ative prey tactic coefficient χ satisfying (3.12) or (3.13) changes the stability of e∗ = (u∗, v∗) completely. Furthermore, we have the stability change of e∗ for M > σd1 d2 if χ < 0. Theorem 3.9. Assume p(u∗) > M > 0. (1) If max{M,σ1} < σ < min {d2M d1 , σ2 } , then e∗ = (u∗, v∗) becomes unstable for χ < D1 < 0. (2) If M < σ < σ1 or σ2 < σ < d2M d1 , then e∗ = (u∗, v∗) remains unstable for any χ < 0. 10 Y. SONG, T. ZHANG, J. LI EJDE-2022/37 Table 1. The stability/instability of (u∗, v∗) for different χ and other parameters. χ < 0 (u ∗ , v ∗ ) is u n st a b le . (u ∗ , v ∗ ) is u n st a b le fo r χ < D 3 < 0. (u ∗ , v ∗ ) is st a b le fo r 0 > χ > D 3 . D 2 ≤ χ < 0, (u ∗ , v ∗ ) is u n st ab le . χ ≤ D 1 < 0, (u ∗ , v ∗ ) is u n st ab le . w h en σ ∗ < σ < σ ∗ , χ ≤ D 1 < 0, (u ∗ , v ∗ ) is u n st ab le . w h en M < σ < σ 1 o r σ 2 < σ < d 2 M d 1 , (u ∗ , v ∗ ) is u n st a b le fo r a n y χ < 0 . χ > 0 (u ∗ , v ∗ ) is st ab le . (u ∗ , v ∗ ) is st a b le . (u ∗ , v ∗ ) is st ab le . w h en σ ∗ < σ < σ ∗ , (u ∗ , v ∗ ) is st ab le fo r an y χ > 0. w h en M < σ < σ 1 o r σ 2 < σ < d 2 M d 1 , (u ∗ , v ∗ ) is st ab le , fo r χ > D 1 . χ = 0 (u ∗ , v ∗ ) is st ab le . (u ∗ , v ∗ ) is st a b le . (u ∗ , v ∗ ) is st ab le . w h en σ ∗ < σ < σ ∗ , (u ∗ , v ∗ ) is st ab le . w h en M < σ < σ 1 o r σ 2 < σ < d 2 M d 1 , (u ∗ , v ∗ ) is u n st a b le . σ = d2 d1 M > M ,d2 > d1 > 0 σ > d2 d1 M , d2, d1 > 0 d2 d1 M > σ > M , d2, d1 > 0 p(u∗) < M p(u∗) > M M > 0, σ > M Comparing the conditions in Theorems 3.3 and 3.9, the negative prey tactic coefficient χ can exacerbate the instability of e∗ = (u∗, v∗). Summarizing the above analysis, the stability/instability of (u∗, v∗) can be listed in Table 1 for different EJDE-2022/37 DYNAMICAL BEHAVIOR IN A REACTION-DIFFUSION SYSTEM 11 χ and other parameters. For convenience, we denote σ∗ = max{M,σ1}, σ∗ = min { d2M d1 , σ2 } , D3 := −2 √ d1d2(p(u∗)−M)σ p(u∗)α(v∗)v∗ in Table 1. The results for system (1.1) satisfies the following conditions. Theorem 3.10. Let d1, d2 > 0 and Ω is a bounded domain with smooth boundary. (1) If M ≤ 0, (u∗, v∗) is locally asymptotically stable for any χ. (2) If M > 0 and σ ≤M , (u∗, v∗) is unstable for any χ. (3) If σ > M > 0 and p(u∗) < M , (u∗, v∗) is unstable for any χ. Figure 1. u and v components of system (1.1) with χ = 0 and ini- tial value (u0(x), v0(x)) = (0.9512+0.1∗rand; 0.9512+0.1∗rand). The positive equilibrium (u∗, v∗) = (0.9512, 0.9512) is asymptoti- cally stable. Figure 2. u and v components of system (1.1) with χ = 1.5579 and initial value (u0(x), v0(x)) = (0.9512 + 0.1 ∗ rand; 0.9512 + 0.1 ∗ rand). The positive equilibrium (u∗, v∗) = (0.9512, 0.9512) is asymptotically stable. 4. Conclusion and simulation In this paper, we discuss a diffusive predator-prey model with prey-taxis subject to homogeneous Neumann boundary conditions. The global existence and bound- edness of nonnegative solution of (1.1) is obtained for every prey tactic. The global stability of positive constant equilibrium remains for small prey tactic, see Theorem 12 Y. SONG, T. ZHANG, J. LI EJDE-2022/37 Figure 3. u and v components of system (1.1) with χ = −8 and initial value (u0(x), v0(x)) = (0.9512 + 0.1 ∗ rand; 0.9512 + 0.1 ∗ rand). The positive equilibrium (u∗, v∗) becomes unstable. Figure 4. u and v components of system (1.1) with χ = 0 and initial value (u0(x), v0(x)) = (0.9512 + 0.1 ∗ rand; 0.9512 + 0.1 ∗ rand). The positive equilibrium (u∗, v∗) = (0.9512, 0.9512) is un- stable. Figure 5. u and v components of system (1.1) with χ > D1 = 298.99725 and initial value (u0(x), v0(x)) = (0.9512 + 0.1 ∗ rand; 0.9512 + 0.1 ∗ rand). The positive equilibrium (u∗, v∗) = (0.9512, 0.9512) is asymptotically stable. EJDE-2022/37 DYNAMICAL BEHAVIOR IN A REACTION-DIFFUSION SYSTEM 13 Figure 6. u and v components of system (1.1) with χ = −0.00001 < 0 and initial value (u0(x), v0(x)) = (0.9512 + 0.1 ∗ rand; 0.9512 + 0.1 ∗ rand). The positive equilibrium (u∗, v∗) = (0.9512, 0.9512) is unstable. 3.1. From Theorems 3.3, 3.4, 3.5, 3.6, 3.7 and 3.9, we find the positive prey tactic coefficient can maintain the stability of the positive constant equilibrium, while negative prey tactic coefficient can lead to the instability of the positive constant equilibrium. The results are applicable in the case of linear functional response g(u) = p − bu and Holling-Tanner type p(u) = u u+a , which satisfy (A1) and (A2). Taking p = 1, a = 0.1, b = 0.1, d1 = 1, d2 = 3, M = 0.7237, we can illustrate the above results for different value of σ: (1) σ = 0.83. Then d2M d1 = 2.1710, σ1 = 0.8287, σ2 = 5.6878. In this case, max{M,σ1} < σ < min{d2Md1 , σ2}, then the constant positive equilibrium solution (u∗, v∗) is locally asymptotically stable for χ = 0 by Theorem 3.3, see Figure 1. According to Theorem 3.5, the positive equilibrium (u∗, v∗) remains stable for positive prey tactic, see Figure 2; while (u∗, v∗) becomes unstable for negative prey tactic from Theorem 3.9, see Figure 3. (2) σ = 0.76. In this case, M < σ < σ1, then the constant positive equilibrium e∗ = (u∗, v∗) is unstable for χ = 0, by Theorem 3.3, see Figure 4. While (u∗, v∗) becomes stable for positive prey tactic from Theorem 3.6, see Figure 5. We observe that (u∗, v∗) is unstable for any χ < 0 from Theorem 3.9, see Figure 6. Acknowledgments. The authors would like to thank the anonymous referees for their valuable comments and suggestions, which have highly improved the paper’s quality. This work was supported by the State Key Laboratory of Synthetical Automation for Process Industries Fundamental Research Funds (No. 2013ZCX02) and by the Basic Scientific Research Expenses Project of Heilongjiang Provincial Colleges and by Universities in 2020 (2020-KYYWF-0359). References [1] H. Amann; Nonhomogeneous linear and quasilinear elliptic and parabolic boundary value problems, Vieweg+Teubner Verlag, 133 (1993), 9–126. [2] C. Cosner, D. L. DeAngelis, J.S. Ault, D.B. Olson; Effects of spatial grouping on the func- tional response of predators, Theoretical Population Biology, 56 (1999), 65–75. [3] A. Chakraborty, M. Singh, D. Lucy, P. Ridland; Predator-prey model with prey-taxis and diffusion, Mathematical and Computer Modelling, 46 (2007), 482–498. 14 Y. SONG, T. ZHANG, J. LI EJDE-2022/37 [4] Y. Cai, Q. Cao, Z. Wang; Asymptotic dynamics and spatial patterns of a ratio-dependent predator-prey system with prey-taxis, Applicable Analysis, 9 (2019), 1–18. [5] C. Chen, L. Hung, C. Li, J. Llibre; Nonexistence of traveling wave solutions, exact and semi- exact traveling wave solutions for diffusive lotka-volterra systems of three competing species, Communications on Pure and Applied Analysis, 15 (2017), 1451–1469. [6] S. Chen, J. Shi, J. Wei; Global stability and hopf bifurcation in a delayed diffusive leslie-gower predator-prey system, International Journal of Bifurcation and Chaos, 22 (2012), 331–517. [7] P. Drabek, M. Zahradnikova; Traveling waves for unbalanced bistable equations with density dependent diffusion, Electronic Journal of Differential Equations, 2021 (2021), no. 76, 1-21. [8] Y. Du, S. B. Hsu; A diffusive predator-prey model in heterogeneous environment, Journal of Differential Equations, 203 (2004), 331–364. [9] X. He, S. Zheng; Global boundedness of solutions in a reaction-diffusion system of predator- prey model with prey-taxis, Applied Mathematics Letters, 49 (2015), 73–77. [10] B. D. Hassard, N. D. Kazarinoff, Y.-H. Wan; Theory and applications of Hopf bifurcation, Cambridge Univ. Press, Cambridge, 1981. [11] H. Y. Jin, Z. A. Wang; Global stability of prey-taxis systems, Journal of Differential Equa- tions, 262 (2017), 1257–1290. [12] P. Kareiva, G. Odell; Swarms of predators exhibit “preytaxis”if individual predators use area-restricted search, The American Naturalist, 130 (1987), 233–270. [13] A. Korobeinikov; A Lyapunov function for Leslie-Gower predator-prey models, Applied Math- ematics Letters, 14 (2001), 697–699. [14] W. Ko, K. Ryu; Non-constant positive steady-states of a diffusive predator-prey system in homogeneous environment, Journal of Mathematical Analysis and Applications, 327 (2007), 539–549. [15] Y. Kuang, K. Wang; Coexistence and extinction in a data-based ratio-dependent model of an insect community, Mathematical Biosciences and Engineering, 17 (2020), 3274–3293. [16] J. M. Lee, T. Hillen, M. A. Lewis; Continuous traveling waves for prey-taxis, Bulletin of Mathematical Biology, 70 (2008), 654–676. [17] J. M. Lee, T. Hillen, M. A. Lewis; Pattern formation in prey-taxis systems, Journal of Biological Dynamics, 3 (2009), 551–573. [18] P. H. Leslie, J. C. Gower; The properties of a stochastic model for the predator-prey type of interaction between two species, Biometrika, 47 (1960), 219–234. [19] R. Peng, M. X. Wang; Global stability of the equilibrium of a diffusive Holling-Tanner prey- predator model, Applied Mathematics Letters, 20 (2007), 664–670. [20] K. J. Painter, T. Hillen; Volume-filling and quorum-sensing in models for chemosensitive movement, Canadian Applicable Mathematics Quartery, 10 (2002), 501–543. [21] Y. H. Pang, M. Wang; Qualitative analysis of a ratio-dependent predator-prey system with diffusion, Proceedings of the Royal Society of Edinburgh Section A Mathematics, 133 (2003), 919–942. [22] H. L. Smith; Monotone dynamical systems: an introduction to the theory of competitive and cooperative systems, Ams Ebooks Program, 41 (1995), 174. [23] Y. W. Song, T. Zhang; Spatial pattern formations in diffusive predator-prey systems with non- homogeneous dirichlet boundary conditions, Journal of Applied Analysis and Computation, 10 (2020), 165–177. [24] S. Su, G. B. Zhang; Global stability of traveling waves for delay reaction-diffusion systems without quasi-monotonicity, Electronic Journal of Differential Equations, 2020 (2020), no. 46, 1-18. [25] J. T. Tanner; The stability and intrinsic growth rates of prey and predator populations, Ecology, 56 (1975), 855–867. [26] Y. Tao; Global existence of classical solutions to a predator-prey model with nonlinear prey- taxis, Nonlinear Analysis: Real World Applications, 11 (2010), 2056–2064. [27] X. Wang, X. Zou; Pattern formation of a predator-prey model with the cost of anti-predator behaviors, Mathematical Biosciences and Engineering, 15 (2018), 775–805. [28] Z. Wang, T. Hillen; Classical solutions and pattern formation for a volume filling chemotaxis model, Chaos, 17 (2007), 37–108. EJDE-2022/37 DYNAMICAL BEHAVIOR IN A REACTION-DIFFUSION SYSTEM 15 Yingwei Song Department of Mathematics, Northeastern University, Shenyang 110006, China. School of Mathematical Sciences, Harbin Normal University, Harbin 150025, China Email address: songyingwei001@163.com Tie Zhang (corresponding author) Department of Mathematics, Northeastern University, Shenyang 110006, China Email address: ztmath@163.com Jinpeng Li Department of Mathematics, University College London, London WC1E 6BT, United Kingdom Email address: lijinpengmath@126.com 1. Introduction 2. Existence of global solutions 3. Effect of prey-taxis on the dynamics 3.1. Global stability of positive constant steady state 3.2. Effect of prey-taxis on the stability of (u*,v*) 4. Conclusion and simulation Acknowledgments References