Electronic Journal of Differential Equations, Vol. 2020 (2020), No. 52, pp. 1–21. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu EXISTENCE OF WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS WITHOUT THE AMBROSETTI-RABINOWITZ CONDITION XIAOHUI WANG, PEIHAO ZHAO Abstract. In this article, we study the existence of the weak solution for superlinear elliptic equations and systems without the Ambrosetti-Rabinowitz condition. The Ambrosetti-Rabinowitz condition guarantees the boundedness of the PS sequence of the functional I for the corresponding problem. We establish the existence of the weak solution for the superlinear elliptic equation by using (PS)c form of the Mountain pass lemma, and the existence of the weak solution for the superlinear elliptic system by using (PS)∗c form of the Linking theorem. 1. Introduction and statement of main results In this article, we investigate the existence of the nontrivial weak solution for the superlinear elliptic problems. We first consider the p-Laplacian equation −∆pu = λf(x, u) in Ω, u = 0 on ∂Ω, (1.1) where p > 1, λ > 0, Ω ⊂ Rn is a bounded domain, f : Ω × R → R is a continuous function, and for 1 < p <∞, the p-Laplacian operator is ∆pu = div(|Du|p−2Du) for u ∈W 1,p(Ω). We shall say the function f satisfies the well-known Ambrosetti-Rabinowitz (AR) condition, if there are constants θ > p and r > 0 such that 0 < θF (x, t) ≤ f(x, t)t for all |t| ≥ r and x ∈ Ω, where F (x, t) = ∫ t 0 f(x, s)ds. Since 1973 when Ambrosetti and Rabinowitz [2] established the Mountain pass lemma under the AR condition, many researchers have studied the superlinear el- liptic problems under the AR condition. The AR condition guarantees the bound- edness of the PS sequence of the functional I given by the corresponding problem, which plays a key role in the application of the critical point theory. Although the AR condition is convenient, it is very restrictive and excludes a lot of nonlinear 2010 Mathematics Subject Classification. 35J20, 35J47, 35J15, 35A15. Key words and phrases. Superlinear elliptic system; weak solution; mountain pass lemma; linking theorem. c©2020 Texas State University. Submitted May 8, 2018. Published May 27, 2020. 1 2 X. WANG, P. ZHAO EJDE-2020/52 problems. Therefore, many researchers have been studied various problems without the AR condition. In 2004, Schechter and Zou [20] established the existence of nontrivial weak solution for the problem (1.1) without the AR condition when p = 2. In this paper, for a general p (1 < p <∞), we will establish the existence of the nontrivial weak solution for the p-Laplacian superlinear elliptic boundary value problem (1.1) without the AR condition. The AR condition implies that there exist positive constants c1 and c2 such that F (x, t) ≥ c1|t|θ − c2 for all (x, t) ∈ Ω× R. Although this condition is weaker, it still eliminates many superlinear problems. A much weaker condition implies that superlinearity is either lim t→+∞ F (x, t) |t|p = +∞ a.e. in Ω, or lim t→−∞ F (x, t) |t|p = +∞ a.e. in Ω. Our first objective is to establish the existence of the nontrivial weak solution for the p-Laplacian superlinear elliptic equation (1.1) under the weaker condition than the AR condition in this paper. Let us state the main result for the elliptic equation as follows. In the next theorem we use the following assumptions: (H1) f ∈ C0(Ω× R,R), f(x, 0) = 0, lim t→0 f(x, t) |t|p−2t = 0 uniformly a.e. in Ω; (H2) There exist positive constants a and b such that |f(x, t)| ≤ a+ b|t|q−1 ∀(x, t) ∈ Ω× R, where q ∈ [1, p∗), p∗ = { np n−p if 1 < p < n, +∞ if p ≥ n; (H3) Either lim t→+∞ F (x, t) |t|p = +∞, or lim t→−∞ F (x, t) |t|p = +∞, uniformly a.e. in Ω; (H4) There exist µ > p and r > 0 such that µF (x, t)− tf(x, t) ≤ C(|t|p + 1) for all |t| ≥ r and x ∈ Ω. Theorem 1.1. If f satisfies (H1)–(H4), then for each λ > 0, problem (1.1) has at least one nontrivial solution. Secondly, we consider the non-cooperative elliptic system −4u = Hu(x, u, v) x ∈ Ω, −4v = −Hv(x, u, v) x ∈ Ω, u(x) = v(x) = 0 x ∈ ∂Ω, (1.2) EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 3 where Ω ⊂ Rn (n ≥ 3) is a smooth bounded domain, H : Ω × R2 → R is a C1 function, Hu denotes the partial derivative of H with respect to the variable u. We write z := (u, v), we suppose H(x, 0) ≡ 0 and Hz(x, 0) ≡ 0, then z = 0 is a trivial solution for this system. We will also establish the existence of the nontrivial solution for the elliptic system (1.2) in this paper. Roughly speaking, we are mainly interested in the class of Hamiltonians H such that H(x, u, v) ∼ |u|p + |v|q +R(x, u, v) with lim |z|→∞ R(x, u, v) |u|p + |v|q = 0, where 1 < p < 2∗ := 2n n−2 and q > 1. For elliptic system, we shall say H satisfies the AR condition, if there exist µ > 2, ν > 1 and R ≥ 0 such that 1 µ Hu(x, z)u+ 1 ν Hv(x, z)v ≥ H(x, z) whenever |z| ≥ R, with the provision that ν = µ if q > 2. In 1995, by using variational method, Costa and Magalhaes [6] established the existence of the nontrivial weak solution for the subcritical non-cooperative elliptic system without the AR condition. In 2004, Lam and Lu [14] obtained the ex- istence of the nontrivial weak solution for the critical and subcritical superlinear cooperative elliptic system without the AR condition. In 2003, De Figueiredo and Ding [7] obtained the existence of the nontrivial weak solution for the supercrit- ical superlinear non-cooperative elliptic system when 2 < p < 2∗ under the AR condition. As we mentioned above, many researchers have studied the existence of the non- trivial weak solution for the superlinear elliptic systems, such as, the subcritical non-cooperate elliptic system without the AR condition, the critical and subcriti- cal superlinear cooperative elliptic system without the AR condition and the super- critical superlinear non-cooperative elliptic system under the AR condition. Our another aim in this paper is to prove the existence of the nontrivial weak solution for the supercritical superlinear non-cooperative elliptic system without the AR condition, that is, we are going to study the system (1.2) without the AR condition when p ∈ (2, 2∗) and q ∈ (2∗,+∞). We would like to mentioned that the main difficulty is to establish the bounded- ness of the (PS)∗c sequence for the non-cooperative elliptic system without the AR condition. Let us state the main result for the elliptic system, using the following assump- tions: (H5) There exist p ∈ (2, 2∗) and q ∈ (2∗,+∞) such that |Hu(x, u, v)| ≤ γ0 ( 1 + |u|p−1 + |v| q 2−1 ) , |Hv(x, u, v)| ≤ γ0 ( 1 + |u|p−1 + |v|q−1 ) , for all (x, z). In all hypotheses on H the γi denote positive constants independent of (x, z); (H6) limz→∞H(x, z)/|z| = +∞ uniformly in Ω; (H7) There exist µ > 2 and R1 > 0 such that µH(x, z)− zHz(x, z) ≤ C(|z|p + 1) whenever |z| ≥ R1; 4 X. WANG, P. ZHAO EJDE-2020/52 (H8) For p and q as above, H(x, z) ≥ γ1(|u|p + |v|q)− γ2 for all (x, z); (H9) H(x, 0, v) ≥ 0 and Hu(x, u, 0) = o(|u|) uniformly with respect to x, as u→ 0. Theorem 1.2. Suppose H satisfies (H5)–(H8). Then the superlinear elliptic system (1.2) has at least one nontrivial weak solution. The rest of this paper is organized as follows. In section 2, we will discuss the superlinear elliptic equation (1.1) by a variational method, and establish the existence of the nontrivial weak solution for this superlinear elliptic equation. Fur- thermore, we will investigate the superlinear non-cooperative elliptic system (1.2) by variational method in section 3, and establish the existence of the nontrivial weak solution for this superlinear non-cooperative elliptic system. 2. Superlinear elliptic equation In this section, we establish the existence of the nontrivial weak solution for the superlinear elliptic boundary value problem (1.1) of p-Laplacian type. 2.1. Preliminaries. Throughout this section, let Ω be a bounded domain in Rn. For 1 < p < +∞, we denote by ‖u‖ = ( ∫ Ω |∇u|pdx )1/p the norm in the Sobolev space W 1,p 0 (Ω), by ‖ ·‖∗ the norm in W−1,p′(Ω) which is the dual space of W 1,p 0 (Ω), by ‖u‖p = ( ∫ Ω |u|pdx )1/p the usual Lp norm, by |E| the n-dimensional Lebesgue measure of a set E ⊂ Rn. Moreover, we use “→ ” and “ ⇀ ” denote the strong and weak convergence respectively, “ ↪→ ” and “ ↪→↪→ ” denote imbedding and compact imbedding respectively. We denote the subsequence of a sequence {un} as {un} to simplify the notion unless specified. And X denotes a Banach space. Definition 2.1. We shall say that the convex function A : X → R is uniformly convex on the set (convex) S ⊂ X, if for any ε1 > 0, there exists δ(ε1) > 0 such that A (x+ y 2 ) ≤ 1 2 A(x) + 1 2 A(y)− δ(ε1), for x, y ∈ S with ‖x−y‖ > ε1. If A is uniformly convex on every ball of X, we shall say that A is locally uniformly convex, i.e., if for any ε2 > 0, there exists δ(ε2) > 0 such that x, y ∈ X, |A(x)| ≤ 1, |A(y)| ≤ 1 and |A(x− y)| > ε2, then∣∣A(x+ y 2 )∣∣ < 1− δ(ε2). Remark 2.2 ([18]). X is uniformly convex if and only if its norm is locally uni- formly convex. Remark 2.3 ([18]). The Banach space W 1,p 0 (Ω) with norm ‖u‖ = ( ∫ Ω |∇u|pdx )1/p is uniformly convex. Remark 2.4 ([3]). Every uniformly convex Banach space is reflexive. That is, the Banach space W 1,p 0 (Ω) is reflexive. Remark 2.5. [24] Let X be a reflexive Banach space, {xn} is a bounded sequence in X. Then {xn} has weak convergent subsequence. That is, the bounded sequence in reflexive Banach space W 1,p 0 (Ω) has weak convergent subsequence. EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 5 Definition 2.6. Let I be a functional defined in Banach space X. We say that I is weakly lower semicontinuous, if for any sequence {xn} such that xn ⇀ x weakly, then we have lim inf n→∞ I(xn) ≥ I(x). Definition 2.7. An operator I ′ : X → X∗ satisfies the (S+) condition, if for every sequence {xn} ⊂ X such that xn ⇀ x and lim sup n→+∞ 〈I ′(xn), xn − x〉 ≤ 0, we have xn → x strongly. We would like to mentioned that the (S+) condition is used to prove that the weak convergent sequence obtained is actually strongly convergent. Next, we verify that the relevant functional satisfies the (S+) condition. Proposition 2.8. Let X be a Banach space. We denote I(x) = ‖x‖p, where p ≥ 1, x ∈ X, then I : X → R is C1 and I ′ : X → X∗ satisfies the (S+) condition. Proof. It is easy to verify that I : X → R is a C1 functional. Let {xn} be a sequence in X such that xn ⇀ x and lim sup n→+∞ 〈I ′(xn), xn − x〉 ≤ 0. Claim: xn → x in X. Indeed, since {xn} is weakly convergent, it is bounded. That is, there is a large enough R > 0 such that ‖xn‖ < R. In view of Remark 2.3, we obtain that I is locally uniformly convex. Then I is locally bounded, and therefore, I(xn) is bounded. For a subsequence {xn}, we assume that I(xn)→ c. Since ‖ · ‖ is continuous and convex, we know that I is weakly lower semicontinuous. Also by the definition of weakly lower semicontinuous, we have I(x) ≤ lim inf I(xn) = c. On the other hand, since I is convex, its graphic lies above the tangent hyper- plane at xn, that is, I(x) ≥ I(xn) + 〈I ′(xn), x− xn〉. Using that lim sup n→+∞ 〈I ′(xn), xn − x〉 ≤ 0, we deduce that I(x) ≥ c. Then I(x) = c. Also we have that xn+x 2 ⇀ x, and again by weakly lower semicontinuity, we obtain c = I(x) ≤ lim inf I (x+ xn 2 ) . (2.1) If we suppose that {xn} does not convergence strongly to x, then there exists an ε > 0 and a subsequence {xn} that verifies ‖x − xn‖ ≥ ε. Using the uniform convexity of I over ball B(0, R), we obtain that there exists a δ(ε) > 0 such that 1 2 I(x) + 1 2 I(xn)− I (x+ xn 2 ) ≥ δ(ε). Taking n→ +∞, we have lim sup I (x+ xn 2 ) ≤ c− δ(ε), which contradicts (2.1). Then the desired conclusion follows from the claim. � 6 X. WANG, P. ZHAO EJDE-2020/52 Definition 2.9. Let (X, ‖·‖X) be a real Banach space with dual space (X∗, ‖·‖X∗), and I ∈ C1(X,R). For c ∈ R, we shall say I satisfies the (PS)c condition, if for any sequence {xn} ⊂ X such that I(xn)→ c and I ′(xn)→ 0, we have that {xn} is strongly convergent in X. Theorem 2.10 (Mountain pass lemma [24]). Let X be a real Banach space, I ∈ C1(X,R) satisfies (1) I(0) ≤ 0; (2) There exist constants ρ, α > 0 such that I(u) ≥ α, when ‖u‖ = ρ; (3) There exists an e ∈ E \Bρ such that I(e) < 0. Denote c = infγ∈Γ max0≤t≤1 I(γ(t)), where Γ = {γ ∈ C([0, 1];X) : γ(0) = 0, γ(1) = e}. Then c > 0 and there is a sequence {xn} ⊂ X such that I(xn)→ c, I ′(xn)→ 0. Furthermore, if f satisfies the (PS)c condition, then c is the critical value of I. 2.2. Existence of a nontrivial weak solution to the elliptic equation. In this subsection, we establish the existence of the nontrivial weak solution for the elliptic equation. We firstly introduce the energy functional corresponding to the elliptic equation (1.1). If Ω ⊂ Rn is a bounded domain and f satisfies (H1) and (H2), then we define functional in W 1,p 0 (Ω), Iλ(u) = 1 p ∫ Ω |∇u|pdx− λ ∫ Ω F (x, u)dx. (2.2) For any λ ∈ R1, a straightforward computation yields that Iλ ∈ C1(W 1,p 0 (Ω),R), and 〈I ′λ(u), v〉 = ∫ Ω |∇u|p−2∇u∇v dx− λ ∫ Ω f(x, u)v dx, (2.3) for any u ∈ W 1,p 0 (Ω). Next, we prove that the functional Iλ satisfies the mountain pass geometry as follows. Lemma 2.11. If λ > 0 and f satisfies (H1)–(H3), then (1) Iλ(u) is unbounded from below in W 1,p 0 (Ω); (2) u = 0 is a strictly local minimum for Iλ(u). Proof. For any M > 0, it follows from (H3) that there is a CM > 0 such that F (x, t) ≥Mtp − CM for all t ≥ 0 and all x ∈ Ω. (2.4) Indeed, for any M > 0, there is a s0 > 0 such that F (x, t) tp ≥M whenever t > s0. That is, F (x, t) ≥Mtp whenever t > s0. Furthermore, thanks to F being continuous on Ω× [0, s0], we have max x∈Ω,0≤t≤s0 {F (x, t)−Mtp} ≤ CM . Also since F (x, t)−Mtp + max x∈Ω,0≤t≤s0 {F (x, t)−Mtp} ≥ 0, EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 7 we obtain F (x, t) ≥ Mtp − CM , for any x ∈ Ω and 0 ≤ t ≤ s0. To sum up, we obtain (2.4). Taking φ ∈ W 1,p 0 (Ω) with φ > 0, and t ≥ 0. Then for any λ > 0, we have Iλ(tφ) = 1 p tp ∫ Ω |∇φ|pdx− λ ∫ Ω F (x, tφ)dx ≤ 1 p tp‖φ‖p − λtpM ∫ Ω φpdx+ λCM |Ω| = tp (1 p ‖φ‖p − λM ∫ Ω φpdx ) + λCM |Ω|. If M is large enough such that 1 p ‖φ‖p − λM ∫ Ω φpdx < 0, then limt→+∞ Iλ(tφ) = −∞, which is equivalent to (1). On the other hand, for any ε > 0, by using (H1) and (H2), it is easy to see that there exists a Cε > 0 such that |f(x, t)| ≤ ε|t|p−1 + Cε|t|q−1 for all (x, t) ∈ Ω× R. That is, |F (x, t)| ≤ ε|t|p + Cε|t|q for all (x, t) ∈ Ω× R, (2.5) where q ∈ (p, p∗). Indeed, in view of (H1), for any ε > 0, there is a δ > 0 such that |f(x, t)| |t|p−1 < ε for any |t| < δ. That is, |f(x, t)| < ε|t|p−1 for any |t| < δ. Furthermore, from (H2), we have |f(x, t)| ≤ a+ b|t|q−1 ≤ a|t|q−1 + b|t|q−1 = (a+ b)|t|q−1 for |t| > 1, |f(x, t)| ≤ a+ b|t|q−1 = (a|t|1−q + b)|t|q−1 ≤ (a|δ|1−q + b)|t|q−1 for δ ≤ |t| ≤ 1. Therefore, for any ε > 0, there is a Cε > 0 such that |f(x, t)| ≤ ε|t|p−1 + Cε|t|q−1 for all (x, t) ∈ Ω× R, (2.6) where Cε = max{(a + b), a|δ|1−q + b}. It follows from (2.5) and the Poincaré inequality that ‖u‖pp ≤ 1 λ1 ‖u‖p, where 0 < λ1 = inf u∈W 1,p 0 (Ω),u6=0 ‖u‖p ‖u‖pp . 8 X. WANG, P. ZHAO EJDE-2020/52 And therefore, for any λ > 0 and ε > 0 small enough such that 1 p − λε λ1 > 0, the Hölder inequality implies Iλ(u) = 1 p ‖u‖p − λ ∫ Ω F (x, u)dx ≥ 1 p ‖u‖p − λε ∫ Ω |u|pdx− λCε ∫ Ω |u|qdx ≥ 1 p ‖u‖p − λε ∫ Ω |u|pdx− λCε|Ω| p−q p (∫ Ω |u|pdx )q/p ≥ (1 p − λε λ1 ) ‖u‖p − λCε|Ω| p−q p ( 1 λ1 ‖u‖p )q/p = (1 p − λε λ1 ) ‖u‖p − λCε λ q/p 1 |Ω| p−q p ‖u‖q = (1 p − λε λ1 − λCε λ q/p 1 |Ω| p−q p ‖u‖q−p ) ‖u‖p ≥ 1 2 (1 p − λε λ1 ) ‖u‖p, (2.7) provided ‖u‖ = ρ is sufficiently small such that λCε λ q/p 1 |Ω| p−q p ‖u‖q−p < 1 2 (1 p − λε λ1 ) , when q ∈ (p, p∗). Therefore u = 0 is a strictly local minimum for Iλ(u). � Lemma 2.12. Assume f satisfies (H1)–(H3) and 0 < λ0 < µ0. Then Iλ(u) pos- sesses uniform mountain pass geometric structure around u = 0 for λ ∈ [λ0, µ0], i.e., there is an e ∈ W 1,p 0 (Ω) such that Iλ(e) < 0 for any λ ∈ [λ0, µ0], and there exist constants ρ, α > 0 such that Iλ(u) ≥ α for any λ ∈ [λ0, µ0], and u ∈W 1,p 0 (Ω) with ‖u‖ = ρ. Proof. Fix ε > 0 small enough, in view of (2.7), we have Iλ(u) ≥ 1 2 (1 p − µ0ε λ1 ) ‖u‖p, for any λ ∈ [λ0, µ0] and u ∈W 1,p 0 (Ω). Thus there is a ρ = ρ(µ0, ε) > 0, taking α = 1 2 (1 p − µ0ε λ1 ) ‖ρ‖p, we have Iλ(u) ≥ α, for any λ ∈ [λ0, µ0] and u ∈W 1,p 0 (Ω) with ‖u‖ = ρ. Let us take φ ∈W 1,p 0 (Ω) with φ > 0, and M > 0 large enough such that 1 p ||φ||p − λ0M ∫ Ω φpdx < 0. As a consequence of (2.4), for any t > 0, we have Iλ0 (tφ) ≤ 1 p tp‖φ‖p − λ0t pM ∫ Ω φpdx+ λ0CM |Ω| ≤ tp (1 p ‖φ‖p − λ0M ∫ Ω φpdx ) + λ0CM |Ω|. EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 9 Furthermore, taking e = t0φ with t0 large enough such that Iλ0 (e) < 0, for any 0 < λ0 < λ, we have Iλ(e) < Iλ0(e) < 0. This means that Iλ(e) < 0. � By Lemmas 2.11, 2.12 and the Mountain pass lemma (Theorem 2.10), there is a (PS)c sequence {un} ⊂W 1,p 0 (Ω) satisfies Iλ(un)→ c, I ′λ(un)→ 0. (2.8) Next, we prove that the (PS)c sequence is actually bounded. Lemma 2.13. Assume f satisfies (H1)–(H4), then the (PS)c sequence {un} ⊂ W 1,p 0 (Ω) for the functional Iλ defined in (2.2) is bounded. Proof. Suppose towards a contradiction that ‖un‖ → +∞. (2.9) Denote wn = un ‖un‖ . It is obvious that wn ∈ W 1,p 0 (Ω) with ‖wn‖ = 1, and therefore, it follows from the Remark 2.5 that there exists a w ∈W 1,p 0 (Ω) such that wn ⇀ w in reflexive Banach space W 1,p 0 (Ω). Since Ω is bounded, the Sobolev’s compact imbedding theorem implies that wn → w in Lq(Ω) and L1(Ω), and therefore wn(x)→ w(x) a.e. in Ω. Set Ω0 = {x ∈ Ω, w(x) 6= 0}. Then lim n→+∞ un ‖un‖ = lim n→+∞ wn = w 6= 0 in Ω0. And in view of (2.9), we have |un| → +∞ a.e. in Ω0. By (H3), it is easy to see that lim n→+∞ F (x, un) |un|p = +∞ a.e. in Ω0, which implies lim n→+∞ F (x, un) |un|p |wn|p = +∞ a.e. in Ω0. (2.10) It follows from (H3) that there is a N0 > 0 such that F (x, un) |un|p > 1, (2.11) for any x ∈ Ω and |un| ≥ N0. Since F is continuous on Ω × [−N0, N0], there is a M > 0 such that |F (x, un)| ≤M for all (x, un) ∈ Ω× [−N0, N0]. (2.12) Combining (2.11) with (2.12), we deduce that there is a constant C such that F (x, un) ≥ C for all (x, un) ∈ Ω× R, which shows that F (x, un)− C ‖un‖p ≥ 0. (2.13) Thanks to (2.8), we have c = Iλ(un) + o(1) = 1 p ‖un‖p − λ ∫ Ω F (x, un)dx+ o(1). 10 X. WANG, P. ZHAO EJDE-2020/52 So we obtain ‖un‖p = pc+ pλ ∫ Ω F (x, un)dx+ o(1). (2.14) In accordance with (2.8) and (2.14), we obtain∫ Ω F (x, un)dx→ +∞. (2.15) Next, we claim that |Ω0| = 0. In fact, if |Ω0| 6= 0, then by using (2.10), (2.14) and the Fatou’s lemma, we have +∞ = ∫ Ω0 lim inf n→+∞ F (x, un) |un|p |wn|pdx− ∫ Ω0 lim sup n→+∞ C ‖un‖p = ∫ Ω0 lim inf n→+∞ (F (x, un) |un|p |wn|p − C ‖un‖p ) dx ≤ lim inf n→+∞ ∫ Ω0 (F (x, un) |un|p |wn|p − C ‖un‖p ) dx ≤ lim inf n→+∞ ∫ Ω (F (x, un) |un|p |wn|p − C ‖un‖p ) dx = lim inf n→+∞ ∫ Ω F (x, un) |un|p |wn|pdx− lim sup n→+∞ ∫ Ω C ‖un‖p dx = lim inf n→+∞ ∫ Ω F (x, un) |un|p |wn|pdx− lim sup n→+∞ C|Ω| ‖un‖p = lim inf n→+∞ ∫ Ω F (x, un) |un|p |wn|pdx = lim inf n→+∞ ∫ Ω F (x, un)dx pc+ pλ ∫ F (x, un)dx+ o(1) . (2.16) Therefore, it follows from (2.15) and (2.16) that +∞ ≤ 1 pλ . This is a contradiction, which implies that |Ω0| = 0. Hence we obtain that w(x) = 0 a.e. in Ω. From (2.8), we have Iλ(un) = 1 p ‖u‖p − λ ∫ Ω F (x, un)dx→ c. Then Iλ(un) ‖un‖p = 1 p − λ ∫ Ω F (x, un) |un|p |wn|pdx, that is, ∫ Ω F (x, un) |un|p |wn|pdx→ 1 pλ , Again by (2.8), we have 〈I ′λ(un), un〉 = ‖un‖p − λ ∫ Ω f(x, un)undx = o(1), where o(1)→ 0, as n→∞. Then 1− λ ∫ Ω unf(x, un) |un|p |wn|pdx = 〈I ′λ(un), un〉 ‖un‖p ≤ ‖I ′ λ(un)‖ · ‖un‖ ‖un‖p = ‖I ′λ(un)‖ ‖un‖p−1 → 0, that is, ∫ Ω unf(x, un) |un|p |wn|pdx→ 1 λ . EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 11 Therefore, ∫ Ω µF (x, un)− unf(x, un) |un|p |wn|pdx→ µ pλ − 1 λ . However, the hypothesis (H4) implies lim sup µF (x, un)− unf(x, un) |un|p |wn|p ≤ lim supC |un|p + 1 |un|p |wn|p = 0. Therefore, µ pλ − 1 λ ≤ 0, which leads to a contradiction. Hence {un} is bounded, i.e., there is a C > 0 such that ‖un‖ ≤ C < +∞. � Lemma 2.14. Assume f satisfies (H2). Then the (PS)c sequence {un} ⊂W 1,p 0 (Ω) for the functional Iλ defined in (2.2) has a convergent subsequence. Proof. Let {un} ⊂ W 1,p 0 (Ω) be a (PS)c sequence for the functional Iλ. Using Lemma 2.13, we deduce that {un} is bounded. Therefore, there exists a u ∈ W 1,p 0 (Ω) such that un ⇀ u in W 1,p 0 (Ω). (2.17) Furthermore, the Sobolev’s compact imbedding implies un → u in Lq(Ω). Denote εn = ‖I ′λ(un)‖∗. It is easy to check that εn → 0 and |〈I ′λ(un), v〉| = ∣∣∣ ∫ Ω |∇un|p−2∇un∇v dx− λ ∫ Ω f(x, un)v dx ∣∣∣ ≤ εn‖v‖, (2.18) for any v ∈W 1,p 0 (Ω). Thanks to (H2), we have∫ Ω (f(x, un)− f(x, u))(un − u)dx→ 0. (2.19) In fact, it follows from the Hölder inequality that∣∣ ∫ Ω (f(x, un)− f(x, u))(un − u)dx ∣∣ ≤ (∫ Ω |f(x, un)− f(x, u)|pdx )1/p(∫ Ω |un − u|qdx )1/q . Since ‖un‖ ≤ C (see Lemma 2.13) and f is continuous on Ω × [−C,C], there is a M > 0 such that |f(x, un)| ≤M for all (x, un) ∈ Ω× [−C,C]. Therefore,(∫ Ω |f(x, un)− f(x, u)|pdx )1/p ≤ (∫ Ω (2M)pdx )1/p = 2M |Ω|1/p,(∫ Ω |un − u|qdx )1/q → 0, since un → u in Lq(Ω). Hence∫ Ω (f(x, un)− f(x, u))(un − u)dx ≤ 2M |Ω|1/p (∫ Ω |un − u|qdx )1/q → 0, 12 X. WANG, P. ZHAO EJDE-2020/52 as n→ +∞. Taking v = un − u in (2.18), and it follows from (2.19) that 〈I ′(un), un − u〉 = ∫ Ω |∇un|p−2∇un∇(un − u)dx = 〈I ′(un), un − u〉+ ∫ Ω f(x, un)(un − u)dx ≤ εn‖un − u‖+ ∫ Ω f(x, un)(un − u)dx→ 0. By using the (S+) property of I ′λ, we conclude that un → u in W 1,p 0 (Ω). � roof of Theorem 1.1. Firstly, in view of Lemmas 2.11, 2.12 and the Mountain pass lemma (Theorem 2.10), there is a (PS)c sequence {un} ⊂ W 1,p 0 (Ω) that satisfies I(un)→ c and I ′(un)→ 0. Secondly, in accordance to Lemma 2.14, we deduce that {un} converges strongly to some function u ∈W 1,P 0 (Ω). Clearly, u is a weak solution for the problem (1.1). This completes the proof. � 3. Existence of the nontrivial weak solution for the superlinear elliptic system In this section, we establish the existence of the nontrivial solution for the su- percritical superlinear (i.e., p ∈ (2, 2∗), q ∈ (2∗,+∞)) elliptic system (1.2) without the AR condition. 3.1. Preliminaries. The key point is to show the boundedness of the (PS)∗c se- quence of the energy functional. We denote by | · |t the usual Lt(Ω) norm for all t ∈ [1,∞]. For q > 2∗, let Vq = H1 0 (Ω) ∩ Lq(Ω) and the Banach space Vq equipped with the norm ‖v‖Vq = (|∇v|22 + |v|2q) 1 2 . Let Eq be the product space H1 0 (Ω) × Vq with elements denoted by z = (u, v) and the norm in Eq by ‖z‖q = (|∇u|22+‖v‖2Vq ) 1 2 . We also denote |z| = |u|+ |v|. Eq has the direct sum decomposition Eq = E−q ⊕ E+, z = z− + z+, where E−q = {0}×Vq and E+ = H1 0 (Ω)×{0}. For simplicity, write z+ = u, z− = v. If Ω ⊂ Rn is a smooth bounded domain and H satisfies (H5), then we define the functional on Eq as I(z) := 1 2 ∫ Ω ( |∇u|2 − |∇v|2 ) dx− ∫ Ω H(x, z)dx. (3.1) By a straightforward computation, we obtain that I is a C1 functional, and 〈I ′(z), w〉 = ∫ Ω ∇u∇ϕdx− ∫ Ω Hu(x, z)ϕdx+ ∫ Ω Hv(x, z)ψdx. (3.2) It is not difficult to verify that the critical point of I is the solution of the elliptic system (1.2). Next, we show that the Frechet derivative of the functional I is weakly sequence continuous . Lemma 3.1. Assume (H5) holds. Then I ′ is weakly sequence continuous, that is, I ′(zn) ⇀ I ′(z), as zn ⇀ z. EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 13 Proof. Suppose zn ⇀ z in Eq. We claim that I ′(zn) ⇀ I ′(z), that is, 〈I ′(zn), w〉 → 〈I ′(z), w〉, for any w = (ϕ,ψ) ∈ Eq. Since zn ⇀ z, we have un ⇀ u in H1 0 , and vn ⇀ v in Vq. Thus (un, ϕ)→ (u, ϕ), that is,∫ Ω ∇un∇ϕdx→ ∫ Ω ∇u∇ϕdx. Similarly, we have ∫ Ω ∇vn∇ψdx→ ∫ Ω ∇v∇ψdx. Therefore, ∫ Ω (∇un∇ϕ−∇vn∇ψ) dx→ ∫ Ω (∇u∇ϕ−∇v∇ψ) dx. Next, we verify the following two equalities lim n→∞ ∫ Ω Hu(x, zn)ϕdx = ∫ Ω Hu(x, z)ϕdx, for any ϕ ∈ H1 0 (Ω), (3.3) lim n→∞ ∫ Ω Hv(x, zn)ψdx = ∫ Ω Hv(x, z)ψdx, for any ψ ∈ Vq. (3.4) It follows from the Sobolev’s compact imbedding theorem and the Interpolation theorem that un → u in Lt for any t ∈ [1, 2∗), vn → v in Lt for any t ∈ [1, q). By (H5), we have |Hu(x, zn)ϕ| ≤ γ0 ( |ϕ|+ |un|p−1|ϕ|+ |vn| q 2−1|ϕ| ) and ∫ Ω ( |ϕ|+ |un|p−1|ϕ|+ |vn| q 2−1|ϕ| ) dx ≤ ∫ Ω |ϕ|dx+ (∫ Ω |un|(p−1) p p−1 dx ) p−1 p (∫ Ω |ϕ|pdx )1/p + (∫ Ω |vn|( q 2−1)·2∗dx )1/2∗(∫ Ω |ϕ|2 ∗ dx )1/2∗ = |ϕ|1 + |un|p−1 p |ϕ|p + |vn| q 2−1 2∗( q 2−1) |ϕ|2∗ . Thanks to ϕ ∈ H1 0 (Ω) ↪→ L2∗ , and(q 2 − 1 ) 2∗ = (q 2 − 1 ) 2∗ 2∗ − 1 < (q 2 − 1 ) 2 < q. Then we obtain (3.3). Furthermore, (3.4) is obvious for ψ ∈ L∞. In fact,∣∣∣ ∫ Ω Hv(x, zn)ψdx− ∫ Ω Hv(x, z)ψdx ∣∣∣ = ∣∣∣ ∫ Ω (Hv(x, zn)−Hv(x, z))ψdx ∣∣∣ ≤ |ψ|∞ ∫ Ω |Hv(x, zn)−Hv(x, z)|dx→ 0. 14 X. WANG, P. ZHAO EJDE-2020/52 Generally, for ψ ∈ Vq, there is ψm ∈ L∞ such that ψm → ψ(m → ∞) in Lq, since L∞ is dense in Lq. In the light of (H5), we have |Hv(x, u, v)| ≤ γ0 ( 1 + |u|p−1 + |v|q−1 ) . And zn is bounded in Eq, then∣∣ ∫ Ω Hv(x, zn)ψdx ∣∣ = ∣∣ ∫ Ω Hv(x, zn)(ψm + (ψ − ψm))dx ∣∣ ≤ ∣∣ ∫ Ω Hv(x, zn)ψmdx ∣∣+ ∣∣ ∫ Ω Hv(x, zn)(ψ − ψm)dx ∣∣ ≤ ∣∣ ∫ Ω Hv(x, zn)ψmdx ∣∣+ c1 ( |ψ − ψm|1 + |un|p−1 p |ψ − ψm|p + |vn|q−1 q |ψ − ψm|q ) ≤ ∣∣ ∫ Ω Hv(x, zn)ψmdx ∣∣+ c2 (|ψ − ψm|1 + |ψ − ψm|p + |ψ − ψm|q) . Therefore, we obtain (3.4). Then 〈I ′(zn), w〉 → 〈I ′(z), w〉 for all w ∈ Eq. � Next, we introduce the Linking theorem, which is the basic tool for the existence of the nontrivial weak solution for the elliptic system. Let E be a Banach space with the norm ‖ · ‖. Suppose E has the direct sum decomposition E = E1⊕E2, where E1 and E2 are both infinite dimension. Assume (e1 n) and (e2 n) are the basis of E1 and E2 respectively. Let Xn := span{e1 1, . . . , e 1 n} ⊕ E2, Xm := E1 ⊕ span{e2 1, . . . , e 2 m}, and (Xm)⊥ denote the supplement of Xm in E. For a functinal I ∈ C1(E,R), let In := I ∣∣ Xn denote the restriction of I to Xn. Definition 3.2. Let E be a Banach space, and I ∈ C1(E,R). We shall say {zj} ⊂ E is a (PS)∗c sequence, if zj ∈ Xnj satisfies I(zj)→ c, I ′nj (zj)→ 0, as nj → ∞. Furthermore, we shall say I satisfies (PS)∗c condition, if any (PS)∗c sequence has a convergent subsequence. Definition 3.3. Let E be a Banach space, Q,Q0 and S are the closed subset of E with Q0 ⊂ Q. We say (Q,Q0) links with S, if (1) Q0 ∩ S = ∅; (2) For any continuous map γ : Q→ E satisfies γ |Q0 = id ∣∣ Q0 , we have γ(Q) ∩ S 6= ∅. Remark 3.4 ([24]). Let (Q,Q0) link with S. Define the subset family of E as Γ = {γ ∈ C(Q,X) : γ |Q0= id |Q0}. If I is a C1 functional on E, set c = inf γ∈Γ sup x∈γ(Q) I(x), (3.5) then under suitable conditions, we can demonstrate c is the critical value of I. EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 15 Theorem 3.5 (Linking theorem [24]). Assume E is a Banach space, Q,Q0 and S are the closed subset of E with Q0 ⊂ Q, and (Q,Q0) links with S. Moreover, assume I ∈ C1(E,R) satisfies (1) supx∈Q I(x) < r < +∞; (2) There exists a constant β > α such that sup x∈Q0 I(x) ≤ α, inf x∈S I(x) ≥ β. Then there is a sequence {xn} ⊂ E such that I(xn)→ c, I ′n(xn)→ 0, where c is defined in (3.5). 3.2. Existence of a nontrivial weak solution for the elliptic system. Now we set E ′ = E−q , E 2 = E+ and e1 n = e−n , e 2 n = e+ n for all n ∈ N , and therefore, Eq = E1 ⊕ E2. We will show that the functional I defined in (3.1) satisfies the linking geometry. Lemma 3.6. Suppose H satisfies (H5) and (H9). Then there exist constants r and ρ > 0 such that inf I(∂B+ r ) ≥ ρ, where B+ r = Br(0) ∩ E+. Proof. Recalling (H5) and (H9), for any ε > 0, there is Cε > 0 such that H(x, u, 0) ≤ ε|u|2 + Cε|u|2 ∗ . In fact, it follows from (H5) that |Hu(x, u, 0)| ≤ γ0 ( 1 + |u|p−1 ) . (3.6) Furthermore, in the light of (H9), we have Hu(x, u, 0) = o(|u|) as u→ 0. Then for any ε > 0, there is a constant c > 0 such that |Hu(x, u, 0)| ≤ ε|u| whenever |u| < c. And there exists C > 0 such that |Hu(x, u, 0)| ≤ γ0 ( 1 + |u|p−1 ) ≤ C|u|p−1 whenever |u| > c. To sum up, we have |Hu(x, u, 0)| ≤ ε|u|+ Cε|u|p−1. That is, |H(x, u, 0)| ≤ ε|u|2 + Cε|u|p < ε|u|2 + Cε|u|2 ∗ . Therefore, I(u) := 1 2 ∫ Ω |∇u|2dx− ∫ Ω H(x, u, 0)dx ≥ 1 2 ∫ Ω |∇u|2dx− ε|u|22 − Cε|u|2 ∗ 2∗ . Then we obtain the conclusion. � Assume e ∈ E+ with |∇e|22 = 1, and let Q = {(se, v) : 0 ≤ s ≤ r1, ‖v‖q ≤ r2}. Lemma 3.7. Suppose H satisfies (H8) and (H9). Then there are constants r1, r2 > 0 with r1 > r such that I(z) ≤ 0 for all z ∈ ∂Q. 16 X. WANG, P. ZHAO EJDE-2020/52 Proof. In view of (H9) and H(x, 0, v) ≥ 0, we have I(z) := −1 2 ∫ Ω |∇v|2dx− ∫ Ω H(x, 0, v)dx ≤ 0, when z ∈ E−q . By (H8), we obtain I((se, v)) = s2 2 ∫ Ω |∇e|2dx− 1 2 ∫ Ω |∇v|2dx− ∫ Ω H(x, se, v)dx ≤ s2 2 − 1 2 |∇v|22 − ∫ Ω (γ1(|se|p + |v|q)− γ2) dx ≤ s2 2 − 1 2 |∇v|22 − c1 ∫ Ω (|se|p + |v|q) dx+ c2. Since p > 2, we obtain the conclusion. � Next, we establish the boundedness of the (PS)∗c sequence, which plays an im- portant role in the existence theory of the nontrivial weak solution. Lemma 3.8. Assume H satisfies (H6) and (H7). Then the (PS)∗c sequence {zn} ⊂ Eq is bounded, where zn = (un, vn). Proof. Without loss of generality, suppose ‖zn‖q → +∞. By ‖zn‖2q = |∇un|22 + |∇vn|22 + |vn|2q, we assume that |∇un|2 → +∞, |∇vn| |∇un|2 → a < 1. Setting Yn = zn ‖zn‖q , then Yn ∈ Eq with ‖Yn‖q = 1. Therefore, there is Y ∈ Eq such that Yn ⇀ Y in Eq. Then we have Yn(x)→ Y (x) a.e. in Ω. Denote Ω0 = {x ∈ Ω, Y (x) 6= 0}. Then we have lim n→+∞ zn ‖zn‖q = lim n→+∞ Yn = Y 6= 0 a.e. in Ω0, (3.7) which implies |zn| → +∞ a.e. in Ω0. It follows from (H6) that lim n→+∞ H(x, zn) |zn|2 |Yn|2 = +∞ a.e. in Ω0. Again by using (H6), there is N0 > 0 such that, for any x ∈ Ω, we have H(x, zn) |zn|2 > 1 whenever |zn| ≥ N0. (3.8) Since H is continuous on Ω × [−N0, N0] × [−N0, N0], there exists an M > 0 such that |H(x, zn)| ≤M, (3.9) for any (x, zn) ∈ Ω× [−N0, N0]× [−N0, N0]. From (3.8) and (3.9), we deduce that there exists constant C such that H(x, zn) ≥ C for all (x, zn) ∈ Ω× R× R. EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 17 This implies that H(x, zn)− C ‖zn‖2q ≥ 0. Since I(zn) = 1 2 ∫ Ω ( |∇un|2 − |∇vn|2 ) dx− ∫ Ω H(x, zn)dx = c+ o(1), we have ∫ Ω ( |∇un|2 − |∇vn|2 ) dx = 2c+ 2 ∫ Ω H(x, zn)dx+ o(1). In view of |∇un|2 → +∞, we have ∫ Ω H(x, zn)dx→ +∞. Therefore, +∞ = ∫ Ω0 lim inf n→+∞ H(x, zn) |zn|2 |Yn|2dx− ∫ Ω0 lim sup n→+∞ C ‖zn‖2q dx = ∫ Ω0 lim inf n→+∞ (H(x, zn) |zn|2 |Yn|2 − C ‖zn‖2q ) dx ≤ lim inf n→+∞ ∫ Ω0 (H(x, zn) |zn|2 |Yn|2 − C ‖zn‖2q ) dx ≤ lim inf n→+∞ ∫ Ω (H(x, zn) |zn|2 |Yn|2 − C ‖zn‖2q ) dx = lim inf n→+∞ ∫ Ω H(x, zn) ‖zn‖2q dx− lim sup n→+∞ ∫ Ω C ‖zn‖2q dx = lim inf n→+∞ ∫ Ω H(x, zn) ‖zn‖2q dx− lim sup n→+∞ C|Ω| ‖zn‖2q = lim inf n→+∞ ∫ Ω H(x, zn) ‖zn‖2q dx = lim inf n→+∞ ∫ Ω H(x, zn)dx |∇u|22 + |∇v|22 + |v|2q = lim inf n→+∞ ∫ Ω H(x, zn)dx 2c+ 2 ∫ Ω H(x, zn)dx+ 2|∇v|22 + |v|2q + o(1) = 1 2 , which leads to a contradiction. Then |Ω0| = 0, and therefore, Y (x) = 0 a.e. in Ω0. Since I(zn) = 1 2 ∫ Ω ( |∇un|2 − |∇vn|2 ) dx− ∫ Ω H(x, zn)dx, we have I(zn) ‖zn‖2q = 1 2 − ∫ Ω H(x, zn) |zn|2 |Yn|2dx, that is, ∫ Ω H(x, zn) |zn|2 |Yn|2dx→ 1 2 . Moreover, thanks to 〈I ′(zn), zn〉 = ∫ Ω |∇un|dx− ∫ Ω |∇vn|2dx− ∫ Ω Hz(x, zn)zn dx. 18 X. WANG, P. ZHAO EJDE-2020/52 we have 1− ∫ Ω Hz(x, zn)zn ‖zn‖2q dx = 〈I ′(zn), zn〉 ‖zn‖2q leq ‖I ′(zn)‖ · ‖zn‖q ‖zn‖2q = ‖I ′(zn)‖ ‖zn‖2q → 0, that is, ∫ Ω znHz(x, zn) |zn| |Yn|2dx→ 1. Therefore, ∫ Ω µH(x, zn)− znHz(x, zn) |zn|2 |Yn|dx→ µ 2 − 1. Then it follows from (H7) that lim sup µH(x, zn)− znHz(x, zn) |zn|2 |Yn|2 ≤ lim supC |zn|2 + 1 |zn|2 |Yn|2 = 0; that is, µ 2 − 1 ≤ 0. Hence, µ ≤ 2. This is a contradiction. Therefore, {zn} is bounded in Eq. � Lemma 3.9. Let {zn} ⊂ Xn be a (PS)∗c sequence. Then there exists z ∈ Eq such that along a subsequence, zn ⇀ z with I ′(z) = 0 and I(z) ≥ c. Proof. Since {zn} ⊂ Xn is a (PS)∗c subsequence, it follows from Lemma 3.8 that the (PS)∗c sequence is bounded, that is, {zn} is bounded. Thus, {zn} has weakly convergent subsequence, might as well suppose zn ⇀ z in Eq. Then for any 1 ≤ s < 2∗, the imbedding theorem implies that zn → z in (Ls(Ω))2. As a result, zn(x)→ z(x) a.e. in Ω. Moreover, by using Lemma 3.1, we know I ′ is weakly sequence continuous. Hence, we obtain I ′(z) = 0. Let w = (ϕ,ψ) = (un − u, 0) in (3.2) and by I ′n(zn)→ 0, we have (∇un,∇un −∇u)L2 = I ′n(zn)(un − u, 0) + ∫ Ω Hu(x, zn)(un − u)dx = o(1) + ∫ Ω Hu(x, zn)(un − u)dx. By using (H5), Hölder’s inequality and 2q q+2 < 2 < 2∗, we have∣∣ ∫ Ω Hu(x, zn)(un − u)dx ∣∣ ≤ ∫ Ω γ0 ( 1 + |un|p−1 + |vn| q 2−1 ) |un − u|dx ≤ γ0 ( |un − u|1 + |un|p−1 p |un − u|p + |vn| q 2−1 q |un − u| 2q q+2 ) = o(1). Therefore, (∇un,∇un −∇u)L2 = o(1), that is, |∇un|22 → |∇u|22. Then un → u in H1 0 (Ω). EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 19 Let pn : Eq → Xn denote the projection. Observe that Pnz → z in Eq for all z ∈ Eq. Moreover, using again (H5) and Hölder’s inequality, we deduce∣∣ ∫ Ω Hv(x, zn)(v − Pnv)dx ∣∣ ≤ C ( |v − Pnv|1 + |un|p−1 p |v − Pnv|p + |vn|q−1 q |v − Pnv|q ) → 0. On the other hand, lettin w = (ϕ,ψ) = (0, vn−Pnv) in (3.2), and by I ′n(zn)→ 0, we obtain I ′n(zn)(0, vn − Pnv) = − ∫ Ω ∇vn∇(vn − Pnv)dx+ ∫ Ω Hv(x, zn)(vn − Pnv)dx = − ∫ Ω ∇vn∇(vn − v + v − Pnv)dx+ ∫ Ω Hv(x, zn)(vn − v + v − Pnv)dx = −(∇vn∇vn −∇v)L2 − ∫ Ω ∇vn∇(v − pn)dx+ ∫ Ω Hv(x, zn)(vn − v)dx + ∫ Ω Hv(x, zn)(v − Pnv)dx = −(∇vn∇vn −∇v)L2 + ∫ Ω Hv(x, zn)(vn − v)dx+ o(1). Then (∇vn∇vn −∇v)L2 = ∫ Ω Hv(x, zn)(vn − v)dx+ o(1) = ∫ Ω Hz(x, zn)(zn − z)dx+ ∫ Ω Hu(x, zn)(un − u)dx+ o(1) = ∫ Ω Hz(x, zn)zndx− ∫ Ω Hz(x, zn)zdx+ o(1). It follows from the Lebesgue’s theorem and the weak sequential continuity of Hz that |∇v|22 − lim sup n→∞ |∇vn|22 = lim inf n→∞ (∫ Ω Hz(x, zn)zndx− ∫ Ω Hz(x, zn)zdx ) ≥ ∫ Ω lim inf n→∞ (Hz(x, zn)zn −Hz(x, zn)z) dx = 0, that is, |∇v|22 ≥ lim sup n→∞ |∇vn|22, which together with the weak lower semicontinuity of the norm implies that |∇v|2 ≤ lim sup n→∞ |∇vn|2. So |∇vn|2 → |∇v|2, that is, vn → v in H1 0 (Ω). Observe that I(z)− I(zn) = 1 2 ( |∇u|22 − |∇un|22 ) − 1 2 ( |∇v|22 − |∇vn|22 ) + ∫ Ω H(x, zn)dx− ∫ Ω H(x, z)dx. 20 X. WANG, P. ZHAO EJDE-2020/52 The Lebesgue’s theorem then yields I(z)− C = lim inf n→∞ ∫ Ω H(x, zn)dx− ∫ Ω H(x, z)dx ≥ ∫ Ω lim inf n→∞ H(x, zn)dx− ∫ Ω H(x, z)dx = 0. Then we have I(z) ≥ C. � Proof of Theorem 1.2. From the above discussion, it follows from Lemmas 3.6 and 3.7 that I has the linking geometry. Let Qn := Q ∩Xn and define cn := inf γ∈Γn sup x∈γ(Qn) I(x), where Γn := {γ ∈ C(Qn, Xn) : γ |∂Qn= id}. Then ρ ≤ cn ≤ k := sup I(γ(Q)). Therefore, by the Linking theorem, there is zn ∈ Xn such that |I(zn)− cn| ≤ 1 n and ‖I ′n(zn)‖ ≤ 1 n . So we obtain a (PS)∗c sequence {zn} ⊂ Eq with c ∈ [ρ, k]. Lemma 3.9 implies zn ⇀ z with I ′(z) = 0 and I(z) ≥ c. As a result, the Theorem 1.2 is obtained. � Acknowledgements. This research was supported by the National Science Foun- dation of China (No.11471147). The authors would like to thank the referees for helpful suggestions. References [1] R. Adams, J. Fournier; Sobolev Spaces, Academic Press, New York (2003). [2] A. Ambrosetti, P. H. Rabinowitz; Dual variational methods in critical point theory and applications, Journal of Functional Analysis, 14, No. 4(1973), 349-381. [3] H. Brezis; Functional Analysis, Sobolev Spaces and Partial Differential Equations, Springer, New York (2011). [4] P. Cl ment, M. Garc a-Huidobro, Man sevich; Mountain pass type solutions for quasilinear elliptic equations, Calculus of Variations and Partial Differential Equations, 11, No. 1(2000), 33-62. [5] M. L. M. Carvalho, J. V. A. Goncalves, E. D. D. Silva; On quasilinear elliptic problems without the Ambrosetti-Rabinowitz condition, Journal of Mathematical Analysis and Appli- cations, 426, No. 1(2015), 466-483. [6] D. G. Costa, C. A. Magalhaes; A variational approach to noncooperative elliptic systems, Nonlinear Analysis Theory Methods and Applications, 25, No. 7(1995), 699-715. [7] D. G. De Figueiredo, Y. H. Ding; Strongly indefinite functionals and multiple solutions of elliptic systems, Transactions of the American Mathematical Society, 355, No. 7(2003), 2973- 2989. [8] D. G. De Figueiredo, P. L. Felmer; On superquadratic elliptic systems, Transactions of the American Mathematical Society, 343, No. 1(1994), 99-116. [9] G. Dinca, P. Jebelean, J. Mawhin; A result of Ambrosetti-Rabinowitz type for p-Laplancian, Qualitative problems for differential equations and control theory, World Sci. Publ., River Edge, NJ, (1995), 231-242. [10] L. C. Evans; Partial Differential Equations(second edition), University of Califolia, Berkeley, (2010). [11] D. Gilbarg, N. Trudinge; Elliptic Partial Differential Equation of Second Order, Springer, New York (2001). [12] Z. Hu, L. Wang, P. H. Zhao; A priori estimates and existence for quasilinear elliptic equations with nonlinear Neumann boundary conditions, Electronic Journal of Differential Equations, 2016, No. 187 (2016), 1-8. EJDE-2020/52 WEAK SOLUTIONS TO SUPERLINEAR ELLIPTIC SYSTEMS 21 [13] P. L. Lions; On the existence of positive solutions of semilinear elliptic equations, Siam Review, 24, No. 4 (1982), 441-467. [14] N. Lam, G. Lu; Elliptic equations and systems with subcritical and critical exponential growth without the Ambrosetti-Rabinowitz condition, Journal of Geometric Analysis, 24, No. 1(2014), 118-143. [15] Z. L. Liu, Z. Q. Wang; On the Ambrosetti-Rabinowitz superlinear condition, Advanced Non- linear Studies, 4, No. 4(2004), 563-574. [16] G. B. Li, C. Y. Yang; The existence of a nontrivial solution to a nonlinear elliptic boundary value problem of p-Laplacian type without the Ambrosetti-Rabinowitz condition, Nonlinear Analysis, 72, (2010), 4602-4613. [17] O. H. Miyagaki, M. A. S. Souto; Superlinear problems without Ambrosetti and Rabinowitz growth condition, Journal of Differential Equations, 245, No. 12(2008), 3628-3638. [18] P. D. N pol , M. C. Mariani; Mountain pass solutions to equations of p-Laplacian type, Nonlinear Analysis, 54, No. 7(2003), 1205-1219. [19] M. Struwe; Variational Methods: Applications to Nonlinear Partial Differential Equations and Hamiltonian Systems(fourth edition), Springer, 2012. [20] M. Schechter, W. M. Zou; Superlinear problem, Pacific Journal of Mathematics, 214, No. 1(2004), 145-160. [21] J. Vazquez; A strong maximum principle for some quasilinear elliptic equations, Applied Mathematics and Optimization, 12, No. 1(1984), 191-202. [22] L. Wang, P. H. Zhao; Infinitely many sign-changing solutions for concave-convex elliptic problem with nonlinear boundary condition, Electronic Journal of Differential Equations, No. 266(2015), 1-9. [23] Z. Q. Wu, J. X. Yin, C. P. Wang; Introduction to elliptic and parabolic equations, Science Press, Beijing (2003). [24] C. K. Zhong, X. L. Fan, W. Y. Cheng; Introduction to Nonlinear Functional Analysis, Lanzhou University Press, Lanzhou, 2004. Xiaohui Wang School of Mathematics and Statistics, Lanzhou University, Lanzhou 730000, China Email address: xiaohuiwang1@126.com Peihao zhao School of Mathematics and Statistics, Lanzhou University, Lanzhou 730000, China Email address: zhaoph@lzu.edu.cn 1. Introduction and statement of main results 2. Superlinear elliptic equation 2.1. Preliminaries 2.2. Existence of a nontrivial weak solution to the elliptic equation 3. Existence of the nontrivial weak solution for the superlinear elliptic system 3.1. Preliminaries 3.2. Existence of a nontrivial weak solution for the elliptic system Acknowledgements References