Electronic Journal of Differential Equations, Vol. 2020 (2020), No. 119, pp. 1–25. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES WITH SIGN-CHANGING WEIGHT FUNCTIONS ANU RANI, SARIKA GOYAL Abstract. This article concerns the existence of multiple solutions of the polyharmonic system involving critical nonlinearities with sign-changing weight functions (−∆)mu = λf(x)|u|r−2u+ β β + γ h(x)|u|β−2u|v|γ in Ω, (−∆)mv = µg(x)|v|r−2v + γ β + γ h(x)|u|β |v|γ−2v in Ω, Dku = Dkv = 0 for all |k| ≤ m− 1 on ∂Ω, where (−∆)m denotes the polyharmonic operators, Ω is a bounded domain in RN with smooth boundary ∂Ω, m ∈ N, N ≥ 2m+ 1, 1 < r < 2 and β > 1, γ > 1 satisfying 2 < β + γ ≤ 2∗m with 2∗m = 2N N−2m as a critical Sobolev exponent and λ, µ > 0. The functions f , g and h : Ω → R are sign-changing weight functions satisfying f , g ∈ Lα(Ω) and h ∈ L∞(Ω) respectively. Using the variational methods and Nehari manifold, we prove that the system admits at least two nontrivial solutions with respect to parameter (λ, µ) ∈ R2 + \ {(0, 0)}. 1. Introduction Let Ω be a bounded domain in RN with smooth boundary ∂Ω, m ∈ N, N ≥ 2m+ 1. We consider the following polyharmonic system involving concave-convex nonlinearities with critical exponent and sign-changing weight functions (−∆)mu = λf(x)|u|r−2u+ β β + γ h(x)|u|β−2u|v|γ in Ω, (−∆)mv = µg(x)|v|r−2v + γ β + γ h(x)|u|β |v|γ−2v in Ω, Dku = Dkv = 0 for all |k| ≤ m− 1 on ∂Ω, (1.1) where 1 < r < 2, β > 1, γ > 1 satisfying 2 < β + γ ≤ 2∗m with 2∗m = 2N N−2m as a critical Sobolev exponent and λ, µ are the parameter such that (λ, µ) ∈ R2 +\{(0, 0)}. 2010 Mathematics Subject Classification. 35A15, 35B33, 35J91. Key words and phrases. Polyharmonic operator system; sign-changing weight functions; critical exponent; Nehari manifold; concave-convex nonlinearities. c©2020 Texas State University. Submitted August 31, 2020. Published December 10, 2020. 1 2 A. RANI, S. GOYAL EJDE-2020/119 Here ∆m denotes the polyharmonic operators which is defined as ∆mu = { ∆j(∆ju) if m = 2j, j = 1, 2, . . . ∇ · (∆j−1∇∆j−1u) if m = 2j − 1, j = 1, 2, . . . . To construct our problem more precise, we give the following assumptions on the weight functions f , g and h: (A1) f , g ∈ Lα(Ω) with α = β+γ β+γ−r , f± = max{±f, 0} 6≡ 0 in Ω and g± = max{±g, 0} 6≡ 0 in Ω i.e. (f and g are possibly sign-changing on Ω); (A2) h ∈ L∞(Ω) and h+ = max{h, 0} 6≡ 0 in Ω. When β = γ, β + γ = 2∗m, λ = µ, u = v and f ≡ g, problem (1.1) reduces to the polyharmonic equation (−∆)mu = λf(x)|u|r−2u+ h(x)|u|2 ∗ m−2u in Ω, Dku = 0 for all |k| ≤ m− 1 on ∂Ω, which was investigated in [30] when f and h are continuous functions. Recently, a lot of attention has been directed to the study of biharmonic and polyharmonic equations, both from concrete applications and for pure mathematical point of view. Such models naturally arise in many applications, such as micro electro-mechanical system, phase field models of multi-phase systems, in thin film theory, nonlinear surface diffusion on solids, interface dynamics, flow in Hele-Shaw cells, and the deformation of a nonlinear elastic beam (see [17, 27]). Starting with the pioneering work of Ambrosetti et al. [3] on Laplacian involving convex concave type nonlinearities, an enormous amount of work has been examined by authors such as Bartsch-Willem [4], Figueiredo et al [11], Brown and Zhang [10], Hamidi [21] and Hsu [23] in this direction. Brézis and Nirenberg [8] studied the problem with critical nonlinearity −∆u = u N+2 N−2 + λu, u > 0 in Ω, u = 0 on ∂Ω, (1.2) where N ≥ 3. They showed that for N ≥ 4, (1.2) has positive solution if and only if λ ∈ (0, λ1). For N = 3 and Ω = B1 is unit ball in RN , problem (1.2) has a positive solution if and only if λ ∈ (λ1 4 , λ1), where λ1 > 0 is first eigenvalue of −∆ in Ω. If Ω is star shaped, then (1.2) has no solution for λ ≤ 0. Moreover, a great amount of mathematical effort has been demonstrated by many authors involving biharmonic equation with critical nonlinearity (see [5, 12, 13, 15, 26, 29]). Pucci-Serrin [28] considered the polyharmonic equation with critical nonlinearity (−∆)mu = |u|2 ∗ m−2u+ λu in Ω, Dku = 0 for all |k| ≤ m− 1 on ∂Ω . (1.3) They found that if N ≥ 4m and Ω = B1, then (1.3) has positive solution, for all λ ∈ (0, λ (m) 1 ), where λ (m) 1 is the first eigenvalue of polyharmonic operator (−∆)m. If N = 2m + 1 and Ω = B1, then (1.3) admits the existence of a nonnegative, nontrivial solution if (λ ∈ (2m − 1 2 )λ (m−1) 1 , λ (m) 1 ). If λ < 0 and Ω is star shaped, then (1.3) has the trivial solution. Later Edmunds et al [15] extended the results of problem (1.3) for biharmonic operator (m = 2) and showed that (1.3) has a nontrivial solution if λ ∈ (0, λ1) and N ≥ 8. When N = 5, 6 or 7, problem (1.3) has a nontrivial solution, for all λ ∈ (λ̄, λ1), where λ̄ = λ1 − S|Ω|− 4 N and S is EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 3 the best constant for Sobolev embedding of H2 0 (Ω) in L 2N N−4 (Ω). Also, Grunau [19] studied (1.3) in case of ball and proved that, if 2m + 1 ≤ N ≤ 4m − 1, (1.3) has a positive solution for λ ∈ (λ̄, λ (m) 1 ) for some λ̄ = λ̄(N,m) ∈ (0, λ (m) 1 ). Thereafter, Gazzola [18] contributed for polyhamonic operators with critical growth. During the previous decades many authors have paid attention to semilinear and quasilinear elliptic equations involving sign-changing weight functions with subcritical and critical nonlinearity using Nehari manifold. Reader is referred to [1, 2, 6, 9, 20, 31, 34, 35] and references therein. Further, Hsu [22, 24] proved the multiplicity results for elliptic system and quasilinear elliptic system involving convex-concave nonlinearities with sign-changing weight function respectively. Ji and Wang [25] studied the p-biharmonic equation involving subcritical nonlinearity with sign-changing weight function and showed the existence of two nontrivial so- lution by Nehari manifold and fibering map analysis. Recently, in 2014, Shang and Li [30] investigated the multiplicity of nontrivial solutions of polyharmonic equa- tion with critical exponents and sign-changing weight functions. To the best of our knowledge, there is no result so far concerning polyharmonic system involving critical nonlinearities with sign-changing weight functions. Apart from this, the results obtained here are new for linear case (m = 2). In this article, using the Nehari manifold and fibering map analysis, we establish the existence of at least two nontrivial solutions for a polyharmonic system involv- ing critical nonlinearities with sign-changing weight functions with respect to the pair of parameters λ, µ belongs to a suitable subset of R2. Since the embedding Hm 0 (Ω) ↪→ L2∗ m(Ω) is not compact, so the corresponding energy functional does not satisfy the Palais-Smale condition in general. Therefore, it is difficult to obtain the critical points of energy functional by simple arguments, which are based on the compactness of the Sobolev embedding. To overcome this difficulty, we extract a Palais-Smale sequence in the Nehari manifold and show that the weak limit of this sequence is the required solution of problem (1.1). To state our main results, we introduce Λ1 := ( 2− r (β + γ − r)|h|∞ ) 2 β+γ−2 (β + γ − r β + γ − 2 )− 2 2−r S 2(β+γ−r) (2−r)(β+γ−2) > 0, (1.4) where S is the best constant that will be introduced in next section. Then we obtain the following existence results. Theorem 1.1. Assume that (A1), (A2) hold. If 1 ≤ r < 2 < N m , 2 < β + γ ≤ 2∗m, and λ, µ > 0 satisfy 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ1, then system (1.1) has at least one nontrivial solution in Hm 0 (Ω)×Hm 0 (Ω). Theorem 1.2 (Second nontrivial solution in subcritical case). Assume that (A1), (A2) hold. If 1 ≤ r < 2 < N m , 2 < β + γ < 2∗m, and λ, µ > 0 satisfy 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, then system (1.1) has at least two nontrivial solution in Hm 0 (Ω)×Hm 0 (Ω). To obtain the second nontrivial in critical case β+γ = 2∗m, we need the following extra assumptions on f , g and h: (A3) There exist a0, b0 and r0 > 0 such that B(x0, 2r0) ⊂ Ω and f(x) ≥ a0, g(x) ≥ b0 for all x ∈ B(0, 2r0); 4 A. RANI, S. GOYAL EJDE-2020/119 (A4) there exists δ0 > 0 such that |h|∞ = h(0) = maxx∈Ω h(x), h(x) > 0 for all x ∈ B(0, 2r0) and h(x) = h(0) + o(|x|δ0) as x→ 0. Theorem 1.3 (Second nontrivial solution in critical case). Assume that (A1)–(A4) hold. If 1 ≤ r < 2 < N/m, and λ, µ > 0 satisfy 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, then system (1.1) has at least two nontrivial solution in Hm 0 (Ω)×Hm 0 (Ω). The article is organized as follows: In section 2, variational setting of problem (1.1) and some preliminary results are introduced. In section 3, we show that the Palais-Smale condition holds for the energy functional associated with (1.1) at energy level in a suitable range related to the best Sobolev constant. Some results about the Nehari manifold and fibering map analysis are discussed in section 4. In section 5, we prove the existence of Palais-Smale sequences and proof of Theorems 1.1 and 1.2. We give the detail of proof of Theorem 1.3 in section 6. Notation. • Lp(Ω), 1 ≤ p < ∞, denote Lebesgue spaces; the norm Lp is denoted by | · |p; • Qλ,µ(u, v) = ∫ Ω (λf(x)|u|r + µg(x)|v|r)dx; • B(x0, r) = Br(x0) = {x ∈ RN : |x− x0| < r} is the ball in RN ; • O(εt) denotes |O(εt)/εt| ≤ C as ε→ 0 for t ≥ 0; • on(1) denotes on(1)→ 0 as n→∞; • O1(εt) denotes that there exist the constants C1, C2 > 0 such that C1ε t ≤ O1(εt) ≤ C2ε t as ε small enough. C, Ci’s are positive constants. 2. Preliminaries In this section, we firstly define the function space corresponding to problem (1.1), posed in framework of Sobolev space H := Hm 0 (Ω) ×Hm 0 (Ω) with standard norm ‖(u, v)‖ = ( ‖Dmu‖2 + ‖Dmv‖2)1/2, where ‖Dmu‖2 = { ‖(−∆) m 2 u‖2 if m = 2j, j = 1, 2, . . . , ‖∇(−∆) m−1 2 u‖2 if m = 2j − 1, j = 1, 2, . . . . Then H is a Hilbert space. Definition 2.1. A pair of functions (u, v) ∈ H is said to be a weak solution of (1.1) if for all (φ1, φ2) ∈ H, (i) when m is even,∫ Ω (−∆) m 2 u(−∆) m 2 φ1 + ∫ Ω (−∆) m 2 v(−∆) m 2 φ2 − λ ∫ Ω f(x)|u|r−2uφ1 − µ ∫ Ω g(x)|v|r−2vφ2 − β β + γ ∫ Ω h(x)|u|β−2u|v|γφ1 − γ β + γ ∫ Ω h(x)|u|β |v|γ−2vφ2 = 0 ; EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 5 (ii) when m is odd,∫ Ω ∇(−∆) m−1 2 u · ∇(−∆) m−1 2 φ1 + ∫ Ω ∇(−∆) m−1 2 u · ∇(−∆) m−1 2 φ2 − λ ∫ Ω f(x)|u|r−2uφ1 − µ ∫ Ω g(x)|v|r−2vφ2 − β β + γ ∫ Ω h(x)|u|β−2u|v|γφ1 − γ β + γ ∫ Ω h(x)|u|β |v|γ−2vφ2 = 0. Now, we define the energy functional Iλ,µ : H → R associated with problem (1.1) as Iλ,µ(u, v) = 1 2 ‖(u, v)‖2− 1 r ∫ Ω (λf(x)|u|r +µg(x)|v|r)dx− 1 β + γ ∫ Ω h(x)|u|β |v|γdx. Then Iλ,µ is well defined in H and Iλ,µ ∈ C1(H,R). Moreover, the critical points of the functional Iλ,µ are the weak solutions of (1.1). Further, we will prove a lemma which will be used to prove the second solution in critical case. For this, let S be the best Sobolev constant defined as S := inf u∈Hm0 (Ω)\{0} ‖Dmu‖2 ( ∫ Ω |u|β+γ) 2 β+γ , (2.1) where β+ γ = 2∗m. Then it is well known that S is achieved if and only if Ω = RN , by the function U(x) = C N−2m 4m N,m (1 + |x|2) N−2m 2 (see[33]). Moreover, all the minimizers of S are obtained by Uε(x) = ε 2m−N 2 U( x ε ) = C N−2m 4m N,m ε N−2m 2 (ε2 + |x|2) N−2m 2 , where ε > 0. (2.2) The normalizing constant CN,m := C(N,m) = ∏m j=1−m(N − 2j) and is chosen in such a way that Uε(x) solves the equation (−∆)mu = |u|2 ∗ m−2u in RN , and satisfies ‖Uε(x)‖2 = |Uε(x)|2 ∗ m 2∗ m = S N 2m . Now, consider the minimization problem Sβ,γ = inf (u,v)∈H\{(0,0)} ‖Dmu‖2 + ‖Dmv‖2 ( ∫ Ω |u|β |v|γdx) 2 β+γ . (2.3) Then we establish the following relationship between Sβ,γ and S, using an idea from [2]. Lemma 2.2. For the constants Sβ,γ and S given in (2.1) and (2.3), it holds Sβ,γ = [(β γ ) γ β+γ + (γ β ) β β+γ ] S. (2.4) In particular, the constant Sβ,γ is achieved for Ω = RN . 6 A. RANI, S. GOYAL EJDE-2020/119 Proof. Let {wn} ⊂ Hm 0 (Ω) be a minimizing sequence for S. Then take the se- quences un = swn and vn = twn in Hm 0 (Ω), where s, t > 0. By definition of Sβ,γ , we have Sβ,γ ≤ ‖(un, vn)‖2 ( ∫ Ω |un|β |vn|γdx) 2 β+γ . Therefore Sβ,γ ≤ (s2 + t2)S s 2β β+γ t 2γ β+γ = [ ( s t ) 2γ β+γ + ( t s ) 2β β+γ ] S. Now, define a function Υ : R+ → R+ such that Υ(x) = x 2γ β+γ + x −2β β+γ . Then Υ( st ) = ( st ) 2γ β+γ + ( ts ) 2β β+γ and Υ attains its minimum at x0 = ( β γ ) 1 2 . So, we have min x∈R+ Υ(x) = Υ(x0) = (β γ ) γ β+γ + (γ β ) β β+γ . Choosing s, t such that s t = ( β γ ) 1 2 and letting n→∞ yields Sβ,γ ≤ [(β γ ) γ β+γ + (γ β ) β β+γ ] S. (2.5) On the other hand, let {(un, vn)} be a minimizing sequence for Sβ,γ . Define an = snvn for some sn > 0 such that ∫ Ω |un|β+γdx = ∫ Ω |an|β+γdx. Then Young’s inequality implies that∫ Ω |un|β |an|γdx ≤ β β + γ ∫ Ω |un|β+γdx+ γ β + γ ∫ Ω |an|β+γdx = ∫ Ω |an|β+γdx = ∫ Ω |un|β+γdx. Thus, using this we obtain ‖(un, vn)‖2 ( ∫ Ω |un|β |vn|γdx) 2 β+γ = s 2γ β+γ n [ ‖Dmun‖2 ( ∫ Ω |un|β |an|γdx) 2 β+γ + ‖Dmvn‖2 ( ∫ Ω |un|β |an|γdx) 2 β+γ ] ≥ s 2γ β+γ n ‖Dmun‖2 ( ∫ Ω |un|β+γdx) 2 β+γ + s 2γ β+γ−2 n ‖Dman‖2 ( ∫ Ω |an|β+γdx) 2 β+γ ≥ ( s 2γ β+γ n + s 2γ β+γ−2 n ) S ≥ Υ(x0)S. On passing to the limit as n→∞, we obtain Sβ,γ ≥ [(β γ ) γ β+γ + (γ β ) β β+γ ] S. (2.6) Hence, from (2.5) and (2.6), we obtain the required result. � Definition 2.3. Let J : X → R be a C1 functional on a Banach space X. • For c ∈ R, a sequence {uk} ⊂ X is a Palais-Smale sequence at level c ((PS)c) in X for J if J(uk) = c+ ok(1) and J ′(uk)→ 0 in X−1 as k →∞. • We say J satisfies (PS)c condition if for any Palais-Smale sequence {uk} in X for J has a convergent subsequence. EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 7 3. The Palais-Smale condition Lemma 3.1. Suppose that {(un, vn)} ⊂ H is a (PS)c-sequence for Iλ,µ such that (un, vn) ⇀ (u, v) weakly in H. Then I ′λ,µ(u, v) = 0 and there exists a positive constant P0 depending on m, N , r and S such that Iλ,µ(u, v) ≥ −P0((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ). Proof. Let {(un, vn)} be a (PS)c-sequence in H, then by using the standard argu- ment, one can easily obtain I ′λ,µ(u, v) = 0, i.e. 〈I ′λ,µ(u, v), (u, v)〉 = 0. Using this, Hölder’s and Young’s inequalities, we obtain Iλ,µ(u, v) = (1 2 − 1 β + γ ) ‖(u, v)‖2 − (1 r − 1 β + γ )∫ Ω (λf(x)|u|r + µg(x)|v|r)dx ≥ m N ‖(u, v)‖2 − (β + γ − r) r(β + γ) S−r/2 × [ ω 2 2−r (2− r 2 )( λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) + ω−2/r 2 ‖(u, v)‖2 ] = m N ‖(u, v)‖2 − m N ‖(u, v)‖2 − P0 ( ((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2 2−r + (µ|g|α) 2 2−r ) , where P0 = (β + γ − r)(2− r) 2r(β + γ) S−r/2ω 2 2−r , ω = (N(β + γ − r) 2m(β + γ) S−r/2 )r/2 . This completes the proof. � Lemma 3.2. If {(un, vn)} ⊂ H is a (PS)c-sequence for Iλ,µ, then {(un, vn)} is bounded in H. Proof. Let {(un, vn)} be a (PS)c-sequence for Iλ,µ in H, then we assume by con- tradiction that ‖(un, vn)‖ → ∞ as n→∞. Define (ûn, v̂n) := (un, vn) ‖(un, vn)‖ = ( un ‖(un, vn)‖ , vn ‖(un, vn)‖ ) . Then {(ûn, v̂n)} is a bounded sequence. So, up to a subsequence (ûn, v̂n) ⇀ (û, v̂) weakly inH. This implies that ûn → û, v̂n → v̂ strongly in Ls(Ω) for all 1 ≤ s < 2∗m and Qλ,µ(ûn, v̂n) = Qλ,µ(û, v̂) + on(1). (3.1) Since {(un, vn)} is a (PS)c-sequence for Iλ,µ and ‖(un, vn)‖ → ∞ as n → ∞, we obtain 1 2 ‖(ûn, v̂n)‖2 − ‖(un, vn)‖r−2 r Qλ,µ(ûn, v̂n) − ‖(un, vn)‖β+γ−2 β + γ ∫ Ω h(x)|ûn|β |v̂n|γdx = on(1), and ‖(ûn, v̂n)‖2 − ‖(un, vn)‖r−2Qλ,µ(ûn, v̂n) − ‖(un, vn)‖β+γ−2 ∫ Ω h(x)|ûn|β |v̂n|γdx = on(1). (3.2) From (3.1) and (3.2), we can deduce that ‖(ûn, v̂n)‖2 = 2(β + γ − r) r(β + γ − 2) ‖(un, vn)‖r−2Qλ,µ(ûn, v̂n) + on(1). (3.3) 8 A. RANI, S. GOYAL EJDE-2020/119 Since 1 ≤ r < 2 and ‖(un, vn)‖ → ∞, then (3.3) implies ‖(ûn, v̂n)‖2 → 0 as n→∞, which is a contradiction to the fact that ‖(ûn, v̂n)‖ = 1. � Lemma 3.3. Iλ,µ satisfies the (PS)c-condition with c satisfying c ∈ (0, c∞), where c∞ = m N S N 2m β,γ |h| −N−2m 2m∞ − P0 ( (λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) , and P0 is given in Lemma 3.1. Proof. Let {(un, vn)} ⊂ H be a (PS)c-sequence for Iλ,µ with 0 < c < c∞. Then by Lemma 3.2, {(un, vn)} is a bounded sequence in H. Hence, up to a subsequence, (un, vn) ⇀ (u, v) weakly in H. So un ⇀ u and vn ⇀ v weakly in Hm 0 (Ω), un → u and vn → v strongly in Ls(Ω) for all 1 ≤ s < 2∗m and un → u, vn → v pointwise a.e. in Ω. Thus∫ Ω (λf(x)|un|r + µg(x)|vn|r)dx = ∫ Ω (λf(x)|u|r + µg(x)|v|r)dx+ on(1). (3.4) Also, I ′λ,µ(u, v) = 0, follows from Lemma 3.1. Now, define (ũn, ṽn), where ũn = un − u, ṽn = vn − v. Then by Brézis-Lieb Lemma [7] and Vitali theorem, we have ‖(ũn, ṽn)‖2 = ‖(un, vn)‖2 − ‖(u, v)‖2 + on(1), (3.5)∫ Ω h(x)|ũn|β |ṽn|γdx = ∫ Ω h(x)|un|β |vn|γdx− ∫ Ω h(x)|u|β |v|γdx+ on(1). (3.6) Using Iλ,µ(un, vn) = c+ on(1), I ′λ,µ(un, vn) = on(1), (3.4) and (3.6), we obtain 1 2 ‖(ũn, ṽn)‖2 − 1 β + γ ∫ Ω h(x)|ũn|β |ṽn|γdx = c− Iλ,µ(u, v) + on(1), (3.7) and ‖(ũn, ṽn)‖2 − ∫ Ω h(x)|ũn|β |ṽn|γdx = 〈I ′λ,µ(u, v), (un − u, vn − v)〉+ on(1) = on(1). Therefore, we assume that ‖(ũn, ṽn)‖2 → l, ∫ Ω h(x)|ũn|β |ṽn|γdx→ l. (3.8) If l = 0, then proof is complete. If l > 0, then by definition of Sβ,γ and (3.8), we obtain Sβ,γ l 2 β+γ ≤ Sβ,γ lim n→∞ ( |h|∞ ∫ Ω |un|β |vn|γdx )2/2∗ m ≤ |h| 2 β+γ ∞ lim n→∞ ‖(ũn, ṽn)‖2 = |h| 2 β+γ ∞ l. As β + γ = 2∗m, so the above relation gives l ≥ S N 2m β,γ |h| − (N−2m) 2m∞ . Now, by (3.7), (3.8) and Lemma 3.1, we obtain c = (1 2 − 1 β + γ ) l + Iλ,µ(u, v) ≥ m N S N 2m β,γ |h| −N−2m 2m∞ − P0((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) = c∞, which is a contradiction to c < c∞. The proof is complete. � EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 9 4. Nehari manifold for (1.1) Since the energy functional Iλ,µ is not bounded below on H, it is appropriate to consider the functional on the Nehari manifold Nλ,µ = {(u, v) ∈ H \ {(0, 0)} : 〈I ′λ,µ(u, v), (u, v)〉 = 0}. Thus, (u, v) ∈ Nλ,µ if and only if 〈I ′λ,µ(u, v), (u, v)〉 = ‖(u, v)‖2 −Qλ,µ(u, v)− ∫ Ω h(x)|u|β |v|γdx = 0. (4.1) It is easy to see that Nλ,µ contains every nonzero solution of (1.1). In fact, we will show later that local minimizers of Nλ,µ are the critical points of Iλ,µ. Lemma 4.1. The energy functional Iλ,µ is coercive and bounded below on Nλ,µ. Proof. Let (u, v) ∈ Nλ,µ, then by (4.1), Hölder inequality and Sobolev embedding theorem, we have Iλ,µ(u, v) = β + γ − 2 2(β + γ) ‖(u, v)‖2 − β + γ − r r(β + γ) Qλ,µ(u, v) ≥ β + γ − 2 2(β + γ) ‖(u, v)‖2 − β + γ − r r(β + γ) S−r/2 ( (λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2−r 2 ‖(u, v)‖r. (4.2) Since 1 < r < 2. Thus, Iλ,µ is coercive. Now, consider the function ρ : R → R as ρ(t) = at2 − btr. Then one can easily see that ρ′(t) = 0 if and only if t = ( br2a ) 1 2−r := t∗ and ρ′′(t∗) > 0. So ρ attains its minimum at t∗. Moreover, ρ(t) ≥ ρ(t∗) = −(2− r)( b 2 ) 2 2−r ( r a ) r 2−r . Taking a = β + γ − 2 2(β + γ) , b = β + γ − r r(β + γ) S−r/2((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2−r 2 , t = ‖(u, v)‖ in the function ρ, we obtain Iλ,µ(u, v) ≥ ρ(‖(u, v)‖) ≥ ρ(t∗). Hence, Iλ,µ is bounded below on Nλ,µ. � The Nehari manifold is closely related to the fibering map introduced by Drábek and Pohozaev in [14]. For each (u, v), we define Ψ(u,v) : t→ Iλ,µ(tu, tv) given by Ψ(u,v)(t) = Iλ,µ(tu, tv) = t2 2 ‖(u, v)‖2 − tr r Qλ,µ(u, v)− tβ+γ β + γ ∫ Ω h(x)|u|β |v|γdx, Ψ′(u,v)(t) = t‖(u, v)‖2 − tr−1Qλ,µ(u, v)− tβ+γ−1 ∫ Ω h(x)|u|β |v|γdx, Ψ′′(u,v)(t) = ‖(u, v)‖2 − (r − 1)tr−2Qλ,µ(u, v)− (β + γ − 1)tβ+γ−2 ∫ Ω h(x)|u|β |v|γdx. 10 A. RANI, S. GOYAL EJDE-2020/119 It is observed that Ψ′(u,v)(t) = 0 if and only if (tu, tv) ∈ Nλ,µ. Thus (u, v) ∈ Nλ,µ if and only if Ψ′(u,v)(1) = 0. Therefore it is natural to split Nλ,µ into three parts corresponding to local minima, local maxima and points of inflexion respectively as N±λ,µ := {(u, v) ∈ Nλ,µ : Ψ′′(u,v)(1) ≷ 0}, N 0 λ,µ := {(u, v) ∈ Nλ,µ : Ψ′′(u,v)(1) = 0}. For each (u, v) ∈ Nλ,µ, we have one of the following 3 equalities Ψ′′(u,v)(1) =  2‖(u, v)‖2 − rQλ,µ(u, v)− (β + γ) ∫ Ω h(x)|u|β |v|γdx , (2− r)‖(u, v)‖2 − (β + γ − r) ∫ Ω h(x)|u|β |v|γdx , (β + γ − r)Qλ,µ(u, v)− (β + γ − 2)‖(u, v)‖2. (4.3) Lemma 4.2. If (u0, v0) is the local minimizer for Iλ,µ on Nλ,µ and (u0, v0) /∈ N 0 λ,µ. Then I ′λ,µ((u0, v0)) = 0 in H−1, where H−1 denotes the dual space of H. Proof. If (u0, v0) is a local minimizer for Iλ,µ on Nλ,µ, then (u0, v0) is a solution of the problem: minimize Iλ,µ(u, v) subject to Φλ,µ(u, v) : 〈I ′λ,µ(u, v), (u, v)〉 = 0. Hence, by Lagrange multipliers, there exists θ ∈ R such that I ′λ,µ((u0, v0)) = θΦ′λ,µ((u0, v0)). Thus, 〈I ′λ,µ(u0, v0), (u0, v0)〉 = θ〈Φ′λ,µ(u0, v0), (u0, v0)〉. Since (u0, v0) ∈ Nλ,µ, it follows that 〈Φ′λ,µ(u0, v0), (u0, v0)〉 = (2− r)‖(u0, v0)‖2 − (β + γ − r) ∫ Ω h(x)|u0|β |v0|γdx 6= 0, as (u0, v0) /∈ N 0 λ,µ. Hence, we have θ = 0. � Lemma 4.3. We have the following (i) If (u, v) ∈ N+ λ,µ ∪N 0 λ,µ, then Qλ,µ(u, v) > 0. (ii) If (u, v) ∈ N−λ,µ ∪N 0 λ,µ, then ∫ Ω h(x)|u|β |v|γdx > 0. The proof of the above lemma follows directly from (4.3). Now, we show that N+ λ,µ andN−λ,µ are nonempty. For this we define some notations. For each (u, v) ∈ H with ∫ Ω h(x)|u|β |v|γdx > 0 tmax = ( (2− r)‖(u, v)‖2 (β + γ − r) ∫ Ω h(x)|u|β |v|γdx ) 1 β+γ−2 > 0, and for Qλ,µ(u, v) > 0, tmax = ( (β + γ − r)Qλ,µ(u, v) (β + γ − 2)‖(u, v)‖2 ) 1 2−r > 0. Lemma 4.4. Suppose that 0 < (λ|f |α) 2 2−r +(µ|g|α) 2 2−r < Λ1 and (u, v) ∈ H. Then we have the following: (i) If ∫ Ω h(x)|u|β |v|γdx > 0 and Qλ,µ(u, v) ≤ 0, then there exists a unique t− > tmax such that (t−u, t−v) ∈ N−λ,µ and Iλ,µ(t−u, t−v) = supt≥tmax Iλ,µ(tu, tv). (ii) If ∫ Ω h(x)|u|β |v|γdx > 0 and Qλ,µ(u, v) > 0, then there exists a unique 0 < t+ < tmax < t− such that (t+u, t+v) ∈ N+ λ,µ, (t−u, t−v) ∈ N−λ,µ. Moreover, Iλ,µ(t+u, t+v) = inf 0≤t≤tmax Iλ,µ(tu, tv); Iλ,µ(t−u, t−v) = sup t≥tmax Iλ,µ(tu, tv). EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 11 (iii) If Qλ,µ(u, v) > 0 and ∫ Ω h(x)|u|β |v|γ ≤ 0, then there exists a unique 0 < t+ < tmax such that (t+u, t+v) ∈ N+ λ,µ and Iλ,µ(t+u, t+v) = inft≥0 Iλ,µ(tu, tv). (iv) If Qλ,µ(u, v) < 0 and ∫ Ω h(x)|u|β |v|γdx < 0, then there does not exist any critical point. Proof. For (u, v) ∈ H with ∫ Ω h(x)|u|β |v|γdx > 0. Define ξ(u,v)(t) = t2−r‖(u, v)‖2 − tβ+γ−r ∫ Ω h(x)|u|β |v|γdx, for t > 0. We have ξ(u,v)(0) = 0, ξ(u,v)(t)→ −∞ as t→∞. Since ξ′(u,v)(t) = (2− r)t1−r‖(u, v)‖2 − (β + γ − r)tβ+γ−r−1 ∫ Ω h(x)|u|β |v|γdx, we obtain ξ′(u,v)(t) = 0 at t = tmax, ξ′(u,v)(t) > 0 for t ∈ [0, tmax) and ξ′(u,v)(t) < 0 for t ∈ (tmax,∞). So ξ(u,v)(t) attains its maximum at tmax. ξ(u,v)(t) is increasing function for t ∈ [0, tmax) and decreasing for t ∈ (tmax,∞). Moreover, ξ(u,v)(tmax) = ( (2− r)‖(u, v)‖2 (β + γ − r) ∫ Ω h(x)|u|β |v|γdx ) 2−r β+γ−2 ‖(u, v)‖2 − ( (2− r)‖(u, v)‖2 (β + γ − r) ∫ Ω h(x)|u|β |v|γdx ) β+γ−r β+γ−2 ∫ Ω h(x)|u|β |v|γdx = ‖(u, v)‖r( 2− r β + γ − r ) 2−r β+γ−2 (β + γ − 2 β + γ − r )( ‖(u, v)‖β+γ∫ Ω h(x)|u|β |v|γdx ) 2−r β+γ−2 ≥ ‖(u, v)‖r( 2− r β + γ − r ) 2−r β+γ−2 (β + γ − 2 β + γ − r )(S β+γ 2 |h|∞ ) 2−r β+γ−2 . (i) If ∫ Ω h(x)|u|β |v|γdx > 0 and Qλ,µ(u, v) ≤ 0, there is a unique t− > tmax > 0 such that ξ(u,v)(t −) = Qλ,µ(u, v) ≤ 0 and ξ′(u,v)(t −) < 0. 〈I ′λ,µ(t−u, t−v), (t−u, t−v)〉 = (t−)2‖(u, v)‖2 − (t−)rQλ,µ(u, v)− (t−)β+γ ∫ Ω h(x)|u|β |v|γdx = (t−)r(ξ(u,v)(t −)−Qλ,µ(u, v)) = 0. Therefore, (t−u, t−v) ∈ Nλ,µ. Ψ′′(u,v)(t −) = (2− r)(t−)2‖(u, v)‖2 − (β + γ − r)(t−)β+γ ∫ Ω h(x)|u|β |v|γdx = (t−)1+rξ′(u,v)(t −) < 0. Hence, (t−u, t−v) ∈ N−λ,µ. Since for t > tmax, we have Ψ′′(u,v)(t) = (2− r)t2‖(u, v)‖2 − (β + γ − r)tβ+γ ∫ Ω h(x)|u|β |v|γdx = t1+rξ′(u,v)(t) < 0. d2 dt2 Iλ,µ(tu, tv) = (r − 1)tr−2[ξ(u,v)(t)−Qλ,µ(u, v)] + tr−1ξ′(u,v)(t) < 0, when t = t−, d dt Iλ,µ(tu, tv) = tr−1 [ ξ(u,v)(t)−Qλ,µ(u, v) ] = 0, when t = t−. Thus, Iλ,µ(t−u, t−v) = sup t≥tmax Iλ,µ(tu, tv). 12 A. RANI, S. GOYAL EJDE-2020/119 (ii) If ∫ Ω h(x)|u|β |v|γdx > 0 and Qλ,µ(u, v) > 0, then by (4.6), ξ(u,v)(0) = 0 < Qλ,µ(u, v) ≤ S−r/2 ( (λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2−r 2 ‖(u, v)‖r < ( 2− r β + γ − r ) 2−r β+γ−2 (β + γ − 2 β + γ − r )(S β+γ 2 |h|∞ ) 2−r β+γ−2 ‖(u, v)‖r ≤ ξ(u,v)(tmax), for 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ1. There are unique t+ and t− such that 0 < t+ < tmax < t− with ξ(u,v)(t +) = Qλ,µ(u, v) = ξ(u,v)(t −) ξ′(u,v)(t +) > 0 > ξ′(u,v)(t −). This implies (t+u, t+v) ∈ N+ λ,µ, (t−u, t−v) ∈ N−λ,µ and d dt Iλ,µ(tu, tv) = 0, when t = t+ and t = t−, d2 dt2 Iλ,µ(tu, tv) > 0, when t ∈ (0, tmax), d2 dt2 Iλ,µ(tu, tv) < 0, when t ∈ (tmax,∞). Thus, we have Iλ,µ(t+u, t+v) = inf 0≤t≤tmax Iλ,µ(tu, tv), Iλ,µ(t−u, t−v) = sup t≥tmax Iλ,µ(tu, tv). (iii) For (u, v) ∈ H with Qλ,µ(u, v) > 0 and ∫ Ω h(x)|u|β |v|γdx ≤ 0, define ξ(u,v)(t) = t2−β−γ‖(u, v)‖2 − tr−β−γQλ,µ(u, v), for t > 0. We have ξ(u,v)(t)→ −∞ as t→ 0, ξ(u,v)(t)→ 0 as t→∞. Since ξ ′ (u,v)(t) = (2− β − γ)t1−β−γ‖(u, v)‖2 − (r − β − γ)tr−β−γ−1Qλ,µ(u, v), we obtain ξ ′ (u,v)(t) = 0 at t = tmax, ξ ′ (u,v)(t) > 0 for t ∈ (0, tmax) and ξ ′ (u,v)(t) < 0 for t ∈ (tmax,∞). So ξ(u,v)(t) attains its maximum at tmax. ξ(u,v)(t) is increasing function for t ∈ (0, tmax) and decreasing for t ∈ (tmax,∞). Now, using the same argument used in previous parts, there exists a unique 0 < t+ < tmax such that ξ(u,v)(t +) = ∫ Ω h(x)|u|β |v|γdx ≤ 0, ξ ′ (u,v)(t +) > 0. Also, 〈I ′λ,µ(t+u, t+v), (t+u, t+v)〉 = 0. Thus, (t+u, t+v) ∈ Nλ,µ. Further Ψ′′(u,v)(t +) > 0 so (t+u, t+v) ∈ N+ λ,µ. Since 0 < t+ < tmax, then Ψ′′(u,v)(t) > 0. Moreover, for t = t+, d2 dt2 Iλ,µ(tu, tv) > 0 and d dtIλ,µ(tu, tv) = 0. Hence Iλ,µ(t+u, t+v) = inf t≥0 Iλ,µ(tu, tv). (iv) If Qλ,µ(u, v) < 0 and ∫ Ω h(x)|u|β |v|γdx < 0, then Ψ(u,v)(0) = 0, Ψ′(u,v)(t) > 0 for all t > 0. This implies Ψ(u,v) is strictly increasing function and does not have critical point. This completes the proof. � Lemma 4.5. If 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ1, then N 0 λ,µ = ∅. EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 13 Proof. On contrary, assume that there exists (λ, µ) ∈ R2 \ {(0, 0)} with 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ1, such that N 0 λ,µ 6= ∅. Then for (u, v) ∈ N 0 λ,µ, us- ing (4.3), we obtain ‖(u, v)‖2 = β + γ − r 2− r ∫ Ω h(x)|u|β |v|γdx, ‖(u, v)‖2 = β + γ − r β + γ − 2 Qλ,µ(u, v). (4.4) Now, by Young’s inequality and Sobolev embedding theorem, we have∫ Ω h(x)|u|β |v|γdx ≤ |h|∞( β β + γ ∫ Ω |u|β+γdx+ γ β + γ ∫ Ω |v|β+γdx) ≤ |h|∞S− β+γ 2 ‖(u, v)‖β+γ . (4.5) Similarly, by Hölder’s inequality and Sobolev embedding theorem, we obtain Qλ,µ(u, v) ≤ S−r/2((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2−r 2 ‖(u, v)‖r. (4.6) Thus, by (4.4), (4.5) and (4.6), we obtain ‖(u, v)‖ ≥ ( 2− r β + γ − r S β+γ 2 |h|∞ ) 1 β+γ−2 (4.7) and ‖(u, v)‖ ≤ (β + γ − r β + γ − 2 ) 1 2−r S− r 2(2−r) ((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 1 2 . (4.8) On combining (4.7) and (4.8), we have (λ|f |α) 2 2−r + (µ|g|α) 2 2−r ≥ Λ1 := ( 2− r (β + γ − r)|h|∞ ) 2 β+γ−2 (β + γ − r β + γ − 2 )− 2 2−r S 2(β+γ−r) (2−r)(β+γ−2) , which is a contradiction. This completes the proof. � Note that from Lemma 4.5, if 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ1, then Nλ,µ = N+ λ,µ ∪N − λ,µ. Now we define θλ,µ = inf (u,v)∈Nλ,µ Iλ,µ(u, v), θ±λ,µ = inf (u,v)∈N± λ,µ Iλ,µ(u, v). We end this section with the following result. Theorem 4.6. The following facts hold: (i) If 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ1, then θλ,µ ≤ θ+ λ,µ < 0. (ii) If 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, then θ−λ,µ > d0, where d0 is a positive constant depending on λ, µ, r, N , S, |f |α, |g|α and |h|∞. Proof. (i) Assume (u, v) ∈ N+ λ,µ. Then by (4.3), we have 2− r β + γ − r ‖(u, v)‖2 > ∫ Ω h(x)|u|β |v|γdx. (4.9) Using (4.1) and (4.9), we obtain Iλ,µ(u, v) = (1 2 − 1 r ) ‖(u, v)‖2 + (1 r − 1 β + γ ) ∫ Ω h(x)|u|β |v|γdx < [(1 2 − 1 r ) + (1 r − 1 β + γ ) 2− r β + γ − r ] ‖(u, v)‖2 14 A. RANI, S. GOYAL EJDE-2020/119 =− (2− r)(β + γ − 2) 2r(β + γ) ‖(u, v)‖2 < 0. So, from the definitions of θλ,µ, θ+ λ,µ, we can deduce that θλ,µ ≤ θ+ λ,µ < 0. (ii) Let (u, v) ∈ N−λ,µ. Then from (4.3), 2− r β + γ − r ‖(u, v)‖2 < ∫ Ω h(x)|u|β |v|γdx. (4.10) Hölder’s inequality and Sobolev embedding theorem imply that ‖(u, v)‖ > ( 2− r (β + γ − r)|h|∞ ) 1 β+γ−2 S β+γ 2(β+γ−2) for all (u, v) ∈ N−λ,µ. (4.11) By (4.2) and (4.11), it follows that Iλ,µ(u, v) ≥ ‖(u, v)‖r [β + γ − 2 2(β + γ) ‖(u, v)‖2−r − β + γ − r r(β + γ) S−r/2 ( (λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2−r 2 ] > ( 2− r β + γ − r ) r β+γ−2S r(β+γ) 2(β+γ−2) [β + γ − 2 2(β + γ) ( 2− r (β + γ − r)|h|∞ ) 2−r β+γ−2 S (2−r)(β+γ) 2(β+γ−2) − β + γ − r r(β + γ) S−r/2((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2−r 2 ] . Thus, if 0 < (λ|f |α) 2 2−r +(µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, then Iλ,µ(u, v) > d0 for all (u, v) ∈ N−λ,µ, for some positive constant d0 = d0(λ, µ, r,N, S, |f |Lα , |g|Lα , |h|∞). � 5. Proof of Theorems 1.1 and 1.2 In this section, we show the existence of Palais-Smale sequence in N±λ,µ and give the proof of Theorems 1.1 and 1.2. Lemma 5.1. Suppose 0 < (λ|f |α) 2 2−r +(µ|g|α) 2 2−r < Λ1, where Λ1 is same as given in (1.4). Then for every z = (u, v) ∈ Nλ,µ, there exist ε > 0 and a differentiable mapping ζ : B(0, ε) ⊂ H → R+ such that ζ(0) = 1, ζ(w)(z − w) ∈ Nλ,µ and for all w = (w1, w2) ∈ H 〈ζ ′(0), w〉 = 2B(z, w)− rQλ,µ(z, w)− 2P(z, w) (2− r)‖(u, v)‖2 − (β + γ − r) ∫ Ω h(x)|u|β |v|γdx , (5.1) where B(z, w) = ∫ Ω Dmu ·Dmw1dx+ ∫ Ω Dmv ·Dmw2dx, Qλ,µ(z, w) = λ ∫ Ω f(x)|u|r−2uw1dx+ µ ∫ Ω g(x)|v|r−2vw2dx, P(z, w) = ∫ Ω β|u|β−2|v|γuw1dx+ ∫ Ω γ|u|β |v|γ−2vw2dx. Proof. For z = (u, v) ∈ Nλ,µ, define a map ϑz : R×H → R such that ϑz(ζ, w) = 〈I ′λ,µ(ζ(z − w)), ζ(z − w)〉 = ζ2‖(u− w1, v − w2)‖2 − ζr ∫ Ω (λf(x)|u− w1|r + µg(x)|v − w2|r)dx EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 15 − ζβ+γ ∫ Ω h(x)|u− w1|β |v − w2|γdx Then ϑz(1, (0, 0)) = 〈I ′λ,µ(z), z〉 = 0 and d dζ ϑz(1, (0, 0)) = 2‖(u, v)‖2 − r ∫ Ω (λf(x)|u|r + µg(x)|v|r)dx− (β + γ) ∫ Ω h(x)|u|β |v|γdx = (2− r)‖(u, v)‖2 − (β + γ − r) ∫ Ω h(x)|u|β |v|γdx 6= 0. Now, by the Implicit Function Theorem, there exists ε > 0 and a differentiable mapping ζ : B(0, ε) ⊂ H → R+ such that ζ(0) = 1, 〈ζ ′(0), w〉 = 2B(z, w)− rQλ,µ(z, w)− 2P(z, w) (2− r)‖(u, v)‖2 − (β + γ − r) ∫ Ω h(x)|u|β |v|γdx , ϑz(ζ(w), w) = 0 for all w ∈ B(0, ε). Thus, 〈I ′λ,µ(ζ(w)(z − w)), ζ(w)(z − w)〉 = 0 ∀ w ∈ B(0, ε). Therefore ζ(w)(z − w) ∈ Nλ,µ. � Lemma 5.2. Suppose 0 < (λ|f |α) 2 2−r +(µ|g|α) 2 2−r < Λ1, where Λ1 is same as given in (1.4). Then for every z = (u, v) ∈ N−λ,µ, there exist ε > 0 and a differentiable map ζ− : B(0, ε) ⊂ H → R+ such that ζ−(0) = 1 and ζ−(w)(z − w) ∈ N−λ,µ. Moreover, for all (w1, w2) ∈ H 〈(ζ−)′(0), w〉 = 2B(z, w)− rQλ,µ(z, w)− 2P(z, w) (2− r)‖(u, v)‖2 − (β + γ − r) ∫ Ω h(x)|u|β |v|γdx , where B, Qλ,µ and P are defined same as in Lemma 5.1. Proof. By argument used in Lemma 5.1, there exists ε > 0 and a differentiable function ζ− : B(0, ε) ⊂ H → R+ such that ζ−(0) = 1 and ζ−(w)(z − w) ∈ N−λ,µ. Since Ψ′′(u,v)(1) = (2− r)‖(u, v)‖2 − (β + γ − r) ∫ Ω h(x)|u|β |v|γdx < 0. By the continuity of Ψ′′ and ζ−, we have Ψ′′ζ−(w)(z−w)(1) = (2− r)‖ζ−(w)(z − w)‖2 − (β + γ − r) ∫ Ω h(x)|ζ−(w)(z − w)|β |ζ−(w)(z − w)|γ < 0, for ε > 0 is sufficiently small. Thus, ζ−(w)(z − w) ∈ N−λ,µ. � Lemma 5.3. Let 1 ≤ r < 2 < N/m and 2 < β+γ ≤ 2∗m, then the following results hold: (i) If 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ1, then there exists a (PS)θλ,µ-sequence {(un, vn)} ⊂ Nλ,µ in H for Iλ,µ. (ii) If 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, then there exists a (PS)θ−λ,µ - sequence {(un, vn)} ⊂ N−λ,µ in H for Iλ,µ, where Λ1 is same as given in (1.4). 16 A. RANI, S. GOYAL EJDE-2020/119 Proof. (i) By Lemma 4.1 and Ekeland Variational Principle [16], there exists a minimizing sequence {(un, vn)} ⊂ Nλ,µ such that Iλ,µ(un, vn) < θλ,µ + 1 n , Iλ,µ(un, vn) < Iλ,µ(u, v) + 1 n ‖(u, v)− (un, vn)‖, for each (u, v) ∈ Nλ,µ. (5.2) Since θλ,µ < 0 and taking n large, we obtain Iλ,µ(un, vn) = (1 2 − 1 β + γ ) ‖(un, vn)‖2 − (1 r − 1 β + γ )∫ Ω (λf(x)|un|r + µg(x)|vn|r)dx < θλ,µ + 1 n < θλ,µ 2 . (5.3) Thus, we have 0 < −r(β + γ)θλ,µ 2(β + γ − r) < ∫ Ω (λf(x)|un|r + µg(x)|vn|r)dx ≤ S−r/2((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2−r 2 ‖(un, vn)‖r. (5.4) Consequently, (un, vn) 6= (0, 0). Also, (5.3), (5.4) and Hölder’s inequality assert that ‖(un, vn)‖ ≤ [2(β + γ − r) r(β + γ − 2) S−r/2((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2−r 2 ] 1 2−r , (5.5) and ‖(un, vn)‖ ≥ [ − r(β + γ) 2(β + γ − r) θλ,µS r 2 ((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) r−2 2 ]1/r . Now, we show that ‖I ′λ,µ(un, vn)‖H−1 → 0, as n→∞. Using Lemma 5.1 for each zn = (un, vn) to obtain the mapping ζn : B(0, εn)→ R+ for some εn > 0 such that ζn(w)(zn − w) ∈ Nλ,µ. Choose 0 < η < εn. Let z = (u, v) ∈ H with z 6= 0 and take w∗η = ηz ‖z‖ . We set wη = ζn(w∗η)(zn−w∗η). Since wη ∈ Nλ,µ, from (5.2), we obtain Iλ,µ(wη)− Iλ,µ(zn) ≥ − 1 n ‖wη − zn‖. Using Mean Value Theorem, we obtain 〈I ′λ,µ(zn), wη − zn〉+ o(‖wη − zn‖) ≥ − 1 n ‖wη − zn‖. Therefore 〈I ′λ,µ(zn),−w∗η〉+ (ζn(w∗η)− 1)〈I ′λ,µ(zn), zn − w∗η〉 ≥ − 1 n ‖wη − zn‖+ o(‖wη − zn‖). (5.6) Since ζn(w∗η)(zn − w∗η) ∈ Nλ,µ and from (5.6), we obtain − η〈I ′λ,µ(zn), z ‖z‖ 〉+ (ζn(w∗η)− 1)〈I ′λ,µ(zn − wη), zn − w∗η〉 ≥ − 1 n ‖wη − zn‖+ o(‖wη − zn‖). EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 17 Thus, we have 〈I ′λ,µ(zn), z ‖z‖ 〉 ≤ 1 nη ‖wη − zn‖+ 1 η o(‖wη − zn‖) + (ζn(w∗η)− 1) η 〈I ′λ,µ(zn − wη), zn − w∗η〉 (5.7) Since ‖wη − zn‖ ≤ η|ζn(w∗η)|+ |ζn(w∗η)− 1|‖zn‖ and lim η→0 |ζn(w∗η)− 1| η ≤ ‖ζ ′n(0)‖, if we take η → 0 in (5.7) for a fixed n ∈ N and using (5.5) we can find a constant M > 0, free from η such that 〈I ′λ,µ(zn), z ‖z‖ 〉 ≤ M n (1 + ‖ζ ′n(0)‖). Now, we show that ‖ζ ′n(0)‖ is uniformly bounded. From (5.1), (5.7) and by Hölder’s inequality, we have |〈ζ ′n(0)〉| ≤ M1‖(w1, w2)‖ |(2− r)‖(un, vn)‖2 − (β + γ − r) ∫ Ω h(x)|un|β |vn|γdx| , for some M1 > 0. Next we show that |(2− r)‖(un, vn)‖2 − (β + γ − r) ∫ Ω h(x)|un|β |vn|γdx| ≥M2, for some M2 > 0 and n is taking large enough. On the contrary, suppose there exists a subsequence {(un, vn)} such that (2− r)‖(un, vn)‖2 − (β + γ − r) ∫ Ω h(x)|un|β |vn|γdx = on(1). (5.8) From (5.8) and using (un, vn) ∈ Nλ,µ, we have ‖(un, vn)‖2 = β + γ − r 2− r ∫ Ω h(x)|un|β |vn|γdx+ on(1), ‖(un, vn)‖2 = β + γ − r β + γ − 2 Qλ,µ(un, vn) + on(1). By Hölder’s inequality and the Sobolev embedding theorem, we obtain ‖(un, vn)‖ ≥ ( 2− r β + γ − r S β+γ 2 |h|∞ ) 1 β+γ−2 + on(1), ‖(un, vn)‖ ≤ (β + γ − r β + γ − 2 ) 1 2−r S− r 2(2−r) ( (λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 1 2 + on(1). This implies that (λ|f |α) 2 2−r +(µ|g|α) 2 2−r ≥ Λ1, which is a contradiction to the fact that 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ1. Hence 〈I ′λ,µ(un, vn), (u, v) ‖(u, v)‖ 〉 ≤ M n . This completes the proof of (i). (ii) By Lemma 5.2, part (ii) can be shown in similar way as above. � Now, we show the existence of a local minimum for Iλ,µ on N+ λ,µ. 18 A. RANI, S. GOYAL EJDE-2020/119 Theorem 5.4. Let Λ1 be the same defined as in (1.4). If 1 ≤ r < 2 < N m , 2 < β + γ ≤ 2∗m, and 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ1, then Iλ,µ has a minimizer (u1 λ,µ, v 1 λ,µ) in N+ λ,µ and it satisfies the following: (i) Iλ,µ(u1 λ,µ, v 1 λ,µ) = θλ,µ = θ+ λ,µ < 0. (ii) (u1 λ,µ, v 1 λ,µ) is a nontrivial solution of (1.1). (iii) Iλ,µ(u1 λ,µ, v 1 λ,µ)→ (0, 0) as λ→ 0+, µ→ 0+. Proof. By Lemma 5.3 (i), there exists a minimizing sequence {(un, vn)} for Iλ,µ on Nλ,µ such that Iλ,µ(un, vn) = θλ,µ + on(1), I ′λ,µ(un, vn) = on(1) in H−1. (5.9) By coercivity of Iλ,µ on Nλ,µ, we obtain that {(un, vn)} is bounded in H. Therefore up to a subsequence still denoted by {(un, vn)} converges weakly to (u1 λ,µ, v 1 λ,µ) ∈ H. This implies un ⇀ u1 λ,µ, vn ⇀ v1 λ,µ weakly in Hm 0 (Ω), un ⇀ u1 λ,µ, vn ⇀ v1 λ,µ a.e. Ω, un ⇀ u1 λ,µ, vn ⇀ v1 λ,µ strongly in Ls(Ω) ∀1 ≤ s < 2∗m. (5.10) It is easy to see that as n→∞ Qλ,µ(un, vn) = Qλ,µ(u1 λ,µ, v 1 λ,µ) + on(1). (5.11) First we claim that (u1 λ,µ, v 1 λ,µ) is a nontrivial solution of (1.1). From (5.9) and (5.10), one can easily verify that (u1 λ,µ, v 1 λ,µ) is a weak solution of the system (1.1). Since (un, vn) ∈ Nλ,µ and by the definition of Iλ,µ, we have Qλ,µ(un, vn) = r(β + γ − 2) 2(β + γ − r) ‖(un, vn)‖2 − r(β + γ) (β + γ − r) Iλ,µ(un, vn). (5.12) Then letting n→∞ in (5.12) and using (5.9), (5.11) with θλ,µ < 0, we obtain Qλ,µ(u1 λ,µ, v 1 λ,µ) ≥ − r(β + γ) (β + γ − r) θλ,µ > 0. Thus, (u1 λ,µ, v 1 λ,µ) ∈ Nλ,µ is a nontrivial solution of (1.1). Now, we show that (un, vn) → (u1 λ,µ, v 1 λ,µ) strongly in H and Iλ,µ(u1 λ,µ, v 1 λ,µ) = θλ,µ. If (u, v) ∈ Nλ,µ, then Iλ,µ(u, v) = β + γ − 2 2(β + γ) ‖(u, v)‖2 − β + γ − r r(β + γ) Qλ,µ(u, v). (5.13) To prove that Iλ,µ(u1 λ,µ, v 1 λ,µ) = θλ,µ, it is sufficient to recall that (u1 λ,µ, v 1 λ,µ) ∈ Nλ,µ, (5.13) and apply Fatou’s lemma to obtain θλ,µ ≤ Iλ,µ(u1 λ,µ, v 1 λ,µ) = β + γ − 2 2(β + γ) ‖(u1 λ,µ, v 1 λ,µ)‖2 − β + γ − r r(β + γ) Qλ,µ(u1 λ,µ, v 1 λ,µ) ≤ lim inf n→∞ (β + γ − 2 2(β + γ) ‖(un, vn)‖2 − (β + γ − r) r(β + γ) Qλ,µ(un, vn) ) ≤ lim inf n→∞ Iλ,µ(un, vn) = θλ,µ. (5.14) EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 19 This implies that Iλ,µ(u1 λ,µ, v 1 λ,µ) = θλ,µ and limn→∞ ‖(un, vn)‖2 = ‖(u1 λ,µ, v 1 λ,µ)‖2. Let (un, vn) = (un − u1 λ,µ, vn − v1 λ,µ), then by Brézis and Lieb lemma [7] gives ‖(un, vn)‖2 = ‖(un, vn)‖2 − ‖(u1 λ,µ, v 1 λ,µ)‖2 + on(1). Therefore, (un, vn) → (u1 λ,µ, v 1 λ,µ) strongly in H. Moreover, we have (u1 λ,µ, v 1 λ,µ) ∈ N+ λ,µ. Thus, θλ,µ = θ+ λ,µ. On the contrary, if (u1 λ,µ, v 1 λ,µ) ∈ N−λ,µ, then using (4.10) and (5.14), we have that ∫ Ω h(x)|u1 λ,µ|β |v1 λ,µ|γ > 0 and Qλ,µ(u1 λ,µ, v 1 λ,µ) > 0. Thus, from Lemma 4.4, there exist unique t+1 and t−1 such that (t+1 u 1 λ,µ, t + 1 v 1 λ,µ) ∈ N+ λ,µ and (t−1 u 1 λ,µ, t − 1 v 1 λ,µ) ∈ N−λ,µ. In particular, we have t+1 < t−1 = 1. Since d dt Iλ,µ(t+1 u 1 λ,µ, t + 1 v 1 λ,µ) = 0, d2 dt2 Iλ,µ(t+1 u 1 λ,µ, t + 1 v 1 λ,µ) > 0, there exists t+1 < t̄ ≤ t−1 such that Iλ,µ(t+1 u 1 λ,µ, t + 1 v 1 λ,µ) < Iλ,µ(t̄u1 λ,µ, t̄v 1 λ,µ). By Lemma 4.4, we obtain Iλ,µ(t+1 u 1 λ,µ, t + 1 v 1 λ,µ) < Iλ,µ(t̄u1 λ,µ, t̄v 1 λ,µ) ≤ Iλ,µ(t−1 u 1 λ,µ, t − 1 v 1 λ,µ) = Iλ,µ(u1 λ,µ, v 1 λ,µ) = θλ,µ, which is a contradiction. Therefore, using Lemma 4.2, we conclude that (u1 λ,µ, v 1 λ,µ) is a nontrivial solution of (1.1). (iii) Further, from Theorem 4.6 (i) and (4.2), we have 0 > θ+ λ,µ ≥ θλ,µ = Iλ,µ(u1 λ,µ, v 1 λ,µ) > −β + γ − r r(β + γ) S−r/2((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 2−r 2 ‖(u, v)‖r, which implies that Iλ,µ(u1 λ,µ, v 1 λ,µ) → (0, 0) as λ → 0+, µ → 0+. This completes the proof. � Theorem 5.5. If 1 ≤ r < 2 < N m , 2 < β + γ < 2∗m and 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, then Iλ,µ has a minimizer (u2 λ,µ, v 2 λ,µ) in N−λ,µ and satisfies the following: (i) Iλ,µ(u2 λ,µ, v 2 λ,µ) = θ−λ,µ; (ii) (u2 λ,µ, v 2 λ,µ) is a solution of the system (1.1). Proof. Let {(un, vn)} be a minimizing sequence for Iλ,µ on N−λ,µ. Then by Iλ,µ coercive on Nλ,µ and the compact imbedding theorem, there exist a subsequence {(un, vn)} and (u2 λ,µ, v 2 λ,µ) ∈ H such that un ⇀ u2 λ,µ and vn ⇀ v2 λ,µ weakly in Hm 0 (Ω), un → u2 λ,µ and vn → v2 λ,µ strongly in Lr(Ω), Lβ+γ(Ω). This implies Qλ,µ(un, vn) = Qλ,µ(u2 λ,µ, v 2 λ,µ) + on(1),∫ Ω h(x)|un|β |vn|γ = ∫ Ω h(x)|u2 λ,µ|β |v2 λ,µ|γ + on(1). Using (4.10) and (4.11), there exists M3 > 0 such that ∫ Ω h(x)|un|β |vn|γdx > M3. This implies that ∫ Ω h(x)|u2 λ,µ|β |v2 λ,µ|γdx ≥M3. 20 A. RANI, S. GOYAL EJDE-2020/119 Now, we prove that (un, vn)→ (u2 λ,µ, v 2 λ,µ) strongly in H. On contrary, we assume that ‖(u2 λ,µ, v 2 λ,µ)‖ < lim infn→∞ ‖(un, vn)‖. Then using Lemma 4.4, there exists a unique t−2 such that (t−2 u 2 λ,µ, t − 2 v 2 λ,µ) ∈ N−λ,µ. Since (un, vn) ∈ N−λ,µ, Iλ,µ(un, vn) ≥ Iλ,µ(tun, tvn) for all t ≥ 0, we have θ−λ,µ ≤ Iλ,µ(t−u2 λ,µ, t −v2 λ,µ) < lim n→∞ Iλ,µ(t−un, t −vn) ≤ lim n→∞ Iλ,µ(un, vn) = θ−λ,µ. Hence, (un, vn)→ (u2 λ,µ, v 2 λ,µ) strongly in H. This implies Iλ,µ(u2 λ,µ, v 2 λ,µ) = lim n→∞ Iλ,µ(un, vn) = θ−λ,µ. By Lemma 4.2 and (5.14), we say that (u2 λ,µ, v 2 λ,µ) is a nontrivial solution of the system (1.1). Finally, by using the same arguments as in the proof of Theorem 5.4, for all 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, we have that (u2 λ,µ, v 2 λ,µ) is a solution of the system (1.1). � Theorems 1.1 and 1.2 follow from Theorems 5.4 and 5.5 respectively. Also from Theorem 5.4 and 5.5, we obtain that for all 1 < r < 2 < N m , 2 < β+γ < 2∗m, λ, µ > 0 and 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, the system (1.1) has two nontrivial solutions (u1 λ,µ, v 1 λ,µ) ∈ N+ λ,µ and (u2 λ,µ, v 2 λ,µ) ∈ N−λ,µ. Since N+ λ,µ ∩ N − λ,µ = φ, we can conclude that (u1 λ,µ, v 1 λ,µ) and (u2 λ,µ, v 2 λ,µ) are distinct. 6. Proof of Theorem 1.3 In this section, we show the existence of a second weak solution in the critical case β+γ = 2∗m as a limit of Palais-Smale sequence which is obtained by minimizing sequence for Iλ,µ in N−λ,µ. For this, taking ρ > 0 small enough such that Bρ(0) ⊂ Ω and define the function uε(x) = φ(x)Uε(x), where φ(x) ∈ C∞0 (Bρ(0)) is a cut-off function such that φ(x) ≡ 1 in Bρ/2(0) and Uε(x) is same as mentioned in (2.2). Then, we have the following estimates (see [18, 19, 32]). Lemma 6.1. Suppose N ≥ 2m+1. Then the following estimates hold when ε→ 0: ‖uε‖2 = S N 2m +O(εN−2m), (6.1)∫ Ω |uε|2 ∗ mdx = S N 2m +O(εN ), (6.2) ∫ Ω |uε|rdx =  O1(ε (N−2m)r 2 ) if 1 < r < N N−2m , O1(εN− (N−2m)r 2 | ln ε|) if r = N N−2m , O1(εN− (N−2m)r 2 ) if N N−2m < r < 2∗m. (6.3) Lemma 6.2. Suppose that (A1)–(A4) hold with δ0 > N − 2m and N N−2m ≤ r < 2. Then there exists Λ > 0 such that for all 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ there exists (uλ,µ, vλ,µ) in H \ {(0, 0)} such that sup t≥0 Iλ,µ(tuλ,µ, tvλ,µ) < c∞, where c∞ is the constant given in Lemma 3.3. In particular, θ−λ,µ < c∞ for all 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ. EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 21 Proof. By assumption (A4), there exists δ0 > N − 2m such that, for x ∈ B(0, 2ρ0) where 0 < ρ0 ≤ r0 h(x) = h(0) + o(|x|δ0) as x→ 0. Define a functional τ : H → R such that τ(u, v) = 1 2 ‖(u, v)‖2 − 1 β + γ ∫ Ω h(x)|u|β |v|γdx ∀ (u, v) ∈ H. (6.4) Set uε = √ βuε, vε = √ γuε with (uε, vε) ∈ H. The map τ(tuε, tvε) satisfies τ(0) = 0, τ(tuε, tvε) > 0 for t > 0 small and τ(tuε, tvε) < 0 for t > 0 large. Moreover, τ attains its maximum at t0 = ( ‖(uε, vε)‖2∫ Ω h(x)|uε|β |vε|γdx ) 1 β+γ−2 . (6.5) Thus, from (6.1), (6.2), (6.4), (6.5) and (2.4), we have sup t≥0 τ(tuε, tvε) = t20 2 ‖(uε, vε)‖2 − tβ+γ 0 β + γ ∫ Ω h(x)|uε|β |vε|γdx = (1 2 − 1 β + γ ) ‖(uε, vε)‖ 2(β+γ) β+γ−2 ( ∫ Ω h(x)|uε|β |vε|γdx) 2 β+γ−2 = m N [(β γ ) γ β+γ + (γ β ) β β+γ ] N 2m [ ‖uε‖2 ( ∫ Ω h(x)|uε|2∗ mdx) 2 2∗m ] N 2m = m N [(β γ ) γ β+γ + (γ β ) β β+γ ] N 2m [ S N 2m +O(εN−2m) (h(0)S N 2m +O(εN ) +O(εδ0)) 2 2∗m ] N 2m ≤ m N (h(0))− N−2m 2m S N 2m β,γ +O(εN−2m)−O(εδ0). Therefore sup t≥0 τ(tuε, tvε) ≤ m N (h(0))− N−2m 2m S N 2m β,γ +O(εN−2m)−O(εδ0). (6.6) Now, we choose δ1 > 0 such that c∞ > 0 for all 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < δ1. Using the definition of Iλ,µ and λ, µ > 0, we obtain Iλ,µ(tuε, tvε) ≤ t2 2 ‖(uε, vε)‖ 2 for t ≥ 0. Thus, there exists t0 ∈ (0, 1) such that sup 0≤t≤t0 Iλ,µ(tuε, tvε) < c∞ for all 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < δ1. Using β, γ > 1, (6.6) and (6.3), we obtain sup t≥t0 Iλ,µ(tuε, tvε) = sup t≥t0 ( τ(tuε, tvε)− 1 r Qλ,µ(tuε, tvε) ) ≤ m N (h(0))− N−2m 2m S N 2m β,γ +O(εN−2m)− 1 r tr0 ∫ Ω (λf(x)|uε|r + µg(x)|vε|r)dx ≤ m N (h(0))− N−2m 2m S N 2m β,γ +O(εN−2m)− 1 r tr0(a0λβ r/2 + b0µγ r/2) ∫ Ω |uε|rdx 22 A. RANI, S. GOYAL EJDE-2020/119 ≤ m N (h(0))− N−2m 2m S N 2m β,γ +O(εN−2m)− 1 r tr0η(λ+ µ) ∫ Ω |uε|rdx ≤ m N (h(0))− N−2m 2m S N 2m β,γ +O(εN−2m) − 1 r tr0η(λ+ µ) { O1(εN− (N−2m)r 2 | ln ε|) if r = N N−2m O1(εN− (N−2m)r 2 ) if N N−2m < r < 2∗m, where η = min{a0, b0}. Choose δ2 > 0 in such a way that 0 ≤ ε < δ2. Now, take ε = ((λ|f |α) 2 2−r + (µ|g|α) 2 2−r ) 1 N−2m . Then, we have sup t≥t0 Iλ,µ(tuε, tvε) ≤ m N (h(0))− N−2m 2m S N 2m β,γ +O(A(λ, µ)) − η(λ+ µ) r { O1((A(λ, µ)) N 2(N−2m) | ln(A(λ, µ))|) if r = N N−2m O1((A(λ, µ)) N N−2m− r 2 ) if N N−2m < r < 2∗m, (6.7) where A(λ, µ) = (λ|f |α) 2 2−r + (µ|g|α) 2 2−r . Case (i): When r = N N−2m , we can choose δ3 > 0 with 0 < A(λ, µ) < δ3 such that O(A(λ, µ))− η(λ+ µ) r O1((A(λ, µ)) N 2(N−2m) | ln(A(λ, µ))|) < −P0(A(λ, µ)), as λ, µ→ 0, |ln(A(λ, µ))| → +∞. Case (ii): When N N−2m < r < 2∗m, we can choose δ4 > 0 with 0 < A(λ, µ) < δ4 such that O(A(λ, µ))− η(λ+ µ) r O1((A(λ, µ)) N N−2m− r 2 ) < −P0(A(λ, µ)), as 1 + 2 2−r ( N N−2m − r 2 ) < 2 2−r if and only if r > N N−2m . Now, choose Λ = min{δ1, δN−2m 2 , δ3, δ4} > 0. Then using this and (6.7), we have sup t≥0 Iλ,µ(tuε, tvε) < m N (h(0))− N−2m 2m S N 2m β,γ−P0((λ|f |α) 2 2−r +(µ|g|α) 2 2−r ) = c∞, (6.8) for 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ. Next, we show that θ−λ,µ < c∞ for all 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ. From (A3), (A4) and the definition of (uε, vε), we obtain∫ Ω h(x)|uε|β |vε|γdx > 0, Qλ,µ(uε, vε) > 0. Combining this with Lemma 4.4 (ii), definition of θ−λ,µ and (6.8), for all 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < Λ, we obtain that there exists tλ,µ > 0 such that (tλ,µuε, tλ,µvε) ∈ N−λ,µ with θ−λ,µ ≤ Iλ,µ(tλ,µuε, tλ,µvε) < sup t≥0 Iλ,µ(tuε, tvε) < c∞. On taking (uε, vε) = (uλ,µ, vλ,µ), we obtain the desired result which completes the proof. � EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 23 Theorem 6.3. Assume that (A1)–(A4) hold. Then Iλ,µ satisfies the (PS)θ−λ,µ condition for all 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1. Moreover, Iλ,µ has a minimizer (u2 λ,µ, v 2 λ,µ) in N−λ,µ and satisfies the following conditions: (i) Iλ,µ(u2 λ,µ, v 2 λ,µ) = θ−λ,µ > 0; (ii) (u2 λ,µ, v 2 λ,µ) is a nontrivial solution of (1.1), where Λ1 is same as mentioned in (1.4). Proof. By Lemma 5.3 (ii), for 0 < (λ|f |α) 2 2−r +(µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, there exists a (PS)θ−λ,µ -sequence {(un, vn)} ⊂ N−λ,µ in H for Iλ,µ. Then, from Lemma 3.2, we find that {(un, vn)} is bounded in H. Now, using Lemma 6.2 and Lemma 3.3, Iλ,µ satisfies the (PS)θ−λ,µ -condition. Then, there exists (u2 λ,µ, v 2 λ,µ) ∈ H such that up to subsequence (un, vn) → (u2 λ,µ, v 2 λ,µ) in H. Moreover, Iλ,µ(u2 λ,µ, v 2 λ,µ) = θ−λ,µ > 0 and (u2 λ,µ, v 2 λ,µ) ∈ N−λ,µ, since N−λ,µ is a closed set. Using the argument as applied in Theorem 5.4, one can easily obtain that (u2 λ,µ, v 2 λ,µ) is a nontrivial solution of system (1.1) for 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1. � Proof of Theorem 1.3. By Theorem 5.4 and Theorem 6.3, we obtain that for all λ, µ > 0 and 0 < (λ|f |α) 2 2−r + (µ|g|α) 2 2−r < ( r2 ) 2 2−r Λ1, system (1.1) has two distinct solutions (u1 λ,µ, v 1 λ,µ) ∈ N+ λ,µ and (u2 λ,µ, v 2 λ,µ) ∈ N−λ,µ, since N+ λ,µ ∩N − λ,µ = φ. � Next, we show that the solutions (u1 λ,µ, v 1 λ,µ) and (u2 λ,µ, v 2 λ,µ) are not semi-trivial. Using Theorem 5.4 (i) and Theorem 6.3 (i) respectively, we obtain Iλ,µ(u1 λ,µ, v 1 λ,µ) < 0 and Iλ,µ(u2 λ,µ, v 2 λ,µ) > 0. (6.9) We observe that, if (u, 0) (or (0, v)) is a semi-trivial solution of (1.1), then we have (−∆)mu = λf(x)|u|r−2u in Ω, Dku = 0 for all |k| ≤ m− 1 on ∂Ω. (6.10) Then Iλ,µ(u, 0) = 1 2 ‖u‖2 − λ r ∫ Ω f(x)|u|rdx = −2− r 2r ‖u‖2 < 0. (6.11) From (6.9) and (6.11), we obtain that (u2 λ,µ, v 2 λ,µ) is not semi-trivial. Now, we will prove that (u1 λ,µ, v 1 λ,µ) is not semi-trivial. Without loss of generality, we assume that v1 λ,µ ≡ 0. Then u1 λ,µ is a non-trivial solution of (6.10) and ‖(u1 λ,µ, 0)‖2 = ‖u1 λ,µ‖2 = λ ∫ Ω f(x)|u1 λ,µ|rdx > 0. We take w ∈ Hm 0 (Ω) \ {0} such that ‖(0, w)‖2 = ‖w‖2 = µ ∫ Ω g(x)|w|rdx. From Lemma 4.4, there exists a unique 0 < t1 < tmax(u1 λ,µ, w) such that (t1u 1 λ,µ, t1w) ∈ N+ λ,µ, where tmax(u1 λ,µ, w) = ( (β + γ − r) ∫ Ω (λf(x)|u1 λ,µ|r + µg(x)|w|r)dx (β + γ − 2)‖(u1 λ,µ, w)‖2 ) 1 2−r 24 A. RANI, S. GOYAL EJDE-2020/119 = (β + γ − r β + γ − 2 ) 1 2−r > 1. Moreover, Iλ,µ(t1u 1 λ,µ, t1w) = inf 0≤t≤tmax Iλ,µ(tu1 λ,µ, tw). This and the fact that (u1 λ,µ, 0) ∈ N+ λ,µ imply that θ+ λ,µ ≤ Iλ,µ(t1u 1 λ,µ, t1w) ≤ Iλ,µ(u1 λ,µ, w) < Iλ,µ(u1 λ,µ, 0) = θ+ λ,µ, which is a contradiction. Hence, (u1 λ,µ, v 1 λ,µ) is not semi-trivial. This completes the proof. Acknowledgements. S. Goyal was supported by Science and Engineering Re- search Board, Department of Science and Technology, Government of India, Grant number: ECR/2017/002651. References [1] K. Adriouch, E. I. Hamidi; The Nehari manifold for systems of nonlinear elliptic equations, Nonlinear Anal. 64 (2006), no. 10, 2149–2167. [2] C. O. Alves, D. C. de Morais Filho, M. A. S. Souto; On systems of elliptic equations involving subcritical or critical Sobolev exponents, Nonlinear Anal. 42 (2000), 771–787. [3] A. Ambrosetti, H. Brezis, G. Cerami; Combined effects of concave and convex nonlinearities in some elliptic problems, J. Funct. Anal. 122 (1994), no. 2, 519–543. [4] T. Bartsch and M. Willem; On an elliptic equation with concave and convex nonlinearities, Proc. Amer. Math. Soc. 123 (1995), no. 11, 3555–3561. [5] F. Bernis, J. G. Azorero, I. Peral, et al.; Existence and multiplicity of nontrivial solutions in semilinear critical problems of fourth order, Adv. Differential Equations 1 (1996), no. 2, 219–240. [6] Y. Bozhkov, E. Mitidieri; Existence of multiple solutions for quasilinear systems via fibering method, J. Differential Equations 190 (2003), no. 1, 239–267. [7] H. Brézis, E. Lieb; A relation between pointwise convergence of functions and convergence of functionals, Proc. Amer. Math. Soc. 88 (1983), no. 3, 486–490. [8] H. Brézis, L. Nirenberg; Positive solutions of nonlinear elliptic equations involving critical Sobolev exponents, Comm. Pure Appl. Math. 36 (1983), no. 4, 437–477. [9] K. J. Brown, T. F. Wu; A semilinear elliptic system involving nonlinear boundary condition and sign-changing weight function, J. Math. Anal. Appl. 337 (2008), no. 2, 1326–1336. [10] K. J. Brown, Y. Zhang; The Nehari manifold for a semilinear elliptic equation with a sign- changing weight function, J. Differential Equations 193 (2003), no. 2, 481–499. [11] D. G. D. Figueiredo, J. P. Gossez, P. Ubilla; Local “superlinearity” and “sublinearity” for the p-Laplacian, J. Funct. Anal. 257 (2009), no. 3, 721–752. [12] Y. Deng, W. Shuai; Non-trivial solutions for a semilinear biharmonic problem with critical growth and potential vanishing at infinity, Proc. Roy. Soc. Edinburgh Sect. A. 145 (2015), no. 2, 281–299. [13] Y. Deng, G. Wang; On inhomogeneous biharmonic equations involving critical exponents, Proc. Roy. Soc. Edinburgh Sect. A. 129 (1999), no. 5, 925–946. [14] P. Drábek, S. I. Pohozaev; Positive solutions for the p-Laplacian: application of the fibrering method, Proc. Roy. Soc. Edinburgh Sect. A. 127 (1997), no. 4, 703–726. [15] D. E. Edmunds, D. Fortunato, E. Jannelli; Critical exponents, critical dimensions and the biharmonic operator, Arch. Ration. Mech. Anal. 112 (1990), no. 3, 269–289. [16] I. Ekeland; On the variational principle, J. Math. Anal. Appl. 47 (1974), no. 2, 324–353. [17] A. Ferrero, G. Warnault; On solutions of second and fourth order elliptic equations with power-type nonlinearities, Nonlinear Anal. 70 (2009), no. 8, 2889–2902. [18] F. Gazzola, B. Ruf, et al.; Lower-order perturbations of critical growth nonlinearities in semilinear elliptic equations, Adv. Differential Equations 2 (1997), no. 4, 555–572. [19] H. C. Grunau; Positive solutions to semilinear polyharmonic Dirichlet problems involving critical Sobolev exponents, Calc. Var. Partial Differential Equations 3 (1995), no. 2, 243–252. EJDE-2020/119 POLYHARMONIC SYSTEMS INVOLVING CRITICAL NONLINEARITIES 25 [20] E. I. Hamidi; Existence results to elliptic systems with nonstandard growth conditions, J. Math. Anal. Appl. 300 (2004), no. 1, 30–42. [21] E. I. Hamidi; Multiple solutions with changing sign energy to a nonlinear elliptic equation, Commun. Pure Appl. Anal. 3 (2004), no. 2, 253–266. [22] T. S. Hsu; Multiple positive solutions for a critical quasilinear elliptic system with concave- convex nonlinearities, Nonlinear Anal. 71 (2009), no. 7-8, 2688–2698. [23] T. S. Hsu; Multiplicity results for p-laplacian with critical nonlinearity of concave-convex type and sign-changing weight functions, Abstr. Appl. Anal. 2009 (2009). [24] T. S. Hsu; Multiple positive solutions for a quasilinear elliptic system involving concave- convex nonlinearities and sign-changing weight functions, Int. J. Math. Math. Sci. 2012 (2012). [25] C. Ji, W. Wang; On the p-biharmonic equation involving concave-convex nonlinearities and sign-changing weight function, Electron. J. Qual. Theory Differ. Equ. 2012 (2012), no. 2, 1–17. [26] D. Lu, J. Xiao; Multiplicity of solutions for biharmonic elliptic systems involving critical nonlinearity, Bull. Korean Math. Soc. 50 (2013), no. 5, 1693–1710. [27] T. G. Myers; Thin films with high surface tension, SIAM review 40 (1998), no. 3, 441–462. [28] P. Pucci, J. Serrin; Critical exponents and critical dimensions for polyharmonic operators, J. Math. Pures Appl. (9) 69 (1990), no. 1, 55–83. [29] X. Qian, J. Wang, M. Zhu,; Multiple nontrivial solutions for a class of biharmonic elliptic equations with Sobolev critical exponent, Math. Probl. Eng. 2018 (2018), no. 3, 1–12. [30] Y. Shang, W. Li; Multiple nontrivial solutions for a class of semilinear polyharmonic equa- tions, Acta Math. Sci. Ser. B (Engl. Ed.) 34 (2014), no. 5, 1495–1509. [31] M. Squassina; An eigenvalue problem for elliptic systems, New York J. Math. 6 (2000), no. 95, 106. [32] M. Struwe; Variational methods- applications to nonlinear partial differential equations and Hamiltonian systems, Ergeb. Math. Grenzgeb. (3) 34 (1990). [33] C. A. Swanson; The best Sobolev constant, Appl. Anal. 47 (1992), no. 1-4, 227–239. [34] J. Velin; Existence results for some nonlinear elliptic system with lack of compactness, Non- linear Anal. 52 (2003), no. 3, 1017–1034. [35] T. F. Wu; The Nehari manifold for a semilinear elliptic system involving sign-changing weight functions, Nonlinear Anal. 68 (2008), no. 6, 1733–1745. Anu Rani Department of Mathematics, Bennett University, Greater Noida, Uttar Pradesh, India Email address: ar4091@bennett.edu.in Sarika Goyal Department of Mathematics, Bennett University, Greater Noida, Uttar Pradesh, India Email address: sarika1.iitd@gmail.com 1. Introduction Notation 2. Preliminaries 3. The Palais-Smale condition 4. Nehari manifold for (??) 5. Proof of Theorems ?? and ?? 6. Proof of Theorem ?? Acknowledgements References