Electronic Journal of Differential Equations, Vol. 2021 (2021), No. 11, pp. 1–17. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu EXISTENCE OF SOLUTIONS FOR CRITICAL FRACTIONAL p-LAPLACIAN EQUATIONS WITH INDEFINITE WEIGHTS NA CUI, HONG-RUI SUN Abstract. This article concerns the critical fractional p-Laplacian equation with indefinite weights (−∆p)su = λg(x)|u|p−2u+ h(x)|u|p ∗ s−2u in RN , where 0 < s < 1 < p <∞, N > sp and p∗s = Np/(N−sp), the weight functions g may be indefinite, and h changes sign. Specifically, based on the results of asymptotic estimates for an extremal in the fractional Sobolev inequality and the discrete spectrum of fractional p-Laplacian operator, we establish an existence criterion for a nontrivial solution to this problem. 1. Introduction The purpose of this article is to study the existence of nontrivial solutions for the critical fractional p-Laplacian equation (−∆p) su = λg(x)|u|p−2u+ h(x)|u|p ∗ s−2u in RN , (1.1) where 0 < s < 1 < p <∞, N > sp, p∗s = Np/(N −sp) and (−∆p) s is the fractional p-Laplacian operator which is defined as (−∆p) su(x) = 2 lim ε→0+ ∫ RN\Bε(x) |u(x)− u(y)|p−2(u(x)− u(y)) |x− y|N+sp dy, ∀x ∈ RN , the weight functions g may be indefinite and h changes sign. In recent years, there has been an increasing attention to nonlocal diffusion problems, in particular to the ones driven by the fractional Laplacian operator [15]. One of the reasons for this comes from the fact that this operator naturally arises in several physical phenomena like flames propagation [9] and anomalous diffusion [22], in geophysical fluid dynamics [11], or in mathematical finance [13] and so on. It also provides a simple model to describe certain jump Lévy processes in probability theory [3]. For fractional Laplacian problems on bounded domains, an approach was pro- posed by Caffarelli and Silvestre in [10], which allows to transform a nonlocal prob- lem into a local problem via the Dirichlet-Neumann map. By employing the har- monic extension, a large number of existence results have been obtained for Dirichlet problems involving the fractional Laplacian in bounded domain. In particular, to 2010 Mathematics Subject Classification. 35R11, 35J92, 35B33. Key words and phrases. Fractional p-Laplacian; critical exponent; indefinite weight. c©2021 Texas State University. Submitted April 1, 2020. Published March 5, 2021. 1 2 N. CUI, H.-R. SUN EJDE-2021/11 deal with the critical nonlinearity, we refer to [4, 5, 12] and the references therein. Furthermore, Servadei and Valdinoci [25] studied the Brezis-Nirenberg problem (−∆p) su = λ|u|p−2u+ |u|p ∗ s−2u in Ω, u = 0 in RN\Ω, and proved the existence of solutions for the above problem with p = 2. Mosconi et al. [23] deduced the existence of nontrivial solution for the above problem un- der different conditions of N , s, p and λ, by utilizing the asymptotic estimates for the minimizers obtained in [7] and an abstract linking theorem based on the cohomological index [29]. It is worth noting that these results were established for a bounded domain. There are a lot of novel researches and interesting results of fractional Laplacian in the whole space, see [1, 2, 6, 8, 16, 18, 20, 24, 26, 30] and the references therein. For instance, Dipierro et al. [16] considered the existence of a positive solution for the problem (−∆)su = εf(x)uq + u2∗s−1 in RN , where 0 ≤ q < 2∗s − 1, ε > 0 is a small parameter and f is a continuous and compactly supported function. Due to the lack of regularity of the associated energy functional, the case 0 < q < 1 is particularly difficult. Moreover, in [8], Bucur and Medina also investigated the above problem when 1 ≤ q < 2∗s − 1 by using different methods from [16]. Bonder et al. [6] dealt with the following critical equation involving the fractional p-Laplacian (−∆p) su = λf(x)|u|q−2u+K(x)|u|p ∗ s−2u in RN , where p ≤ q < p∗s, 0 ≤ f ∈ L1 loc(RN ) is such that the embedding Ds,p(RN ) ↪→ Lq(fdx;RN ) is compact and K is nonnegative, bounded and has a limit at ∞. By the concentration-compactness principle and mountain pass theorem, they obtained some existence results for two cases: q = p and p < q < p∗s, respectively. Recently, Ambrosio et al. [1] investigated the existence and concentration of positive solutions for the p-fractional Schrödinger equation εsp(−∆p) su+ V (x)|u|p−2u = f(x) + γ|u|p ∗ s−2u in RN , where ε is a small parameter, γ ∈ {0, 1}, V is a continuous positive potential having a local minimum and f is a superlinear continuous function with subcritical growth. The main results were deduced via penalization techniques and suitable variational arguments. However, it is worth emphasizing that, we can find that weight functions were all assumed to be positive in these above mentioned literatures. Motivated by the above analysis, we study the existence of nontrivial solution for problem (1.1), which is closely related to the principle eigenvalue of the problem (−∆p) su = λg(x)|u|p−2u in RN . (1.2) In [14], we established the existence of a sequence of eigenvalues which converges to infinity for problem (1.2), and the principle eigenvalue is simple. The corresponding eigenfunction may be positive in RN (see Lemma 2.1 below), which is a key step to prove the (PS)c condition. In this article we extend the results in [17] for critical classical Laplacian problem to the fractional setting. It is worth mentioning that, as far as we know, there is no result for fractional p-Laplacian in RN with indefinite weights. This problem will present us with two main difficulties: firstly, the lack of an explicit formula of the extremal for the EJDE-2021/11 CRITICAL FRACTIONAL p-LAPLACIAN EQUATION 3 Sobolev embedding Ds,p(RN ) ↪→ Lp ∗ s (RN ). For p = 2, we know that the extremal is of the explicit form cU( |x−x0| ε ) with U(x) = 1 (1 + |x|p′) N−sp p , x ∈ RN , where p′ = p/(p− 1) is the Hölder conjugate of p, c 6= 0, x0 ∈ RN and ε > 0, which was firstly proved by Lieb in [19] and can be devoted to proving the existence of solution. Although it has been conjectured that this extremal has a similar explicit form for the general case, to the best of our knowledge, it is still open until now. However, we can verify an existence of nontrivial solution by utilizing the asymptotic estimates for the extremal, which was obtained by Brasco et al. [7]. Secondly, the nonlinearities contain some indefinite weights, which give us further difficulty. Indeed, the indefinite weight case is more complicated than that of the positive one (see proof of Lemma 3.1), when we consider the convergence in the corresponding topology space, although we can define some spaces, their topology (or norm) are naturally related to indefinite weight functions, to give some technical assistance. Concerning the whole space, it causes the lack of the compact embedding, we will overcome the difficulty by the new concentration-compactness principle obtained in Bonder et al. [6] recently. In this article, we assume that the weight functions g and h satisfy the following conditions: let g = g+ − g− with g+, g− ≥ 0, g+ ∈ L∞(RN ) ∩ L N sp (RN ), g− ∈ L∞(RN ), and (A1) h ∈ L∞(RN ) and h+ 6≡ 0; (A2) there exist constants ρ > 0 and θ > 1 such that h(x) = h(0) + o(|x| N p ) for x ∈ B(0, ρθ); (A3) h(0) = ‖h‖∞ and h(x) > 0 for x ∈ B(0, ρθ); (A4) g(x) ≥ g0 > 0 in B(0, ρθ). Next, we state the main result of this article. Theorem 1.1. Assume that (A1)–(A4) hold, then for any λ ∈ (0, λ+ 1 ), problem (1.1) admits at least a nontrivial solution. The remainder of this paper is organized as follows. In Section 2, we collect some notation and known results, and we prove some preliminaries. In Section 3, by using the concentration-compactness principle and mountain pass theorem, we prove Theorem 1.1. 2. Preliminaries We recall that for any 0 < s < 1, the fractional Sobolev space is defined by Ds,p(RN ) = { u ∈ Lp ∗ s (RN ) : [u]s,p <∞ } , where the term [u]s,p = (∫∫ R2N |u(x)− u(y)|p |x− y|N+sp dx dy )1/p is the so-called Gagliardo seminorm of u. With the induced norm ‖u‖Ds,p(RN ) = [u]s,p, the space Ds,p(RN ) is a uniformly convex Banach space, and there exists a positive constant Cp∗s such that ‖u‖p∗s ≤ Cp∗s [u]s,p for u ∈ Ds,p(RN ) . 4 N. CUI, H.-R. SUN EJDE-2021/11 Let the best Sobolev constant in this inequality be S = inf u∈Ds,p(RN )\{0} [u]ps,p ‖u‖pp∗s . (2.1) Next, we define the fractional (s, p)-gradient of a function u ∈ Ds,p(RN ) as (see [6]) |Dsu(x)|p = ∫ RN |u(x+ t)− u(x)|p |t|N+sp dt. Observe that this (s, p)-gradient is well defined in RN and |Dsu(x)| ∈ Lp(RN ). We introduce a weight function ω(x) = 1 (1 + |x|)sp , x ∈ RN , and define w(x) = max{g−(x), ω(x)}. Let X be the completion of C∞0 (RN ) with respect to the norm ‖u‖X = ( [u]ps,p + ∫ RN w|u|p dx )1/p , then X is a uniformly convex Banach space [14, Lemma 2.1]. Obviously, note that the embedding X ↪→ Ds,p(RN ) is continuous, and since Ds,p(RN ) ↪→ Lσloc(RN ) is compact for σ ∈ [1, p∗s), we deduce that the embedding X ↪→ Lσloc(RN ) is compact for σ ∈ [1, p∗s). Weak solutions of (1.1) coincide with critical points of the C1-functional Jλ : X → R, Jλ(u) = 1 p [u]ps,p − λ p ∫ RN g|u|p dx− 1 p∗s ∫ RN h|u|p ∗ s dx, u ∈ X. (2.2) Lemma 2.1 ([14, Theorem 1.1]). Suppose that g+ ∈ L N sp (RN ) ∩ L∞(RN ), g− ∈ L∞(RN ) and g+, g− ≥ 0. Then there exists a simple eigenvalue λ+ 1 > 0 such that the eigenvalue problem (1.2) has a positive eigenfunction u+ 1 ∈ X associated with λ+ 1 . Moreover, for any λ ∈ (0, λ+ 1 ], we have∫∫ R2N |u(x)− u(y)|p |x− y|N+sp dx dy − λ ∫ RN g|u|p dx ≥ 0, u ∈ X. The following lemma provides the concentration-compactness principle for frac- tional Laplacian operator in unbounded domains. Lemma 2.2 ([6, Theorem 1.1]). Let {uk}k∈N ⊂ Ds,p(RN ) be a weakly convergent sequence with weak limit u. Then there exist two bounded measures µ and ν, an at most enumerable set of indices I, xi ∈ RN , and positive real numbers µi, νi, i ∈ I, such that the following convergence hold weakly∗ in the sense of measures, |Dsuk|p dx ⇀ µ ≥ |Dsu|p dx+ ∑ i∈I µiδxi , |uk|p ∗ s dx ⇀ ν = |u|p ∗ s dx+ ∑ i∈I νiδxi , Sν p/p∗s i ≤ µi, for all i ∈ I. Moreover, if we define µ∞ = lim R→∞ lim sup k→∞ ∫ |x|>R |Dsuk|p dx, ν∞ = lim R→∞ lim sup k→∞ ∫ |x|>R |uk|p ∗ s dx, EJDE-2021/11 CRITICAL FRACTIONAL p-LAPLACIAN EQUATION 5 then lim sup k→∞ ∫ RN |Dsuk|p dx = µ(RN ) + µ∞, lim sup k→∞ ∫ RN |uk|p ∗ s dx = ν(RN ) + ν∞, Sν p/p∗s∞ ≤ µ∞. Now, we estimate the decay of this nonlocal gradient and a compactness conclu- sion with weights. Lemma 2.3 ([6, Coroll. 2.3]). Let φ ∈W 1,∞(RN ) be such that supp(φ) ⊂ B(0, 1), given r > 0 and x0 ∈ RN , we define φr,x0 (x) = φ(x−x0 r ). Then |Dsφr,x0 (x)|p ≤ C min { r−sp, rN |x− x0|−(N+sp) } , where C > 0 depends on N, s, p and ‖φ‖W 1,∞(RN ). Lemma 2.4 ([6, Lemma 2.4]). If a ∈ L∞(RN ), there exist α > 0 and C > 0 such that 0 ≤ a(x) ≤ C|x|−α. Then, if α > sp, the embedding Ds,p(RN ) ↪→ Lp(adx;RN ) is compact. That is, for any bounded sequence {uk}k∈N ⊂ Ds,p(RN ), there exists a subsequence {ukj}j∈N ⊂ {uk}k∈N and a function u ∈ Ds,p(RN ) such that∫ RN a|ukj − u|p dx→ 0 as j →∞. To verify the existence of solution for (1.1), we first need to show some prelimi- nary results. Lemma 2.5. Assume that un ⇀ u in Ds,p(RN ). Then for f ∈ L∞(RN ), there exists a subsequence of {un}, still denoted by {un}, such that for any ϕ ∈ C∞0 (RN ), (i) ∫ RN f |un| p∗s−2unϕdx→ ∫ RN f |u| p∗s−2uϕdx; (ii) ∫ RN f |un| p−2unϕdx→ ∫ RN f |u| p−2uϕdx. Proof. (i) Since un ⇀ u in Ds,p(RN ) and the embedding Ds,p(RN ) ↪→ Lσloc(RN ) is compact for σ ∈ [1, p∗s), we can assume that un → u a.e. in RN , so f |un|p ∗ s−2un → f |u|p∗s−2u a.e. in RN . Since {un} is bounded in Lp ∗ s (RN ), it follows from f ∈ L∞(RN ) that {f |un|p ∗ s−2un} is bounded in L p∗s p∗s−1 (RN ). Thus, we can get that f |un|p ∗ s−2un ⇀ f |u|p∗s−2u in L p∗s p∗s−1 (RN ) (see [27]). For any ϕ ∈ C∞0 (RN ), we obtain∫ RN f |un|p ∗ s−2unϕdx→ ∫ RN f |u|p ∗ s−2uϕdx. (ii) It can be proved by a similar method as (i), so we omit it here. � Lemma 2.6. Suppose that un ⇀ u in Ds,p(RN ), then there exists a subsequence of {un}, denoted still by {un}, such that |un(x)− un(y)|p−2(un(x)− un(y)) |x− y| N+sp p′ ⇀ |u(x)− u(y)|p−2(u(x)− u(y)) |x− y| N+sp p′ 6 N. CUI, H.-R. SUN EJDE-2021/11 in Lp ′ (RN × RN ), where p′ = p p−1 is the Hölder conjugate of p. Proof. For simplicity, we define ξn(x, y) = |un(x)− un(y)|p−2(un(x)− un(y)) |x− y| N+sp p′ . Since {un} is bounded in Ds,p(RN ), we obtain that {ξn} is bounded in Lp ′ (RN × RN ), hence there exists ξ ∈ Lp′(RN × RN ) such that ξn ⇀ ξ in Lp ′ (RN × RN ) up to a subsequence. In addition, since un ⇀ u inDs,p(RN ) and the embeddingDs,p(RN ) ↪→ Lσloc(RN ) is compact for σ ∈ [1, p∗s), we can assume that un → u a.e. in RN . Then it follows that ξn(x, y)→ |u(x)−u(y)|p−2(u(x)−u(y)) |x−y| N+sp p′ a.e. in RN × RN . Combining this with the boundedness of {ξn}, we obtain that ξn(x, y) ⇀ |u(x)− u(y)|p−2(u(x)− u(y)) |x− y| N+sp p′ in Lp ′ (RN × RN ). Thus ξ(x, y) = |u(x)− u(y)|p−2(u(x)− u(y)) |x− y| N+sp p′ . The proof is complete. � Lemma 2.7. Let {un} ⊂ Ds,p(RN ) be a bounded sequence such that Jλ(un) → c and J ′λ(un) → 0 as n → ∞, and let µi, νi be as in Lemma 2.2. Then we have the following estimates: νi ≥ S N sph(xi) − N sp , µi ≥ S N sph(xi) 1− N sp if h(xi) > 0, νi = µi = 0 if h(xi) = 0. Proof. The proof closely follows the technique of [6, Lemma 3.6]. We first show that the set I in Lemma 2.2 is finite. Fix a concentration point xi, let φ ∈ C∞0 (RN , [0, 1]) be such that φ(x) = { 1, if |x| ≤ 1, 0, if |x| ≥ 2, and let φδ(x) = φ(x−xiδ ) for δ > 0. According to J ′λ(un)→ 0 as n→∞, taking the test function ϕ = unφδ, then we have lim n→∞ ( 〈(−∆p) sun, unφδ〉 − λ ∫ RN |un|pφδ dx− ∫ RN h|un|p ∗ sφδ dx ) = lim n→∞ 〈J ′λ(un), unφδ〉 = 0. Thus, it follows from Lemma 2.2, g, h ∈ L∞(RN ) and un → u in Lploc(RN ) that lim n→∞ 〈(−∆p) sun, unφδ〉 = λ ∫ RN g|u|pφδ dx+ ∫ RN hφδdν. (2.3) Next, we verify that lim δ→0 lim n→∞ 〈(−∆p) sun, unφδ〉 = µi. (2.4) We can write 〈(−∆p) sun, unφδ〉 EJDE-2021/11 CRITICAL FRACTIONAL p-LAPLACIAN EQUATION 7 = ∫∫ R2N |un(x)− un(y)|p−2(un(x)− un(y))(un(x)φδ(x)− un(y)φδ(y)) |x− y|N+sp dx dy = ∫∫ R2N |un(x)− un(y)|pφδ(x) |x− y|N+sp dx dy + ∫∫ R2N |un(x)− un(y)|p−2(un(x)− un(y))(φδ(x)− φδ(y))un(y) |x− y|N+sp dx dy = I1 + I2. Clearly, I1 = ∫ RN |D sun|pφδ dx, then by using un ⇀ u in Ds,p(RN ) and Lemma 2.2, we have lim n→∞ I1 = ∫ RN φδdµ. Since µ− ∑ i∈I µiδxi has no atoms and φδ → 0 as δ → 0 for any x 6= xi, i ∈ I, we conclude that limδ→0 limn→∞ I1 = µi. Now,we estimate I2 using Hölder inequality we have I2 ≤ ∫ RN (|Dsun|p) p−1 p (|Dsφδ|p)1/p|un|dy ≤ ‖Dsun‖p−1 p (∫ RN |un|p|Dsφδ|p dx )1/p ≤ c (∫ RN |un|p|Dsφδ|p dx )1/p . Then by Lemmas 2.3 and 2.4, we know that lim sup n→∞ I2 ≤ c (∫ RN |u|p|Dsφδ|p dx )1/p . (2.5) Furthermore, we check that limδ→0 ∫ RN |u| p|Dsφδ|p dx = 0. In fact, thanks to Lemma 2.3 and Hölder inequality, we deduce that∫ RN |u|p|Dsφδ|p dx ≤ C ( δ−sp ∫ |x|<δ |u|p dx+ δN ∫ |x|≥δ |u|p |x|N+sp dx ) ≤ Cδ−sp (∫ |x|<δ |u|p ∗ s dx )p/p∗s |Bδ| sp N + CδN ∞∑ k=0 ∫ 2kδ≤|x|≤2k+1δ |u|p |x|N+sp dx ≤ c1 (∫ |x|<δ |u|p ∗ s dx )p/p∗s + C ∞∑ k=0 2−k(N+sp)δ−sp (∫ |x|≤2k+1δ |u|p ∗ s dx )p/p∗s |B2k+1δ| sp N = c1 (∫ |x|<δ |u|p ∗ s dx )p/p∗s + c2 ∞∑ k=0 2−kN (∫ |x|≤2k+1δ |u|p ∗ s dx )p/p∗s . 8 N. CUI, H.-R. SUN EJDE-2021/11 Since u ∈ Lp ∗ s (RN ), then it is easy to see that limδ→0 ∫ |x|<δ |u| p∗s dx = 0. Given ε > 0, taking k0 ∈ N such that c2 ∑∞ k=k0+1 2−kN < ε, we obtain c2 ∞∑ k=0 2−kN (∫ |x|≤2k+1δ |u|p ∗ s dx )p/p∗s ≤ ε‖u‖pp∗s + c2 k0∑ k=0 2−kN (∫ |x|≤2k0+1δ |u|p ∗ s dx )p/p∗s . Then, lim sup δ→0 c2 ∞∑ k=0 2−kN (∫ |x|≤2k+1δ |u|p ∗ s dx )p/p∗s ≤ ε‖u‖pp∗s . Hence, lim δ→0 ∫ RN |u|p|Dsφδ|p dx = 0, and from (2.5) it follows that limδ→0 limn→∞ I2 = 0. So, the limit equality (2.4) holds. On the other hand, by applying the property of the function φδ, we know that lim δ→0 ∫ RN g|u|pφδ dx = 0 and lim δ→0 ∫ RN hφδdν = h(xi)νi. Then, in view of (2.3), we have h(xi)νi = µi for any i ∈ I, which implies that h(xi) ≥ 0. So, it follows from Lemma 2.2 that νi ≥ S N sph(xi) − N sp , µi ≥ S N sph(xi) 1− N sp if h(xi) > 0, νi = µi = 0 if h(xi) = 0. This completes the proof. � 3. Main results In this section, we prove the existence of solutions for critical fractional p- Laplacian with indefinite weights of problem (1.1). By using the concentration- compactness principle and mountain pass theorem, we prove Theorem 1.1. Firstly, we prove that the functional Jλ satisfies the (PS)c condition for small energy levels. Lemma 3.1. For any λ ∈ (0, λ+ 1 ), the functional Jλ satisfies the (PS)c condition for all c < s N S N sp ‖h‖1− N sp ∞ . Proof. Let {un} be a (PS)c sequence; that is, Jλ(un)→ c and J ′λ(un)→ 0 as n→∞. (3.1) First of all, we show that the sequence {un} is bounded in X. If this is not true, we may suppose that, up to a subsequence, still denoted by {un} such that ‖un‖X →∞ as n→∞. For n ∈ N, let vn = un ‖un‖X , then we can assume that there exists v ∈ X such that vn ⇀ v in X, vn → v in Lploc(RN ), vn → v a.e. in RN . (3.2) Obviously, by (3.1), we have c+ o(1)‖un‖X = Jλ(un)− 1 p∗s 〈J ′λ(un), un〉 = s N ( [un]ps,p − λ ∫ RN g|un|p dx ) . EJDE-2021/11 CRITICAL FRACTIONAL p-LAPLACIAN EQUATION 9 Multiplying both sides by ‖un‖−pX and letting n→∞, we deduce that [vn]ps,p − λ ∫ RN g|vn|p dx→ 0. (3.3) Since the embedding X ↪→ Ds,p(RN ) ↪→ Lp ∗ s (RN ) is continuous and ‖vn‖X = 1, we know that {vn} is bounded in Lp ∗ s (RN ), then {|vn|p} is bounded in L p∗s p (RN ). Combining this with (3.2), we have |vn|p ⇀ |v|p in L p∗s p (RN ). So, in view of g+ ∈ L N sp (RN ), we obtain∫ RN g+|vn|p dx→ ∫ RN g+|v|p dx. (3.4) For λ ∈ (0, λ+ 1 ), by (3.2)-(3.4), we have 0 ≤ [v]ps,p − λ ∫ RN g|v|p dx = [v]ps,p − λ ∫ RN (g+ − g−)|v|p dx ≤ lim inf n→∞ ( [vn]ps,p − λ ∫ RN g+|vn|p dx+ λ ∫ RN g−|vn|p dx ) = 0. (3.5) Thus, the variational characterization of the principle eigenvalue λ+ 1 (see Lemma 2.1) implies that v ≡ 0. Then from (3.4) and (3.5) it follows that∫ RN g+|vn|p dx→ 0, lim n→∞ [vn]ps,p = 0, lim n→∞ ∫ RN g−|vn|p dx = 0. By the Hardy-type inequality [21, Theorem 2], we obtain 0 ≤ ∫ RN ω|vn|p dx ≤ CN,s,p[vn]ps,p, we then deduce that limn→∞ ‖vn‖X = 0, which contradicts that ‖vn‖X = 1. There- fore, {un} is bounded in X. Next, we verify that un → u in X. Since {un} is bounded in X, we can assume that there exists u ∈ X such that un ⇀ u in X. Then applying Lemmas 2.5 and 2.6 and g, h ∈ L∞(RN ), for any ϕ ∈ C∞0 (RN ), we have∫∫ R2N |un(x)− un(y)|p−2(un(x)− un(y))(ϕ(x)− ϕ(y)) |x− y|N+sp dx dy → ∫∫ R2N |u(x)− u(y)|p−2(u(x)− u(y))(ϕ(x)− ϕ(y)) |x− y|N+sp dx dy,∫ RN g|un|p−2unϕdx→ ∫ RN g|u|p−2uϕdx,∫ RN h|un|p ∗ s−2unϕdx→ ∫ RN h|u|p ∗ s−2uϕdx. Hence, in view of 〈J ′λ(un), ϕ〉 → 0, we obtain∫∫ R2N |u(x)− u(y)|p−2(u(x)− u(y))(ϕ(x)− ϕ(y)) |x− y|N+sp dx dy − λ ∫ RN g|u|p−2uϕdx− ∫ RN h|u|p ∗ s−2uϕdx = 0, 10 N. CUI, H.-R. SUN EJDE-2021/11 that is, 〈J ′λ(u), ϕ〉 = 0. From the proof of Lemma 2.7, we know that for any i ∈ I, h(xi)νi = µi, (3.6) νi = 0 if h(xi) = 0, νi ≥ S N sph(xi) − N sp if h(xi) > 0. (3.7) Suppose that νi 6= 0 for some i ∈ I. Becasue un ⇀ u in X, we may obtain that un → u a.e. in RN . On the other hand, we can get that {|un|p} is bounded in L p∗s p (RN ), since the embedding X ↪→ Lp ∗ s (RN ) is continuous. Then, it follows from g+ ∈ L N sp (RN ) that ∫ RN g+|un|p dx→ ∫ RN g+|u|p dx. (3.8) Thus, by (3.1), Lemma 2.2 and (3.8), we deduce that c+ o(1)‖un‖X = Jλ(un)− 1 p∗s 〈J ′λ(un), un〉 = s N ( [un]ps,p − λ ∫ RN g|un|p dx ) = s N ( [un]ps,p − λ ∫ RN g+|un|p dx+ λ ∫ RN g−|un|p dx ) ≥ s N ( [u]ps,p − λ ∫ RN g|u|p dx+ ∑ i∈I µi ) + o(1). Furthermore, combining J ′λ(u) = 0, (3.6) with (3.7), we obtain c+ o(1)‖un‖X ≥ s N ∫ RN h|u|p ∗ s dx+ s N ∑ i∈I h(xi)νi + o(1) ≥ s N ∫ RN h|u|p ∗ s dx+ s N S N sp ∑ i∈I h(xi) 1− N sp + o(1). In view of c < s N S N sp ‖h‖1− N sp ∞ , we obtain that ∫ RN h|u| p∗s dx < 0. However, by the fact J ′λ(u) = 0 and λ ∈ (0, λ+ 1 ), we have ∫ RN h|u|p ∗ s dx = [u]ps,p − λ ∫ RN g|u|p dx ≥ 0, which is a contraction. Thus νi = µi = 0 for any i ∈ I, then it follows from Lemma 2.2 that ∫ RN |un|p ∗ s dx→ ∫ RN |u|p ∗ s dx. EJDE-2021/11 CRITICAL FRACTIONAL p-LAPLACIAN EQUATION 11 This together with the weak convergence of un ⇀ u in Lp ∗ s (RN ) imply that un → u in Lp ∗ s (RN ). By un ⇀ u in X and (3.1), we derive that 〈J ′λ(un)− J ′λ(u), un − u〉 = ∫∫ R2N ( |un(x)− un(y)|p−2(un(x)− un(y)) |x− y|N+sp − |u(x)− u(y)|p−2(u(x)− u(y)) |x− y|N+sp ) (un(x)− un(y)− u(x) + u(y)) dx dy − λ ∫ RN g(|un|p−2un − |u|p−2u)(un − u)dx − ∫ RN h(|un|p ∗ s−2un − |u|p ∗ s−2u)(un − u)dx→ 0. (3.9) Since un ⇀ u in X, un → u in Lp ∗ s (RN ), g+ ∈ L N sp (RN ) and h ∈ L∞(RN ), we have the convergence ∫ RN g+(|un|p−2un − |u|p−2u)(un − u)dx→ 0,∫ RN h(|un|p ∗ s−2un − |u|p ∗ s−2u)(un − u)dx→ 0. (3.10) Then, combining (3.9) with (3.10), we obtain∫∫ R2N ( |un(x)− un(y)|p−2(un(x)− un(y)) |x− y|N+sp − |u(x)− u(y)|p−2(u(x)− u(y)) |x− y|N+sp ) × (un(x)− un(y)− u(x) + u(y)) dx dy + λ ∫ RN g−(|un|p−2un − |u|p−2u)(un − u)dx→ 0. (3.11) Using Hölder inequality, we deduce that∫∫ R2N ( |un(x)− un(y)|p−2(un(x)− un(y)) |x− y|N+sp − |u(x)− u(y)|p−2(u(x)− u(y)) |x− y|N+sp ) × (un(x)− un(y)− u(x) + u(y)) dx dy ≥ [un]ps,p + [u]ps,p − [un]p−1 s,p [u]s,p − [u]p−1 s,p [un]s,p = ( [un]p−1 s,p − [u]p−1 s,p ) ([un]s,p − [u]s,p) and similarly, we have∫ RN g−(|un|p−2un − |u|p−2u)(un − u)dx ≥ ((∫ RN g−|un|p dx ) p−1 p − (∫ RN g−|u|p dx ) p−1 p ) × ((∫ RN g−|un|p dx )1/p − (∫ RN g−|u|p dx )1/p) . Let fb(t) = (tb−1 − βb−1)(t− β) for t ∈ R+, b > 1, we know that fb(t) ≥ 0. Hence, it follows from (3.11) that( [un]p−1 s,p − [u]p−1 s,p ) ([un]s,p − [u]s,p)→ 0, 12 N. CUI, H.-R. SUN EJDE-2021/11((∫ RN g−|un|p dx ) p−1 p − (∫ RN g−|u|p dx ) p−1 p ) × ((∫ RN g−|un|p dx )1/p − (∫ RN g−|u|p dx )1/p) → 0. The function fb also has the following property: if fb(tn)→ 0, then tn → β. Thus, [un]ps,p → [u]ps,p and ∫ RN g−|un|p dx→ ∫ RN g−|u|p dx as n→∞. By the Hardy-type inequality, we derive that ∫ RN ω|un| p dx→ ∫ RN ω|u| p dx as n→ ∞. So, ‖un‖X → ‖u‖X , this together with the weak convergence of un ⇀ u in X implies that un → u in X. � To verify that Jλ has the geometric structure required by the mountain pass theorem, we need to introduce the following results. Lemma 3.2 ([7, Proposition 3.1]). Let 1 < p < ∞, s ∈ (0, 1), N > sp and let S be as in (2.1). Then (i) there exists a minimizer for S; (ii) for every minimizer U ∈ Ds,p(RN ), there exist x0 ∈ RN and a constant sign monotone function u : R→ R such that U(x) = u(|x− x0|); (iii) for every minimizer U , there exists λU > 0 such that∫∫ R2N |U(x)− U(y)|p−2(U(x)− U(y))(ϕ(x)− ϕ(y)) |x− y|N+sp dx dy = λU ∫ RN |U |p ∗ s−2Uϕdx, for ϕ ∈ Ds,p(RN ). In the following, we fix a radially symmetric nonnegative decreasing minimizer U = U(r) for S. Multiplying U by a positive constant if necessary, we may assume that U is a radial solution of (−∆p) sU = |U |p ∗ s−2U. Taking the test function U and applying (2.1) yield [U ]ps,p = ‖U‖p ∗ s p∗s = S N sp . (3.12) For ε > 0, the function Uε(|x|) = ε− N−sp p U ( |x| ε ) is also a minimizer for S satisfying (3.12). Lemma 3.3 ([7, Corollary 3.7]). There exist constants C1, C2 > 0 and θ > 1 such that for all r ≥ 1, C1 r N−sp p−1 ≤ U(r) ≤ C2 r N−sp p−1 and U(θr) U(r) ≤ 1 2 . Now, we give some auxiliary functions and estimate their norms. In what follows, θ is the constant in Lemma 3.3 that depends only on N , s and p. For ε, ρ > 0 and θ > 1, let us set mε,ρ = Uε(ρ) Uε(ρ)− Uε(ρθ) . EJDE-2021/11 CRITICAL FRACTIONAL p-LAPLACIAN EQUATION 13 Moreover, let us define gε,ρ(t) =  0, if 0 ≤ t ≤ Uε(ρθ), mp ε,ρ(t− Uε(ρθ)), if Uε(ρθ) < t ≤ Uε(ρ), t+ Uε(ρ)(mp−1 ε,ρ − 1), if t > Uε(ρ), and Gε,ρ(t) = ∫ t 0 g′ε,ρ(τ)1/pdτ =  0, if 0 ≤ t ≤ Uε(ρθ), mε,ρ(t− Uε(ρθ)), if Uε(ρθ) < t ≤ Uε(ρ), t, if t > Uε(ρ). The functions gε,ρ and Gε,ρ are nondecreasing and absolutely continuous. Consider the radially symmetric non-increasing function uε,ρ(r) = Gε,ρ(Uε(r)), which satisfies uε,ρ(r) = { Uε(r), if r ≤ ρ, 0, if r ≥ ρθ. Then, we have the following estimates for uε,ρ. Lemma 3.4 ([23, Lemma 2.7]). There exists a constant C = C(N, p, s) > 0 such that, for any 0 < ε ≤ ρ 2 , it holds [uε,ρ] p s,p ≤ S N sp + C ( ε ρ )N−sp p−1 , (3.13) ‖uε,ρ‖pp ≥ { 1 C ε sp log(ρε ), if N = sp2, 1 C ε sp, if N > sp2, (3.14) ‖uε,ρ‖ p∗s p∗s ≥ S N sp − C ( ε ρ ) N p−1 . (3.15) Now we define the function vε,ρ(x) = uε,ρ(x) ‖uε,ρ(x)‖p∗s . Lemma 3.5. There exist ε, ρ, t0 > 0 such that for λ ∈ (0, λ+ 1 ), Jλ(t0vε,ρ) < 0 and sup t≥0 Jλ(tvε,ρ) < s N S N sp ‖h‖1− N sp ∞ . Proof. For any λ ∈ (0, λ+ 1 ), by (2.1), we have Jλ(u) ≥ 1 p ( 1− λ λ1 ) [u]ps,p − 1 p∗s ‖h‖∞S− p∗s p [u] p∗s s,p, so we can obtain that Jλ(u) ≥ c, when ‖u‖X is sufficiently small. Furthermore, it is easy from (2.2) to see that limt→∞ Jλ(tvε,ρ) = −∞, hence Jλ(tvε,ρ) attains its maximum at some tε ∈ (0,∞) with ψ′λ(tε) = 0, where ψλ(t) = Jλ(tvε,ρ) = tp p [vε,ρ] p s,p − tp p λ ∫ RN g|vε,ρ|p dx− tp ∗ s p∗s ∫ RN h|vε,ρ|p ∗ s dx. 14 N. CUI, H.-R. SUN EJDE-2021/11 Then, we obtain 0 = ψ′λ(tε) = tp−1 ε ( [vε,ρ] p s,p − λ ∫ RN g|vε,ρ|p dx ) − tp ∗ s−1 ε ∫ RN h|vε,ρ|p ∗ s dx, moreover, combining (A2), (A3) and (A4), we deduce that t p∗s−p ε = [vε,ρ] p s,p − λ ∫ RN g|vε,ρ| p dx∫ RN h|vε,ρ|p ∗ s dx ≤ [vε,ρ] p s,p h(0)‖vε,ρ‖ p∗s p∗s . (3.16) Clearly, we have Jλ(tεvε,ρ) = sup t≥0 Jλ(tvε,ρ) = I1 − I2, (3.17) where I1 = tpε p [vε,ρ] p s,p − t p∗s ε p∗s h(0) ∫ RN |vε,ρ|p ∗ s dx, I2 = tpε p λ ∫ RN g|vε,ρ|p dx− t p∗s ε p∗s ∫ RN (h(0)− h)|vε,ρ|p ∗ s dx. In view of (3.13) and (3.15), we obtain [vε,ρ] p s,p = [uε,ρ] p s,p ‖uε,ρ(x)‖pp∗s ≤ S +O ( ε ρ )N−sp p−1 . For positive numbers a and b, the maximum of ~(t) = a t p p − b t p∗s p∗s for t ≥ 0 is attained at t = (ab ) N−sp sp2 , then, by the assumption (A3) and above inequality, we can deduce that I1 ≤ 1 p ( [vε,ρ] p s,p h(0) ∫ RN |vε,ρ|p ∗ s dx )N−sp sp [vε,ρ] p s,p − h(0) p∗s ( [vε,ρ] p s,p h(0) ∫ RN |vε,ρ|p ∗ s dx ) N sp ∫ RN |vε,ρ|p ∗ s dx = s N ‖h‖1− N sp ∞ ( [vε,ρ] p s,p ) N sp (∫ RN |vε,ρ|p ∗ s dx )1− N sp ≤ s N S N sp ‖h‖1− N sp ∞ . Without loss of generality, we can assume that {tε} is bounded. In fact, since {tε} is bounded from blew, otherwise one concludes easily from (3.17) that Jλ(tεvε,ρ)→ 0 as ε → 0. In addition, according to (3.16), we know that {tε} is bounded from above for ε > 0 small. Next, we estimate the Lκ-norm of uε,ρ, for κ ∈ [1,∞), Lemma 3.3 yields∫ RN |uε,ρ(x)|κ dx ≥ ∫ Bρ(0) |uε,ρ(x)|κ dx = ∫ Bρ(0) |Uε(x)|κ dx = ε− (N−sp)κ p ∫ Bρ(0) ∣∣U(x ε )∣∣κ dx ≥ Cκ1 ε Np−(N−sp)κ p ∫ ρ/ε 1 r− N−sp p−1 κ+N−1dr . EJDE-2021/11 CRITICAL FRACTIONAL p-LAPLACIAN EQUATION 15 Then ∫ RN |uε,ρ(x)|κ dx ≥ cκ  εN− N−sp p κ, if κ > N(p−1) N−sp , εN− N−sp p κ| log ρ ε |, if κ = N(p−1) N−sp , ε N−sp p(p−1) κρN− N−sp p−1 κ, if κ < N(p−1) N−sp . Thus, by assumption (A4), we have ∫ RN g|vε,ρ|p dx ≥ cp  g0ε sp, if N > sp2, g0ε sp| log ρ ε |, if N = sp2, g0ε N−sp p−1 ρN− N−sp p−1 p, if N < sp2. Similarly, in view of assumption (A2), we obtain∫ RN (h(0)− h)|vε,ρ|p ∗ s dx = 1 ‖uε,ρ‖ p∗s p∗s ∫ RN (h(0)− h)|uε,ρ|p ∗ s dx ≥ ε N p ∫ ρ/ε 1 r− N p(p−1) −1dr. Since N p ≥ sp if N ≥ sp2 and N p > N−sp p−1 and N < sp2, then I2 can be dominated by ∫ RN g|vε,ρ| p dx. Thus, we conclude that Jλ(tεvε,ρ) ≤  s N S N sp ‖h‖1− N sp ∞ −K1ε sp, if N > sp2, s N S N sp ‖h‖1− N sp ∞ −K1ε sp| log ρ ε |, if N = sp2, s N S N sp ‖h‖1− N sp ∞ −K1ε N−sp p−1 ρN− N−sp p−1 p, if N < sp2, then Jλ(tεvε,ρ) < s N S N sp ‖h‖1− N sp ∞ if ε > 0 is sufficiently small. This completes the proof. � Proof of Theorem 1.1. For any λ ∈ (0, λ+ 1 ), by (2.1), we have Jλ(u) ≥ 1 p ( 1− λ λ1 ) [u]ps,p − 1 p∗s ‖h‖∞S− p∗s p [u] p∗s s,p, then it follows that Jλ(u) ≥ c > 0, when ‖u‖X is sufficiently small. And since Jλ(0) = 0, 0 is a local minimum of Jλ. In addition, noting that Jλ(tvε,ρ) = tp p ( [vε,ρ] p s,p − λ ∫ RN g|vε,ρ|p dx ) − tp ∗ s p∗s ∫ RN h|vε,ρ|p ∗ s dx→ −∞ as t→ +∞, fix t1 > 0 so large that Jλ(t1vε,ρ) < 0. Now, we construct the set Γ = {γ ∈ C([0, 1], X) : γ(0) = 0, γ(1) = t1vε,ρ}, and let c = inf γ∈Γ sup t∈[0,1] Jλ(γ(t)). In view of Lemma 3.5, it is easy to see that c < s N S N sp ‖h‖1− N sp ∞ , 16 N. CUI, H.-R. SUN EJDE-2021/11 and hence Jλ satisfies the (PS)c condition by Lemma 3.1. Then, c is a critical level of Jλ via the mountain pass theorem [28], and we obtain that problem (1.1) has a nontrivial solution. � Acknowledgement. This work was supported by the NSFC (Grant No 11671181). References [1] V. Ambrosio1, G. M. Figueiredo, T. Isernia; Existence and concentration of positive solutions for p-fractional Schrödinger equations, Ann. Mat. Pura. Appl., 199 (2020), 317–344. [2] S. Antontsev, S. Shmarev, J. Simsen, M. S. Simsen; Differential inclusion for the evolution p(x)-Laplacian with memory, Electron. J. Differential Equations, 2019 (2019), no. 26, 1–28. [3] D. Applebaum; Lévy Processes and Stochastic Calculus, second ed., Camb. Stud. Adv. Math., vol. 116, Cambridge University Press, Cambridge, 2009. [4] B. Barrios, E. Colorado, A. de Pablo, U. Sánchez; On some critical problems for the fractional Laplacian operator, J. Differential Equations, 252 (2012), 6133–6162. [5] B. Barrios, E. Colorado, R. Servadei, F. Soria; A critical fractional equation with concave- convex power nonlinearities, Ann. Inst. H. Poincaré Anal. Non Linéaire, 32 (2015), 875–900. [6] J. F. Bonder, N. Saintier, A. Silva; The concentration-compactness principle for fractional order Sobolev spaces in unbounded domains and applications to the generalized fractional Brezis-Nirenberg problem, Nonlinear Differ. Equ. Appl., 25 (2018), 25 pp. [7] L. Brasco, S. Mosconi, M. Squassina; Optimal decay of extremals for the fractional Sobolev inequality, Calc. Var. Partial Differential Equations, 55 (2016), 32 pp. [8] C. Bucur, M. Medina; A fractional elliptic problem in Rn with critical growth and convex nonlinearities, Manuscr. Math., 158 (2019), 371–400. [9] L. Caffarelli, A. Mellet, Y. Sire; Traveling waves for a boundary reaction-diffusion equation, Adv. Math., 230 (2012), 433-457. [10] L. Caffarelli, L. Silvestre; An extension problem related to the fractional Laplacian, Comm. Partial Differential Equations, 32 (2007), 1245–1260. [11] L. Caffarelli, A. Vasseur; Drift diffusion equations with fractional diffusion and the quasi- geostrophic equation, Ann. Math., 171 (2010), 1903–1930. [12] W. Choi, S. Kim, K. A. Lee; Asymptotic behavior of solutions for nonlinear elliptic problems with the fractional Laplacian, J. Funct. Anal., 266 (2014), 6531–6598. [13] R. Cont, P. Tankov; Financial Modelling with Jump Processes, Chapman & Hall/CRC Fi- nanc. Math. Ser., Chapman & Hall/CRC, Boca Raton, FL, 2004. [14] N. Cui, H. R. Sun; Fractional p-Laplacian eigenvalue problem with indefinite weight in RN , Math. Methods Appl. Sci., 44 (2021), 2585–2599. [15] E. Di Nezza, G. Palatucci, E. Valdinoci; Hitchhiker’s guide to the fractional Sobolev spaces, Bull. Sci. Math., 136 (2012), 521–573. [16] S. Dipierro, M. Medina, I. Peral, E. Valdinoci; Bifurcation results for a fractional elliptic equation with critical exponent in Rn, Manuscr. Math., 153 (2017), 183–230. [17] P. Drábek, Y. X. Huang; Multiplicity of positive solutions for some quasilinear elliptic equa- tion in RN with critical Sobolev exponent, J. Differential Equations, 140 (1997), 106–132. [18] A. Fiscella, P. Pucci; p-fractional Kirchhoff equations involving critical nonlinearities, Non- linear Anal. Real World Appl., 35 (2017), 350–378. [19] E. H. Lieb; Sharp constants in the Hardy-Littlewood-Sobolev and related inequalities, Ann. Math., 118 (1983), 349–374. [20] C. C. Liu, P. P. Li; Boundedness and global solvability for a chemotaxis-haptotaxis model with p-Laplacian diffusion, Electron. J. Differential Equations, 2020 (2020), no. 16, 1–16. [21] V. Maz’ya, T. Shaposhnikova; On the Bourgain, Brezis, and Mironescu theorem concerning limiting embeddings of fractional Sobolev spaces, J. Funct. Anal., 195 (2002), 230-238. [22] A. Mellet, S. Mischler, C. Mouhot; Fractional diffusion limit for collisional kinetic equations, Arch. Ration. Mech. Anal., 199 (2011), 493–525. [23] S. Mosconi, K. Perera, M. Squassina, Y. Yang; The Brezis-Nirenberg problem for the frac- tional p-Laplacian, Calc. Var. Partial Differential Equations, 55 (2016), 25 pp. [24] K. Perera, M. Squassina, Y. Yang; Critical fractional p-Laplacian problems with possibly vanishing potentials, J. Math. Anal. Appl., 433 (2016), 818–831. EJDE-2021/11 CRITICAL FRACTIONAL p-LAPLACIAN EQUATION 17 [25] R. Servadei, E. Valdinoci; The Brezis-Nirenberg result for the fractional Laplacian, Trans. Am. Math. Soc., 367 (2015), 67–102. [26] X. Shang, J. Zhang; Ground states for fractional Schrödinger equations with critical growth, Nonlinearity, 27 (2014), 187–207. [27] M. Willem; Analyse Harmonique Réelle, Hermann, Paris, 1995. [28] M. Willem; Minimax Theorems, Birkhäuser, Boston, 1996. [29] Y. Yang, K. Perera; N-Laplacian problems with critical Trudinger-Moser nonlinearities, Ann. Sci. Norm. Super. Pisa Cl. Sci., 16 (2016), 1123–1138. [30] H. S. Zhan, Z. Feng, Solutions of evolutionary p(x)-Laplacian equation based on the weighted variable exponent space, Z. Angew. Math. Phys., 68 (2017), no. 6, Paper No. 134, 17 pp. Na Cui School of Mathematics and Statistics, Lanzhou University, Lanzhou, Gansu 730000, China Email address: cuin17@lzu.edu.cn Hong-Rui Sun (corresponding author) School of Mathematics and Statistics, Lanzhou University, Lanzhou, Gansu 730000, China Email address: hrsun@lzu.edu.cn 1. Introduction 2. Preliminaries 3. Main results Acknowledgement References