Electronic Journal of Differential Equations, Vol. 2021 (2021), No. 100, pp. 1–24. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS INVOLVING THE FRACTIONAL (p, q)-LAPLACIAN AND CRITICAL NONLINEARITIES HUILIN LV, SHENZHOU ZHENG, ZHAOSHENG FENG Abstract. In this article, we consider the existence of ground state positive solutions for nonlinear Schrödinger equations of the fractional (p, q)-Laplacian with Rabinowitz potentials defined in Rn, (−∆)s1p u+ (−∆)s2q u+ V (εx)(|u|p−2u+ |u|q−2u) = λf(u) + σ|u|q ∗ s2 −2 u. We prove existence by confining different ranges of the parameter λ under the subcritical or critical nonlinearities caused by σ = 0 or 1, respectively. In particular, a delicate calculation for the critical growth is provided so as to avoid the failure of a global Palais-Smale condition for the energy functional. 1. Introduction Let 0 < s1 < s2 < 1, 1 < p < q < n/s2, σ ∈ {0, 1}, ε > 0 be small number and q∗s2 = nq n−s2q be the fractional critical exponent for s2 and q. We are to consider the existence results for the following Schrödinger equations of a fractional (p, q)- Laplacian in Rn: (−∆)s1p u+ (−∆)s2q u+ V (εx)(|u|p−2u+ |u|q−2u) = λf(u) + σ|u|q ∗ s2 −2u, (1.1) where λ > 0 is a parameter specified later for the parameter σ = 0 or 1, V : Rn → R is a continuous function satisfying global Rabinowitz condition, and f : R→ R is a continuous function with subcritical growth. Here, the fractional t-Laplace operator (−∆)st , for s ∈ {s1, s2} and t ∈ {p, q}, is defined as (−∆)stu(x) = 2 lim ε→0 ∫ Rn\Bε(x) |u(x)− u(y)|t−2(u(x)− u(y)) |x− y|n+st dy. Problems of type (1.1) are well known as double-phase equations, appearing in the case of two different materials, where the fractional operator (−∆)st with s ∈ {s1, s2} and t ∈ {p, q} described the geometry of a composite of two materials. Recently, a considerable attention has been devoted to the work on nonlocal prob- lems driven by fractional operators, particularly on fractional p-Laplacian due to both its interesting theoretical structure and concrete applications such as finance, 2010 Mathematics Subject Classification. 35R11, 35A15, 58E05. Key words and phrases. Nonlinear Schrödinger equations; nonlocal (p, q)-Laplacian; critical growth; Rabinowitz potentials; Nehari manifold. ©2021. This work is licensed under a CC BY 4.0 license. Submitted May 18, 2021. Published December 20, 2021. 1 2 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 obstacle problems, phase transitions, optimization, anomalous diffusion, conserva- tion laws, image processing and many others. For more details see [23, 22, 12, 31] and the references therein. In the special case s1 = s2 = 1 and V ≡ 1, Problem (1.1) reduces to the following well-known (p, q)-Laplacian equation in Rn, −∆pu−∆qu+ |u|p−2u+ |u|q−2u = f(x, u), (1.2) where ∆pu = div(|∇u|p−2∇u). As explained in [16], the study of Equation (1.2) can be extended to a more general reaction-diffusion system ut = div(A(u)∇u) + r(x, u) with A(u) = |∇u|p−2 + |∇u|q−2. (1.3) This has a great number of applications in plasma physics, solid state physics, bio- physics and chemical reactions. In such applications, u usually corresponds to the concentration term, div(A(u)∇u) represents the diffusion with diffusion coefficient A(u), and the reaction term r(x, u) relates to source and loss processes. In chemical and biological applications, the reaction term r(x, u) has a polynomial form with respect to the concentration u with variable coefficients, see [16]. The existence results for several classes of equations of (p, q)-Laplacian type defined in bounded domains or in the whole of Rn can be found in [4, 19, 30] and the references therein. It is a well-known fact that as a special model of the differential operator (1.3) it is the following so-called double-phase one Lu := div(|∇u|p−2∇u+ a(x)|∇u|q−2∇u), which is related to the energy functional u 7→ ∫ Ω (|∇u|p + a(x)|∇u|q)dx (1.4) with 1 < p < q and a ∈ L∞(Ω) with a(x) ≥ 0 for almost all x ∈ Ω. Roughly speaking, the integral functional (1.4) is characterized by the fact that the energy density changes its growth properties and ellipticity according to the point in the domain. To be more precise, the modulating potential a(x) dictates the geometry of a composite material made of two different components, with distinct power hard- ening exponents p and q. Following Marcellini’s terminology in [26], the integrand H(x, ξ) = |ξ|p + a(x)|ξ|q for all (x, ξ) ∈ Ω×Rn has different growth near the origin and at infinity (unbalanced growth), that is, |ξ|p ≤ H(x, ξ) ≤ |ξ|q + 1 for a.e. x ∈ Ω and for all ξ ∈ Rn. The study of the double-phase functionals of the form (1.4) has generated con- siderable interests after the initial work by Zhikov in [40] to describe models of strongly anisotropic materials, see for instance [7, 8, 17, 18] and the references therein. While s1 = s2 and σ = 0, the problem (1.1) boils down to the following fractional (p, q)-Laplacian equations (−∆)spu+ (−∆)squ+ V (εx)(|u|p−2u+ |u|q−2u) = f(u) in Rn, (1.5) which has been extensively considered by several authors in recent years. For p = q 6= 2, there has been a source of inspiration around its existence and multiplicity results in the last decade due to two phenomena: the nonlocal character of the operator and its nonlinearity; see for instance [33, 14, 24, 38] and the references therein. Indeed, we would like to stress that standard arguments used to investigate the linear case p = q = 2 seem to be inapplicable to the nonlinear case on account of EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 3 the lack of Hilbertian structure of W s,p(Rn) for p 6= 2. For the reader’s convenience, we refer to [15, 21, 35, 39], where existence and multiplicity results were obtained in the case p = q = 2. However, for p 6= q in the nonlocal framework only few recent papers deal with problems like (1.5). For example, Filippis and Palatucci [20] proved a Hölder regularity result for nonlocal double-phase equations. And Lin and Zheng [25] established the multiplicity and asymptotic behavior of solutions to fractional (p, q)-Kirchhoff type problems with critical Sobolev-Hardy exponent. We refer the readers to some recent papers [5, 6, 13, 27] for other interesting double- phase problems in both local and nonlocal cases. While s1 = s2, σ = 1 and p = q 6= 2, this becomes the following fractional nonlocal Schrödinger equation involving a critical Sobolev exponent (−∆)spu+ V (εx)|u|p−2u = f(u) + |u|p ∗ s−2u in Rn. We refer to [3] for this class of p-fractional Schrödinger equation, where the authors presented the existence and multiplicity results of it. For p 6= q, Ambrosio in [2] es- tablished an existence result for the fractional (p, q)-Laplacian equation with critical growth. The multiplicity results for fractional (p, q)-Laplacian equations involving critical nonlinearities in bounded domains has been obtained in [9]. Motivated by all these works, we are to consider doubly fractional Schrödinger equations of (p, q)- Laplacian with critical growth. More precisely, we are interested in the existence result of positive solutions to Problem (1.1). It is an important observation that the lack of compactness due to the critical exponent greatly increases the methodological difficulties. The main contribution of our work is to deal with the possibility of loss of compactness due to the critical nonlinearity of the double-phase Schrödinger equations in the fractional setting suitably. We then apply this to obtain sufficient existence conditions for equations like (1.1) in all Rn, which generalizes the results of Ambrosio’s paper [4] for the local case. Before stating main result, let us introduce main assumptions imposed on the potential V and the nonlinearity f . Throughout this paper, we assume that V : Rn → R is a continuous function satisfying the following condition by Rabinowitz as in [34]: 0 < V0 = inf x∈RN V (x) < lim inf |x|→∞ V (x) = V∞ ∈ (0,∞], (1.6) and we shall consider it in the two cases of V∞ <∞ and V∞ =∞ in the following. For the nonlinearity f : R→ R, it is assumed that: (A1) f ∈ C0(R,R) and f(t) = 0 for all t ≤ 0; (A2) lim|t|→0 |f(t)|/|t|p−1 = 0; (A3) there exists r ∈ (q, q∗s2) with q∗s2 = nq/(n− s2q), such that lim |t|→∞ |f(t)| |t|r−1 = 0; (A4) there exists θ ∈ (q, q∗s2) such that 0 < θF (t) = θ ∫ t 0 f(τ)dτ ≤ tf(t) for all t > 0; (A5) the map t 7→ f(t)/tq−1 is increasing in (0,∞). Regarding the existence result, we prove that there exists at least one non- negative non-trivial solution to Problem (1.1) in the subcritical for all λ > 0 and for small ε. For the critical case, we prove the existence of at least one non-negative 4 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 non-trivial solution to Problem (1.1) provided small ε and λ large enough. To be more precise, our first result of the paper is the following. Theorem 1.1. Let σ = 0. Assume that (1.6) and (A1)–(A5) hold. Then there exists ε0 > 0 such that, for all ε ∈ (0, ε0), Problem (1.1) with subcritical growth admits at least one positive ground state solution for any λ > 0. The proof of this theorem relies heavily on existence of the corresponding au- tonomous problem motivated from [3]. As usual, the presence of the fractional (p, q)-Laplacian operator leads to more intriguing analysis. Indeed, the arguments used in the study of ((1.5) seem not to be trivially adaptable to handling two totally different components. Therefore, some appropriate technical lemmas (see Lemma 2.4 and Lemma 2.5) and much more delicate estimates will be needed. Further, we consider the existence result of Problem (1.1) for the critical case with σ = 1. Theorem 1.2. Let σ = 1. Assume that (1.6) and (A1)–(A5) hold. Then there exists ε0 > 0 such that, for all ε ∈ (0, ε0), Problem (1.1) with critical growth admits at least one positive ground state solution for any λ ≥ λ∗, where λ∗ is a positive constant. The idea for proving this theorem is also based on suitable variational tech- niques. Owing to the combination of two nonhomogeneous fractional involved op- erators with different scaling properties, it is a rather delicate situation, and more estimates will be needed to achieve our result. Particularly, we would like to point out that the calculations performed for the setting σ = 1 to recover compactness are much more complicated with respect to the case σ = 0 due to the presence of critical exponent. More precisely, the main difficulty in the critical case is that the energy functional fails to satisfy the Palais-Smale condition globally. This is different from the calculations performed as in [29] and the optimal asymptotic behavior of p-minimizers established as in [10]. Instead, we provide some technical results which allow us to avoid unnecessary calculations and prove the Palais-Smale condition for the critical case (see Lemma 4.4 and Lemma 4.5 ). To the best of our knowledge, these are new contributions to show the existence results of Problem (1.1) for double-phase nonlocal Schrödinger equations involved in both subcritical and critical growth. The remainder of this paper is organized as follows. In Section 2, we give some related notations, recall basic facts about the involved fractional Sobolev spaces and provide various useful lemmas. We devoted Section 3 to the proof of existence result for the subcritical case when σ = 0. In Section 4, we deal with the critical case when σ = 1, and finally give the proof of Theorem 1.2. 2. Preliminaries This section is devoted to some well-known facts about the fractional Sobolev spaces and some technical lemmas we will use later. Throughout this paper, C(n, ν, L, · · · ) stands for a universal constant depending only on prescribed quanti- ties and possibly varying from line to line. However, the ones we need to emphasize will be denoted with special symbols, such as C1, C2, C∗, Cξ. For p ∈ [1,∞] and A ⊂ Rn, we denote by |u|Lp(A) the Lp(A)-norm of a function u : Rn → R belonging to Lp(A), and by |u|p its Lp(Rn)-norm. We define Ds,p(Rn) EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 5 as the closure of C∞c (Rn) with respect to the following semi-norm [u]ps,p = ∫ R2n |u(x)− u(y)|p |x− y|n+sp dx dy for s ∈ (0, 1) and p ∈ (1,∞). The fractional Sobolev space W s,p(Rn) is defined as the set of all functions u ∈ Lp(Rn) such that [u]s,p <∞, equipped with the norm ‖u‖ps,p = [u]ps,p + |u|pp. Furthermore, for u, v ∈W s,p(Rn), we put 〈u, v〉s,p = ∫ R2n |u(x)− u(y)|p−2(u(x)− u(y)) |x− y|n+sp (v(x)− v(y)) dx dy. In the following, we denote by BR(x) the ball of radius R with the center at x. While x = 0, we briefly write BR = BR(0). First let us begin with the following Sobolev embedding relation. Lemma 2.1 ([31]). Let s ∈ (0, 1) and p ∈ [1,∞) such that n > sp. Then there exists a constant S∗ > 0 such that, for any u ∈ Ds,p(Rn) it holds |u|pp∗s ≤ S −1 ∗ [u]ps,p. Moreover, W s,p(Rn) is continuously embedded in Lq(Rn) for any q ∈ [p, p∗s] and compactly embedded in Lqloc(Rn) for any q ∈ [1, p∗s). The following compactness result of Lions-type is recalled which will be used in the main proof later. Lemma 2.2 ([3]). Let n > sp and r ∈ [p, p∗s). If {un} is a bounded sequence in W s,p(Rn) with lim n→∞ sup y∈Rn ∫ BR(y) |un|rdx = 0, where R > 0, then un → 0 in Lt(Rn) for all t ∈ (p, p∗s). Lemma 2.3 ([3]). Let u ∈ W s1,p(Rn) ∩ W s2,q(Rn) and φ ∈ C∞c (Rn) such that 0 ≤ φ ≤ 1, φ = 1 in B1 and φ = 0 in Rn\B2. Set φr(x) = φ(xr ). Then lim r→∞ ‖uφr − u‖1 = 0 and lim r→∞ ‖uφr − u‖2 = 0 . For any ε > 0, we define the space related to the potential V (εx), Xε = { u ∈W s1,p(Rn) ∩W s2,q(Rn) : ∫ Rn V (εx)(|u|p + |u|q)dx <∞ } equipped with the norm ‖u‖ε = ‖u‖1 + ‖u‖2, where ‖u‖ti = [u]tsi,t + ∫ Rn V (εx)|u|tdx for all t > 1 and i = {1, 2}. Thanks to the assumption (1.6) and Lemma 2.1, it is easy to check that it holds the following result. Lemma 2.4. The space Xε is continuously embedded into W s1,p(Rn)∩W s2,q(Rn). Moreover, Xε is continuously embedded in Lt(Rn) for any t ∈ [ p, q∗s2 ] , and com- pactly embedded in Ltloc ( Rn ) for any t ∈ [ 1, q∗s2 ) . 6 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 Proof. For any u ∈ Xε, by the assumption (1.6) and the definition of ‖ · ‖i for i = {1, 2} we have min{1, V0}‖u‖ps1,p ≤ ‖u‖ p 1, min{1, V0}‖u‖ps2,q ≤ ‖u‖ p 2. Therefore, it follows that the embedding Xε ↪→W s1,p(Rn)∩W s2,q(Rn) is continu- ous. By Lemma 2.1, it is evident that the embedding Xε ↪→ Lt(Rn) is continuous for any t ∈ [p, q∗s2 ] and the embedding Xε ↪→ Ltloc(Rn) is obviously compact for any t ∈ [1, q∗s2). � In addition, by considering V being coercive, we obtain the following compact- ness lemma. Lemma 2.5. Let V∞ = ∞. Then Xε is compactly embedded in Lt(Rn) for any t ∈ [p, q∗s2). Proof. For t = p, let {un} be a sequence such that un ⇀ 0 in Xε. By Lemma 2.4 we know that Xε ⊂ Lp ( Rn ) . Then un ⇀ 0 in W s1,p(Rn) ∩W s2,q(Rn) and un → 0 in Lp ( BR ) . Therefore, for any ε > 0 there exists n0 > 0 such that∫ BR |u|pdx ≤ ε for any n ≥ n0. (2.1) Since V (x) is coercive, there exists R = Rε > 0 such that 1 V (εx) < ε for any |x| > R. (2.2) Let us set T := sup n∈N ‖un‖ε <∞. (2.3) Hence, for any n ≥ n0, by using (2.1), (2.2) and (2.3) we conclude that∫ Rn |u|pdx = ∫ BR |u|pdx+ ∫ Rn\BR |u|pdx ≤ ε+ ε ∫ Rn\BR V (εx)|u|pdx ≤ ε(1 + T p). Then we have un → 0 in Lp ( Rn ) . As for p < t < q∗s2 , by using the interpolation inequality and Lemma 2.1 we see that |u|t ≤ C[u]αs2,q|u| 1−α p , where 1 t = α p + 1−α q∗s2 , which yields the required result. � Finally, we recall the following splitting lemma which will be very useful in our main proof, which is proved by Ambrosio and Repovš (cf. [4]) following the arguments developed by Brezis and Lieb [11]. Lemma 2.6. Let {un} be a sequence such that un ⇀ u in Xε, and vn = un − u. Then we have (i) [vn]ps1,p + [vn]qs2,q = ( [un]ps1,p + [un]qs2,q ) − ( [u]ps1,p + [u]qs2,q ) + on(1); (ii) ∫ Rn V (εx)(|vn|p+|vn|q)dx = ∫ Rn V (εx) ( (|un|p+|un|q)dx−(|u|p+|u|q) ) dx+ on(1); (iii) ∫ Rn(F (vn)− F (un) + F (u))dx = on(1); EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 7 (iv) sup ‖w‖ε≤1 ∫ Rn |(f(vn)− f(un) + f(u))w|dx = on(1). 3. Subcritical case while σ = 0 3.1. Functional setting in the subcritical case. In this section we consider the problem (−∆)s1p u+ (−∆)s2q u+ V (εx) ( |u|p−2u+ |u|q−2u ) = λf(u) in Rn, u ∈W s1,p(Rn) ∩W s2,q(Rn), u > 0 in Rn. (3.1) Let us define the energy functional associated with (3.1), Iε(u) = 1 p ‖u‖p1 + 1 q ‖u‖q2 − λ ∫ Rn F (u)dx, which is well-defined for all u : Rn → R belonging to the fractional space Xε. Therefore, from the assumptions on f it is easy to check that Iε ∈ C1(Xε,R) and its differential is given by 〈I ′ε(u), v〉 = 〈u, v〉s1,p + 〈u, v〉s2,q + ∫ Rn V (εx)(|u|p−2u+ |u|q−2u)v dx− λ ∫ Rn f(u)v dx for any u, v ∈ Xε. Now we check that Iε possesses a mountain pass geometry (cf. [1]). Lemma 3.1. The functional has a mountain pass geometry shown as follows: (i) there exist α, ρ > 0 such that Iε(u) ≥ α with ‖u‖ε = ρ; (ii) there exists e ∈ Xε with ‖e‖ε > ρ such that Iε(e) < 0. Proof. (i) By growth assumptions (A2) and (A3) on f , we readily see that for any ξ > 0 there exists a constant Cξ > 0 such that |f(t)| ≤ ξ|t|p−1 + Cξ|t|r−1 ∀t ∈ R, (3.2) |F (t)| ≤ ξ p |t|p + Cξ r |t|r ∀t ∈ R. (3.3) Therefore, using V0 ≤ V (εx) and taking ξ ∈ (0, V0 λ ), we have Iε(u) ≥ 1 p ‖u‖p1 + 1 q ‖u‖q2 − λ ξ p |t|pp − λ Cξ r |t|rr ≥ C1‖u‖p1 + 1 q ‖u‖q2 − λ Cξ r |t|rr. By choosing ‖u‖ε = ρ ∈ (0, 1) and using 1 < p < q, we have ‖u‖p1 < 1 which leads to ‖u‖p1 ≥ ‖u‖ q 1. This fact combined with an elementary convex inequality at + bt ≥ 2−t+1(a+ b)t, ∀a, b ≥ 0 and t > 1, and Lemma 2.4 yield Iε(u) ≥ C2‖u‖qε − λ Cξ r |t|rr ≥ C2‖u‖qε − C3‖u‖rε. Therefore, we can find α > 0 such that Iε(u) ≥ α for ‖u‖ε = ρ due to r > q. (ii) By assumption (A4) we can infer that for some C1, C2 > 0 it holds F (t) ≥ C1t θ − C2 for any t > 0. 8 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 By taking u ∈ C∞c (Rn) such that u ≥ 0 and u 6= 0, then we know that Iε(tu) ≤ tp p ‖u‖p1 + tq q ‖u‖q2 − λtθC1 ∫ Rn uθdx+ C3 → −∞ as t→∞, where we have used θ > q > p. This completes the proof. � Therefore, invoking a variant of the mountain pass theorem without Palais-Smale condition (cf. [37]), we can see that there exists a sequence {un} ⊂ Xε such that Iε(un)→ cε and I ′ε(un)→ 0, where cε = inf γ∈Γ max t∈[0,1] Iε(γ(t)) with Γ = {γ ∈ C0([0, 1], Xε) : γ(0) = 0, Iε(γ(1)) < 0}. Further, as in [37] we can use the equivalent characterization of cε, which is more appropriate to our aim, given by cε = inf u∈Xε\{0} max t≥0 Iε(γ(t)). We also introduce the Nehari manifold associated with Iε, which is defined by Nε = {u ∈ Xε\{0} : 〈I ′ε(u), u〉 = 0}. Next, we prove that any Palais-Smale sequence of Iε is bounded. Lemma 3.2. Let {un} be a Palais-Smale sequence of Iε at level c. Then {un} is bounded in Xε. Proof. Let {un} ⊂ Xε be a Palais-Smale sequence at the level c, that is Iε(un) = c+ on(1) and I ′ε(un) = on(1). Using (A4) and considering θ > q > p we can deduce that c(1 + ‖un‖ε) ≥ Iε(un)− 1 θ 〈I ′ε(un), un〉 = (1 p − 1 θ ) ‖un‖p1 + (1 q − 1 θ ) ‖un‖q2 + λ ∫ Rn (1 θ f(un)un − F (un) ) dx ≥ (1 q − 1 θ ) (‖un‖p1 + ‖un‖q2). Next we prove it by contradiction. Assume that ‖un‖ε = ‖u‖1 + ‖u‖2 → ∞, we distinguish it in the following three cases: Case 1. If ‖un‖1 → ∞ or ‖un‖2 → ∞, then for n sufficiently large we have ‖un‖q−p2 ≥ 1, that is‖un‖q2 ≥ ‖un‖ p 2. Thus c(1 + ‖un‖ε) ≥ (1 q − 1 θ ) (‖un‖p1 + ‖un‖p2) ≥ C(‖un‖1 + ‖un‖2)p = C‖un‖pε , which gives a contradiction because p > 1. Case 2. If ‖un‖1 →∞ and ‖un‖2 is bounded, then we have c(1 + ‖un‖1 + ‖un‖2) = c(1 + ‖un‖ε) ≥ (1 q − 1 θ ) ‖un‖p1, which implies that c ( 1 ‖un‖p1 + 1 ‖un‖p−1 1 + ‖un‖2 ‖un‖p1 ) ≥ (1 q − 1 θ ) . EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 9 Passing to the limit as n → ∞, we obtain 0 < 1 q − 1 θ ≤ 0, which leads to a contradiction. Case 3. If ‖un‖2 → ∞ and ‖un‖1 is bounded, we can do it by a similar way as Case 2. Putting the three cases together completes the proof. � 3.2. Autonomous subcritical problem. We devote this subsection to the fol- lowing autonomous problem associated with Problem (3.1) for any µ > 0: (−∆)s1p u+ (−∆)s2q u+ µ ( |u|p−2u+ |u|q−2u ) = λf(u) in Rn, u ∈W s1,p(Rn) ∩W s2,q(Rn), u > 0 in Rn. (3.4) The corresponding energy functional is Iµ(u) = 1 p ‖u‖pµ,1 + 1 q ‖u‖qµ,2 − λ ∫ Rn F (u)dx, which is well-defined on the space Yµ = W s1,p(Rn) ∩W s2,q(Rn) equipped with the norm ‖u‖µ = ‖u‖µ,1 + ‖u‖µ,2, where ‖u‖tµ,i = [u]tsi,t + µ|u|tt for all t > 1 and i = {1, 2}. It is easy to check that Iµ ∈ C1(Yµ,R) and its differential is given by 〈I ′µ(u), v〉 = 〈u, v〉s1,p + 〈u, v〉s2,q + µ ∫ Rn (|u|p−2u+ |u|q−2u)v dx− λ ∫ Rn f(u)v dx for any u, v ∈ Yµ. Let us define the Nehari manifold associated with Iµ as follows: Nµ = {u ∈ Yµ\{0} : 〈I ′µ(u), u〉 = 0}. It follows from (A4) that Iµ(u) = Iµ(u)− 1 q 〈I ′µ(u), u〉 = (1 p − 1 q ) ‖u‖pµ,1 − λ ∫ Rn ( F (u)− 1 q f(u)v ) dx ≥ (1 p − 1 q ) ‖u‖pµ,1 for all u ∈ Nµ. (3.5) We easily check that Iµ has a mountain pass geometry, and we denote by cµ its mountain pass level. Moreover, by the standard arguments from [37] and (3.5) we can show that 0 < cµ = inf u∈Nµ Iµ(u) = inf u∈Yµ\{0} max t≥0 Iµ(tu). With these facts in hand, we arrive to the following lemma. Lemma 3.3. Let t ∈ [ p, q∗s2 ) , and {un} ⊂ Nµ be a minimizing sequence for Iµ. Then, {un} is bounded in Yµ; moreover, there exist a sequence {yn} ⊂ Rn and constants R, β > 0 with lim inf n→∞ ∫ BR(yn) |un|tdx ≥ β > 0. 10 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 Proof. Arguing as in the proof of Lemma 3.2, we immediately see that {un} is bounded in Yµ. To prove the latter conclusion of this Lemma, by contradiction we assume that for any R > 0 it holds lim n→∞ sup y∈Rn ∫ BR(y) |un|tdx = 0. Then, it follows from Lemma 2.2 that un → 0 in Lt(Rn) for all t ∈ ( p, q∗s2 ) . (3.6) We fix ξ ∈ (0, µλ ), and take into account {un} ∈ Nµ and (3.2) to deduce that 0 = 〈I ′µ(un), un〉 ≥ ‖un‖pµ,1 + ‖un‖qµ,2 − λξ|t|pp − λCξ|t|rr ≥ C1‖un‖pµ,1 + ‖un‖qµ,2 − λCξ|t|rr, which combined with (3.6) implies that ‖un‖µ → 0 as n → ∞. This leads to a contradiction because of Iµ(un)→ cµ > 0, which completes the proof. � Next we prove a useful compactness result for the autonomous problem (3.4). Theorem 3.4. Under assumptions (A1)–(A5), Problem (3.4) admits a positive ground state solution. Proof. We see from a variant of the mountain pass theorem without Palais-Smale condition (cf. [37]) that there exists a Palais-Smale sequence {un} ⊂ Yµ for Iµ at the level cµ. Then, using Lemma 3.3 we know that {un} is bounded in Yµ, and we may assume that un ⇀ u in Yµ, un → u in Ltloc(Rn) for all t ∈ [ 1, q∗s2 ) . Now show that the weak limit u is a critical point of Iµ. To this end, we consider the sequence hn(x, y) = |un(x)− un(x)|p−2(un(x)− un(x)) |x− y| n+s1p p′ , and let h(x, y) = |u(x)− u(x)|p−2(u(x)− u(x)) |x− y| n+s1p p′ with p′ = p p−1 . We easily check that {hn} is a bounded sequence in the reflexive Banach space Lp ′(R2n ) with hn → h a.e. in R2n. Then there exists a subsequence, still denoted by {hn}, such that hn ⇀ h in Lp ′(R2n ) , that is to say,∫ R2n hn(x, y)g(x, y) dx dy → ∫ R2n h(x, y)g(x, y) dx dy for all g ∈ Lp ( R2n ) . For any v ∈ C∞c (Rn), by taking g(x, y) = v(x)− v(x) |x− y| n+s1p p ∈ Lp ( R2n ) , we can see that∫ R2n |un(x)− un(y)|p−2(un(x)− un(y)) |x− y|n+s1p (v(x)− v(y)) dx dy EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 11 → ∫ R2n |u(x)− u(y)|p−2(u(x)− u(y)) |x− y|n+s1p (v(x)− v(y)) dx dy. Similarly, we also prove that∫ R2n |un(x)− un(y)|q−2(un(x)− un(y)) |x− y|n+s2q (v(x)− v(y)) dx dy → ∫ R2n |u(x)− u(y)|q−2(u(x)− u(y)) |x− y|n+s2q (v(x)− v(y)) dx dy. Note that ∫ Rn |un|p−2unv dx→ ∫ Rn |u|p−2uv dx,∫ Rn |un|q−2unv dx→ ∫ Rn |u|q−2uv dx,∫ Rn f(un)v dx→ ∫ Rn f(u)v dx and the fact that 〈I ′µ(un), v〉 = on(1), we can deduce that 〈I ′µ(u), v〉 = 0 for all v ∈ C∞c (Rn). By the density of C∞c (Rn) in Yµ, we obtain that u is a critical point of Iµ, which implies that 〈I ′µ(u), u〉 = 0. Next, we prove that Iµ(u) = cµ, which is divided into two cases: Case 1. For u 6= 0, it suffices only to show that ‖un‖pµ,1 → ‖u‖ p µ,1, (3.7) and then similarly, we can see that ‖un‖qµ,2 → ‖u‖ q µ,2. Lemma 2.6 leads to that un → u in Yµ. This together with the fact that Iµ(un) → cµ deduces the desired result. To prove (3.7), we observe that Fatou’s lemma yields ‖u‖pµ,1 ≤ lim inf n→∞ ‖un‖pµ,1. By contradiction, let us assume that ‖u‖pµ,1 < lim sup n→∞ ‖un‖pµ,1. (3.8) We notice that cµ + on(1) = Iµ(un)− 1 q 〈I ′µ(un), un〉 = (1 p − 1 q ) ‖un‖pµ,1 + λ ∫ Rn (1 q f(un)un − F (un) ) dx. (3.9) Recalling that lim sup n→∞ (an + bn) ≥ lim sup an+ n→∞ lim inf n→∞ bn and p < q, using (3.8), (3.9), Fatou’s lemma again and the fact that 〈I ′µ(u), u〉 = 0, it yields that cµ ≥ (1 p − 1 q ) lim sup n→∞ ‖un‖pµ,1 + λ lim inf n→∞ ∫ Rn (1 q f(un)un − F (un) ) dx > (1 p − 1 q ) ‖u‖pµ,1 + λ ∫ Rn (1 q f(u)u− F (u) ) dx = Iµ(u)− 1 q 〈I ′µ(u), u〉 = Iµ(u) ≥ cµ, 12 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 which gives a contradiction. Case 2. For u = 0, we argue it as in the proof of Lemma 3.3, we can find a sequence {yn} ⊂ Rn and constants R, β > 0 such that lim inf n→∞ ∫ BR(yn) |un|qdx ≥ β > 0. Set vn(x) = un(x + yn). From the invariance by translations of Rn, it is clear to show that {vn} ⊂ Yµ is also a bounded Palais–Smale sequence for Iµ at the level cµ, and vn ⇀ v 6= 0 in Yµ. Hence, we can proceed as before to check that {vn} converges strongly in Yµ. Finally, we prove that the ground state obtained above is positive. In fact, thanks to 〈I ′µ(u), u−〉 = 0, assumption (A1) and the inequality |x− y|t−2(x− y)(x− − y−) ≥ |x− − y−|t for all t ≥ 1, for u− = min{u, 0} we have that ‖u−‖pµ,1 + ‖u−‖qµ,2 ≤ 0, which implies that u− = 0, which meads that u ≥ 0 in Rn. By employing a variant of the maximum principle (cf. [32]) we conclude that u > 0 in Rn. This completes the proof. � 3.3. Proof of Theorem 1.1. In this subsection, we concentrate on the existence of the solution to Problem (3.1) and then give the proof of Theorem 1.1 provided that ε is sufficiently small. We shall start with the following inevitable lemma. Lemma 3.5. Let t ∈ [ p, q∗s2 ) , and {un} ⊂ Nε be a sequence such that Iε(un)→ cε and un ⇀ 0 in Xε. Then, one of the following alternatives occurs: (a) un → 0 in Xε; (b) there exist a sequence {yn} ⊂ Rn and constants R, β > 0 such that lim inf n→∞ ∫ BR(yn) |un|tdx ≥ β > 0. Proof. It is natural that (b) fails while (a) happens. Conversely, we assume that (b) does not hold. Then for any R > 0 it holds lim n→∞ sup y∈Rn ∫ BR(y) |un|tdx = 0. By using Lemma 3.2 and Lemma 2.2, it follows that un → 0 in Lt(Rn) for all t ∈ ( p, q∗s2 ) . We argue it as in the proof of Lemma 3.3, and we conclude that ‖un‖ε → 0 as n→∞. Thus we complete the proof. � Before establishing a compactness result for Iε, it is necessary to prove the fol- lowing auxiliary lemma. Lemma 3.6. For V∞ <∞, let {vn} ⊂ Nε be a sequence such that Iε(vn)→ c and vn ⇀ 0 in Xε. If vn 6→ 0 in Xε, then we have c ≥ c∞, where c∞ is the infimum of IV∞ over NV∞ . EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 13 Proof. It follows from Lemma 3.2 that {vn} is bounded in Xε. Let {tn} ⊂ (0,∞) be such that {tnvn} ⊂ NV∞ . Hence, it suffices to prove that lim sup n→∞ tn ≤ 1. In fact, by contradiction we suppose that there exist δ > 0 such that tn ≥ 1 + δ for all n ∈ N. (3.10) Note that {vn} ⊂ Xε is a bounded Palais-Smale sequence for Iε, we easily see that 〈I ′ε(vn), vn〉 = 0, which means that [vn]ps1,p + [vn]qs2,q + ∫ Rn V (εx)(|vn|p + |vn|q)dx− λ ∫ Rn f(vn)vndx = 0. This combined with the fact that tnvn ∈ NV∞ yields tp−qn [vn]ps1,p+[vn]qs2,q+t p−q n V∞ ∫ Rn |vn|pdx+V∞ ∫ Rn |vn|qdx−λ ∫ Rn f(tnvn)vqn (tnvn)q−1 dx = 0, which implies that λ ∫ Rn ( f(tnvn) (tnvn)q−1 − f(vn) vq−1 n ) vqndx ≤ ∫ Rn (V∞ − V (εx))|vn|pdx+ ∫ Rn (V∞ − V (εx))|vn|qdx (3.11) for any required tn and p < q. By using assumption (1.6), for any ζ > 0, there exists a constant R > 0 such that V (εx) ≥ V∞ − ζ for all |x| ≥ R. (3.12) In addition, using the boundedness of {vn} in Xε together with the fact that vn → 0 in Lp(BR), we can deduce that∫ Rn (V∞ − V (εx))|vn|pdx = ∫ BR (V∞ − V (εx))|vn|pdx+ ∫ Rn\BR (V∞ − V (εx))|vn|pdx ≤ V∞ ∫ BR |vn|pdx+ ζ ∫ Rn\BR |vn|pdx ≤ on(1) + ζC. (3.13) Similarly, we also find that∫ Rn (V∞ − V (εx))|vn|qdx ≤ on(1) + ζC. (3.14) Combining (3.11), (3.13) and (3.14) we have∫ Rn ( f(tnvn) (tnvn)q−1 − f(vn) vq−1 n ) vqndx ≤ on(1) + ζC. (3.15) With the help of Lemma 3.5, we can infer that there exist a sequence {yn} ⊂ Rn, such that for the constants R, β > 0 it holds lim inf n→∞ ∫ BR(yn) |vn|tdx ≥ β > 0 (3.16) 14 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 with t ∈ [ p, q∗s2 ) . By considering v̂n = vn(x + yn), then, up to a subsequence we can assume that v̂n ⇀ v̂ in Xε. By formula (3.16) there exists Ω ⊂ Rn with positive measure such that v̂ > 0 in Ω, which combined (3.10) and (3.15) yields the inequality ∫ Ω ( f((1 + δ)v̂n) ((1 + δ)v̂n)q−1 − f(v̂n) v̂q−1 n ) v̂qndx ≤ on(1) + ζC. Therefore, passing to the limit as n→∞, using Fatou’s lemma and (A5), yields 0 < ∫ Ω ( f((1 + δ)v̂) ((1 + δ)v̂)q−1 − f(v̂) v̂q−1 ) v̂qdx ≤ ζC for all ζ > 0, which leads to a contradiction. For the remainder we consider the following two cases. Case 1. For lim supn→∞ tn = 1, up to a subsequence, there exists {tn} such that tn → 1. Considering that Iε(vn)→ c, we have c+ on(1) = Iε(vn) = Iε(vn)− IV∞(tnvn) + IV∞(tnvn) ≥ Iε(vn)− IV∞(tnvn) + c∞. (3.17) Note that Iε(vn)− IV∞(tnvn) = 1− tpn p [vn]ps1,p + 1− tqn q [vn]qs2,q + 1 p ∫ Rn (V (εx)− tpnV∞)|vn|pdx + 1 q ∫ Rn (V (εx)− tqnV∞)|vn|qdx+ λ ∫ Rn (F (tnvn)− F (vn))dx. (3.18) Taking into account that vn → 0 in Lp(BR) as tn → 1, Inequality (3.12) and assumption (1.6) imply that V (εx)− tpnV∞ = (V (εx)− V∞) + (1− tpn)V∞ ≥ −ζ + (1− tpn)V∞ for all |x| ≥ R. Therefore, we conclude that∫ Rn (V (εx)− tpnV∞)|vn|pdx = ∫ BR (V (εx)− tpnV∞)|vn|pdx+ ∫ Rn\BR (V (εx)− tpnV∞)|vn|pdx ≥ (V0 − tpnV∞) ∫ BR |vn|pdx− ζ ∫ Rn\BR |vn|pdx+ (1− tpn)V∞ ∫ Rn\BR |vn|pdx ≥ on(1)− ζC. (3.19) Similarly, we can prove that∫ Rn (V (εx)− tqnV∞)|vn|qdx ≥ on(1)− ζC. (3.20) Using the boundedness of {vn} in Xε, we obtain 1− tpn p [vn]ps1,p = on(1) and 1− tqn q [vn]qs2,q = on(1). (3.21) EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 15 Combining (3.18), (3.19), (3.20) and (3.21) we have Iε(vn)− IV∞(tnvn) ≥ λ ∫ Rn (F (tnvn)− F (vn))dx+ on(1)− ζC. (3.22) Then it suffices to show that∫ Rn (F (tnvn)− F (vn))dx = on(1). (3.23) Employing Lemma 2.6, (3.3) and the boundedness of {vn} in Xε, we can infer that∫ Rn ( F (tnvn)− F (vn) ) dx = ∫ Rn F ((tn − 1)vn)dx+ on(1) ≤ ξ p |tn − 1|p ∫ Rn |vn|pdx+ Cξ r |tn − 1|r ∫ Rn |vn|rdx+ on(1) ≤ on(1). Thus, putting (3.17), (3.22) and (3.23) together we obtain c+ on(1) ≥ on(1)− ζC + c∞, and then passing to the limit as ζ → 0 yields c ≥ c∞. Case 2. For lim supn→∞ tn = t0 < 1, there exists a subsequence, still denoted by {tn} such that tn → t0 and tn < 1 for any n ∈ N. Considering that I ′ε(vn)→ 0, we have c+ on(1) = Iε(vn)− 1 q 〈I ′ε(vn), vn〉 = (1 p − 1 q ) ‖vn‖p1 + λ ∫ Rn (1 q f(vn)vn − F (vn) ) dx. (3.24) Moreover, recalling the fact that tnvn ∈ NV∞ , Inequality (3.13) and assumption (A5) yields c∞ ≤ IV∞(tnvn) = IV∞(tnvn)− 1 q 〈I ′V∞(tnvn), tnvn〉 = (1 p − 1 q ) ‖tnvn‖pV∞,1 + λ ∫ Rn (1 q f(tnvn)tnvn − F (tnvn) ) dx ≤ (1 p − 1 q ) (‖vn‖p1 + on(1) + ζC) + λ ∫ Rn (1 q f(vn)vn − F (vn) ) dx = c+ on(1) + ζC, where we used (3.24) in the last inequality. Therefore, by passing to the limit as ζ → 0 and n→∞ it implies that c ≥ c∞, which completes the proof. � We are now in a position to prove the compactness result as follows. Lemma 3.7. Let {un} ⊂ Nε be such that Iε(un)→ cε, where cε < c∞ for V∞ <∞, and any cε ∈ R for V∞ =∞. Then {un} has a convergent subsequence in Xε. Proof. We argue as before, we immediately see that {un} is bounded in Xε, which means that we can take a sequence {un} such that un ⇀ u in Xε, un → u in Ltloc(Rn) for all t ∈ [ 1, q∗s2 ) . 16 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 Similar to the proof of Theorem 3.4, it is easy to check that I ′ε(u) = 0. Let vn = un − u and Iε(vn) → c. Considering that Iε(un) → cε and Lemma 2.6, we have c+ on(1) = Iε(vn) = Iε(un)− Iε(u) + on(1) = cε − Iε(u) + on(1). (3.25) Arguing it as in the proof of [3, Proposition 3.1], we can see that I ′ε(u) = 0. Note that Iε(u) = Iε(u)− 1 q 〈I ′ε(u), u〉 ≥ 0. (3.26) By using (3.25) and (3.26), and considering V∞ <∞, we can see that c ≤ cε < c∞, which together with Lemma 3.6 yields vn → 0 in Xε, which means that un → u in Xε. On the other hand, by considering V∞ =∞, and using Lemma 2.5 we can infer that vn → 0 in Lt(Rn) for any t ∈ [ p, q∗s2 ) . This combined with assumptions (A2) and (A3) yields ∫ Rn f(vn)vndx = on(1). In addition, in accordance with 〈I ′ε(vn), vn〉 = on(1) we deduce that ‖vn‖p1 + ‖vn‖q2 = on(1), which leads to ‖un − u‖ε = on(1) as n→∞. This completes the proof. � Proof of Theorem 1.1. According to Lemma 3.1, we know that there exists a se- quence {un} ⊂ Xε such that Iε(un)→ cε and I ′ε(un)→ 0, where cε = inf u∈Xε\{0} max t≥0 Iε(tu). By standard arguments, we obtain that {un} is bounded in Xε, which yields that there exists a subsequence {un} such that un ⇀ u in Xε, where u is denoted by its weak limit. Moreover, I ′ε(u) = 0. With the help of Lemma 3.7, it is clear to check that un → u in Xε while V∞ = ∞. Then applying the mountain pass theorem yields the existence result. It is rather clear to check that cε ≤ Iε(u). On the other hand, it follows from Fatou’s lemma that Iε(u) = Iε(u)− 1 q 〈I ′ε(u), u〉 ≤ lim inf n→∞ ( Iε(un)− 1 q 〈I ′ε(un), un〉 ) = cε, which implies that the solution obtained above is a ground state solution, that is, cε = Iε(u). To complete the proof, it suffices to show that cε < c∞ for small ε while V∞ <∞. Without loss of generality, let us suppose that V (0) = V0 = inf x∈Rn V (x) and µ ∈ (V0, V∞). It is clear that c0 < cµ < c∞, where c0 is the infimum of IV0 over NV0 . By Theorem 3.4, there exists w ∈W s1,p(Rn)∩W s2,q(Rn) as a positive ground state of the autonomous problem (3.4). Let φ ∈ C∞c (Rn) be a cut-off function such EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 17 that 0 ≤ φ ≤ 1, φ = 1 in B1, and φ = 0 in Rn\B2. We set φr(x) = φ(xr ) and consider the function wr(x) = φr(x)w(x). Using Lemma 2.3 we see that lim r→∞ ‖wr − w‖ε = 0. (3.27) Take tr > 0 such that Iµ(trwr) = max t≥0 Iµ(twr), which leads to that trwr ∈ Nµ. Now we prove that there exists r large enough such that Iµ(trwr) < c∞. By contradiction we assume Iµ(trwr) ≥ c∞ for any r > 0. By using the assumption (A5), (3.27), trwr ∈ Nµ and w ∈ Nµ, we can infer that tr → 1. Therefore, we conclude that c∞ ≤ lim inf r→∞ Iµ(trwr) = Iµ(wr) = cµ < c∞, which leads to a contradiction. In addition, by assumption (1.6) we obtain that for some ε0 > 0, V (εx) ≤ µ for all ε ∈ (0, ε0). Hence, we deduce that cε ≤ max t≥0 Iε(twr) ≤ max t≥0 Iµ(twr) = Iµ(trwr) < c∞ for all ε ∈ (0, ε0), which completes the proof. � 4. Critical case while σ = 1 4.1. Functional setting in the critical case. In this section we focus on the critical case while σ = 1 (−∆)s1p u+ (−∆)s2q u+ V (εx)(|u|p−2u+ |u|q−2u) = λf(u) + |u|q ∗ s2 −2u in Rn, u ∈W s1,p(Rn) ∩W s2,q(Rn), u > 0 in Rn. (4.1) It is clear that the energy functional associated with (4.1) is Jε(u) = 1 p ‖u‖p1 + 1 q ‖u‖q2 − λ ∫ Rn F (u)dx− 1 q∗s2 |u| q∗s2 q∗s2 and its differential is given by 〈J ′ε(u), v〉 = 〈u, v〉s1,p + 〈u, v〉s2,q + ∫ Rn V (εx)(|u|p−2u+ |u|q−2u)v dx − λ ∫ Rn f(u)v dx− ∫ Rn |u|q ∗ s2 −2uv dx for any u, v ∈ Xε. We also introduce the Nehari manifold associated with Jε, Mε = {u ∈ Xε\{0} : 〈J ′ε(u), u〉 = 0}. It is easy to check that Jε possesses a mountain pass geometry shown as follows (cf. [1]). For simplicity, we here omit the proof because of its similarity to Lemma 3.1. Lemma 4.1. The functional Jε satisfies the following conditions: (i) there exists α, ρ > 0 such that Jε(u) ≥ α with ‖u‖ε = ρ; (ii) there exists e ∈ Xε with ‖e‖ε > ρ such that Jε(e) < 0. 18 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 In view of Lemma 4.1 we define the mountain pass level dε = inf ξ∈Λ max t∈[0,1] Jε(ξ(t)) = inf u∈Xε\{0} max t≥0 Jε(ξ(t)), where Λ = {ξ ∈ C0([0, 1], Xε) : ξ(0) = 0, Jε(ξ(1)) < 0}. Next we prove that any Palais-Smale sequence of Jε is bounded. Lemma 4.2. If {un} is a Palais-Smale sequence of Jε at level c, then {un} is bounded in Xε. Proof. Using assumption (A4) and q∗s2 > θ > q > p, we conclude that c(1 + ‖un‖ε) ≥ Jε(un)− 1 θ 〈J ′ε(un), un〉 = (1 p − 1 θ ) ‖un‖p1 + (1 q − 1 θ ) ‖un‖q2 + λ ∫ Rn (1 θ f(un)un − F (u) ) dx+ (1 θ − 1 q∗s2 ) |u| q∗s2 q∗s2 ≥ (1 q − 1 θ ) (‖un‖p1 + ‖un‖q2). Then we argue it as in the proof of Lemma 3.2, and we easily get the desired result. � 4.2. Autonomous critical problem. First, let us consider the autonomous prob- lem associated with Problem (4.1) as follows: (−∆)s1p u+ (−∆)s2q u+ µ ( |u|p−2u+ |u|q−2u ) = λf(u) + |u|q ∗ s2 −2u in Rn, µ > 0, u ∈W s1,p(Rn) ∩W s2,q(Rn), u > 0 in Rn. (4.2) Therefore, the corresponding energy functional is defined as Jµ(u) = 1 p ‖u‖pµ,1 + 1 q ‖u‖qµ,2 − λ ∫ Rn F (u)dx− 1 q∗s2 |u| q∗s2 q∗s2 , and its differential is given by 〈J ′µ(u), v〉 = 〈u, v〉s1,p + 〈u, v〉s2,q + µ ∫ Rn (|u|p−2u+ |u|q−2u)v dx − λ ∫ Rn f(u)v dx− ∫ Rn |u|q ∗ s2 −2uv dx for any u, v ∈ Yµ. Moreover, the Nehari manifold associated with Jµ is Mµ = {u ∈ Yµ\{0} : 〈J ′µ(u), u〉 = 0}. Arguing it as before, it is standard to check that Jµ has a mountain pass geometry, and we denote by dµ its mountain pass level. Remark 4.3. As in [36] we have the following variational characterization of the infimum of Jµ over Mµ: 0 < dµ = inf u∈Mµ Jµ(u) = inf ξ∈Λ max t∈[0,1] Jε(ξ(t)) = inf u∈Yµ\{0} max t≥0 Jµ(tu). To prove the existence of a nontrivial solution to Problem (4.2), we firstly need to prove the following fundamental result. EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 19 Lemma 4.4. There exists λ∗ > 0 such that dµ ∈ ( 0, s2n S n s2q ∗ ) for all λ ≥ λ∗. Proof. Let v ∈ C∞c (Rn) be a non-zero function such that v ≥ 0 in Rn. Then there exists tλ > 0 such that Jµ(tλv) = maxt≥0 Jµ(tv), which yields 〈J ′µ(tλv), tλv〉 = 0. Therefore, tpλ‖v‖ p µ,1 + tqλ‖v‖ q µ,2 = λ ∫ Rn f(tλv)tλv dx+ t q∗s2 λ |v| q∗s2 q∗s2 . (4.3) Using assumption (A4) we find that tpλ‖v‖ p µ,1 + tqλ‖v‖ q µ,2 ≥ t q∗s2 λ |v| q∗s2 q∗s2 , which combined with p < q < q∗s2 yields that tλ is bounded. Then there exists a subsequence {tλn} such that tλn → t0 ≥ 0 as λn → ∞. Now we prove t0 = 0 by contradiction. Let us suppose that t0 > 0 such that tpλn‖v‖ p µ,1 + tqλn‖v‖ q µ,2 → T ∈ (0,∞), λn ∫ Rn f(tλnv)tλnv dx+ t q∗s2 λn |v| q∗s2 q∗s2 →∞, which certainly goes against (4.3). Thus, tλn → 0 as λn →∞. Next, we write h(t) = tv for t ∈ [0, 1]. Then h ∈ Λ, and we obtain 0 < dµ ≤ max t∈[0,1] Jµ(h(t)) ≤ max t≥0 Jµ(tv) = Jµ(tλv) ≤ tpλ‖v‖ p µ,1 + tqλ‖v‖ q µ,2. Taking λ sufficiently large we obtain tpλ‖v‖ p µ,1 + tqλ‖v‖ q µ,2 < s2 n S n s2q ∗ , which completes the proof. � Lemma 4.5. Let t ∈ [ p, q∗s2 ) , and {un} ⊂ Mµ be a minimizing sequence for Jµ. Then, {un} is bounded in Yµ, and there exist a sequence {yn} ⊂ Rn and constants R, β > 0 such that lim inf n→∞ ∫ BR(yn) |un|tdx ≥ β > 0. Proof. It is easy to see that {un} is bounded in Yµ. We assume by contradiction that for any R > 0 it holds lim n→∞ sup y∈Rn ∫ BR(y) |un|tdx = 0. Then, it follows from Lemma 2.2 that un → 0 in Lt(Rn) for all t ∈ ( p, q∗s2 ) . (4.4) Employing (3.2), (3.3) and (4.4) we can infer that 0 ≤ ∫ Rn f(un)undx ≤ ξ ∫ Rn |un|p + on(1), 0 ≤ ∫ Rn F (un)dx ≤ C ξ ∫ Rn |un|p + on(1). Passing to the limit as ξ → 0, we obtain∫ Rn f(un)undx = on(1) and ∫ Rn F (un)dx = on(1), (4.5) 20 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 which combined with 〈J ′µ(un), un〉 = 0 yields ‖un‖pµ,1 + ‖un‖qµ,2 − |un| q∗s2 q∗s2 = on(1). Moreover, from the boundedness of {un} in Yµ we may assume that ‖un‖pµ,1 + ‖un‖qµ,2 → L ≥ 0 and |un| q∗s2 q∗s2 → L ≥ 0. (4.6) If L = 0, then ‖un‖µ → 0 as n→∞, which is a contradiction because of Jµ(un)→ dµ > 0. Therefore, in the following we assume that L > 0. By taking into account (4.5) and (4.6), we conclude that dµ = Jµ(un) + on(1) = 1 p ‖un‖pµ,1 + 1 q ‖un‖qµ,2 − λ ∫ Rn F (un)dx− 1 q∗s2 |un| q∗s2 q∗s2 ≥ 1 q L− 1 q∗s2 L+ on(1) = s2 n L+ on(1). (4.7) On the other hand, by using Lemma 2.1 it follows that |un|qq∗s2 ≤ S −1 ∗ ([un]qs2,q + µ|un|qq) = S−1 ∗ ‖u‖ q µ,2 ≤ S−1 ∗ (‖un‖pµ,1 + ‖un‖qµ,2). Passing to the limit as n→∞, we find that L q q∗s2 ≤ S−1 ∗ L, which together with (4.7) yields dµ ≥ s2 n S n s2q ∗ . This yields a contradiction in view of Lemma 4.4. Thus we completes the proof. � Let us now prove the existence result for the autonomous critical case. Theorem 4.6. Under assumptions (A1)–(A5), Problem (4.2) admits a positive ground state solution. Proof. This proof follows the argument developed as in Theorem 3.4. We here need to replace (3.9) by dµ + on(1) = Jµ(un)− 1 q 〈J ′µ(un), un〉 = (1 p − 1 q ) ‖un‖pµ,1 + λ ∫ Rn (1 q f(un)un − F (un) ) dx+ (1 q − 1 q∗s2 ) |u| q∗s2 q∗s2 . Recalling that lim sup n→∞ (an + bn + cn) ≥ lim sup n→∞ an + lim inf n→∞ (bn + cn) ≥ lim sup n→∞ an + lim inf n→∞ bn + lim inf n→∞ cn, which implies that dµ ≥ (1 p − 1 q ) lim sup n→∞ ‖un‖pµ,1 + λ lim inf n→∞ ∫ Rn (1 q f(un)un − F (un) ) dx + (1 q − 1 q∗s2 ) lim inf n→∞ |u| q∗s2 q∗s2 . EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 21 We complete the proof by using Lemma 4.5 instead of Lemma 3.3. � 4.3. Proof of Theorem 1.2. By a similar argument as Lemma 4.5, the critical version of Lemma 3.5 is presented here. Lemma 4.7. Assume that dε < s2 n S n s2q ∗ . Let t ∈ [ p, q∗s2 ) , and {un} ⊂ Mε be a sequence such that Jε(un) → dε and un ⇀ 0 in Xε. Then, one of the following alternatives occurs: (i) un → 0 in Xε; (ii) there exist a sequence {yn} ⊂ Rn and constants R, β > 0 such that lim inf n→∞ ∫ BR(yn) |un|tdx ≥ β > 0. For the case of V∞ < ∞, we immediately obtain the following result along the lines of the proof of Lemma 3.6. Lemma 4.8. Assume that V∞ < ∞, and let {vn} ⊂ Mε be a sequence such that Jε(vn) → d and vn ⇀ 0 in Xε. If vn 6→ 0 in Xε, then d ≥ d∞, where d∞ is the infimum of JV∞ over MV∞ . Next, we can give the following compactness result in the critical case. Lemma 4.9. Let {un} ⊂ Mε be such that Jε(un) → dε, where dε < d∞ for V∞ <∞, and dε < s2 n S n s2q ∗ for V∞ =∞. Then {un} has a convergent subsequence in Xε. Proof. We first argue it as in the proof of Lemma 3.7 so that we know that {un} is bounded in Xε, which yields that we may assume that un ⇀ u in Xε. It is clear that J ′ε(u) = 0. Let vn = un − u and Jε(vn) → d. By Brezis-Lieb Lemma in [11] and [28, Lemma 3.3] we find that |vn| q∗s2 q∗s2 = |un| q∗s2 q∗s2 − |u| q∗s2 q∗s2 + on(1). We replace (3.25) by d+ on(1) = Jε(vn) = Jε(un)− Jε(u) + on(1) = dε − Jε(u) + on(1) and notice that Jε(u) = Jε(u)− 1 q 〈J ′ε(u), u〉 = 1 p ‖u‖p1 + λ ∫ Rn (1 q f(u)u− F (u) ) dx+ (1 q − 1 q∗s2 ) |u| q∗s2 q∗s2 ≥ 0, where we used assumption (A4) in the last inequality. Therefore, for the case of V∞ <∞ we deduce that d ≤ dε < d∞, which together with Lemma 4.8 yields vn → 0 in Xε, that is to say, un → u in Xε. For the case of V∞ =∞, by using Lemma 2.5 we can infer that vn → 0 in Lt(Rn) for any t ∈ [ p, q∗s2 ) . This combined with assumptions (A2) and (A3) implies that∫ Rn f(vn)vn dx = on(1), 22 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 which together with 〈J ′ε(vn), vn〉 = 0 yields ‖vn‖p1 + ‖vn‖q2 − |vn| q∗s2 q∗s2 = on(1). Considering the boundedness of {vn}, we may assume that ‖vn‖p1 + ‖vn‖q2 → L ≥ 0 and |vn| q∗s2 q∗s2 → L ≥ 0. If L > 0, we follow the lines of the proof of Lemma 4.5 to get d ≥ s2 n S n s2q ∗ . Note that d ≤ dε < s2 n S n s2q ∗ , which yields a contradiction. This leads to that L = 0, which completes the proof. � We are finally ready to give the proof of Theorem 1.2. Proof of Theorem 1.2. The proof follows the same lines as in the proof of Theorem 1.1, once one replace Lemma 3.1, Lemma 3.2, Theorem 3.4 and Lemma 3.7 by Lemma 4.1, Lemma 4.2, Theorem 4.6 and Lemma 4.9, respectively. � Acknowledgements. This research was supported by National Natural Science Foundation of China 12071021. References [1] A. Ambrosetti, P. H. Rabinowitz; Dual variational methods in critical point theory and applications, J. Functional Anal., 14 (1973), 349–381. [2] V. Ambrosio; Fractional p&q Laplacian problems in RN with critical growth, Z. Anal. An- wend., 39 (3) (2020), 289–314. [3] V. Ambrosio, T. Isernia; Multiplicity and concentration results for some nonlinear Schrödinger equations with the fractional p-Laplacian, Discrete Contin. Dyn. Syst., 38 (11) (2018), 5835– 5881. [4] V. Ambrosio, D. Repovš; Multiplicity and concentration results for a (p, q)-Laplacian problem in RN , Z. Angew. Math. Phys., 72 (33) (2021), art. 33. [5] C. O. Alves, V. Ambrosio, T. Isernia; Existence, multiplicity and concentration for a class of fractional p&q Laplacian problems in RN , Commun. Pure Appl. Anal., 18 (4) (2019), 2009–2045. [6] S. C. Q. Arruda, R. G. Nascimento; Existence and multiplicity of positive solutions for singular p&q-Laplacian problems via sub-supersolution method, Electron. J. Differ. Eqs., 2021 (2021), art. 25, 1–11. [7] P. Baroni, M. Colombo, G. Mingione; Regularity for general functionals with double phase, Calc. Var. Partial Differ. Equ., 57 (2) (2018), art. 62. [8] L. Beck, G. Mingione; Lipschitz bounds and nonuniform ellipticity, Commun. Pure Appl. Math., 73 (5) (2020), 944–1034. [9] M. Bhakta, D. Mukherjee; Multiplicity results for (p, q) fractional elliptic equations involving critical nonlinearities, Adv. Differ. Equ., 24 (3-4) (2019), 185–228. [10] L. Brasco, S. Mosconi, M. Squassina; Optimal decay of extremals for the fractional Sobolev inequality, Calc. Var. Partial Differ. Equ., 55 (2) (2016), Art. 23. [11] H. Brezis, E. Lieb; A relation between pointwise convergence of functions and convergence of functionals, Proc. Amer. Math. Soc., 88 (3) (1983), 486–490. [12] L. Caffarelli; Non-local diffusions, drifts and games, Nonlinear partial differential equations, Abel Symp., 7 (2012), 37–52. [13] P. Candito, S. A. Marano, K. Perera; On a class of critical (p, q)-Laplacian problems, NoDEA Nonlinear Differential Equations Appl., 22 (2015), 1959–1972. [14] A. D. Castro, T. Kuusi, G. Palatucci; Local behavior of fractional p-minimizers, Ann. Inst. H. Poincaré Anal. Non Lineaire, 33 (5) (2016), 1279–1299. [15] X. Chang, Z.-Q. Wang; Ground state of scalar field equations involving a fractional laplacian with general nonlinearity, Nonlinearity, 26 (2013), 479–494. EJDE-2021/100 EXISTENCE RESULTS FOR NONLINEAR SCHRÖDINGER EQUATIONS 23 [16] L. Cherfils, Y. Il’yasov; On the stationary solutions of generalized reaction diffusion equations with p&q-Laplacian, Commun. Pure Appl. Anal., 4 (1) (2005), 9–22. [17] M. Colombo, G. Mingione; Bounded minimisers of double phase variational integrals, Arch. Ration. Mech. Anal., 218 (1) (2015), 219–273. [18] M. Colombo, G. Mingione; Regularity for double phase variational problems, Arch. Ration. Mech. Anal., 215 (2) (2015), 443–496. [19] M. E. Filippakis, N. S. Papageorgiou; Resonant (p, q)-equations with Robin boundary condi- tion, Electron. J. Differ. Eqs., 2018 (2018), art. 1, 1–24. [20] C. D. Filippis, G. Palatucci; Hölder regularity for nonlocal double phase equations, J. Differ. Equ., 267 (1) (2019), 547–586. [21] R. L. Frank, E. Lenzmann, L. Silvestre; Uniqueness of radial solutions for the fractional Laplacian, Commun. Pure Appl. Math., 69 (2016), 1671–1726. [22] G. Gilboa, S. Osher; Nonlocal operators with applications to image processing, Multiscale Model. Simul., 7 (3) (2008), 1005–1028. [23] S. Levendorski; Pricing of the American put under Levy processes, Int. J. Theor. Appl. Finance, 7 (3) (2004), 303–335. [24] Q. Li, Z. Yang, Z. Feng; Multiple solutions of a p-Kirchhoff equation with singular and critical nonlinearities, Electron. J. Differ. Eqs., 2017 (2017), art. 84, 1–14. [25] X. Lin, S. Zheng; Multiplicity and asymptotic behavior of solutions to fractional (p, q)- Kirchhoff type problems with critical Sobolev-Hardy exponent, Electron. J. Differ. Eqs., 2021 (2021), art. 66, 1–20. [26] P. Marcellini; Regularity and existence of solutions of elliptic equations with p, q-growth conditions, J. Differ. Equ., 90 (1) (1991), 1–30. [27] P. Marcellini; Regularity under general and p, q-growth conditions, Discrete Contin. Dyn. Syst., 13 (7) (2020), 2009–2031. [28] C. Mercuri, M. Willem; A global compactness result for the p-Laplacian involving critical nonlinearities, Discrete Contin. Dyn. Syst., 28 (2) (2010), 469–493. [29] S. Mosconi, K. Perera, M. Squassina, Y. Yang; The Brezis-Nirenberg problem for the frac- tional p-Laplacian, Calc. Var. Partial Differ. Equ., 55 (4) (2016), art. 105. [30] D. Mugnai, N. S. Papageorgiou; Wang’s multiplicity result for superlinear (p, q)-equations without the Ambrosetti-Rabinowitz condition, Trans. Amer. Math. Soc., 366 (9) (2014), 4919–4937. [31] E. D. Nezza, G. Palatucci, E. Valdinoci; Hitchhiker’s guide to the fractional Sobolev spaces, Bull. Sci. Math., 136 (5) (2012), 521–573. [32] L. M. D. Pezzo, A. Quaas; A Hopf’s lemma and a strong minimum principle for the fractional p-Laplacian, J. Differ. Equ., 263 (1) (2017), 765–778. [33] L. M. D. Pezzo, A. Quaas; Spectrum of the fractional p-Laplacian in RN and decay estimate for positive solutions of a Schrödinger equation, Nonlinear Anal., 193 (2020), art. 111479. [34] P. Rabinowitz; On a class of nonlinear Schrödinger equations, Z. Angew. Math. Phys., 43 (2) (1992), 270–291. [35] M. Souza; On a class of nonhomogeneous fractional quasilinear equations in Rn with expo- nential growth, NoDEA Nonlinear Differential Equations Appl., 22 (2015), 499–511. [36] A. Szulkin, T. Weth; The method of Nehari manifold, Handbook of nonconvex analysis and applications, Int. Press, 2010. [37] M. Willem; Minimax Theorems, Progress in Nonlinear Differential Equations and their Ap- plications, Vol. 24, Birkhäuser Inc., Boston, 1996. [38] M. Xiang, B. Zhang, V. D. Rǎdulescu; Superlinear Schrödinger-Kirchhoff type problems involving the fractional p-Laplacian and critical exponent, Adv. Nonlinear Anal., 9 (1) (2020), 690–709. [39] J. Yang, H. Chen, Z. Feng; Multiple positive solutions to the fractional Kirchhoff problem with critical indefinite nonlinearities, Electron. J. Differ. Eqs., 2020 (2020), art. 101, 1–21. [40] V. V. Zhikov; Averaging of functionals of the calculus of variations and elasticity theory, Izv. Akad. Nauk SSSR, Ser. Mat., 50 (4) (1986), 675–710. Huilin Lv Department of Mathematics, Beijing Jiaotong University, Beijing 100044, China Email address: huilinlv@bjtu.edu.cn 24 H. LV, S. ZHENG, Z. FENG EJDE-2021/100 Shenzhou Zheng (corresponding author) Department of Mathematics, Beijing Jiaotong University, Beijing 100044, China Email address: shzhzheng@bjtu.edu.cn Zhaosheng Feng School of Mathematical and Statistical Sciences, University of Texas Rio Grande Val- ley, Edinburg, TX 78539, USA Email address: zhaosheng.feng@utrgv.edu 1. Introduction 2. Preliminaries 3. Subcritical case while =0 3.1. Functional setting in the subcritical case 3.2. Autonomous subcritical problem 3.3. Proof of Theorem ?? 4. Critical case while =1 4.1. Functional setting in the critical case 4.2. Autonomous critical problem 4.3. Proof of Theorem ?? Acknowledgements References