Electronic Journal of Differential Equations, Vol. 2020 (2020), No. 01, pp. 1–17. ISSN: 1072-6691. URL: http://ejde.math.txstate.edu or http://ejde.math.unt.edu FRACTIONAL SCHRÖDINGER-POISSON SYSTEMS WITH WEIGHTED HARDY POTENTIAL AND CRITICAL EXPONENT YU SU, HAIBO CHEN, SENLI LIU, XIANWEN FANG Communicated by Claudianor O. Alves Abstract. In this article we consider the fractional Schrödinger-Poisson sys- tem (−∆)su− µ Φ(x/|x|) |x|2s u+ λφu = |u|2 ∗ s−2u, in R3, (−∆)tφ = u2, in R3, where s ∈ (0, 3/4), t ∈ (0, 1), 2t + 4s = 3, λ > 0 and 2∗s = 6/(3 − 2s) is the Sobolev critical exponent. By using perturbation method, we establish the existence of a solution for λ small enough. 1. Introduction In this article we consider the fractional Schrödinger-Poisson system (−∆)su− µΦ(x/|x|) |x|2s u+ λφu = |u|2 ∗ s−2u, in R3, (−∆)tφ = u2, in R3, (1.1) where s ∈ (0, 3/4), t ∈ (0, 1), 2t+ 4s = 3, λ > 0 and 2∗s = 6/(3− 2s) is the Sobolev critical exponent. The function Φ and the parameter µ satisfy the condition (A1) If 0 6 Φ ∈ L 3 2s (S2) then µ ∈ (0,Λ(Φ)), where Λ(Φ) is defined in Lemma 2.1 below. In quantum mechanics a zero spin relativistic particle of charge e and mass m in the Coulomb field of an infinitely heavy nucleus of charge Z is described by the Hamiltonian (see e.g. [23, 28]) H(p, x) = (p2 +m2)1/2 − Ze2 |x| . Fall and Felli [16], extended the study of H(p, x) to H̃(p, x) = (p2 +m2)s − Φ( x |x| ) |x|2s . 2010 Mathematics Subject Classification. 35B38, 35J47. Key words and phrases. Fractional Schrödinger-Poisson system; weighted Hardy potential; critical exponent. c©2020 Texas State University. Submitted October 24, 2019. Published January 6, 2020. 1 2 Y. SU, H. CHEN, S. LIU, X. FANG EJDE-2020/01 They considered the operator H̃(i∇, x) = (−∆ +m2)s − Φ( x |x| ) |x|2s . For Φ ≡ 0, the operator H̃(i∇, x) becomes (−∆ +m2)s, for more details, we refer to [2, 3]. In this article, we study H̃(i∇, x), i. e. when m = 0. In nonrelativistic molecular physics, the interaction between an electric charge and the dipole moment D ∈ RN of a molecule is described by an inverse square potential with an anisotropic coupling strength. In particular the Schrödinger equa- tion for the wave function of an electron interacting with a polar molecule (supposed to be point-like) can be written as H = − ~ 2m ∆ + e x ·D |x|3 − E, where D is the dipole moment of the molecule, e and m denote the charge and the mass of the electron (see [26]). We consider the operator Ld := −∆− 2me|D| ~ x · d |x|3 , in RN , where N > 3, being |D| the magnitude of the dipole moment D, and d = D |D| denotes the orientation of D. The Laplace operator with dipole-type potential (purely angular multiples of radial inverse square potentials): LΦ := −∆− µΦ(x/|x|) |x|2 , in RN , where N > 3. We consider the more general class of operator H̃(i∇, x) with zero mass, Ls,Φ := (−∆)s − µΦ(x/|x|) |x|2s . If Φ ≡ 0, then Ls,Φ becomes the fractional Laplacian operator (−∆)s. If Φ ≡ 1, then Ls,Φ is the fractional Laplacian operator with Hardy potential (−∆)s − µ |x|2s . If s = 1, then Ls,Φ becomes LΦ. From the mathematical point of view, a reason of interest in potentials of the type Φ( x |x| )/|x| 2s relies in their criticality with respect to the differential operator (−∆)s; indeed, they have the same homogeneity as the fractional Laplacian (−∆)s, hence they cannot be regarded as a lower order perturbation term. We mention that the operator with singular potentials have been widely studied, see for example [1, 13, 14, 15, 17, 18, 19, 20, 21, 34, 36, 38, 39] and references therein. On the other hand, if µ = 0 and λ = 1, then system (1.1) becomes the system: (−∆)su+ φu = |u|2 ∗ s−2u, in R3, (−∆)tφ = u2, in R3. (1.2) For this system there are four cases: (i) s = t = 1; (ii) s = 1 and t 6= 1; (iii) s = t 6= 1; (iv) s 6= 1, t 6= 1, and 2t+ 4s > 3. (i) For s = t = 1, system (1.2) reduces to the Schrödinger–Poisson system −∆u+ φu = |u|p−2u, in R3, −∆φ = u2, in R3. (1.3) EJDE-2020/01 FRACTIONAL SCHRÖDINGER-POISSON SYSTEM 3 The Schrödinger-Poisson system arises in many mathematical physical context, such as in quantum electrodynamics, to describe the interaction between a charge particle interacting with the electromagnetic field, and in semiconductor theory, in nonlinear optics and in plasma physics (see [8] for more details in the physics aspects). System (1.3) has been investigated by many researchers, see for example: [6, 11] for |u|p−2u (4 < p < 6), [47] for |u|p−2u (3 < p < 4), [40] for |u|p−2u (1 < p < 2). (ii) For s = 1 and t 6= 1, Mercuri, Moroz and Van Schaftingen [32] studied the Schrödinger–Poisson–Slater equation −∆u+ (Iα ∗ |u|q) |u|q−2u = |u|p−2u, in R3, (1.4) where p > 1, q > 1, α ∈ (0, N), and Iα = Γ( 3−α 2 ) 2απ 3 2 Γ(α2 ) · 1 |x|3−α is the Riesz potential. Equation (1.4) is formally equivalent to the following system (see [32, Page 146, Line 10]): −∆u+ φ|u|q−2u = |u|p−2u, in R3, (−∆)α/2φ = |u|q, in R3. (1.5) The existence of nonnegative ground state solution of (1.4) was established when the parameters satisfy: either q > 3 + α and 2(α+ 2q) α+ 2 < p < 6, or q < 3 + α and 6 < p < 2(α+ 2q) α+ 2 . See [32, Theorem 3]. Moreover, some qualitative properties of the solutions were estudied. Li, Gao and Zhu [27] proved the existence of sign-changing solution of Kirchhoff type system with Hartree-type nonlinearity −(a+ b ∫ R3 |∇u|2dx)∆u+ λV (x)u+ φ|u|q−2u = f(u), in R3, (−∆)α/2φ = l|u|q, in R3, (1.6) where a > 0, b, l > 0, α ∈ (0, 3), q ∈ [2, 3 + α), λ > 0 is a parameter, and the functions V and f satisfy suitable assumptions. For recent works, we refer to [4] and the references therein. (iii) For s = t 6= 1, the author in [43] applied the fountain theorem to prove the existence of infinitely many solutions to the system (−∆)su+ V (x)u+ φu = f(x, u), in R3, (−∆)sφ = γsu 2, in R3. (iv) For s 6= 1, t 6= 1 and 2t+ 4s > 3, Zhang, do Ó and Squassina [46] investigated the system (−∆)su+ λφu = f(u), in R3, (−∆)tφ = λu2, in R3, (1.7) where f is a nonlinearity of Berestycki-Lions type. The authors proved that system (1.7) admits a positive radial solution if λ > 0 small enough. Liu and Zhang [30] 4 Y. SU, H. CHEN, S. LIU, X. FANG EJDE-2020/01 proved the existence of positive ground state solution for ε > 0 sufficiently small to the system ε2s(−∆)su+ V (x)u+ φu = f(u) + |u|2 ∗ s−2u, in R3, ε2t(−∆)tφ = u2, in R3, where V ∈ C1(R3,R+) and f is subcritical. For more details and recent works, we refer to [5, 31, 37, 41, 42] and the references therein. In [30, 41, 42, 46] the authors considered the existence of solution for system (1.7) in which 2t + 4s > 3. Therefore, it is natural to ask whether system (1.7) admits a solution for 2t+ 4s = 3. To the best of our knowledge, there is no result on such question in the current literature. Motivated by the above facts, we study the existence of solution for system (1.1). 1.1. Statement of results. We introduce the energy functional associated with system (1.1) by Iλ(u) = 1 2 ‖u‖2Φ + λ 4 ∫ R3 ∫ R3 |u(x)|2|u(y)|2 |x− y|3−2t dxdy − 1 2∗s ∫ R3 |u|2 ∗ sdx. where the norm ‖ · ‖Φ is defined in Section 2. There is a difficulty in applying variational methods to the functional Iλ. (Difficulty) Since 2∗s < 4, we know that the functional Iλ does not satisfy Mountain Pass Theorem, the boundedness of the Palais-Smale sequence for Iλ is hard to obtain. To overcome this difficult, by using the perturbation method in [12, 25], for λ small enough, we look the system (1.1) as a perturbation of the equation (−∆)su− µΦ(x/|x|) |x|2s u = |u|2 ∗ s−2u, in R3. (1.8) We introduce the energy functional associated with equation (1.8) by I0(u) = 1 2 ‖u‖2Φ − 1 2∗s ∫ R3 |u|2 ∗ sdx. For λ small enough, we look at the functional Iλ as a perturbation of functional I0, Iλ(u) := I0(u) +Kλ(u), where Kλ(u) = λ 4 ∫ R3 ∫ R3 |u(x)|2|u(y)|2 |x− y|3−2t dxdy. Let S be the set of ground state critical points of I0. The perturbation method mainly consists of two aspects: (1) the mountain pass type critical point of I0 is a ground state in Ds,2 rad(R3); (2) the set S is compact in Ds,2 rad(R3). If conditions (1) and (2) are satisfied, then there exists a Palais-Smale sequence of Iλ near the set S for sufficiently small λ. Since the perturbation method is based on the properties of set S, we have to study the existence of ground state solution to equation (1.8) first. The following is our first result. Theorem 1.1. Let s ∈ (0, 3/4) and (Φ1) hold. Then (1.8) has a ground state solution v ∈ Ds,2 rad(R3). EJDE-2020/01 FRACTIONAL SCHRÖDINGER-POISSON SYSTEM 5 There is a difficulty in the proof of Theorem 1.1. Because of the lack of com- pactness of the Sobolev embedding Ds,2(R3) ↪→ L2∗ s (R3), and the functional I0 is invariant under the weighted dilation, it is hard to show that the Palais-Smale sequence of I0 has a convergence subsequence. For Φ ≡ 1, let us review the method in [45]. Applying their method, one has the existence of ground state solution to the equation (−∆)su− µ u |x|2s = |u|2 ∗ s−2u, in R3. (1.9) Similar to Section 3 in [45], there exists a Palais-Smale sequence {un} for the energy functional corresponding to (1.9) such that un ⇀ u in Ds,2(R3) with u solving (1.9). However, it may occur that u ≡ 0. The key step of [45] was to rule out the ”vanishing” of the Palais-Smale sequence by using the limit equation of (1.9). Taking vn(x) = λ 3−2s 2 n un(λnx+ xn) where λn > 0, xn ∈ R3 and xn λn →∞ as n→ +∞, they derived that vn ⇀ v in Ds,2(R3) and∫ R3 vkφ |x+ xk λk |2s → 0 as k → +∞ (1.10) for any φ ∈ Ds,2(R3). Then v weakly solves (−∆)sv = |v|2 ∗ s−2v, in R3. (1.11) Using the limit equation (1.11), they ruled out the “vanishing” of the Palais-Smale sequence for the energy functional corresponding to (1.9). Naturally, we would hope to overcome our difficulty by using the method in [45], but unfortunately, for Φ satisfies condition (Φ1), the behavior of lim n→∞ Φ ( x+ xn λn |x+ xn λn | ) , as xn λn →∞, is unknown, so we do not have any information of the expression lim n→∞ ∫ R3 Φ ( x+ xn λn |x+ xn λn | ) vnϕ |x+ xn λn |2s dx. (1.12) Clearly, their method does not apply to our case, since the expression (1.12) is not equal to zero. For the above reason, we use another way to prove Theorem 1.1. The cru- cial point is the utilization of the uniform decay estimate for radial function v ∈ Ds,2 rad(R3) (see [9]) |v(x)| 6 C |x| 3−2s 2 ‖v‖s 6 C |x| 3−2s 2 . We consider the function I0 : Ds,2 rad(R3)→ R defined by J0 = I0|Ds,2rad(R3). The principle of symmetric criticality implies that the critical points of J0 are also critical points for I0. Moreover, in Theorem 1.1 we have the following results: (1) the mountain pass type critical point of I0 is a ground state in Ds,2 rad(R3); (2) the set S is compact in Ds,2 rad(R3). Combining the above results and the perturbation method, we give our second result as follows. 6 Y. SU, H. CHEN, S. LIU, X. FANG EJDE-2020/01 Theorem 1.2. Let s ∈ (0, 3/4), t ∈ (0, 1), 2t+ 4s = 3 and (Φ1) hold. Then there exists λ0 such that system (1.1) has a radially symmetric solution for all λ ∈ (0, λ0). This article is organized as follows. In Section 2, we present some notations. In Section 3, we give the proof of Theorem 1.1. In Section 4, we show the proof of Theorem 1.2. 2. Preliminaries For s ∈ (0, 1), the space Ds,2(R3) is the completion of C∞0 (R3) with respect to the norm ‖u‖2s = ∫ R3 ∫ R3 |u(x)− u(y)|2 |x− y|3+2s dxdy. We denote by Ds,2 rad(R3) the space of radial functions in Ds,2(R3). We could define the best constants Ss := inf u∈Ds,2(R3)\{0} ‖u‖2s ( ∫ R3 |u|2∗ sdx) 2 2∗s . (2.1) We know that Ss can be attained in R3 (see [33]). For u ∈ Ds,2(R3), we have the Hardy inequality (see [22]): Cs,2 ∫ R3 |u|2 |x|2s dx 6 ∫ R3 ∫ R3 |u(x)− u(y)|2 |x− y|3+2s dxdy, (2.2) with Cs,2 := 2 ∫ 1 0 r2s−1|1− r 3−2s 2 |2Φs,2(r)dr. We introduce the measure dϑ induced by Lebesgues measure on the unit sphere S2 ⊂ R3. We denote by ‖ · ‖Lq(S2) the quantity ‖Φ‖qLq(S2) = ∫ S2 |Φ(ϑ)|qdϑ. Lemma 2.1 ([24]). Let s ∈ (0, 1) and 0 6 Φ ∈ L 3 2s (S2). Then∫ R3 ∫ R3 |u(x)− u(y)|2 |x− y|3+2s dxdy > Λ(Φ) ∫ R3 Φ(x/|x|)|u|2 |x|2s dx, where u ∈ Ds,2(R3) and Λ(Φ) = Cs,2|S2| 2s 3 ‖Φ‖−1 L 3 2s (S2) . By using Lemma 2.1 and µ ∈ (0,Λ(Φ)), ‖u‖Φ = (∫ R3 ∫ R3 |u(x)− u(y)|2 |x− y|3+2s dxdy − µ ∫ R3 Φ(x/|x|)|u|2 |x|2s dx )1/2 is an equivalent norm in Ds,2(RN ). Lemma 2.2 ([29]). Let t, r > 1 and 0 < α < 3 with 1 t + 1 r + 3−α 3 = 2, f ∈ Lt(R3) and h ∈ Lr(R3). There exists a sharp constant C(α, t, r) > 0, independent of f, h such that ∫ R3 ∫ R3 |f(x)||h(y)| |x− y|3−α dxdy 6 C(α, t, r)‖f‖t‖h‖r. If t = r = 6 3+α , then C(α, t, r) = C(α) = π 3−α 2 Γ(α2 ) Γ(3 + α 2 ) {Γ( 3 2 ) Γ(3) }−α/3 . EJDE-2020/01 FRACTIONAL SCHRÖDINGER-POISSON SYSTEM 7 A measurable function u : R3 → R belongs to the Morrey space Lq,$(R3) with q ∈ [1,∞) and $ ∈ (0, N ] if ‖u‖qLq,$(R3) = sup R>0,x∈R3 R$−3 ∫ B(x,R) |u(y)|qdy <∞. Lemma 2.3 ([35]). For s ∈ (0, 1), there exists C > 0 such that for ι and ϑ satisfying 2 2∗ s 6 ι < 1, 1 6 ϑ < 2∗s, we have (∫ R3 |u|2 ∗ sdx )1/(2∗ s) 6 C‖u‖ιs‖u‖1−ι Lϑ, ϑ(N−2s) 2 (R3) , for any u ∈ Ds,2(R3). 2.1. Formulation of system (1.1). Considering u ∈ Ds,2(R3), we define the linear functional Fu(v) : Dt,2(R3)→ R by Fu(v) = ∫ R3 u2vdx. Using 2t+ 4s = 3, Holder’s inequality, and (1.9), we obtain |Fu(v)| 6 ∫ R3 |u| 6−4s 3−2s |v|dx = ∫ R3 |u| 3+2t 3−2s |v|dx 6 (∫ R3 |u| 3+2t 3−2s · 6 3+2t dx ) 3+2t 6 (∫ R3 |v| 6 3−2t dx ) 3−2t 6 6 ( 1 Ss ‖u‖2s ) 3+2t 6 ( 1 St ‖v‖2t ) 3−2t 6 , which implies that Fu is well defined and continuous in Dt,2(R3). From the Lax-Milgram theorem it follows that, for every u ∈ Ds,2(R3), there exists a unique φu,t ∈ Dt,2(R3) such that∫ R3 (−∆)t/2φu,t(−∆)t/2vdx = ∫ R3 u2vdx, which implies that φu,t is a weak solution of (−∆)tφu,t = u2, and the representation formula holds, φu,t = ∫ R3 |u(y)|2 |x− y|3−2t dy. (2.3) It follows from (2.3) that system (1.1) can be rewritten as the equivalent form (−∆)su− µΦ(x/|x|) |x|2s u+ λ ( I2t ∗ |u|2 ) u = |u|2 ∗ s−2u in R3. 8 Y. SU, H. CHEN, S. LIU, X. FANG EJDE-2020/01 3. Perturbed equation We look at equation (1.1) as a perturbation of the equation (−∆)su− µΦ(x/|x|) |x|2s u = |u|2 ∗ s−2u, in R3. (3.1) We introduce the energy functional associated with (1.8) by I0(u) = 1 2 ‖u‖2Φ − 1 2∗s ∫ R3 |u|2 ∗ sdx. Define J0 = I0|Ds,2rad(R3). 3.1. Mountain pass geometry and Nehari manifold. Lemma 3.1. Under the assumptions of Theorem 1.1, the functional J0 possesses the mountain pass geometry. Proof. We prove that J0 satisfies all the conditions in Mountain Pass Theorem. (i) For any u ∈ Ds,2 rad(R3) \ {0}, we have J0(u) > 1 2 ‖u‖2Φ − 1 2∗s · S 2∗ s s ‖u‖2 ∗ s Φ . Because of 2 < 2∗s, there exists a sufficiently small positive number ρ such that ς := inf ‖u‖s=ρ J0(u) > 0 = J0(0). (ii) Given u ∈ Ds,2 rad(R3) \ {0}, we have J0(tu) = t2 2 ‖u‖2Φ − t2 ∗ s 2∗s ∫ R3 |u|2 ∗ sdx < 0, for t large enough. We choose tu > 0 corresponding to u such that J0(tu) < 0 for all t > tu and ‖tuu‖s > ρ. � Define c0 = inf Υ∈Γ0 max t∈[0,1] J0(Υ(t)), where Γ0 = {Υ ∈ C([0, 1], Ds,2 rad(R3))|Υ(0) = 0, J0(Υ(1)) < 0}. By Lemma 3.1, we obtain that for all λ > 0, there exists {un} ⊂ Ds,2 rad(R3) such that J0(un)→ c0 > 0 and J ′0(un)→ 0 as n→∞. It is also easy to see that {un} is uniformly bounded in Ds,2 rad(R3). The Nehari manifold on Ds,2 rad(R3) is defined by N = {u ∈ Ds,2 rad(R3)|〈J ′0(u), u〉 = 0, u 6= 0}, ¯̄c0 = inf u∈N J0(u), c̄0 = inf u∈Ds,2rad(R3) max t>0 J0(tu). With minor changes in the proof of [44, Theorem 4.2], we can show that ¯̄c0 = c̄0 = c0. EJDE-2020/01 FRACTIONAL SCHRÖDINGER-POISSON SYSTEM 9 The following result implies the non-vanishing property of (PS)c0 sequence. Lemma 3.2. Assume that the assumptions of Theorem 1.1 hold. Let {un} be a (PS)c0 sequence of J0 with c0 > 0. Then lim n→∞ ∫ R3 |un|2 ∗ sdx > 0. Proof. It is easy to see that {un} is uniformly bounded in Ds,2 rad(R3). Then there exists a constant 0 < C <∞ such that ‖un‖Φ 6 C. Suppose on the contrary that lim n→∞ ∫ R3 |un|2 ∗ sdx = 0. (3.2) According to (3.2) and the definition of (PS)c0 sequence, we obtain c0 + o(1) = 1 2 ‖un‖2Φ and ‖un‖2Φ = o(1); these imply that c0 = 0, which contradicts with 0 < c0. � 3.2. Ground state solution. Proof of Theorem 1.1. We divide the proof of Theorem 1.1 into two Steps. Step 1. Since {un} is a bounded sequence in Ds,2 rad(R3), up to a subsequence, we assume that un ⇀ u in Ds,2 rad(R3), un → u a.e. in R3, un → u in Lrloc(R3) for all r ∈ [2, 2∗s). According to Lemmas 2.3 and 3.2, there exists C > 0 such that for any n, we obtain ‖un‖L2,3−2s(R3) > C > 0. On the other hand, since {un} is bounded in Ds,2 rad(R3) and Ds,2 rad(R3) ↪→ Ds,2(R3) ↪→ L2∗ s (R3) ↪→ L2,3−2s(R3), we have ‖un‖L2,3−2s(R3) 6 C, for some C > 0 independent of n. Hence, there exists a positive constant which we denote again by C such that for any n, we obtain C 6 ‖un‖L2,3−2s(R3) 6 C −1. Combining the definition of Morrey space and above inequality, we deduce that for any n ∈ N there exist σn > 0 and xn ∈ R3 such that 1 σ2s n ∫ B(xn,σn) |un(y)|2dy > ‖un‖2L2,3−2s(R3) − C 2n > C1 > 0. Let vn(x) = σ 3−2s 2 n un(σnx). We know that J0(vn) = J0(un)→ c0 and J ′0(vn)→ 0 as n→∞. Thus there exists v such that vn ⇀ v in Ds,2 rad(R3), vn → v a.e. in R3, vn → v in Lrloc(R3) for all r ∈ [2, 2∗s). Then∫ B( xnσn ,1) |vn(y)|2dy = 1 σ2s n ∫ B(xn,σn) |un(y)|2dy > C1 > 0. (3.3) 10 Y. SU, H. CHEN, S. LIU, X. FANG EJDE-2020/01 Step 2. We will show that {xnσn } is bounded. Suppose on the contrary that xn σn →∞ as n → ∞. By using the boundedness of {un} in Ds,2 rad(R3), we have ‖vn‖s = ‖un‖s 6 C2. Applying the uniform decay estimates of radial functions, we have |vn(x)| 6 C |x| 3−2s 2 ‖vn‖s 6 C3 |x| 3−2s 2 , where C3 = CC2. For any √ C1 |B(0,1)| > ε > 0, there exists M > 0, such that for n > M , we have |vn(x)| 6 C3 |xnσn − 1| 3−2s 2 6 ε, for all x ∈ Bc(0, |xn σn − 1|). Since B(xnσn , 1) ⊂ Bc(0, |xnσn − 1|), we obtain∫ B( xnσn ,1) |vn(y)|2dy 6 ε2 ∫ B( xnσn ,1) dy = ε2|B( xn σn , 1)| = ε2|B(0, 1)| < C1. This contradicts (3.3). Hence, {xnσn } is bounded, and there exists R > 0 such that∫ B(0,R) |vn(y)|2dy > ∫ B( xnσn ,1) |vn(y)|2dy > C1 > 0. Since the embedding Ds,2 rad(R3) ↪→ Lrloc(R3), r ∈ [2, 2∗s) is compact, we deduce that∫ B(0,R) |v(y)|2dy > C1 > 0. Hence, v 6≡ 0. From vn ⇀ v weakly in Ds,2 rad(R3) and limn→∞〈J ′0(vn), ϕ〉 = o(1), we obtain 〈J ′λ(v), ϕ〉 = 0. Since v 6≡ 0, we know that v ∈ N . Now, we show that vn → v strongly in Ds,2 rad(R3). Applying the Brézis-Lieb lemma [10], we obtain c0 = lim n→∞ J0(vn)− lim n→∞ 1 2∗s 〈J ′0(vn), vn〉 = lim n→∞ (1 2 − 1 2∗s ) ‖vn‖2Φ > (1 2 − 1 2∗s ) ‖v‖2Φ = J0(v) > ¯̄c0 = c0. (3.4) Thus, the inequalities above have to be equalities. We know that lim n→∞ ‖vn‖2Φ = ‖v‖2Φ. By using Brézis-Lieb lemma [10] again, we have lim n→∞ ‖vn‖2Φ − lim n→∞ ‖vn − v‖2Φ = ‖v‖2Φ, which implies that lim n→∞ ‖vn − v‖2Φ = 0. EJDE-2020/01 FRACTIONAL SCHRÖDINGER-POISSON SYSTEM 11 Using (3.4) again, we know that J0(v) = c0. This implies that v attain the minimum of J0 at c0. � 4. Proof of Theorem 1.2 First, we summarize the proof of Theorem 1.1 as follows. Lemma 4.1. The energy functional J0 satisfies the following properties: (1) there exist ρ, ς > 0 such that if ‖u‖s = ρ, then J0(u) > ς, and e ∈ Ds,2 rad(R3) exists such that ‖e‖s > ρ and J0(e) < 0; (2) there exists a critical point v of J0 such that J0(v) = c0 := min Υ∈Γ0 max t∈[0,1] J0(Υ(t)), where Γ0 = {Υ ∈ C([0, 1], Ds,2 rad(R3))|Υ(0) = 0, J0(Υ(1)) < 0}; (3) c0 := inf{J0(u)|J ′0(u) = 0, u ∈ Ds,2 rad(R3) \ {0}}; (4) there exists a path Υ0(t) ∈ Γ0 passing through v at t = t0 and satisfying J0(v) > J0(Υ0(t)) for all t 6= t0; (5) the set S := {u ∈ Ds,2 rad(R3)|J ′0(u) = 0, J0(u) = c0} is compact in Ds,2 rad(R3) endowed with the strong topology up to dilations in R3. Proof. It is easy to see that J0 is invariant by dilations. As a byproduct of the proof of Theorem 1.1, the weak convergence of the dilated subsequence can be upgraded into strong convergence. As a direct consequence, we show that the set S is compact in Ds,2 rad(R3) endowed with the strong topology up to dilations in R3. The proof is completed. � We introduce the energy functional associated with system (1.1) by Iλ(u) = 1 2 ‖u‖2Φ + λ 4 ∫ R3 ∫ R3 |u(x)|2|u(y)|2 |x− y|3−2t dxdy − 1 2∗s ∫ R3 |u|2 ∗ sdx. Define Jλ = Iλ|Ds,2rad(R3). As in [12, 25], we define a modified mountain pass energy level of Jλ cλ := min Υ∈ΥM max t∈[0,1] Jλ(Υ(t)), where ΥM = {Υ ∈ Γ0| sup t∈[0,1] ‖Υ(t)‖s 6M}, M = 2{sup u∈S ‖u‖s, sup t∈[0,1] ‖Υ(t)‖s}. Clearly, by the choice of M , Υ0 ∈ ΓM and thus, c0 = min Υ∈ΓM max t∈[0,1] J0(Υ(t)). However, since ΓM ( Γ0, the standard mountain pass theorem cannot be applied, so other arguments are needed to prove that cλ is a critical value. Lemma 4.2. Let λ > 0. Then limλ→0 cλ = c0. 12 Y. SU, H. CHEN, S. LIU, X. FANG EJDE-2020/01 Proof. Since λ > 0, it is easy to see that cλ > c0. On the other hand, we can take e = Tv in property (1), where T > (2∗s 2 ) 1 2∗s−2 . Then Υ0(t) ∈ C([0, 1], Ds,2(R3)) is defined as Υ0(t) = te = tTv, and t0 = 1 T in property (4). We know that lim λ→0 cλ = lim λ→0 Jλ(Υ0(t)) 6J0(Υ0(t)) + lim λ→0 λ 4 ∫ R3 ∫ R3 |Tv(y)|2|Tv(x)|2 |x− y|3−2t dxdy =J0(v) = c0. The proof is complete. � For any positive number d, we denote Bd(u) := {v ∈ Ds,2 rad(R3)|‖u− v‖s 6 d}, and for any subset A of Ds,2 rad(R3), we set Ad := ∪u∈ABd(u). Lemma 4.3. Set d1 = √ 2 · 2∗s · c0/(2∗s − 2). Let d ∈ (0, d1). For any u ⊂ Sd, we have u 6≡ 0. Proof. For any v ⊂ Sd, we have ‖v‖2s > ‖v‖2Φ = 2 · 2∗s 2∗s − 2 ( J0(v)− 1 2∗s 〈J ′0(v), v〉 ) = 2 · 2∗s · c0 2∗s − 2 , which gives ‖v‖s > d1. (4.1) By the definition of Sd and u ⊂ Sd, there exists some v ⊂ Sd such that ‖u− v‖s 6 d < d1. (4.2) Applying the triangle inequality, we have ‖v‖s = ‖v − u+ u‖s 6 ‖v − u‖s + ‖u‖s. which gives ‖v‖s − ‖v − u‖s 6 ‖u‖s. (4.3) Putting (4.1) and (4.2) into (4.3), we obtain ‖u‖s > d1 − ‖v − u‖s > 0. The proof is complete. � Lemma 4.4. Let d > 0 be a fixed number and let {uj} ⊂ Sd. Then there exists {σj}, such that ‖ūj‖s = ‖uj‖s, where ūj(x) = σ 3−2s 2 j uj(σjx). And, up to a subsequence, ūj ⇀ ū ∈ S2d. EJDE-2020/01 FRACTIONAL SCHRÖDINGER-POISSON SYSTEM 13 Proof. Let {uj} ⊂ Sd. By the definition of Sd and Lemma 4.1(5), there exists wj ∈ S such that ‖uj − wj‖s 6 d. By property (5) again, there exists {σj}, such that w̄j ∈ S, one can prove that w̄j → w̄ ∈ S, where w̄j(x) = σ 3−2s 2 j wj(σjx) and ūj(x) = σ 3−2s 2 j uj(σjx). Also ‖ūj‖s = ‖uj‖s, ‖ūj − w̄j‖s = ‖uj − wj‖s 6 d. Hence, for j large enough, we obtain ‖ūj − w̄‖s = ‖ūj − w̄j + w̄j − w̄‖s 6 ‖ūj − w̄j‖s + ‖w̄j − w̄‖s 6 2d. Thus, {ūj} is bounded and, up to a subsequence, ūj ⇀ ū in Ds,2 rad(R3). Since B2d(w̄) is weakly closed in Ds,2 rad(R3), we obtain ū ∈ B2d(w̄) ⊂ S2d. The proof is complete. � Lemma 4.5. Set d ∈ (0, d1). Suppose that there exist sequences λj > 0, λj → 0, and {uj} ⊂ Sd satisfying lim j→∞ Jλj (uj) 6 c0 and lim j→∞ J ′λj (uj) = 0. Then there is sequence {σj} such that ‖ūj‖s = ‖uj‖s, where ūj(x) = σ 3−2s 2 j uj(σjx). And, up to a subsequence, {ūj} converges to some ū ∈ S. Proof. Considering that limj→∞ J ′λj (uj) = 0 and that {uj} is bounded. By Lemma 4.4, up to a subsequence, ūj ⇀ ū ∈ S2d, and by the choice of d1, ū 6= 0. We may readily verify that lim j→∞ Jλj (ūj) = lim j→∞ Jλj (uj) 6 c0 and lim j→∞ J ′λj (ūj) = 0. For all ϕ ∈ Ds,2 rad(R3), we obtain |〈J ′λj (ūj), ϕ〉| = |〈J ′ λj (uj), ϕ̄〉| 6 ‖J ′ λj (uj)‖s−1‖ϕ̄‖s = o(1)‖ϕ̄‖s, where ϕ̄ = σ − 3−2s 2 j ϕ( xσj ). Since ‖ϕ̄‖s = ‖ϕ‖s, we obtain J ′λj (ūj)→ 0 as j →∞. Now, we have 〈J ′0(ū), ϕ〉 = lim j→∞ 〈J ′λj (ūj), ϕ〉 − λj ∫ R3 ∫ R3 |ūj(y)|2ūj(x)ϕ(x) |x− y|3−2t dxdy = 0. Hence, J ′0(ū) = 0. Furthermore, applying Lemma 2.2 and ūj ∈ S2d for all j, we obtain lim j→∞ 〈J ′0(ūj), ϕ〉 = lim j→∞ 〈J ′λj (ūj), ϕ〉 − λj ∫ R3 ∫ R3 |ūj(y)|2ūj(x)ϕ(x) |x− y|3−2t dxdy = o(1)‖ϕ‖s. On the other hand, c0 > lim j→∞ Jλj (ūj) = lim j→∞ J0(ūj) + λj 4 ∫ R3 ∫ R3 |ūj(y)|2|ūj(x)|2 |x− y|3−2t dxdy = lim j→∞ J0(ūj). (4.4) 14 Y. SU, H. CHEN, S. LIU, X. FANG EJDE-2020/01 So {ūj} is a (PS)m sequence for J0 with m := limj→∞ J0(ūj). Thus, up to a subsequence, ūj ⇀ ū and J0(ū) = 1 2 ‖ū‖2Φ − 1 2∗s ∫ R3 |ū|2 ∗ sdx = (1 2 − 1 2∗s ) ‖ū‖2Φ 6 (1 2 − 1 2∗s ) lim inf j→∞ ‖ūj‖2Φ = lim inf j→∞ ( J0(ūj)− 1 2∗s 〈J ′0(ūj), ūj〉 ) = m. Then, property (3) implies m > J0(ū) > c0, which, combined with (4.4), yields m = J0(ū) = c0, which implies ū ∈ S. � Now, we set mλ := max t∈[0,1] Jλ(Υ0(t)). Then cλ 6 mλ. It is easy to see that limλ→0mλ 6 c0, this inequality together with Lemma 4.2, provides limλ→0 cλ = limλ→0mλ = c0. We also define Jmλλ = {u ∈ Ds,2 rad(R3)|Jλ(u) 6 mλ}. Lemma 4.6. For any d2, d3 > 0 satisfying d3 < d2 < d1, there are constants ι > 0 and λ̃ > 0 depending on d2, d3 such that for λ ∈ (0, λ̃), we have ‖J ′λ(u)‖s−1 > ι, for all u ∈ Jmλλ ∩ (Sd2\Sd3). Proof. To the contrary, suppose that for some d2, d3 > 0 satisfying d3 < d2 < d1, there exist sequences {λj} with limj→∞ λj = 0 and {uj} ∈ J mλj λj ∩ (Sd2 \Sd3), such that lim j→∞ Jλj (uj) 6 c0 and lim j→∞ J ′λj (uj) = 0. By using property (5), there exists a sequence {σj} such that {ūj} ∈ J mλj λj ∩ (Sd2 \ Sd3), lim j→∞ Jλj (ūj) 6 c0, lim j→∞ J ′λj (ūj) = 0, where ūj(x) = σ 3−2s 2 j uj(σjx). Hence, we can apply Lemma 4.4 and assert the existence of some ū ∈ S such that ūj → ū in Ds,2 rad(R3). As a consequence, dist(ūj ,S)→ 0 as j →∞, contradicting the relation ūj 6∈ Sd3 . � Lemma 4.7. For d > 0, there exists δ > 0, such that if λ > 0 with λ small enough, t ∈ [0, 1], Jλ(Υ0(t)) > cλ − δ implies Υ0(t) ∈ Sd. Since the proof of the above lemma is quite similar to that of [25, Propositions 4], we omit it here. Lemma 4.8. To each 0 < d < d1 corresponds a number λ0 > 0 such that for all λ ∈ (0, λ0), there exists a sequence {uj} ∈ Jmλλ ∩ Sd such that J ′λ(uj) → 0 as j →∞. Since the proof of the above lemma is quite similar to that of [12, Propositions 5.3], we omit it here. EJDE-2020/01 FRACTIONAL SCHRÖDINGER-POISSON SYSTEM 15 Proof of Theorem 1.2. Taking d ∈ (0, d1), by Lemma 4.8, there exists λ0 > 0 such that for all λ ∈ (0, λ0), there exists a Palais Smale sequence {uλj } ⊂ S d 2 . By using property (5), there exists sequence {σj} such that {ūλj } ⊂ S d 2 where ūλj (x) = σ 3−2s 2 j uλj (σjx). It is easy to see that {ūλj } is bounded in Ds,2 rad(R3). Then by Lemma 4.4, up to a subsequence, there exists some ūλ ∈ S d2 ·2 = Sd such that ūλj ⇀ ūλ. Then we obtain J ′λ(ūλ) = 0, and by the choice of d, ūλ 6≡ 0. Hence ūλ is a nontrivial critical point of Jλ. The principle of symmetric criticality implies that the critical points of Jλ are also critical points for Iλ. � 5. Open problem In Theorem 1.1, we obtain a ground state solution v ∈ Ds,2 rad(R3) to equation (1.8). Since Ds,2 rad(R3) ⊂ Ds,2(R3), so v just is a ground state solution in Ds,2 rad(R3), not a ground state solution in Ds,2(R3). It is natural to ask: Does there exist a ground state solution to equation (1.8) in Ds,2(R3)? Acknowledgements. This research was supported by the National Natural Sci- ence Foundation of China 11671403, and by the University-level key projects of Anhui University of science and technology QN2019101. References [1] L. Abatangelo, S. Terracini; Solutions to nonlinear Schrödinger equations with singular elec- tromagnetic potential and critical exponent, J. Fixed Point Theory Appl., 10 (2011), no. 1, 147–180. [2] V. Ambrosio; Ground states solutions for a non-linear equation involving a pseudo-relativistic Schrödinger operator, J. Math. Phys., 57 (2016), no. 5, 051502, 18 pp. [3] V. Ambrosio; Periodic solutions for the non-local operator (−∆ + m2)s −m2s with m ≥ 0, Topol. Methods Nonlinear Anal., 49 (2017), 75–104. [4] V. Ambrosio; An existence result for a fractional Kirchhoff–Schrödinger–Poisson system, Z. Angew. Math. Phys., 69 (2018), no. 2, Art. 30, 13 pp. [5] V. Ambrosio; Multiplicity and concentration results for a class of critical fractional Schrödinger-Poisson systems via penalization method, Commun. Contemp. Math., doi: 10.1142/S0219199718500785. [6] A. Azzollini, A. Pomponio; Ground state solutions for the nonlinear Schrödinger–Maxwell equations, J. Math. Anal. Appl., 345 (2008), no. 1, 90–108. [7] J. Bellazzini, M. Ghimenti, C. Mercuri, V. Moroz, J. Van Schaftingen; Sharp Gagliardo– Nirenberg inequalities in fractional Coulomb–Sobolev spaces, Trans. Amer. Math. Soc., (2017), doi.org/10.1090/tran/7426. [8] V. Benci, D. Fortunato; An eigenvalue problem for the Schrödinger–Maxwell equations, Topol. Methods Nonlinear Anal., 11 (1998), no. 2, 283–293. [9] H. Berestycki, P. L. Lions; Nonlinear scalar field equations. I. Existence of a ground state, Arch. Rational Mech. Anal., 82 (1983), no. 4, 313–345. [10] H. Brézis, E. H. Lieb; A relation between pointwise convergence of functions and convergence of functionals, Proc. Amer. Math. Soc., 88 (1983), no. 3, 486–490. [11] G. Cerami, G. Vaira; Positive solutions for some non–autonomous Schrödinger-Poisson sys- tems, J. Differential Equations, 248 (2010), no. 3, 521–543. [12] G. Cerami, X. Zhong, W. Zou; On some nonlinear elliptic PDEs with Sobolev–Hardy critical exponents and a Li-Lin open problem, Calc. Var. Partial Differential Equations, 54 (2015), no. 2, 1793–1829. [13] G. Che, H. Shi, Z. Wang; Existence and concentration of positive ground states for a 1- Laplacian problem in RN , Appl. Math. Lett., 100 (2020), 106045. [14] G. Che, H. Chen, T. Wu; Existence and multiplicity of positive solutions for fractional Lapla- cian systems with nonlinear coupling, J. Math. Phys., 60 (2019), 081511. 16 Y. SU, H. CHEN, S. LIU, X. FANG EJDE-2020/01 [15] S. Dipierro, L. Montoro, I. Peral, B. Sciunzi; Qualitative properties of positive solutions to nonlocal critical problems involving the Hardy-Leray potential, Calc. Var. Partial Differential Equations, 55 (2016), no. 4, Art. 99, 29. [16] M. M. Fall, V. Felli; Sharp essential self-adjointness of relativistic Schrödinger operators with a singular potential, J. Funct. Anal. 267 (2014), no. 6, 1851–1877. [17] M. M. Fall, V. Felli; Unique continuation properties for relativistic Schrödinger operators with a singular potential, Discrete Contin. Dyn. Syst., 35 (2015), no. 12, 5827–5867. [18] V. Felli, A. Ferrero, S. Terracini; Asymptotic behavior of solutions to Schrödinger equations near an isolated singularity of the electromagnetic potential, J. Eur. Math. Soc. (JEMS) 13 (2011), no. 1, 119–174. [19] V. Felli, A. Ferrero, S. Terracini; On the behavior at collisions of solutions to Schrödinger equations with many-particle and cylindrical potentials, Discrete Contin. Dyn. Syst., 32 (2012), no. 11, 3895–3956. [20] V. Felli, E. M. Marchini, S. Terracini; On the behavior of solutions to Schrödinger equations with dipole type potentials near the singularity, Discrete Contin. Dyn. Syst., 21 (2008), no. 1, 91–119. [21] V. Felli, E. M. Marchini, S. Terracini; On Schrödinger operators with multisingular inverse- square anisotropic potentials, Indiana Univ. Math. J., 58 (2009), no. 2, 617–676. [22] R. L. Frank, R. Seiringer; Non–linear ground state representations and sharp Hardy inequal- ities, J. Funct. Anal. 255 (2008), no. 12, 3407–3430. [23] I. W. Herbst; Spectral theory of the operator (p2 + m2)1/2 − Ze2 r , Comm. Math. Phys., 53 (1977), no. 3, 285–294. [24] T. Hoffmann-Ostenhof, A. Laptev; Hardy inequalities with homogeneous weights, J. Funct. Anal., 268 (2015), no. 11, 3278–3289. [25] W. Jeong, J. Seok; On perturbation of a functional with the mountain pass geometry: appli- cations to the nonlinear Schrödinger–Poisson equations and the nonlinear Klein–Gordon– Maxwell equations, Calc. Var. Partial Differential Equations, 49 (2014), no. 1–2, 649–668. [26] J. M. Lévy-Leblond; Electron capture by polar molecules, Phys. Rev. 153 (1967), 1–4. [27] F. Li, C. Gao, X. Zhu; Existence and concentration of sign–changing solutions to Kirchhoff– type system with Hartree-type nonlinearity, J. Math. Anal. Appl., 448 (2017), no. 1, 60–80. [28] E. H. Lieb; The stability of matter: from atoms to stars, Bull. Amer. Math. Soc. (N.S.) 22 (1990), no. 1, 1–49. [29] E. H. Lieb, M. Loss; Analysis, Graduate Studies in Mathematics, vol. 14, American Mathe- matical Society, Providence, 2001. [30] Z. Liu, J. Zhang; Multiplicity and concentration of positive solutions for the fractional Schrödinger–Poisson systems with critical growth, ESAIM Control Optim. Calc. Var. 23 (2017), no. 4, 1515–1542. [31] Z. Liu, M. Squassina, J. Zhang; Ground states for fractional Kirchhoff equations with critical nonlinearity in low dimension, NoDEA Nonlinear Differential Equations Appl., 24 (2017), 50. [32] C. Mercuri, V. Moroz, J. Van Schaftingen; Groundstates and radial solutions to nonlinear Schrödinger-Poisson-Slater equations at the critical frequency, Calc. Var. Partial Differential Equations 55 (2016), no. 6, 146. [33] T. Mukherjee, K. Sreenadh; Fractional Choquard equation with critical nonlinearities, NoDEA Nonlinear Differential Equations Appl., 24 (2017), no. 6, 63. [34] B. Noris, M. Nys, S. Terracini; On the Aharonov-Bohm operators with varying poles: the boundary behavior of eigenvalues, Comm. Math. Phys., 339 (2015), no. 3, 1101–1146. [35] G. Palatucci, A. Pisante; Improved Sobolev embeddings, profile decomposition, and concentration-compactness for fractional Sobolev spaces, Calc. Var. Partial Differential Equa- tions, 50 (2014), no. 3-4, 799–829. [36] Y. Su, H. Chen; Fractional Kirchhoff-type equation with Hardy-Littlewood-Sobolev critical exponent, Comput. Math. Appl., 78 (2019), no. 6, 2063–2082. [37] Y. Su, L. Wang, T. Han; Schrödinger–Poisson system with Hardy-Littlewood-Sobolev critical exponent, Math. Methods Appl. Sci., 42 (2019), no. 14, 4815–4838. [38] Y. Su; New result for nonlinear Choquard equations: doubly critical case, Appl. Math. Lett., 102 (2020), 106092. [39] Y. Su, H. Chen; The minimizing problem involving p–Laplacian and Hardy-Littlewood- Sobolev upper critical exponent, Electron. J. Qual. Theory Differ. Equ. (2018), No. 74. EJDE-2020/01 FRACTIONAL SCHRÖDINGER-POISSON SYSTEM 17 [40] J. Sun; Infinitely many solutions for a class of sublinear Schrödinger-Maxwell equations, J. Math. Anal. Appl., 390 (2012), no. 2, 514–522. [41] K. Teng; Existence of ground state solutions for the nonlinear fractional Schrödinger–Poisson system with critical Sobolev exponent, J. Differential Equations, 261 (2016), no. 6, 3061–3106. [42] K. Teng, R. P. Agarwal; Existence and concentration of positive ground state solutions for nonlinear fractional Schrödinger–Poisson system with critical growth, Math. Methods Appl. Sci., 41 (2018), no. 17, 8258–8293. [43] Z. Wei; Existence of infinitely many solutions for the fractional Schrödinger-Poisson equa- tions, arXiv:1508.03088v1. [44] M. Willem; Minimax theorems, Progress in Nonlinear Differential Equations and their Ap- plications, vol. 24, Birkhäuser Boston, 1996. [45] J. Yang, F. Wu; Doubly critical problems involving fractional Laplacians in RN , Adv. Non- linear Stud. 17 (2017), no. 4, 677–690. [46] J. Zhang, J. M. do Ó, M. Squassina; Fractional Schrödinger–Poisson systems with a general subcritical or critical nonlinearity, Adv. Nonlinear Stud., 16 (2016), no. 1, 15–30. [47] L. Zhao, F. Zhao; On the existence of solutions for the Schrödinger-Poisson equations, J. Math. Anal. Appl., 346 (2008), no. 1, 155–169. Yu Su (corresponding author) School of Mathematics and Big Data, Anhui University of Science and Technology, Huainan, 232001 Anhui, China. School of Mathematics and Statistics, Central South University, Changsha, 410083 Hunan, China Email address: yusumath@qq.com Haibo Chen School of Mathematics and Statistics, Central South University, Changsha, 410083 Hunan, China Email address: math chb@163.com Senli Liu School of Mathematics and Statistics, Central South University, Changsha, 410083 Hunan, China Email address: jasonliu0615@163.com Xianwen Fang School of Mathematics and Big Data, Anhui University of Science and Technology, Huainan, 232001 Anhui, China Email address: xwfang@aust.edu.cn 1. Introduction 1.1. Statement of results 2. Preliminaries 2.1. Formulation of system (??) 3. Perturbed equation 3.1. Mountain pass geometry and Nehari manifold 3.2. Ground state solution 4. Proof of Theorem ?? 5. Open problem Acknowledgements References