E extracta mathematicae Vol. 31, Núm. 1, 89 – 107 (2016) On the Moduli Space of Donaldson-Thomas Instantons Yuuji Tanaka Graduate School of Mathematics, Nagoya University Furo-cho, Chikusa-ku, Nagoya, 464-8602, Japan yu2tanaka@gmail.com Presented by Oscar Garćıa Prada Received May 19, 2015 Abstract: In alignment with a programme by Donaldson and Thomas, Thomas [48] constructed a deformation invariant for smooth projective Calabi-Yau threefolds, which is now called the Donaldson-Thomas invariant, from the moduli space of (semi-)stable sheaves by using algebraic geometry techniques. In the same paper [48], Thomas noted that certain perturbed Hermitian-Einstein equations might possibly produce an analytic theory of the invariant. This article sets up the equations on symplectic 6-manifolds, and gives the local model and structures of the moduli space coming from the equations. We then describe a Hitchin-Kobayashi style correspondence for the equations on compact Kähler threefolds, which turns out to be a special case of results by Álvarez-Cónsul and Garćıa-Prada [1]. Key words: gauge theory; the Donaldson-Thomas theory. AMS Subject Class. (2010): 53C07. 1. Introduction In [21], Donaldson and Thomas suggested higher-dimensional analogues of gauge theories, and proposed the following two directions: gauge theories on Spin(7) and G2-manifolds; and gauge theories in complex 3 and 4 dimen- sions. The first ones could be related to “Topological M-theory” proposed by Nekrasov and others [39], [15]. The second ones are a “complexification” of the lower-dimensional gauge theories. In this direction, Thomas [48] con- structed a deformation invariant of smooth projective Calabi-Yau threefolds from the moduli space of (semi-)stable sheaves, which he called the holomor- phic Casson invariant because it can be viewed as a complex analogue of the Taubes-Casson invariant [47]. It is now called the Donaldson-Thomas invari- ant (D-T invariant for short), and further developed by Joyce-Song [28] and Kontsevich-Soibelman [32], [33], [34]. Later, Donaldson and Segal [20] further promoted the programme, taking into account the progress made after the proposal. Recently, more breakthroughs concerning the “categorification” of the D-T invariant by using perverse sheaves were made by a group led by Joyce [7], [27], [8], [9], [4], also by Kiem-Li [29]. 89 90 y. tanaka Let us mention here a conjecture (called the MNOP conjecture) posed by Maulik-Nekrasov-Okounkov-Pandharipande [37], [38], which insists that the rank one D-T invariants (“counting” of ideal sheaves on a Calabi-Yau threefold) can be determined by only the Betti numbers and the Gromov- Witten invariants. Assuming the conjecture is true, one can observe that the rank one D-T invariants are symplectic invariants, as the Gromov-Witten invariants are symplectic invariants. One might further speculate that the full D-T invariants defined by Joyce and Song could be also symplectic invariants. One of our goals is to work toward proving this by using a gauge-theoretic equation (we call it the Donaldson-Thomas equation) on a compact symplectic 6-manifold, which ought to be an analytic counterpart of the notion of stable holomorphic vector bundles, as the problem is analytic in nature. Perhaps, one might think of that a gauge-theoretic equation which would describe the D-T invariant could be the Hermitian-Einstein equations, as the Hitchin-Kobayashi correspondence [17], [18], [50], [51] (see also [31], [36]) insists that there is a one-to-one correspondence between the existence of the Hermitian-Einstein connection and the Mumford-Takemoto stability of an irreducible vector bundle over a compact Kähler manifold. However, the Hermitian-Einstein equations do not form an elliptic system even with a gauge fixing equation in complex dimension three and more (see Section 2.1), so this might cause a little problem. In order to work out this issue, Donaldson and Thomas [48] suggested a perturbation of the Hermitian-Einstein equations described below. This perturbation was also brought in by Baulieu-Kanno-Singer [3] and Iqbal- Nekrasov-Okounkov-Vafa [26] in String Theory context. Let Z be a compact symplectic 6-manifold with symplectic form ω, P a principal U(r)-bundle on Z, and E the associated unitary vector bundle on Z. The equations we consider are ones for a connection A of P and an Ad(P)-valued (0,3)-form u on Z of the following form: F 0,2 A + ∂̄ ∗ Au = 0 , F 1,1 A ∧ ω 2 + [u,ū] + 2πiµ(E)IdEω 3 = 0 , where F 0,2 A and F 1,1 A are the (0,2) and (1,1) components of the curvature FA of A, and µ(E) := 1 r ∫ Z c1(E) ∧ ω2. Here we picked up an almost complex structure compatible with ω to get the splitting of the space of the complexified two forms. We call the equations the Donaldson-Thomas equations (D-T equations for short) and a solution moduli space of donaldson-thomas instantons 91 to the equations a Donaldson-Thomas instanton (D-T instanton for short). These equations with a gauge fixing equation form an elliptic system. We aim at developing an analytic theory concerning the D-T invariant by using the moduli space coming from these equations. In [44], [45], we studied some analytic properties of solutions to the equa- tions on compact Kähler threefolds. In [44], we proved that a sequence of solutions to the D-T equation has a subsequence which smoothly converges to a solution to the D-T equation outside a closed subset of the Hausdorff dimension two. In [45], we proved some of singularities which appeared in the above weak limit can be removed. In this article, we describe the infinitesimal deformation and the Kuran- ishi model of the moduli space of D-T instantons by using familiar techniques in gauge theory, for example, the corresponding results for the anti-self-dual instantons in real four dimensions were studied by Atiyah-Hitchin-Singer [2] (see also [22], [19]), and for the Hermitian-Einstein connections by Kim [30] (see also [31], [36]). We then describe a Hitchin-Kobayashi style correspon- dence for the D-T instanton on compact Kähler threefolds, which turns out to be a special case of results by Álvarez-Cónsul and Garćıa-Prada [1]. The organisation of this article is as follows. In Section 2, we briefly recall the Hermitian-Einstein connections, subsequently, we introduce the D- T equations on symplectic 6-manifolds. We also mention a relation between the D-T equations and the complex anti-self-dual equations by dimensional reduction argument. In Section 3, we give the Kuranishi model of the space of the D-T instantons. In Section 4, we describe a Hitchin-Kobayashi style correspondence for the D-T instanton on compact Kähler threefolds. 2. The Donaldson-Thomas instantons 2.1. The Hermitian-Einstein connections on compact Kähler manifolds. We first recall the notion of the Hermitian-Einstein connections on compact Kähler manifolds. General references for the Hermitian-Einstein connections are [31] and [36]. Let X be a compact Kähler manifold of complex dimension n with Kähler form ω, E a hermitian vector bundle over X with hermitian metric h. A metric preserving connection A of E is said to be a Hermitian-Einstein connection if A satisfies the following equations: F 0,2 A = 0 , iΛF 1,1 A = 2nπµ(E)IdE , (2.1) where F 0,2 A and F 1,1 A are the (0,2) and (1,1) components of the curvature FA 92 y. tanaka of A, Λ := (ω)∗, and µ(E) := 1 r ∫ X c1(E) ∧ ωn−1. The existence of a solution to the equations (2.1) is related to the notion of stability for holomorphic vector bundles. In fact, Donaldson [17], [18] and Uhlenbeck-Yau [50], [51] proved that there is a one-to-one correspondence between the existence of the Hermitian-Einstein connection and the Mumford- Takemoto stability of an irreducible vector bundle over a compact Kähler manifold (see also [31], [36]). The infinitesimal deformation of a Hermitian-Einstein connection A was studied by Kim [30] (see also [31], [41]), and it is described by the following: 0 −→ Ω0(X,u(E)) dA−−→ Ω1(X,u(E)) d+ A−−→ Ω+(X,u(E)) D̄′ A−−→ A0,3(X,u(E)) D̄A−−→ A0,4(X,u(E)) D̄A−−→ · · · D̄A−−→ A0,n(X,u(E)) −→ 0, (2.2) where A0,q(X,u(E)) := C∞(u(E) ⊗ A0,q), u(E) = End(E,h) is the bundle of skew-Hermitian endomorphisms of E, A0,p is the space of real (0,p)-forms (see [42, pp. 32-33]) over X, defined by A0,p ⊗R C = Λ0,p ⊕ Λp,0, Ω+(X,u(E)) := A0,2(X,u(E)) ⊕ Ω0(X,u(E))ω = { ϕ + ϕ̄ + fω : ϕ ∈ Ω0,2(X,u(E)), f ∈ Ω0(X,u(E)) } , D̄A : A 0,p(X,u(E)) → A0,p+1(X,u(E)) is defined by D̄Aα = ∂̄Aα 0,p + ∂Aα0,p for α = α0,p + α0,p, where α0,p ∈ Ω0,p(X,u(E)), and d+A := π + ◦ dA , D̄′A := D̄A ◦ π 0,2, where π+ and π0,2 are respectively the orthogonal projections from Ω2 to Ω+ and A0,2. Kim proved that (2.2) is an elliptic complex if A is a Hermitian-Einstein connection. However, it is obviously not the Atiyah-Hitchin-Singer type com- plex [2] if n ≥ 3, since there are additional terms such as A0,3(X,u(E)) and so on. Hence, the Hermitian-Einstein connections would not work for an an- alytic construction of the Donaldson-Thomas invariant just as it is. But, in moduli space of donaldson-thomas instantons 93 [48], Thomas noted a perturbed Hermitian-Einstein equation, which basically corresponds to a “holding” of the extra term A0,3(X,u(E)) in (2.2) (we shall see it in Section 3.1), could possibly work for an analytic definition of the Donaldson-Thomas invariant. We introduce that perturbed equation in the next subsection. 2.2. The Donaldson-Thomas instantons on compact symplectic 6-manifolds. Let Z be a compact symplectic 6-manifold with symplectic form ω, and E a unitary vector bundle of rank r over Z. We take an almost complex structure on Z compatible with the symplectic form ω. Then the almost complex structure induces the splitting of the complexified two forms as Λ2 ⊗ C = Λ2,0 ⊕ Λ0,2 ⊕ Λ1,1. We consider the following equations for a connection A of E, which preserves the hermitian structure of E, and a u(E)-valued (0,3)-form u on Z. F 0,2 A + ∂̄ ∗ Au = 0 , (2.3) F 1,1 A ∧ ω 2 + [u,ū] + 2πiµ(E)IdE ω 3 = 0 , (2.4) where F 0,2 A and F 1,1 A are the (0,2) and (1,1) components of the curvature FA of A, and µ(E) := 1 r ∫ Z c1(E) ∧ ω2. We call these equations (2.3), (2.4) the Donaldson-Thomas equations, and a solution (A,u) to these equations a Donaldson-Thomas instanton (D-T instanton for short). One may think of these equations as the Hermitian-Einstein equations with a perturbation u. However, we think of u as a Higgs field, namely, a new vari- able. One of advantages of bringing in the new field u is that the Donaldson- Thomas equations form an elliptic system after fixing a gauge transformation, despite the fact that the Hermitian-Einstein equations on compact Kähler threefolds do not form it in the same way. These equations (2.3), (2.4) were also studied in physics such as in [3]. In that context, these equations are interpreted as a bosonic part of dimensional reduction equations of the N = 1 super Yang-Mills equation in 10 dimensions to 6 dimensions (see also [26], [40]). The equations in the Kähler case. If the almost complex struc- ture is integrable, then we have ∂̄AF 0,2 A = 0 by the Bianchi identity. Hence 94 y. tanaka ∂̄A∂̄ ∗ Au = 0 by (2.3), thus we have ∂̄ ∗ Au = 0 on compact Kähler threefolds. Therefore, the Donaldson-Thomas equations (2.3), (2.4) becomes ∂̄∗Au = 0, F 0,2 A = 0 , F 1,1 A ∧ ω 2 + [u,ū] + 2πiµ(E)IdE ω 3 = 0 . The above equations could be thought of as a generalization of the Hitchin equation on Riemann surfaces [24] to Kähler threefolds in the same way as the Vafa-Witten equations on Kähler surfaces as mentioned in [46]. In Section 4 to this article, we describe the corresponding Hitchin-Kobayashi correspondence in this setting, which turns out to be a special case of results by Álvarez-Cónsul and Garćıa-Prada [1]. 2.3. The complex ASD and the Donaldson-Thomas instantons. In this section, we see that the Donaldson-Thomas equations on Calabi-Yau threefolds can be thought of as the dimensional reduction of the complex ASD equations on Calabi-Yau fourfolds, this was pointed out by Tian [49], and it is analogous to the Hitchin pair [24]. Complex ASD equations on Calabi-Yau fourfolds. Let X be a compact Calabi-Yau fourfold with Kähler form ω and holomorphic (4,0)-form θ. We assume the normalization condition θ ∧ θ̄ = 16 4! ω4 on ω and θ. Let E be a hermitian vector bundle over X. By using the holomorphic (4,0)-form θ, we define the complex Hodge operator ∗θ : Λ0,2 → Λ0,2 by tr(ϕ ∧ ∗θψ) = ⟨ϕ,ψ⟩θ̄ , ϕ,ψ ∈ Λ0,2. Then ∗2θ = 1, and the space of (0,2)-forms further decomposes into Λ 0,2 = Λ 0,2 + ⊕ Λ 0,2 − , where Λ 0,2 + = {ϕ ∈ Λ 0,2 : ∗θϕ = ϕ} , Λ 0,2 − = {ϕ ∈ Λ 0,2 : ∗θϕ = −ϕ}. Note that the operator ∗θ is an anti-holomorphic map, hence Λ 0,2 + and Λ 0,2 − are real subspaces of Λ0,2. We consider the following equations for connections of E: (1 + ∗θ)F 0,2 A = 0 , iΛF 1,1 A = 8πµ(E)IdE, (2.5) where µ(E) := 1 r ∫ X c1(E) ∧ ω3. moduli space of donaldson-thomas instantons 95 We call these equations complex ASD equations, and a solution to these equa- tions a complex ASD instanton. These were brought in by Donaldson and Thomas in [21]. These equations with a gauge fixing equation form an elliptic system. Analytic properties of the complex ASD instantons were studied by Tian [49]. Note that the complex ASD instantons are special cases of Spin(7)-instan- tons on Spin(7)-manifolds (see [43, §3.1]). More recently, Donaldson-Thomas style invariants for Calabi-Yau four- folds, which concerns the moduli space of the solutions to the above complex ASD equations, were defined by Borisov-Joyce [5], Cao [10] and Cao-Leung [11] (see also [12], [13], [14]). Dimensional reduction. We describe a relation between the Donald- son-Thomas equations (2.3), (2.4) and the complex ASD equations (2.5) by dimensional reduction argument. This was pointed out by Tian [49]. Let Z be a compact Calabi-Yau threefold with Kähler form ω0 and holo- morphic (3,0)-form θ0, and T 2 a torus of complex dimension one. We consider the direct product of Z and T2, and denote it by X, namely, X := Z × T2. We define a Kähler form ω and a holomorphic (4,0)-form on X by ω := ω0 + dz ∧ dz̄ , θ := θ0 ∧ dz , where dz is the standard flat (1,0) form on T2. Let E be a hermitian vector bundle with structure group SU(r) over Z, and p : X = Z × T2 → Z. We then consider T2-invariant solutions to the complex ASD equations (2.5) on p∗E → X. Then these solutions satisfy the Donaldson-Thomas equations on Z. In fact, if we write a connection A on X = Z × T2 as AX = A + ϕdz + ϕ̄dz̄, where A is the Z-component of the connection AX and ϕ ∈ Γ(Z,su(E)), then the curvature becomes FAX = FA + dAϕ ∧ dz + dAϕ̄ ∧ dz̄ + [ϕ,ϕ̄]dz ∧ dz̄ . Hence, if we put u := ϕθ̄0 ∈ Ω0,3(Z,su(E)), then A and u satisfy the Donaldson-Thomas equations, provided that this AX is a T 2-invariant so- lution to the complex ASD equations. 3. Local model for the moduli space of Donaldson-Thomas instantons Let Z be a compact symplectic 6-manifold with symplectic form ω, (E,h) a hermitian vector bundle over Z with hermitian metric h. 96 y. tanaka We denote by A(E) = A(E,h) the set of all connections of E which pre- serve the hermitian structure of E, and put C(E) := A(E) × Ω0,3(Z,u(E)). We denote by G(E) = G(E,h) the gauge group, the group of unitary auto- morphism of (E,h), where the action of the gauge group on C(E) is defined by g(A,u) = (A − (dAg)g−1,g−1ug). These spaces C(E), G(E) can be seen as Fréchet spaces with C∞-norms, but we shall use Sobolev completions of them in Section 3.2. We denote by Γ(A,u) the stabilizer at (A,u) ∈ C(E) of the gauge group G(E), namely, Γ(A,u) := { g ∈ G(E) : g(A,u) = (A,u) } . We call (A,u) ∈ C(E) irreducible if Γ(A,u) coincides with the centre of the structure group of E, and reducible otherwise. We denote by C∗(E) the set of all irreducible pair (A,u) ∈ C(E). Note that the action of G(E) is not free on C∗(E), but the action of Ĝ(E) = G(E)/U(1) is free on C∗(E). We denote by D(E) the set of all D-T instantons of E, and by D∗(E) the set of all irreducible D-T instantons of E. We call M(E) = D(E)/G(E) the moduli space of the Donaldson-Thomas instantons. 3.1. Linearization. The infinitesimal deformation of a D-T instanton (A,u) is described by the following sequence: 0 −→ Ω0(Z,u(E)) D(A,u) −−−−−−→ Ω1(Z,u(E)) ⊕ A0,3(Z,u(E)) D+ (A,u) −−−−−−→ Ω+(Z,u(E)) −→ 0 , (3.1) where D(A,u)(s) = (dAs, [ũ,s]) , ũ = u + ū , D+ (A,u) (α,υ) = d+Aα + Λ 2([u,ῡ] + [υ,ū]) + D̄∗Aυ for s ∈ Ω0(Z,u(E)) and (α,υ) ∈ Ω1(Z,u(E)) ⊕ A0,3(Z,u(E)). If (A,u) is a D-T instanton, then (3.1) is a complex. In fact, D+ (A,u) D(A,u) = 0 follows directly from the equations (2.3), (2.4). The complex (3.1) can be seen as “holding” of the A0,3(Z,u(E))-term in (2.2), namely, it is equivalent to con- sider the following complex instead of (3.1): 0 −→ Ω0(X,u(E)) dA−−→ Ω1(X,u(E)) d+ A−−→ Ω+(X,u(E)) D̄′ A−−→ A0,3(X,u(E)) −→ 0 . (3.2) moduli space of donaldson-thomas instantons 97 This is the same as that of the Hermitian-Einstein connections in Section 2.2, but it still makes sense in the almost complex setting. Hence the following just reduces to the case in (3.2), and it was proved by Reyes Carrión [41]. Proposition 3.1. If (A,u) ∈ D(E), then the complex (3.1) is elliptic. We denote by Hi (A,u) = Hi (A,u) (Z,u(E)) the i-th cohomology of the complex (3.1) for i = 0,1,2. The complex (3.2) has the associated Dolbeault complex as Kim [30] de- scribed it in the Kähler case (see also [31, Chapter VII, §2]): 0 −−−−→ Ω0 dA−−−−→ Ω1 d+ A−−−−→ Ω+ D̄′ A−−−−→ A0,3 D̄A−−−−→ 0yj0 yj1 yj2 yj3 0 −−−−→ Ω0,0 ∂̄A−−−−→ Ω0,1 ∂̄A−−−−→ Ω0,2 ∂̄A−−−−→ Ω0,3 ∂̄A−−−−→ 0 , (3.3) where j0 is injective, j1 is bijective, j2 is surjective with the kernel {βω : β ∈ Ω0}, and j3 is bijective. Hence the index of the complex (3.2), thus that of the complex (3.1), can be expressed by that of the Dolbeault complex above, which is given by ∫ Z Â(Z)∧ch ( K 1/2 Z ) ∧ch(u(E)) (see [23, §3.5]). In the Kähler case, the index can be computed as ∫ Z c1(Z) ∧ ( r − 1 2 c1(E) 2 − rc2(E) ) + r2 3∑ i=0 (−1)i dimH0,i(Z) . Note that the index is zero if Z is a Calabi-Yau threefold. 3.2. Kuranishi model and the local description of the moduli space. We denote by Ck(E), C∗k(E), Dk(E), D ∗ k(E) the L 2 k-completions of C(E), C∗(E), D(E), D∗(E) respectively, and by Gk+1(E) the L2k+1-completion of G(E). We take k sufficiently large so that Gk+1 becomes a Hilbert Lie group acting smoothly on Ck(E), the quotient topology Ck(E)/Gk+1(E) becomes Hausdorff (see e.g. [22, §3]), and to use implicit function theorems for the Sobolev spaces. A general reference for the Sobolev spaces and the implicit function theorems on them for our purpose is, for example, [52]. 98 y. tanaka Slice. We define slice S(A,u),ε at (A,u) in Ck(E) by S(A,u),ε := { (α,υ) ∈ L2k ( u(E) ⊗ ( Λ1 ⊕ A0,3 )) : D∗(A,u)(α,υ) = 0 , ||(α,υ)||L2k ≤ ε } . This set S(A,u),ε is transverse to the Gk+1-orbit through (A,u) as kerD∗(A,u) is orthogonal, with respect to the L2-norm in L2k ( u(E) ⊗ (Λ1 ⊕ A0,3) ) , to ImD(A,u). There is a natural map P(A,u),ε : S(A,u),ε −→ Ck(E)/Gk+1(E) , (α,υ) 7−→ [(A + α,u + υ ′)] , where υ′ = j3(υ), and j3 : A 0,3 → Ω0,3 is the map in (3.3). In the following, we take (A,u) ∈ C∗k(E) for simplicity. Proposition 3.2. Let (A,u) ∈ C∗k(E). Then there exists ε > 0 such that S(A,u),ε is diffeomorphic to P(A,u),ε ( S(A,u),ε ) in C∗k(E)/Ĝk+1(E). Proof. This is a familiar claim in gauge theory, the proof is a modification of known results for the ASD and the Hermitian-Einstein connections (cf. [16, Theorem 6], [22, Theorem 3.2 and Theorem 4.4], [31, Chapter VII, §4, Theorem 4.16], and [36, Proposition 4.2.1]). We divide the proof into two steps: Step 1. We consider a map f(A,u) : S(A,u),ε × Ĝk+1(E) → C∗k(E) defined by f(A,u)((α,υ),g) = g(A+α,u+υ ′). Then the differential of f(A,u) at ((0,0), id) is given by Df(A,u)|((0,0),id)((β,φ),s) = (β,φ) + D(A,u)(s) . As ImD(A,u) and kerD ∗ (A,u) are L2-orthogonal in L2k ( u(E) ⊗ (Λ1 ⊕ A0,3) ) , Df(A,u)|((0,0),id) is injective if (A,u) is irreducible. On the other hand, associated to the operator D∗(A,u)D(A,u) : L 2 k+1(u(E) ⊗ Λ 0)/u(1) −→ L2k−1(u(E) ⊗ Λ 0)/u(1) , where L2k+1(u(E) ⊗ Λ 0)/u(1) = { s ∈ L2k+1(u(E) ⊗ Λ 0) : ∫ Z tr (s) volg = 0 } , there exist the Green operator G0 : L2k(u(E)⊗Λ 0)/u(1) → L2k(u(E)⊗Λ 0)/u(1) and the harmonic projection H0 : L2k(u(E) ⊗ Λ 0)/u(1) → L2k(u(E) ⊗ Λ 0)/u(1) with the identity: Id = H0 + D∗(A,u)D(A,u) ◦ G 0 moduli space of donaldson-thomas instantons 99 (see e.g. [52, Chapter IV, §5]). From the identity, we obtain D∗(A,u)((γ,χ) − D(A,u)G 0D∗(A,u)(γ,χ)) = 0 for any (γ,χ) ∈ L2k(u(E) ⊗ (Λ 1 ⊕ A0,3)). Thus, for a given (γ,χ) ∈ L2k(u(E) ⊗ (Λ1⊕A0,3)), we take (β,φ) = (γ,χ)−D(A,u)G0D∗(A,u)(γ,χ), s = G 0D∗ (A,u) (γ,χ) to get (γ,χ) = (β,φ) + D(A,u)(s). Therefore Df(A,u)|((0,0),id) is surjective. We then use an inverse mapping theorem for the Hilbert spaces (see e.g. [35, Chapter 6]) to deduce that around (A,u), C∗k(E) is locally diffeomorphic to a neighbourhood of ((A,u), id) in S(A,u),ε × Ĝk+1(E). Step 2. We then prove that if for (α1,υ1),(α2,υ2) ∈ S(A,u),ε there exists g ∈ Gk+1(E) such that (A + α1, ũ + υ1) = g(A + α2, ũ + υ2) , (3.4) then cg is close to idE in L 2 k+1 for some c ∈ U(1). Since we assume that (A,u) is irreducible, we can take c ∈ U(1) so that g′ = cg − idE ∈ ker ( D(A,u) )⊥ . From (3.4), we get dAg ′ = α1g ′ − g′α2 + (α1 − α2) , [ũ,g′] = g′υ1 − υ2g′ + υ1 − υ2 . Hence, D(A,u)g ′ = ( α1g ′ − g′α2 + α12,g′υ1 − υ2g′ + υ12 ) , (3.5) where α12 = α1 − α2,υ12 = υ1 − υ2. Since g′ lies in ( kerD(A,u) )⊥ , there exists a constant C > 0 independent of (A,u) and g′ such that ||g′||L2 k+1 ≤ C||D(A,u)g′||L2 k . Thus, using (3.5), we obtain ||g′||L2 k+1 ≤ C ( ||g′||L2 k ( ||α1||L2 k + ||α2||L2 k + ||υ2||L2 k ) + ||α12||L2 k + ||υ12||L2 k ) . Hence, ||g′||L2 k+1 ≤ C 1 − 3εC ( ||α12||L2 k + ||υ12||L2 k ) for ε < 1/3C. Thus, we get ||cg − idE||L2 k+1 < C′ε for ε small, where C′ is a positive constant. From this, the assertion of the lemma is reduced to Step 1. Remark 3.3. By modifying the proof of Lemma 3.2, one can prove that for (A,u) ∈ Ck(E), there exists ε > 0 such that S(A,u),ε/Γ̂(A,u) is diffeomorphic to P(A,u) ( S(A,u),ε/Γ̂(A,u) ) in Ck(E)/Ĝk+1(E), where Γ̂(A,u) = Γ(A,u)/U(1), fol- lowing, for example, [22, Theorem 4.4]. 100 y. tanaka Kuranishi model. This is also a familiar picture in gauge theory. We describe it for the Donaldson-Thomas instanton case, modifying known results in the ASD and Hermitian-Einstein connections (cf. [16, Proposition 8], [31, Chapter VII, §4, Theorem 4.20], and [36, Proposition 4.5.3]). We take (A,u) ∈ Dk(E), and consider a deformation (A + α,u + υ′) ∈ Dk(E), where (α,υ) ∈ L2k ( u(E) ⊗ (Λ1 ⊕ A0,3) ) . Then, (α,υ) satisfies the following: d+Aα + π +(α ∧ α) + Bu(υ) + Λ2[υ,ῡ] + D̄∗Aυ + ∗̄α∗̄υ = 0 , (3.6) where Bu(υ) := Λ 2([u,ῡ] + [υ,ū]). Associated to the operator D+ (A,u) (D+ (A,u) )∗ : L2k(u(E) ⊗ Λ +) −→ L2k(u(E) ⊗ Λ +) , there exist the Green operator G2 : L2k(u(E) ⊗ Λ +) → L2k(u(E) ⊗ Λ +) and the harmonic projection H : L2k(u(E) ⊗ Λ +) → L2k(u(E) ⊗ Λ +) with the identity: Id = H + D+ (A,u) (D+ (A,u) )∗ ◦ G2 (see e.g. [52, Chapter IV, §5]). Using these, we define a map K(A,u) : L 2 k(u(E) ⊗ (Λ 1 ⊕ A0,3)) −→ L2k(u(E) ⊗ (Λ 1 ⊕ A0,3)) by K(A,u)(α,υ) := ( α + ( d+A )∗ ◦ G2 ◦ (π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ) , υ + ( D̄′A + (B ∗ u) ′) ◦ G2 ◦ (π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ)) , where (B∗u) ′ = B∗u ◦ πω, B∗u : Ω0ω → A0,3 is the adjoint of Bu, and πω is the orthogonal projection from Ω2 to Ω0ω. Lemma 3.4. A pair (α,υ) ∈ L2k(u(E) ⊗ (Λ 1 ⊕ A0,3)) satisfies (3.6) if and only if it satisfies D+ (A,u) K(A,u)(α,υ) = 0 and H ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ ) = 0 . moduli space of donaldson-thomas instantons 101 Proof. Using the identity Id = H + D+ (A,u) ( D+ (A,u) )∗ ◦ G2, we rewrite the left-hand side of (3.6) as d+A ( α + (d+A) ∗ ◦ G2 ◦ ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ )) + Bu(υ) + D̄∗A ( υ + ( D̄′A + (B ∗ u) ′) ◦ G2 ◦ (π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ)) + H ◦ ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ ) = D+ (A,u) K(A,u) + H ◦ ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ ) . (3.7) Hence, if D+ (A,u) K(A,u) = 0 and H ◦ ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ ) = 0, then (3.6) holds. Conversely, if (3.6) holds, then from (3.7) we get D+ (A,u) K(A,u) + H ◦ ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ ) = 0 . Thus, ( D+ (A,u) )∗ D+ (A,u) K(A,u) = 0. This implies ∥∥D+ (A,u) K(A,u) ∥∥ L2 k−1(u(E)⊗Λ +) = 0, hence, D+ (A,u) K(A,u) = 0 and H ◦ ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ ) = 0. We put Sd (A,u),ε := { (α,υ) ∈ S(A,u),ε : (α,υ) satisfies (3.6) } , and denote by Hi (A,u) (Z,u(E)) (i = 0,1,2) the harmonic spaces of the complex (3.1). Lemma 3.5. K(A,u) ( Sd (A,u),ε ) ⊂ H1 (A,u) (Z,u(E)) . Proof. From the definition of the map K(A,u), we have D∗(A,,u)K(A,,u)(α,υ) = D ∗ (A,u)(α,υ) + D∗(A,u)(D + (A,u) )∗ ( G2 ◦ ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ )) for (α,υ) ∈ Sd (A,u),ε . This is equal to 0, because D∗ (A,u) (α,υ) = 0 for (α,υ) ∈ Sd (A,u),ε , and D∗ (A,u) ( D+ (A,u) )∗ = 0 as D+ (A,u) D(A,u) = 0. From Lemma 3.4, we also have D+ (A,u) K(A,u) = 0. Thus Lemma 3.5 holds. From Lemmas 3.4 and 3.5, we deduce the following. Lemma 3.6. A pair (α,υ) ∈ L2k ( u(E) ⊗ (Λ1 ⊕ A0,3) ) lies in Sd (A,u),ε if and only if K(A,u)(α,υ) ∈ H1(A,u)(Z,u(E)) and H ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ ) = 0 . 102 y. tanaka We now prove the following. Theorem 3.7. Let (A,u) ∈ D∗(E). Then there exists a neighbourhood U of 0 in H1 (A,u) (Z,u(E)) such that around [(A,u)] the moduli space M∗(E) = D∗(E)/Ĝ(E) is locally modeled on the zero set of a real analytic map κ(A,u) : U → H2(A,u)(Z,u(E)) with κ(A,u)(0) = 0, and the first derivative of κ(A,u) at 0 also vanishes. Proof. From the definition of the map K(A,u), we have K(A,u)(0) = 0. Since the differential of K(A,u) at 0 is identity, we can deduce, from the inverse mapping theorem on the Hilbert spaces (see e.g. [35, Chapter 6]), that there exist a neighbourhood U of 0 in H1 (A,u) (Z,u(E)) and a map K−1 (A,u) : U −→ L2k ( u(E) ⊗ ( Λ1 ⊕ A0,3 )) such that K−1 (A,u) is a diffeomorphism between U and K−1 (A,u) (U). We then define a map κ(A,u) : U → H2(A,u) by κ(A,u) = ψ ◦ K −1 (A,u) , where ψ : H1 (A,u) → H2 (A,u) is defined by ψ(α,υ) = H ( π+(α ∧ α) + Λ2[υ,ῡ] + ∗̄α∗̄υ ) . We now take ε sufficiently small so that all the following hold. Firstly, from Lemma 3.6, the zero set of κ(A,u) is mapped by K −1 (A,u) diffeomorphically to an open subset in Sd (A,u),ε . Next, from Proposition 3.2, Sd (A,u),ε is diffeo- morphic to p(A,u),ε(S d (A,u),ε ) in D∗k(E)/Ĝk+1(E). Hence, the zero set of κ(A,u) is diffeomorphic to a neighbourhood of [(A,u)] in D∗k(E)/Ĝk+1(E). More- over, from the elliptic regularity, the harmonic elements are actually smooth, therefore the neighbourhood of [(A,u)] in D∗k(E)/Ĝk+1(E) is isomorphic to a neighbourhood of [(A,u)] in M∗(E). The assertions that κ(A,u) = 0 and the derivative of κ(A,u) at 0 is zero just follow from the definition κ(A,u) = ψ ◦K −1 (A,u) and the fact that the differential of K(A,u) at 0 is the identity. From Theorem 3.7, one can deduce that M∗(E) is smooth around [(A,u)] if H2 (A,u) (Z,u(E)) = 0. But, as in the case of the Hermitian-Einstein connections (cf. [30], [31, Chapter VII, §4], [25, Chapter 2, §2.1], [36, Chapter 4, §4.5]), it can be improved in the following way. Firstly, we note that, corresponding to the decomposition of u(r) = iR ⊕ su(r), the bundle u(E) naturally decom- poses into R and u(E)0 over Z, where u(E)0 is the bundle of trace-free skew- Hermitian endmorphisms of E, and there is a subcomplex of the complex (3.1), moduli space of donaldson-thomas instantons 103 which is defined by using the bundle u(E)0 instead of u(E). The decomposition is preserved by the operators of the complex, hence it induces a corresponding splitting of Hi (A,u) (Z,u(E)) (i = 0,1,2). For (αc,υc) ∈ Λ1(Z) ⊕ A0,3(Z), it is always H ( π+(αc ∧αc) + Λ2[υc, ῡc] + ∗̄αc∗̄υc ) = 0, hence the map κ(A,u) values in H2(Z,u(E)0). In particular, we obtain the following. Corollary 3.8. Around [(A,u)] ∈ M∗(E) with H2 (A,u) (Z,u(E)0) = 0, the moduli space M∗(E) is smooth. Remark 3.9. Around (A,u) ∈ D(E), which is not irreducible, one can prove that H1 (A,u) (Z,u(E)) and H2 (A,u) (Z,u(E)) are Γ(A,u)-invariant, and the map κ(A,u) is Γ(A,u)-equivariant. Hence, combining the claim in Remark 3.3, one can deduce that around [(A,u)] the moduli space M(E) is locally modeled on κ−1 (A,u) (0)/Γ(A,u). 4. The Hitchin-Kobayashi correspondence for the Donaldson-Thomas instantons on compact Kähler threefolds Perhaps one might ask what kind of a Hitchin-Kobayashi style correspon- dence would hold for the Donaldson-Thomas instanton on compact Kähler threefolds. In this section, we describe this, which actually follows from a result by Álvarez-Cónsul and Garćıa-Prada [1]. Let Z be a compact Kähler threefold, and E = (E,h) a Hermitian vector bundle over Z with Hermitian metric h. If (A,u) is a D-T instanton on E, then the connection A defines a holomorphic structure ∂̄A on E as F 0,2 A = 0, thus, we can think of E as a locally free sheaf O(E,∂̄A). In addition, the End(E)-valued (0,3)-form u is naturally identified with a section of the bundle End(E) ⊗ K−1Z , so ∗̄u is a section of the bundle End(E) ⊗ KZ. The equation ∂̄∗Au = 0 implies ∂̄A∗̄u = 0, hence, φ := ∗̄u is a holomorphic section of End(E) ⊗ KZ. We then consider a pair (E,φ) consisting of a torsion-free sheaf E and a holomorphic section φ of End(E) ⊗ KZ. A subsheaf F of E is said to be a φ-invariant if φ(F) ⊂ F ⊗KZ. We define a slope µ(F) of a coherent subsheaf F of E by µ(F) := 1 rank(F) ∫ Z c1(det F) ∧ ω2 . Definition 4.1. A pair (E,φ) consisting of a torsion-free sheaf E and a holomorphic section φ of End(E)⊗KZ is called semi-stable if µ(F) ≤ µ(E) for 104 y. tanaka any φ-invariant coherent subsheaf F with rank(F) < rank(E). A pair (E,φ) is called stable if µ(F) < µ(E) for any φ-invariant coherent subsheaf F with rank(F) < rank(E). Definition 4.2. A pair (E,φ) consisting of a torsion-free sheaf E and a holomorphic section φ of End(E)⊗KZ is said to be poly-stable if it is a direct sum of stable sheaves with the same slopes in the sense of Definition 4.1. Then the correspondence can be stated as a one-to-one correspondence between a pair (E,φ), where E is a locally-free sheaf on a Kähler threefold Z and a holomorphic section φ of End(E) ⊗ KZ, which is stable in the sense of Definition 4.1; and the existence of a solution to the Donaldson-Thomas equations on E. This fits into a setting studied by Álvarez-Cónsul and Garćıa- Prada [1] (see also [6]), and it is stated as a special case of their results as the case of a twisted quiver bundle with one vertex and one arrow, whose head and tail conincide, and with twisting sheaf the anti-canonical bundle. We state it in our setting as follows. Theorem 4.3. ([1]) Let Z be a compact Kähler threefold with Kähler form ω. Let (E,φ) be a pair consisting of a locally-free sheaf E on Z and a holomorphic section φ of End (E)⊗KZ. Then, (E,φ) is poly-stable if and only if E admits a unique Hermitian metric h satisfying ΛFh + Λ 3[φ,φ̄h] + 6πiµ(E)IdE = 0 , where Fh is the curvature form of h, and Λ := (∧ω)∗. Note that the equation ∂̄∗Au = 0 in the Donaldson-Thomas equations on a compact Kähler threefold is implicitly addressed in Theorem 4.3 by saying that φ = ∗̄u is a holomorphic section of End(E) ⊗ KZ. One more remark is that a proof of the Hitchin-Kobayashi correspondence using the Mehta- Ramanathan argument for the Vafa-Witten equations in [46] could also apply to the Donaldson-Thomas instanton on smooth projective threefold as men- tined in [46]. Acknowledgements I would like to thank Mikio Furuta, Ryushi Goto, Ryoichi Kobayashi, Hiroshi Ohta for valuable comments, and referees for many useful advice. I am also grateful to Katrin Wehrheim for wonderful encouragement. A part of this article was written when I had visited Beijing International Center for Mathematical Research, Peking University in 2008 – 2009, moduli space of donaldson-thomas instantons 105 I am very grateful to Gang Tian and the institute for their support and hospitality. A part of revision was made during my visit to Institut des Hautes Études Scientifiques in February to March of 2012. I would like to thank the institute for the support and giving me an excellent research environment. Last but not least, I would like to thank Dominic Joyce for enlightening me on these subjects over the years. This work was partially supported by JSPS Grant-in-Aid for Scientific Research No. 15H02054. References [1] L. Álvarez-Cónsul, O. Garćıa-Prada, Hitchin-Kobayashi correspon- dence, quivers and vortices, Comm. Math. Phys. 238 (2003), 1 – 33. [2] M.F. Atiyah, N.J. Hitchin, I.M. Singer, Self-duality in four-dimen- sional Riemannian geometry, Proc. Roy. Soc. London Ser. A 362 (1978), 425 – 461. [3] L. Baulieu, H. Kanno, I.M. Singer, Special quantum field theories in eight and other dimensions, Comm. Math. Phys. 194 (1998), 149 – 175. [4] O. Ben-Bassat, C. Brav, V. Bussi, D. Joyce, A ’Darboux Theorem’ for shifted symplectic structures on derived Artin stacks, with applications, Geom. Topol. 19 (2015), 1287 – 1359. [5] D. Borisov, D. Joyce, Virtual fundamental classes for moduli spaces of sheaves on Calabi-Yau four-folds, arXiv:1504.00690. [6] S.B. Bradlow, O. Garćıa-Prada, I. Mundet i Riera, Relative Hitchin-Kobayashi correspondences for principal pairs, Q. J. Math. 54 (2003), 171 – 208. [7] C. Brav, V. Bussi, D. Dupont, D. Joyce, B. Szendroi, Symmetries and stabilization for sheaves of vanishing cycles, With an appendix by Jörg Schürmann, J. Singul. 11 (2015), 85 – 151. [8] C. Brav, V. Bussi, D. Joyce, A Darboux theorem for derived schemes with shifted symplectic structure, arXiv:1305.6302. [9] V. Bussi, D. Joyce, S. Meinhardt, On motivic vanishing cycles of critical loci, arXiv:1305.6428. [10] Y. Cao, Donaldson-Thomas theory for Calabi-Yau four-folds, arXiv:1309.4230. [11] Y. Cao, N.C. Leung, Donaldson-Thomas theory for Calabi-Yau fourfolds, arXiv:1407.7659. [12] Y. Cao, N.C. Leung, Orientability for gauge theories on Calabi-Yau mani- folds, arXiv:1502.01141. [13] Y. Cao, N.C. Leung, Relative Donaldson-Thomas theory for Calabi-Yau four-folds, arXiv:1502.04417. [14] Y. Cao, N.C. Leung, Remarks on mirror symmetry of Donaldson-Thomas theory for Calabi-Yau 4-folds, arXiv:1506.04218. [15] R. Dijkgraaf, S. Gukov, A. Neitzke, C. Vafa, Topological M-theory as unification of form theories of gravity, Adv. Theor. Math. Phys. 9 (2005), 603 – 665. 106 y. tanaka [16] S.K. Donaldson, An application of gauge theory to four dimensional topol- ogy, J. Differential Geom. 18 (1983), 279 – 315. [17] S.K. Donaldson, Anti self-dual Yang-Mills connections over complex alge- braic surfaces and stable vector bundles, Proc. London Math. Soc. (3) 50 (1985), 1 – 26. [18] S.K. Donaldson, Infinite determinants, stable bundles and curvature, Duke Math. J. 54 (1987), 231 – 247. [19] S.K. Donaldson, P.B. Kronheimer, “ The Geometry of Four- Manifolds ”, Oxford Mathematical Monographs, The Clarendon Press, Oxford University Press, New York, 1990. [20] S. Donaldson, E. Segal, Gauge theory in higher dimensions, II, in “ Surveys in Differential Geometry, Volume XVI ”, Int. Press, Somerville, MA, 2011, 1 – 41. [21] S.K. Donaldson, R.P. Thomas, Gauge theory in higher dimensions, in “The Geometric Universe”, Oxford University Press, Oxford, 1998, 31 – 47. [22] D.S. Freed, K.K. Uhlenbeck, “ Instantons and Four-Manifolds ”, Second edition, Springer-Verlag, New York, 1991. [23] P. Gilkey, “ Invariance Theory, the Heat Equation, and the Atiyah-Singer Index Theorem ”, Second edition, CRC Press, Boca Raton, FL, 1995. [24] N. Hitchin, The self-duality equations on a Riemann surface, Proc. London Math. Soc. (3) 55 (1987), 59 – 126. [25] M. Itoh, H. Nakajima, Yang-Mills connections and Einstein-Hermitian metrics, in “ Kähler Metric and Moduli Spaces ”, Academic Press, Boston, MA, 1990, 395 – 457. [26] A. Iqbal, N. Nekrasov, A. Okounkov, C. Vafa, Quantum foam and topological strings, J. High Energy Phys. 4 (2008), 011. [27] D. Joyce, A classical model for derived critical loci, J. Differential Geom. 101 (2015), 289 – 367. [28] D. Joyce, Y. Song, “ A Theory of Generalized Donaldson-Thomas Invari- ants ”, Mem. Amer. Math. Soc. 217 (2012), no. 1020. [29] Y-H. Kiem, J. Li, Categorification of Donaldson-Thomas invariants via Per- verse Sheaves, arXiv:1212.6444. [30] H. Kim, Moduli of Hermite-Einstein vector bundles, Math. Z. 195 (1987), 143 – 150. [31] S. Kobayashi, “ Differential Geometry of Complex Vector Bundles ”, Publica- tions of the Mathematical Society of Japan, 15; Princeton University Press, Princeton, NJ; Iwanami Shoten, Tokyo; 1987. [32] M. Kontsevich, Y. Soibelman, Stability structures, motivic Donaldson- Thomas invariants and cluster transformations, arXiv:0811.2435. [33] M. Kontsevich, Y. Soibelman, Cohomological Hall algebra, exponen- tial Hodge structures and motivic Donaldson-Thomas invariants, Commun. Number Theory Phys. 5 (2011), 231 – 352. [34] M. Kontsevich, Y. Soibelman, Wall-crossing structures in Donaldson- Thomas invariants, integrable systems and mirror symmetry, in “ Homological Mirror Symmetry and Tropical Geometry ”, Lect. Notes moduli space of donaldson-thomas instantons 107 Unione Mat. Ital., 15, Springer, Cham, 2014, 197 – 308. [35] S. Lang, “ Real Analysis ”, Second edition, Addison-Wesley Publishing Com- pany, Reading, MA, 1983. [36] M. Lübke, A. Teleman, “ The Kobayashi-Hitchin Correspondence ”, World Scientific Publishing Co., Inc., River Edge, NJ, 1995. [37] D. Maulik, D.N. Nekrasov, A. Okounkov, R. Pandharipande, Gromov-Witten theory and Donaldson-Thomas theory, I, Compos. Math. 142 (2006), 1263 – 1285. [38] D. Maulik, D.N. Nekrasov, A. Okounkov, R. Pandharipande, Gromov-Witten theory and Donaldson-Thomas theory, II, Compos. Math. 142 (2006), 1286 – 1304. [39] N. Nekrasov, Z-theory: chasing m/f theory, C. R. Phys. 6 (2005), 261 – 269. [40] N. Nekrasov, H. Ooguri, C. Vafa, S-duality and topological strings, J. High Energy Phys. 10 (2004), 009. [41] R. Reyes Carrión, A generalization of the notion of instanton, Differential Geom. Appl. 8 (1998), 1 – 20. [42] S.M. Salamon, “ Riemannian Geometry and Holonomy Groups ”, Pitman Research Notes in Mathematics Series, 201; Longman Scientific & Technical, Harlow; John Wiley & Sons, Inc., New York; 1989. [43] Y. Tanaka, A construction of Spin(7)-instantons, Ann. Global Anal. Geom. 42 (2012), 495 – 521. [44] Y. Tanaka, A weak compactness theorem of the Donaldson-Thomas instan- tons on compact Kähler threefolds, J. Math. Anal. Appl. 408 (2013), 27 – 34. [45] Y. Tanaka, A removal singularity theorem of the Donaldson-Thomas in- stanton on compact Kähler threefolds, J. Math. Anal. Appl. 411 (2014), 422 – 428. [46] Y. Tanaka, Stable sheaves with twisted sections and the Vafa-Witten equations on smooth projective surfaces, Manuscripta Math. 146 (2015), 351 – 363. [47] C.H. Taubes, Casson’s invariant and gauge theory, J. Differential Geom. 31 (1990), 547 – 599. [48] R.P. Thomas, A holomorphic Casson invariant for Calabi-Yau 3-folds, and bundles on K3 fibrations, J. Differential Geom. 54 (2000), 367 – 438. [49] G. Tian, Gauge theory and calibrated geometry, I, Ann. of Math. (2) 151 (2000), 193 – 268. [50] K. Uhlenbeck, S-T. Yau, On the existence of Hermitian-Yang-Mills con- nections in stable vector bundles, Comm. Pure Appl. Math. 39 (suppl.) (1986), S257 – S293. [51] K. Uhlenbeck, S-T. Yau, A note on our previous paper: On the existence of Hermitian-Yang-Mills connections in stable vector bundles, Comm. Pure Appl. Math. 42 (1989), 703 – 707. [52] R.O. Wells, Jr., “ Differential Analysis on Complex Manifolds ”, Third edi- tion, with a new appendix by Oscar Garcia-Prada, Graduate Texts in Math- ematics, 65, Springer, New York, 2008.