2_denizhowe.dvi EUROPEAN JOURNAL OF PURE AND APPLIED MATHEMATICS Vol. 5, No. 3, 2012, 282-301 ISSN 1307-5543 – www.ejpam.com Performance of Information Complexity Criteria in Structural Equation Models with Applications Eylem Deniz Howe1,∗, Hamparsum Bozdogan2, and Gülay Kıroğlu1 1 Mimar Sinan Fine Arts University, Istanbul Turkey 2 Department of Statistics, Operations, and Management Science, The University of Tennessee, Knoxville, TN, 37996, USA Abstract. A common problem in structural equation modeling is that of model selection. Many re- searchers have addressed this problem, but many methods have provided mixed benefits until recently. Akaike’s well-known criteria, AIC , has been applied in the context of structural equation modeling, but the effectiveness of many other information criteria have not been studied in a convincing manner. In this paper, we compare the SEM model selection prowess of several AIC-type and ICOM P-type crite- ria. We also introduce two new large sample consistent forms of Bozdogan’s ICOM P criteria - one of which is robust to model misspecification. To study the empirical performance of the information criteria, we use a well-known SEM simulation protocol, and demonstrate that most of the information-theoretic criteria select the “pseudo true” model with very high frequencies. We also demonstrate, however, that the performance of AIC is inversely related to the sample size. Finally, we apply the new criteria to select an analytical model for a real dataset from a retail marketing study of consumer behavior. Our results show the versatility of the new proposed method where both the goodness-of fit and the complexity of the model is taken into account in one criterion function. 2010 Mathematics Subject Classifications: 62H12,62H15,62H20,62H25,62P15,62F35 Key Words and Phrases: Structural Equation Model, Information Criteria, AIC-type Criteria, ICOM P- type Criteria 1. Introduction Structural Equation Modeling (SEM) has been a popular tool in social and behavioral sciences since the early 1990’s for the causal modeling of complex, multivariate data sets. Presently, it has penetrated engineering, management and information sciences, and genomic research, to mention a few. It has changed the perspective of researchers on how to do good statistical modeling and model selection. It has also fostered the curiosity of social ∗Corresponding author. Email addresses: eylemdeniz�gmail. om (E. Howe), bozdogan�utk.edu (H. Bozdogan), gkiroglu�msu.edu.tr (G. Kıroğlu) http://www.ejpam.com 282 c© 2012 EJPAM All rights reserved. E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 283 scientists about the underlying constructs or factors in many applications. According to [11], the SEM approach has been regarded, perhaps, as the most significant and effective statistical revolution in social sciences. In SEM one of the fundamental problems is how to evaluate different models and how to select among the rival models [18]. In the literature of SEM, there are many alternative fit indices available to the researchers. An excellent recent book by [22], “Linear Causal Mod- eling with Structural Equations” devotes a fair amount of discussion on many goodness-of-fit indices. Mulaik [22] further discusses information theoretic measures of model discrepancy such as AIC, see, e.g., [1, 2, 27, 4]. He devotes a section on the information theoretic mea- sure of complexity (ICOMP) developed in [6, 7]. However, [22] does not provide numerical examples or evaluate the empirical performance of these model selection criteria in the SEM framework. The first author in [14], evaluated the performance of several information criteria for measurement models and SEMs. In this paper, our objective is to to extend that research and to present some new results on ICOMP which is consistent with respect to sample size and robustness against model misspecification in SEM and in other high dimensional model fitting. Therefore, in this paper we revisit and discuss information criteria used to gauge model fit in structural equation modeling (SEM). This is based on the objective that statistical methods of fit are related to sample size, any statistical test is increasingly likely to imply rejection of a model as sample size increases, even if model misfit is trivial in magnitude. Various practical fit indices have been proposed over the years - and information criteria are among the most widely used. We review prior information criteria, and propose a couple of new information criteria based on information measure of complexity of a covariance matrix called ICOM P- type criteria. We demonstrate the empirical performance of ICOM P-type criteria along with AIC , Con- sistent Akaike Information Criterion (CAIC), Consistent Akaike Information Criterion with Fisher Information (CAIC F) both due to [4], and a Bayesian Model Selection (BMS) crite- rion developed by [10]. The rest of the paper is organized as follows. Section 2 provides the requisite background on SEM, which is then followed by Section 3 regarding AIC-type and ICOM P-type criteria for model selection. Here, we define ICOM P under the correct and misspecified models. The basic idea is that one can use the difference between ICOMP(misspecified model) and ICOMP(correctly specified model) as an indication of possible departures from the distribu- tional form of the model. This brings out the most important weakness of Akaike-type criteria for model selection, that AIC depends crucially on the assumption that the specified family of models includes the pseudo true model. In general, this may not be the case. Finally, re- sults with both simulated and real datasets are shown in Section 5. Our simulation results are based on three models based on a slight modification of Fan’s simulation protocol [15] to achieve convergence in LISREL. E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 284 2. Structural Equation Model with Latent Variables General SEMs consist of two models referred to as measurement and structural equation (or latent variable) models. The structural equation model defines relationship between la- tent variables. This model is a path model adapted to latent variables. The measurement model is a confirmatory factor (CF) model which defines relationship between observed and latent variables. Latent variables in the measurement model are the factors in the CF model. Following [19] in matrix notation, we define the general SEM model with these two models by three equations: Structural Equation model: η (r×1) = Bη (r×r)(r×1) + Γξ (r×s)(s×1) + ζ (r×1) Measurement model for y: y (p×1) = Λyη (p×r)(r×1) + ǫ (p×1) Measurement model for x : x (q×1) = Λxξ (q×s)(s×1) + δ (q×1) where η is a (r×1) vector of latent endogenous (or dependent) variables, ξ is (s×1) vector of latent exogenous (or independent) variables, ζ is (r × 1) vector of latent errors in equations, β is an (r × r) coefficient matrix for the latent endogenous variables, and Γ is a (r × s) co- efficient matrix for the latent exogenous variables. The structural model specifies the causal relationships among the latent endogenous variables in β , between the exogenous and en- dogenous variables in Γ, and describes unexplained residuals of the latent factors in ζ. The usual assumptions for the structural model are that: E(η) = E(ξ) = E(ζ) = 0, ζ is uncorrelated with ξ, and (I −β) is nonsingular. The covariance matrix, E(ξξ′) = Φ, (1) is an (s× s) covariance matrix of the latent exogenous variables, and E(ζζ′) = Ψ (2) is an (r × r) covariance matrix of the latent errors in equations. The measurement model specifies how the observed variables, x and y, are determined through Λx and Λy by the latent variables, ξ and η, respectively. The ǫ and δ terms represent the residuals in x and y unexplained by ξ and η. The usual assumptions for the measurement model are: E(η) = E(ξ) = E(ǫ) = E(δ) = 0 . Furthermore, ǫ is uncorrelated with ξ, η and δ. Likewise, δ is uncorrelated with ξ, η and ǫ. The covariances matrices in the case of the measurement model are: E(ǫǫ′) = Θǫ = Σ(ǫ), (3) a (p× p) covariance matrix of ǫ, and E(δδ′) = Θδ = Σ(δ), (4) a (q× q) covariance matrix of δ. E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 285 According to the measurement and structural equation models, the implied full covariance matrix, Σ(θ), for the general SEM is given by Σ(θ) = � Λy(I − B)−1(ΓΦΓ′ +Ψ)[(I − B)−1]′Λ′y +Θε Λy(I − B)−1ΓΦΛ′x (Λy(I − B)−1ΓΦΛ′x ) ′ ΛxΦΛ ′ x +Θδ � . (5) The elements of the general implied covariance matrix are functions of B, Γ, Λy , Λx , Φ, Ψ, Θε and Θδ. To be able to use SEM, we need to specify the pattern of the elements of these eight matrices. There are three kinds of specifications. These are: • Fixed parameters- that have been assigned specific values, • Constrained parameters- that are unknown but equal to one or more other parameters, and • Free parameters- that are unknown and not constrained to be equal to any other pa- rameter. For more on these, see [3, 19], and others. The fundamental hypothesis for these structural equations procedures is that the covari- ance matrix of the observed variables is a function of a set of parameters. If the model is correct and if we know the parameters, then the population covariance matrix can be exactly reproduced. The hypothesis in this case is: Σ = Σ(θ). (6) In (6), Σ is the population covariance matrix of the observed variables, θ is a vector containing the free parameters of the model, and Σ(θ) is the covariance matrix written as a function of the covariance matrix implied by a specific model [3]. 3. Information Theoretic Model Selection Criteria One of the fundamental difficulties in statistical analysis and data mining is the choice of an appropriate model, estimating and determining the order or dimension of a model. In gen- eral statistical modeling and model evaluation problems, the concept of information theoretic measure of complexity plays an important role. At the philosophical level, complexity involves notions such as connectivity patterns and the interactions of model components. Without a measure of overall model complexity, prediction of model behavior and assessing model quality is difficult. This requires detailed statistical analysis and computation to choose the best fitting model among a portfolio of competing models for a given finite sample [4]. Based on Akaike’s AIC , many model-selection procedures that take the form of a penalized likelihood plus a penalty term) have been proposed. For a general multivariate linear or nonlinear model defined by Statistical model = Signal + N oise (7) E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 286 a summary diagram for AIC and ICOMP in terms of a loss function is give by Loss = Lack of Fit +Lack of Parsimony « −→ AIC +Profusion of Complexity    −→ ICOM P 3.1. Akaike-Type Information Criteria Since Akaike (1973), entropic information criteria has had a substantial impact in statis- tical model evaluation problems. Its introduction furthered the recognition of the importance of good modeling and model fitting in the statistical science. As a result, many important statistical modeling techniques have been developed in various fields of statistics such as in biostatistics, control theory, econometrics, engineering, medical data mining, psychometrics, and many others. Akaike Information Criterion (AIC) AIC , which is one of the first information criterion developed by Akaike, can be thought of as a generalized form of the maximum likelihood. AIC can be formulated as maximizing generalized entropy, or equivalently minimizing Kullback-Leibler (KL) information. See, e.g. [21], and [20]. It is obtained from asymptotic unbiased estimation of the logarithm of the average expected likelihood of a model and it is defined by AIC = −2 log L(bθk) + 2k, (8) where θ is a k-dimensional unknown parameter vector, bθ , denotes the maximum likelihood estimator of θ , L(bθ ), is the maximized likelihood function of the model, and k is the number of unknown parameters estimated in the model. Consistent Akaike Information Criterion (CAIC) Without violating Akaike’s principles and using the results in mathematical statistics, Bozdo- gan [4] extended AIC analytically in two ways. These extensions make AIC asymptotically consistent, and penalize overparameterization more stringently, so as to pick the simplest of the true models whenever there is nothing to be lost in doing so. CAIC is defined by CAIC = −2 log L(bθk) + k(log(n) + 1). (9) Using a correction factor based on the sample size (n) CAIC is an attempt to overcome the tendency of the AIC in overestimating the complexity (i.e., number of parameters) of the underlying model [17]. E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 287 Consistent Akaike Information Criterion with Fisher Information (CAICF) Also in [4], a different estimator for negative twice the expected entropy was proposed. Sim- ilar to CAIC , this approach extends AIC analytically to make it consistent without deviat- ing from Akaike’s original premise. In this manner [4] penalizes overparameterization more strongly, in particular, in large samples. CAIC F is defined by CAIC F = −2 log L(bθ) + k(log(n)+ 2)+ log |F̂ |, (10) where F̂ is the estimated Fisher information matrix (FIM) of the model. Bayesian Model Selection Criterion (BMS) Bozdogan and Ueno [10] provided yet a further extension of AIC called Bayesian Model Selection Criterion (BMS), given by BMS = −2 log L(bθ ) + k log(n)+ 2( nk n− k− 2 ) + log |F̂ |. (11) As we note, these extended forms AIC all attempt to repair the inconsistency problem of the AIC . 3.2. The General Form of Information Complexity: ICOMP Motivated from considerations similar to those in AIC , here we give details a new entropic or information-theoretic measure of complexity called ICOM P of [6, 7, 9] as a decision rule for model selection in SEM. The development and construction of ICOM P is based on a generalization of the covari- ance complexity index originally introduced in [29]. Instead of penalizing the number of free parameters directly, ICOM P penalizes the covariance complexity of the model. It is defined by ICOM P = −2 log L(θ̂k) + 2C(ÔCov(θ̂k)), (12) where L(θ̂k) is the maximized likelihood function, θ̂k is the maximum likelihood estimate of the parameter vector θk under the model Mk with k unknown parameter, and C represents a real-valued complexity measure. As explained in [7, 8, 9], there are several forms and theoretical justifications of ICOM P. For brevity, here, we will recapitulate and use one of the most general forms of ICOM P referred to as ICOM P(I F IM). ICOM P(I F IM) exploits the well-known asymptotic optimality properties of the maximum likelihood estimators (MLEs), and uses the information-based complexity of the estimated inverse-Fisher information matrix (I F IM or F̂−1) of a model. This is known as the Cramér-Rao lower bound (CRLB) matrix. See, e.g., [12] and [23, 24, 25]. As such, ICOM P(I F IM) is an approximation to the sum of two KL distances. This approach provides us an achievable accuracy of the parameter estimates of the model by considering the entire parameter space. E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 288 For a multivariate normal linear or nonlinear structural model we define the general form of ICOM P(I F IM) as ICOM P(F̂−1) = −2 log L(θ̂k) + 2C1(F̂ −1), (13) where C1 denotes the maximal informational complexity of F̂−1 and is given by C1(F̂ −1) = s 2 log[ t r(F̂−1) s ]− 1 2 log |F̂−1|, (14) and where s = dim(F̂−1) = rank(F̂−1). The first component of ICOM P(I F IM) measures the lack of fit of the model, and the second component measures the complexity of the accuracy of the estimated parameters and implicitly adjusts for the number of free parameters included in the model. It is a measure of the state of disorder of a model for a given data set. For more on this, and for some immediate physical motivation, we refer the readers to the interesting book by [16], entitled “Physics from Fisher Information.” The trace of I F IM in the complexity measure involves only the diagonal elements anal- ogous to variances while the determinant involves also the off-diagonal elements analogous to covariances. Therefore, ICOM P(I F IM) contrasts the trace and the determinant of I F IM , and this amounts to a comparison of the geometric and arithmetic means of the eigenvalues of I F IM given by ICOM P(F̂−1) = −2 log L(θ̂k) + s log � λa/λg � . (15) We note that ICOM P(I F IM) now looks in appearance like the CAIC , M DL [26], and the Bayesian criterion SBC [28], except for using log � λa/λg � instead of using log(n), where log(n) denotes the natural logarithm of the sample size n. In this sense, ICOM P(I F IM) resembles a penalized likelihood method similar to AIC and AIC-type criteria, except that the penalty depends on the curvature of the log likelihood function via the scalar C1 complexity value of the estimated I F IM . 3.3. Consistent and Misspecification Forms of ICOMP Bozdogan [9] suggested several other different approaches to generalize and derive the ICOM P criteria by maximizing the posterior expected utility (PEU) when the models are mis- specified (which is the case in practice). Here, we give derived forms of these generalized ICOM P criteria as follows. ICOM P � F̂−1 � PEU_Miss = −2 log L � θ̂ � + k+ 2 � t r � F̂−1R̂ � + C1 � F̂−1 �� (16) In (16), whereas F̂−1 is I F IM in inner-product form, R̂ is I F IM in outer-product form. If the model is correctly specified, these two forms of Fisher information matrices would be equal to one another. That is, if F̂−1 = R̂ , then t r(F̂−1R̂) = t r � Ik � = k. In this case, we have ICOM P � F̂−1 � PEU = −2 log L(θ̂ ) + 3k+ 2C1(F̂ −1) E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 289 = AIC3 + 2C1(F̂ −1) (17) Bozdogan [9] approximates the term t r(F̂−1R̂) in equation (16) by t r � F̂−1R̂ � = nk n− k− 2 which corrects the bias for small as well as large sample sizes. Thus, equation (16) reduces to ICOM P � F̂−1 � PEU_Miss = −2 log L(θ̂ ) + k+ 2( nk n− k− 2 ) + 2C1(F̂ −1). (18) This form of ICOM P is useful in cases where we cannot determine the closed form expression of R̂, outer-product form of I F IM in many modeling situations and SEM is a case in point example. Consistent ICOM P that maximizes the posterior expected utility (PEU)is given by C ICOM P(F̂−1)PEU = −2 log L(θ̂ |X )+ k+ 2k log(n)+ 2C1(F̂ −1) = CAIC + 2C1(F̂ −1). (19) Further, consistent ICOM P that maximizes the posterior expected utility (PEU) and also guards us against misspecification is given by C ICOM P(F̂−1)PEU_Miss = −2 log L(θ̂ |X )+ k+ 2 log(n) nk n− k− 2 + 2C1(F̂ −1). (20) A model with minimum ICOMP-type and AIC-type criteria are chosen to be the best model among all possible competing alternative models. With ICOM P-type criteria, complexity is viewed not as the number of parameters in the model, but as the degree of interdependence (i.e., the correlational structure among the pa- rameter estimates). By defining complexity in this way, ICOM P-type criteria provide a more judicious penalty term than AIC , M DL, SBC , or CAIC . The lack of parsimony is automatically adjusted by C1(F̂ −1) across the competing alternative portfolio of models as the parameter spaces of these models are constrained in the model selection process. For more on ICOM P- type criteria, we refer the readers to [9] and [13]. 4. Derived Forms of Information Criteria in Structural Equation Models 4.1. AIC-type Criteria in SEM Under the assumption that the observed variables are continuous and have interval scales, and multivariate normal, that is, z = (y ′, x ′)′ ∼ N(p+q)(0,Σ(θ)) (21) and using the maximum likelihood estimators, we obtain AIC , CAIC , CAIC F and BMS for the SEM. These are given in [13] as follows. AIC = n(p+ q) log(2π) + n log |Σ̂|+ ntr(Σ̂−1S) + 2k, (22) E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 290 CAIC = n(p+ q) log(2π)+ n log |Σ̂|+ ntr(Σ̂−1S) + k � log(n)+ 1 � , (23) CAIC F = n(p+ q) log(2π)+ n log |Σ̂|+ ntr(Σ̂−1S) + k � log(n)+ 2 � + log |F̂ |, (24) BMS = n(p+ q) log(2π)+ n log |Σ̂|+ ntr(Σ̂−1S) + k log(n)+ ( nk n− k− 2 ) + log |F̂ |, (25) where k is the number of parameters in SEM. 4.2. ICOMP-type Criteria in SEM Let Σ̂(θ̂) denote the maximum likelihood estimator (MLE) of the implied covariance ma- trix Σ(θ) given by Σ̂(θ̂) (p+q)×(p+q) = � Λ̂y(I − B̂)−1(Γ̂Φ̂Γ̂′+ Ψ̂)(I − B̂′)−1Λ̂y + Θ̂ǫ Λ̂y(I − B̂)−1Γ̂Φ̂Λ̂′x Λ̂x Φ̂Γ̂ ′(I − B̂′)−1Λ̂′y Λ̂x Φ̂Λ̂ ′ x + Θ̂δ � . (26) The first ICOM P criterion in SEM is given by ICOM P = n(p+ q) log(2π)+ n log |Σ̂|+ ntr(Σ̂−1S) + 2C1(Σ̂(θ̂)). (27) We can also use the estimated I F IM , the covariance of the model parameters, for the general SEM as given by F̂−1 =   1 n Σ̂(θ̂) 0 0′ 2 n D (p+q) +[Σ̂(θ̂)⊗ Σ̂(θ̂))] D (p+q) + ′   (28) to obtain several other new forms of ICOM P based on the work of [5]. Note that in one dimension equation in (28) reduces to the I F IM of the normal distribu- tion N(µ,σ2) given by F̂−1 = � 1 n σ̂2 0 0′ 2 n σ̂4 � (29) which checks and shows the correctness of the formula in (28) and that θ̂ has the consistency property. In (28), the matrix D+p is the Moore-Penrose inverse of the duplication matrix Dp. The duplication matrix Dp is a unique p2× 1 2 p � p+ 1 � matrix, and so its Moore-Penrose inverse is D+p = � D′pDp �−1 D′p which is a 1 2 p � p+ 1 � × p2 matrix. Further, note that the duplication matrix Dp is implicitly defined by Dpvech(Σ̂(θ̂)) = vec(Σ̂(θ̂)) where vech(Σ̂) vectorizes the distinct elements of Σ̂ by vertically stacking those on and below the principal diagonal. Consequently vech(Σ̂(θ̂ )) = D+p vec(Σ̂(θ̂)). E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 291 Based on the above results, we establish ICOM P(I F IM) for the SEM with latent variables given by ICOM P(F̂−1) = n(p+ q) log(2π)+ n log |Σ̂|+ ntr(Σ̂−1S) + 2C1(F̂ −1). (30) We further note that with C1(F̂ −1), we do not actually need to construct F̂−1, since it is a scalar measure of complexity. As derived in [13], using the definition of complexity, we get a computationally convenient open form of the expression for C1(F̂ −1)— in the sense that all the required inputs are available as a part of the standard output of most SEM packages — is C1(F̂ −1) = s 2 log   1 n t r(Σ̂) + 1 2n  t r(Σ̂2) + (t r(Σ̂))2 + 2 p+q∑ j=1 � σ̂ j j �2   s   − 1 2 (p+ q+ 2) log ��Σ̂ ��+ 1 2 � (p+ q) + 1 2 � p+ q �� p+ q+ 1 �� log(n) (31) + 1 4 � p+ q �� p+ q− 1 � log(2), where s = dim(F̂−1) = rank(F̂−1) = 1 2 (p+ q)(p+ q+ 3). (32) Therefore, C1(F̂ −1) avoids the construction of I F IM which is very attractive. This shows the scalability properties of the ICOM P criterion. The other new forms of ICOM P criteria for SEM are: ICOM P � F̂−1 � PEU = n(p+ q) log(2π)+ n log |Σ̂|+ ntr(Σ̂−1S) + 3k+ 2C1(F̂ −1), (33) and ICOM P � F̂−1 � PEU_Miss =n(p+ q) log(2π)+ n log |Σ̂|+ ntr(Σ̂−1S) + k (34) + 2( nk n− k− 2 ) + 2C1(F̂ −1). The consistent and misspecification and robust forms of ICOM P-type criteria are obtained in a similar fashion which are given by C ICOM P(F̂−1)PEU = n(p+q) log(2π)+ n log |Σ̂|+ ntr(Σ̂−1S)+ k(1+2 log(n))+2C1(F̂ −1), (35) and C ICOM P(F̂−1)PEU_Miss =n(p+ q) log(2π) + n log |Σ̂|+ ntr(Σ̂−1S) + k (36) + 2 log � nk n− k− 2 � + 2C1(F̂ −1). E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 292 Note that these two above forms of ICOM P criteria are both consistent and misspecification resistant. Comparing AIC , CAIC , CAIC F , BMS, and the different forms of ICOM P-type criteria, we see that the difference between these criteria are in their penalty terms. For convenience, we summarize these model selection criteria in Table 1, which we will use in our numerical ex- amples. In the next section we give two numerical examples to demonstrate the performance Table 1: Summary of Information Criteria and Their Penalties. Information Criteria Penalty term AIC 2k CAIC k � log(n)+ 1 � CAIC F k � log(n)+ 2 � + log |F̂ | BMS k log(n)+ ( nk n−k−2 ) + log |F̂ | ICOM P 2C1(Σ̂(θ̂ )) ICOM P(I F IM) 2C1(F̂ −1) C ICOM P(I F IM)PEU k(1+ 2 log(n))+ 2C1(F̂ −1) C ICOM P(I F IM)PEU_Miss k+ 2 log � nk n−k−2 � + 2C1(F̂ −1) of these information criteria in SEM. 5. Numerical Results 5.1. A Large Scale Monte Carlo Simulation Study In this simulation study, we used the widely known general SEM simulation protocol given in [15]. This model is formed from 1 latent exogenous variable, 2 latent endogenous vari- ables, 2 observed exogenous variables, and 4 observed endogenous variables. We slightly modify the structure of [15] in θε matrix to achieve convergence in LISREL. We generated 500 sample covariance matrices for four sample sizes (100, 400, 1000, and 4000). Each sam- ple covariance matrix was analyzed for each of the three analytic models given below using LISREL. Figure 1 shows the population parameters of the model and the path diagram as expressed in LISREL notation, with parameters shown here. Λx = � 1.00 0.50 � ,Λy =   1.00 0 0.95 0 0 1.00 0 0.90   ,Γ = � −0.60 −0.25 � ,Φ = � 7 � , B = � 0 0 0.60 0 � , Ψ = � 5.00 0 0 4.00 � ,θδ = � 3.00 0 0 2.50 � ,θε =   3.00 0 0 0 0 3.00 0 0 0 0 4.00 0 0 0 0 4.00   E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 293 Once the model parameters are specified, the implied population covariance matrix is ob- Figure 1: True model with population parameters and model spe i� ation onditions. tained through Equation (5). For the analysis of this covariance matrix, we used three differ- ent analytic models, given below. • Overfitting model: AM1 (15 free parameters) Λx (1,1) = Λy(1,1) = Λx (3,2) = 1 Γ,Φ, B,Ψ,θδ ,θε are estimated freely Λx = � 1.00 λ1 � ,Λy =   1.00 0 λ2 0 0 1.00 0 λ3   ,Γ = � γ1 γ2 � ,Φ = � φ � , B = � 0 0 β 0 � , Ψ = � ψ1 0 0 ψ2 � ,Θδ = � θδ1 0 0 θδ2 � ,Θε =   θε1 0 0 0 0 θε2 0 0 0 0 θε3 0 0 0 0 θε4   • Pseudo true model: AM2 (13 free parameters) Λx (1,1) = Λy(1,1) = Λx (3,2) = 1 θε(1,1) = θε(2,2),θε(3,3) = θε(4,4) Γ,Φ, B,Ψ,θδ are estimated freely Λx = � 1.00 λ1 � ,Λy =   1.00 0 λ2 0 0 1.00 0 λ3   ,Γ = � γ1 γ2 � ,Φ = � φ � , B = � 0 0 β 0 � , E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 294 Ψ = � ψ1 0 0 ψ2 � ,Θδ = � θδ1 0 0 θδ2 � ,Θε =   θε1 0 0 0 0 θε1 0 0 0 0 θε3 0 0 0 0 θε3   • Underfitting model: AM3 (11 free parameters) Λx (1,1) = Λy(1,1) = Λx (3,2) = 1 Λx (2,1) = Λy(2,1) Γ(2,1) = 0 θε(1,1) = θε(2,2),θε(3,3) = θε(4,4) Φ, B,Ψ,θδ are estimated freely Λx = � 1.00 λ1 � ,Λy =   1.00 0 λ1 0 0 1.00 0 λ3   ,Γ = � γ1 0 � ,Φ = � φ � , B = � 0 0 β 0 � , Ψ = � ψ1 0 0 ψ2 � ,Θδ = � θδ1 0 0 θδ2 � ,Θε =   θε1 0 0 0 0 θε1 0 0 0 0 θε3 0 0 0 0 θε3   Based on the above AM structures above, we carried out a large scale Monte Carlo simulation study. Since we have three AMs and four different sample sizes n= 100,400,1000, and 4000, we have in total 12 different simulation experiments. For each simulation experiment, we performed 500 replications and scored each of the information criteria to study their empirical performance based on their hit ratios. One of the important characteristics of AM3 is that it is a misspecified model according to [15]; that is misspecified as it relates to underfitting . We note that his misspecification does not indicate to distributional misspecification since there are many other ways we can misspecify a model. The steps of our Monte Carlo simulation are as follows, 1. The covariance matrices are generated with population parameters given in Figure 1, using PRELIS. 2. Each generated sample covariance matrix is analyzed by means of the three analytic models using LISREL. Goodness of fit indices and implied covariance matrices from LISREL outputs are saved. 3. LISREL outputs are passed to program are read into MATLAB, which computes both AIC-type and ICOM P-type criteria scores, and AMs selection frequencies. Our MATLAB modules are available from the authors upon request which marries LISREL results with MATLAB. E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 295 5.1.1. Results of Monte Carlo Simulation Experiments Table 2: Corre t model sele tion frequen y by riteria (in %). Information Criteria n AM1 AM2 AM3 AIC 100 16 84 0 400 23.4 76.6 0 1000 41.6 58.6 0 4000 85.6 14.4 0 CAIC 100 0.4 98.8 0.8 400 0.2 99.8 0 1000 1.2 98.8 0 4000 5.8 94.2 0 CAIC F 100 0 98.4 1.6 400 0.2 99.8 0 1000 0.6 99.4 0 4000 3.6 96.4 0 BMS 100 0 98.6 1.4 400 0.2 99.8 0 1000 1.0 99.0 0 4000 5.6 94.4 0 ICOM P 100 98.6 1.4 0 400 99.8 0.2 0 1000 100 0 0 4000 100 0 0 ICOM P(I F IM) 100 37 63 0 400 48.8 51.2 0 1000 65.4 34.6 0 4000 95.4 4.6 0 ICOM P(I F IM)PEU 100 13.8 86.2 0 400 22.8 77.2 0 1000 41.2 58.8 0 4000 85.4 14.6 0 ICOM P(I F IM)PEU_Miss 100 9.2 90.8 0 400 20.8 79.2 0 1000 40.2 59.8 0 4000 85.4 14.6 0 C ICOM P(I F IM)PEU 100 0 80.6 19.4 400 0 100 0 1000 0 100 0 4000 0.2 99.8 0 C ICOM P(I F IM)PEU_Miss 100 0 52.4 47.6 400 0 100 0 1000 0 100 0 4000 0.2 99.8 0 E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 296 In our simulation study, we intentionally did not include the nine traditional goodness of fit indices which were scored in [15], as they do not have the provision of taking into account different type of model misspecifications. Therefore, it is misleading even to score and report their results under such circumstances. In Table 2, we summarize our result of percent hit ratios of the three AMs, AIC CAIC , CAIC F , BMS, ICOM P, ICOM P(I F IM), ICOM P(I F IM)PEU , ICOM P(I F IM)PEU_Miss, C ICOM P(I F IM)PEU , and C ICOM P(I F IM)PEU_Miss criteria, for each sample size. We note that CAIC , CAIC F , BMS, ICOM P, ICOM P(I F IM), ICOM P(I F IM)PEU , ICOM P(I F IM)PEU_Miss, C ICOM P(I F IM)PEU , and C ICOM P(I F IM)PEU_Miss all hit the pseudo true AM2 with very high frequencies. Specifically the performances of CAIC , CAIC F , BMS, C ICOM P(I F IM)PEU , and C ICOM P(I F IM)PEU_Miss are outstanding and all above 90%. We further observe that AIC ’s performance is not satisfactory as the sample size increases. For sample sizes n = 100, AIC picks AM2 84% of the time, and as n gets large, AIC ’s hit percentage diminishes, and deteriorates. We know that AIC is not a consistent criterion. Specifically, AIC leans toward the AM1 which is an overfitting model which is the behaviour that is often demonstrated in the literature about AIC . Of course one should note that, the large scale Monte Carlo simulation experiment we performed is only using one model set here. This can be easily extended to a large dimensional other model settings to further study the performances of these criteria in SEM. This requires high speed computation and computational capability on a super computer and a stand alone SEM software to carry out the task that is limited at this point. 5.2. A Real Data Example in Market Research In this example, we apply the information criteria to a real data set from a soft drink company in Turkey. For propriety reasons we cannot disclose the name of the company. The data set in this study consists of a sample of n = 135 marketing surveys to study the product quality and to build a predictive operating model for the soft drink company. The goal of this survey was to enhance the market positioning and determine the influence of the investment of the company on their marketing campaign. For this data set we have seven characteristics which are measured to establish the companies objectives. These seven characteristics are: refreshing, tastes great, good with meal, exhilarating, feels good, worth the money, and brands having products of good quality. Since we do not know a priori the generating model for this real data set, first we applied an Exploratory Factor Analysis (EFA) to learn which factors were related to the original vari- ables. According to EFA results, three latent variables were named as “taste”, “feeling”, and “quality”, based on pre-determined factors. In the EFA model, variables which were loaded with factor 1 were: “refreshing”, “tastes great”, and “good with meal”. These were taken as exogenous variables and related to the latent exogenous variable “taste”. Variables “exhilarating”, and “feels good” were loaded with factor 2. These were taken as exogenous variables and related to the latent exogenous variable; “feeling.” Finally, vari- ables “worth money” and “quality products” were loaded with factor 3. These were taken as endogenous variables and related to the latent endogenous variable “quality”. E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 297 Figure 2: Full SEM with 17 free parameters (Model 1). Figure 3: Full SEM with 15 free parameters (Model 2). Figure 4: Full SEM with 13 free parameters (Model 3). E. Howe, H. Bozdogan, G. Kıroğlu / Eur. J. Pure Appl. Math, 5 (2012), 282-301 298 Since the information criteria used in the study were not applied to the just identified models, over-identified models were chosen. In accordance with the t-rule [19], one of the rules of identification must have free parameters under 28 to be over identified. Accordingly, three different models with free parameters, 17, 15, and 13 were compared to assess the information criteria. The path diagram of each of these three models, obtained from LISREL, are given in Figures 2, 3, and 4. Model 1 was fit to the data with 17 free parameters in Figure 2 with P−value = 0.29> 0.05. In this model, the diagonal elements of the covariance of latent exogenous variables and a single path coefficient between latent and observed endogenous were fixed. The other parameters were estimated freely. Model 2 was fit to the data with 15 free parameters shown in Figure 3 with P − value = 0.42 > 0.05. It was restricted by setting to 1 a single path coefficient in each latent variable, and equating diagonal elements of the covariance matrix of latent variables. In the case of Model 3, the model was fit to the data with 13 free parameters shown in Figure 4 with P − value = 0.00009 < 0.05. It also was restricted by setting to 1 as single path coefficient in each latent variable, and equating the measurement errors of the observed variables belonging to each latent variable. All information criteria scores for each of these three SEMs, obtained from LISREL and our MATLAB module, are reported in Table 3. According to the minimum of the information Table 3: The values of riteria for three General SEMs. Model Selection Criteria Model 1 Model 2 Model 3 AIC 4218.1 4216.6 4298.8 CAIC 4284.5 4275.2 4349.5 CAIC F 4333.8 4322.4 4389.0 BMS 4319.6 4309.5 4377.6 ICOM P 4187.9 4190.5 4275.9 ICOM P(I F IM) 4245.7 4248.6 4328.3 ICOM P(I F IM)PEU 4279.7 4278.6 4354.3 ICOM P(I F IM)PEU_Miss 4285.2 4282.9 4357.5 C ICOM P(I F IM)PEU 4429.4 4410.8 4468.8 C ICOM P(I F IM)PEU_Miss 4456.8 4432.0 4484.8 criteria, we choose the “best” model among those compared, as the model that achieves the best balance of fit with respect to parameter cardinality. In this case, all criteria are minimized at Model 2. These are indicated by the boldfaced scores in Table 3. We note that for this data set all criteria agreed. In general, this would not typically be the case with other real data sets. Based the results above, we can use Model 2 as our operating predictive SEM model in the subsequent surveys for market campaign to study the product quality. In passing we note that, although we used the EFA model to determine the SEM structure for this real data set, one can also use an expert pattern analysis method using the genetic algorithm with information complexity to obtain the “best” factor pattern structure to establish a confirmatory factor analysis to derive the SEMs. This new approach is well explained in [30] for further reading. We have not pursued this approach in this paper. REFERENCES 299 6. Conclusions and Discussion In this paper, we showed the performance of several information criteria, some old and new ones in general SEMs, under different sample sizes, and different models. Based on our results from this specific large scale Monte Carlo simulation experiment, for general SEM, CAIC , CAIC F , BMS, C ICOM P(I F IM)PEU and C ICOM P(I F IM)PEU_Miss criteria show the best performance. On the other hand, the performance of AIC seems to degrade as sample sizes increase - it tends to select overfitting models. This behavior is based on the sensitivity of the log likelihood function to sample size. Because CAIC , CAIC F , BMS, C ICOM P(I F IM)PEU and C ICOM P(I F IM)PEU_Miss criteria are consistent with respect to sample size (penalty term includes log(n)), they become more accurate as the sample size gets larger. Our sim- ulation results demonstrates the performance and the versatility of these criteria. In sum- mary, we recommend that the information criteria: CAIC F , BMS, C ICOM P(I F IM)PEU and C ICOM P(I F IM)PEU_Miss, be used in SEM. In the real example, we showed how one can build an operating predictive SEM in market research. We are currently studying other forms of misspecification such as the distributional misspecification, presence of high multicollinear- ity, and error variance heteroscedasticity within the SEM framework using the information criteria along with other applications. Our results will be reported elsewhere. ACKNOWLEDGEMENTS This research was supported by the Scientific and Technological Research Council of Turkey (TUBITAK) for the first author at the Department of Statistics, Operations, and Management Science at the University of Tennessee as a Visiting Scholar under the supervision of Professor Bozdogan. The first author extends her gratitude and thanks to Professor Bozdogan for the hospitality and conducive research atmosphere provided. References [1] H Akaike. Information theory and an extension of the maximum likelihood principle. In B.N. Petrox and F. Csaki, editors, Second International Symposium on Information Theory., pages 267–281, Budapest, 1973. Academiai Kiado. [2] H Akaike. Factor analysis and AIC. Psychometrica, 52(3):317–332, 1987. [3] K A Bollen. Structural Equations with Latent Variables. John Wiley,New York, 1989. [4] H Bozdogan. Model selection and akaike’s information criteria (AIC): the general theory and its analytical extensions. Psychometrica, 52:317–332, 1987. [5] H Bozdogan. A new information theoretic measure of complexity index for model eval- uation in general structural equation models with latent variables. Rutgers the State Univarsity, June 13-16 1991. Symposium on Model Selection in Covariance Structures at the Joint Meeting of Psychometric Society and The Classification Society. [6] H Bozdogan. Choosing the number of clusters, subset selection of variables, and outlier detection in the standard mixture-model cluster analysis. In E. Diday et al., editor, New REFERENCES 300 Approaches in Classification and Data Analysis, pages 169–177. Springer-Verlag, New York, 1994. [7] H Bozdogan. Akaike’s information criterion and recent developments in information complexity. Journal of Mathematical Psychology, 44:62–91, 2000. [8] H Bozdogan. Statistical Data Mining Knowledge Discovery, chapter Intelligent Statisti- cal Data Mining with Information Complexity and Genetic Algorithms., pages 15–56. ChapmanHall,CRC, 2004. [9] H Bozdogan. A new class of information complexity (ICOMP) criteria with an applica- tion to customer profiling and segmentation. Journal of the School of Business Adminis- tration, 39(2):370–398, 2010. [10] H Bozdogan and M Ueno. A unified approach to information-theoretic and bayesian model selection criteria. Crete, Greece, 2000. The 6th World Meeting of the Interna- tional Society for Bayesian Analysis. [11] N Cliff. Some cautions concerning the application of causal modeling methods. Multi- variate Behavioral Research, 18:115–126, 1983. [12] H Cramér. Mathematical Methods of Statistics. Princeton University Press, Princeton, New Jersey, 1946. [13] E Deniz, H Bozdogan, and S Katraggadda. Structural equation modeling (SEM) of cate- gorical and mixed-data using the novel gifi transformations and information complexity (ICOMP) criterion. Journal of the School of Business Administration, 40(1):86–123, 2011. [14] E A Deniz. Information Criteria in Structural Equation Models. unpublished ph.d. thesis, Mimar Sinan Fine Arts University, 2007. [15] X Fan and X Fan. Using SAS for monte carlo simulation research in SEM. Structural Equation Modeling, 12(2):299–333, 2005. [16] B Frieden. Physics from Fisher Information. Cambridge University Press, Cambridge, UK, 1998. [17] N Gheissari and A Bab-Hadiashar. Detecting cylinders in 3D range data using model selection criteria. Proceeding of Fifth International Conference on 3-D Digital Imaging and Modelling, 2005. [18] K Jöreskog. Testing Structural Equation Models., chapter Testing Structural Equation Models. Sage Publication, London, 1993. [19] K Jöreskog and D Sörbom. LISREL 7 A Guide to the Program and Applications 2nd Edition. SPSS Inc, Chicago, 1989. REFERENCES 301 [20] S Kullback. Information Theory and Statistics. Dover Publications, Dover, New York, 1968. [21] S Kullback and R Leibler. On information and sufficiency. Annals of Mathematical Statis- tics, 22:79–86, 1951. [22] S A Mulaik. Linear Causal Modeling with Structural Equations. Chapman and Hall, 2009. [23] C R Rao. Information and accuracy attainable in the estimation of statistical parameters. Bulletin of the Calcutta Mathematical Society, 37:81, 1945. [24] C R Rao. Minimum variance and the estimation of several parameters. In Proceedings of the Cambridge Philosophical Society., volume 43, page 280, 1947. [25] C R Rao. Sufficient statistics and minimum variance estimates. In Proceedings of the Cambridge Philosophical Society., volume 45, page 213, 1948. [26] J Rissanen. Modeling by shortest data description. Automatica, pages 465–471, 1978. [27] Y Sakamoto, M Ishiguro, and G Kitagawa. Akaike Information Criterion Statistics. Dor- drecht, The Netherlands: Reidel, 1986. [28] G Schwarz. Estimating the dimension of a model. Annals of Statistics, pages 461–464, 1978. [29] M Van Emden. An analysis of complexity. In Mathematical Centre Tracts., volume 35. Mathematisch Centrum, 1971. [30] H Y Yang and H Bozdogan. Learning factor patterns in exploratory factor analysis using the genetic algorithm and information complexity as the fitness function. Journal of Pattern Recognition Research, 2:307–326, 2011.