Int. J. Anal. Appl. (2022), 20:11 Nonconvex Vector Optimization and Optimality Conditions for Proper Efficiency E. Kiyani1,2,∗, S. M. Vaezpour1, J. Tavakoli2 1Department of Mathematics and Computer Science, Amirkabir University of Technology, 424, Hafez Avenue, 15914 Tehran, Iran 2School of Arts and Sciences, University of British Columbia, Okanagan 333 University Way, Kelowna, B.C., Canada ∗Corresponding author: kiyani-e@ut.ac.ir Abstract. In this paper, we consider, a new nonlinear scalarization function in vector spaces which is a generalization of the oriented distance function. Using the algebraic type of closure, which is called vector closure, we introduce the algebraic boundary of a set, without assuming any topology, in our context. Furthermore, some properties of this algebraic boundary set are given and present the concept of the oriented distance function via this set in the concept of vector optimization. We further investigate Q-proper efficiency in a real vector space, where Q is some nonempty (not necessarily convex) set. The necessary and sufficient conditions for Q-proper efficient solutions of nonconvex optimization problems are obtained via the scalarization technique. The scalarization technique relies on the use of two different scalarization functions, the oriented distance function and nonconvex separation function, which allow us to characterize the Q-proper efficiency in vector optimization with and without constraints. 1. Introduction Many works have been done with vector optimization problems under real linear spaces without any particular topology [1,2,4–14]. However, only a few authors focus on nonconvex vector optimization problems [5,7,13–15]. Inspired by this fact, the main purpose of this paper is to study some optimality conditions on Q-proper efficiency of general nonconvex optimization problems in a real linear vector space without topology, by using nonlinear scalarization functions. Received: Aug. 7, 2019. 2010 Mathematics Subject Classification. 90C26, 90C29, 90C30. Key words and phrases. vector optimization; algebraic interior; vector closure; Gerstewitz’s function. https://doi.org/10.28924/2291-8639-20-2022-11 ISSN: 2291-8639 © 2022 the author(s). https://doi.org/10.28924/2291-8639-20-2022-11 2 Int. J. Anal. Appl. (2022), 20:11 Efficiency is one of the most important concepts in vector optimization. This concept has been studied in many papers [1,2,4,5,8]. Kuhn and Tucker and Geoffrion [16,17], introduced the concept of proper efficiency. Since then, different definitions of proper efficient points have been introduced by the other authors. Wang and Li [18, 19] studied the Benson and Borwein proper efficiency in finite-dimensional Euclidean spaces. Borwein [20] has proposed a definition for extending Geoffrion’s concept of proper efficiency to the vector maximization problem in which the domination cone could be any nontrivial, closed convex cone. Adán and Novo [1,2,8] used the vector closure to define the concept of Benson proper efficiency of vector optimization problems and they proved scalarization theorems. Also, they investigated weak and proper efficiency of vector optimization problems with generalized convex set-valued maps involving relative algebraic interior and vector closure of ordering cone in linear spaces. Ha [21] presented the notion of Q-minimal solution of vector optimization problems via topological concepts, where Q is some nonempty open (not necessarily convex) cone. Q-minimal points were characterized by the Hiriart-Urruty function. Scalarization techniques play a vital role in sketching the numerical algorithms and duality results [5,7,8,10,12,22,23]. During the last three decades, many authors have been interested in extending scalarization approaches in vector optimization. Nonlinear scalarization approaches have been widely used as efficient methods to study several optimization problems in recent years. In vector optimization, two types of nonlinear scalarization functions are most widely used, the Ger- stewitz function [24] and the oriented distance function [21]. The Gerstewitz function is a nonlinear scalarization function most commonly used in optimization problems with vector-valued or set-valued maps [5, 13, 15, 22, 25]. This function was introduced by different names such as the Gerstewitz function, nonlinear scalarization function, shortage function, and smallest strictly monotonic func- tion, [25–28]. The properties of the Gerstewitz function in a topological vector space with a closed convex (solid) cone, have been studied in [26–29]. Hernández and Rodríguez-Marín [30] presented an extension of the Gerstewitz function and characterized some topological properties to obtain a non- convex scalarization and optimality conditions for set-valued optimization problems. The nonconvex separation functional in real linear spaces without considering a topology has been presented by La Torre, Popovici, and Rocca [31,32] . They showed that weakly cone-convex vector-valued functions can be characterized in terms of weakly convexity and weakly quasiconvexity of the Gerstewitz scalar- ization functions. The authors in [13,15] extend the Gerstewitz function from the topological spaces to real linear via algebraic concepts. Beside the Gerstewitz function, the oriented distance function is a common scalarization function in vector spaces introduced by Hiriart–Urruty [33]. The oriented distance function has been used to study well-posedness and stability for vector optimization problems in [35–38]. The generalized version Int. J. Anal. Appl. (2022), 20:11 3 of the oriented distance function introduced by Crespi et al. [35] can be used to characterize optimality conditions of set-valued vector optimization. In this paper, we propose a new definition of distance function by using vector closure. For this a new definition of the boundary set is presented which can be used to define the new form of the oriented distance function. The aim of this work is to provide a necessary and sufficient conditions of Q-Global Borwein Vectorial Proper Efficient (Q-GBOV) solutions in a real vector space. We use algebraic concepts such as algebraic interior and vectorial closure to define and characterize Q-GBOV. The necessary and sufficient conditions for Q-proper efficient solutions of nonconvex optimization problems are obtained via scalarization by oriented distance function and nonlinear scalarization function in a vector space. As the reader sees, some arguments developed for Global Borwein’s proper efficiency are still valid for Q-GBOV. Some results in this paper, are the generalization of several results given in [1,2,5,8,13,39]. The remainder of the paper is organized as follows: In Section 2, we introduce an algebraic boundary set in a vector space and study its properties. We discuss the notion of Q-proper efficiency where Q is not necessarily a convex set in Section 3. Section 4 is devoted to the scalarization functions; We describe a new nonlinear scalarization functions and explain how to use these functions to obtain optimality conditions. Finally, constrained problems in real vector spaces have been discussed in section 5. We use the results of previous sections to obtain optimality conditions for the constrained problems without convexity assumption. The results of this paper can be also used to develop a vector optimization on vector spaces, which can be applied to any numerical and theoretical scalar optimization. 2. Preliminaries Throughout the paper, X and Y are real spaces and A is a subset of X. Furthermore, we consider K ⊆ Y be a pointed convex proper cone which introduces a partial order on Y by the equivalence relation y1 ≤ y2 ⇔ y2 − y1 ∈ K. K is called pointed if K ∩ (−K) = {0}. The cone generated by A is denoted by cone(A). Moreover, a nonempty set F ⊂ Y is said to be free disposal with respect to a convex cone K ⊂ Y if F + K = F. The algebraic interior of A and the vectorial closure of A are denoted by cor(A) and vcl(A), respectively and these are defined as follows [1] cor(A) = {x ∈ A : ∀x ′ ∈ X , ∃λ ′ > 0; ∀λ ∈ [0,λ ′ ], x + λx ′ ∈ A}, vcl(A) = {b ∈ X : ∃x ∈ X ; ∀λ ′ > 0, ∃λ ∈ [0,λ ′ ]; b + λx ∈ A}. When cor(A) 6= ∅ we say that A is solid, A is algebraically open if cor(A) = A and A is called vectorially closed if A = vcl(A). It is known that, if cor(K) 6= ∅, then cor(K)∪{0} is a convex cone, in addition cor(K) + K = cor(K) and cor(cor(K)) = cor(K) for solid nontrivial convex cone K [8]. For each q ∈ Y , q-vector closure of A in the direction q is denoted by vclq(A) and define as follows: vclq(A) = {x ∈ X : ∀λ ′ > 0, ∃λ ∈ [0,λ ′ ] ; x + λq ∈ A}. 4 Int. J. Anal. Appl. (2022), 20:11 In fact it can be shown that vclq(A) = {x ∈ X : ∃λn ≥ 0, λn → 0 ; x + λnq ∈ A, ∀n ∈ N}. Obviously, A ⊂ vclq(A) ⊂∪q∈Y vclq(A) = vcl(A). Now, for an arbitrary functional h : Y →<∪{±∞}, define S(h,r,R) := {y ∈ Y : h(y) Rr}, ∀r ∈ Re, ∀R ∈{≤,<,>,≥}, S(h,r, =) := {y ∈ Y : h(y) = r}, ∀r ∈ Re ∪{±∞}, for a free disposal Q. The following proposition shows that there is e ∈ corK such that vcle(Q) + (0, +∞)e = cor(Q). Proposition 2.1. [15] Suppose that Q is free disposal with respect to an algebraic solid convex cone K. Then vcle(Q) + cor(K) = vcle(Q) + (0, +∞)e = cor(Q), where e ∈ cor(K). The boundary of a subset A of a topological space X is the set of points which can be approached both from A and from the outside of A. More precisely, it is the set of points in the closure of A not belonging to the interior of A. The algebraic boundary of a set A in a vector space can be defined by using algebraic type of interior and closure. Definition 2.1. The algebraic boundary of a set A denoted by bd(A) and is defind as follows bd(A) = vcl(A)\cor(A). which is the set of points in the vectorial closure of A not belonging to the algebraic interior of A. It is clear that bd(tA) = tbd(A) for t > 0 because vcl(tA) = tvcl(A) and cor(tA) = tcor(A), cor(tA) = {x ∈ tA : ∀x ′ ∈ X , ∃λ ′ > 0; ∀λ ∈ [0,λ ′ ], x + λx ′ ∈ tA}, cor(tA) = {1 t x ∈ A : ∀x ′ ∈ X , ∃λ ′ > 0; ∀λ ∈ [0,λ ′ ], 1 t x + λ t x ′ ∈ A} which lead to 1 t x ∈ cor(Q) for a free disposal Q. Also, we have vcl(tA) = {b ∈ X : ∃x ∈ X ; ∀λ ′ > 0, ∃λ ∈ [0,λ ′ ]; b + λx ∈ tA}. vcl(tA) = {1 t b ∈ X : ∃x t ∈ X ; ∀λ ′ > 0, ∃λ ∈ [0,λ ′ ]; 1 t b + λx t ∈ A}. Proposition 2.2. For Q ⊂ Y , we have vcl(Y\Q) = Y\cor(Q). Proof. Assume by contradiction that x ∈ vcl(Y\Q) and x ∈ cor(Q). x ∈ vcl(Y\Q) implies that for all λ ′ > 0 there exist x ′ ∈ X and λ ∈ [0,λ ′ ] such that x + λx ′ ∈ Y\Q. (2.1) Int. J. Anal. Appl. (2022), 20:11 5 On the other hand, for x ∈ cor(Q) we have ∀x ′′ ∈ X, ∃λ ′′ > 0, ∀λ ∈ [0,λ ′′ ], x + λx ′′ ∈ Q. (2.2) If we consider x ′′ = x ′ and λ ′ = λ ′′ then we can write ∃λ ′ = λ ′′ > 0, ∀λ ∈ [0,λ ′′ ], x + λx ′′ = x + λx ′ ∈ Q, (2.3) which contradicts 2.1. Therefore x /∈ cor(Q). � From Proposition 2.2, it is now clear that bd(Y\Q) = bd(Q). 3. Proper Efficiency Definition of Global Borwein vectorial proper efficient solutions in vector spaces, for the first time, introduced in [8] where the concept of vector closure has been used. The notion of Q-minimal points which Q is some nonempty open (not necessarily convex) cone was presented by Ha [21]. Necessary and sufficient conditions for these Q-minimal points were character- ized by the Hiriart-Urruty function. Now, we are in the position to introduce a general concept of Weak Proper Efficiency, Proper Effi- ciency, and Global Borwein Proper Efficiency via algebraic concepts. Let Y and Z be two real spaces that are partially ordered by nontrivial ordering convex cones K and M, respectively. Let f : X → Y and g : X → Z be two maps on X. Consider the following unconstrained and constrained problems: (UP ) Min{f (x) : x ∈ X}, (CP ) Min{f (x) : x ∈ X, g(x) ∈ (−M)}, and the following vector optimization problem: (P ) Min{f (x) : x ∈ S}, where the feasible set S can be either S = X or S = {x ∈ X; g(x) ∈ (−M)}. Definition 3.1. A point x0 ∈ S is called a Q-Proper Efficient solution (Q-EF) of (P) if (f (S) − f (x0)) ∩ (−Q\{0}) = ∅. (3.1) If Q is a solid set and 0 /∈ cor(Q), then x0 is called Q-Weak Proper Eficient solution (Q-WEF) of (P) when (f (S) − f (x0)) ∩ (−cor(Q)) = ∅. (3.2) 6 Int. J. Anal. Appl. (2022), 20:11 Definition 3.2. A point x0 is a Q-GBOV for (P) with respect to Q if vcl(cone(f (S) − f (x0))) ∩ (−Q\{0}) = ∅. (3.3) It is easy to see that if x0 is a Q-GBOV for (P), then x0 is also a (Q-EF) and a (Q-WEF) for (P). 4. Scalarization In this section, we will present the necessary and sufficient optimality conditions for Q-Global Borwein Vectorial Proper Efficient solutions of vector optimization problems. A useful approach for solving a vector problem is to reduce it to a scalar problem. In general, scalarization means the replacement of a vector optimization problem by a suitable scalar problem which tends to be an optimization problem with a real valued objective function. The main idea of this section obtained from [13, 15]. In [13] the Gerstewitz function is generated by a general convex cone in a real space and the authors investigated some properties of this function such as sub-additive and positively homogeneous. However, similar to [15], in this section we consider the nonconvex separation function which is an extension of the Gerstewitz function and generated by a subset of a linear space instead of a convex cone. The main properties of the nonconvex separation functional were extended from the topological framework to the linear setting via suitable algebraic counterparts [15]. Now, let e ∈ cor(K). The Gerstewitz function heQ(y) : Y −→R is defined by heQ(y) := inf{t ∈R : y ∈ te −Q}, (4.1) where Q ⊂ Y . It has been proved that heQ is finite [31, Remark 2.3], whenever Q is a vectorially closed and algebraic solid proper convex cone, and in this case, one has heQ(y) = sup{h(y) : h ∈ Q +,h(e) = 1}, ∀y ∈ Y , where Q+ denotes the positive polar cone of Q [5]. In the following, some properties of the Gerstewitz function are addressed. One can find [15, Theorem 4.1, Theorem 4.2]. Specifically, these theorems are the generalization of [13, Lemma 2.8, Lemma 2.9]. Theorem 4.1. [15] Consider e ∈ Y \{0} and ∅ 6= Q ⊂ Y . We have the following properties of heQ i) S(heQ, 0,≤) = (−∞, 0]e −vcleQ, ii) S(heQ, 0,<) = (−∞, 0)e −vcleQ, iii) S(heQ, 0, =) = (−vcleQ)\((−∞, 0)e −vcleQ), iv) S(heQ, 0,≥) = Y\((−∞, 0)e −vcleQ). Theorem 4.2. [15] Consider e ∈ Y \{0} and ∅ 6= Q ⊂ Y . If vcle(Q) is a cone, then heQ is positively homogeneous. It means that heQ(αy) ≤ αh e Q(y), where y ∈ Y and α > 0. Int. J. Anal. Appl. (2022), 20:11 7 In Theorem 4.3, we prove heQ is sub-additive whenever Q is closed under addition. This theorem will be used in the sequel. Theorem 4.3. Consider e ∈ Y \ {0} and let ∅ 6= Q ⊂ Y be closed under addition and heQ is finite. Then heQ(y1 + y2) ≤ h e Q(y1) + h e Q(y2), for all y1,y2 ∈ Y , except for these make it indeterminate form ∞−∞. Proof. From definition of heQ given in ( 4.1) and Lemma 3 in [15], we have yi ∈ heQ(yi )e −vcle(Q), i = 1, 2. We can use the fact that he vcle(Q) = heQ to obtain yi ∈ hevcle(Q)(yi )e −vcle(Q), i = 1, 2. Then obviously, y1 + y2 ∈ (hevcle(Q)(y1) + h e vcle(Q) (y2))e −vcle(Q), which implies he vcle(Q) (y1 + y2) ≤ hevcle(Q)(y1) + h e vcle(Q) (y2), and this yields heQ(y1 + y2) ≤ h e Q(y1) + h e Q(y2). � Definition 4.1. Let Q ⊂ Y . A distance function is defined by d(y,Q) = inf{λ ∈ R≥0 : y ∈ vcl(λQ)}, and d(y,∅) = +∞. For a set Q ⊂ Y let the oriented distance function 4Q : Y → R∪{±∞} be defined as 4Q(y) = d(y,Q) −d(y,Y \Q). One can find the main properties of the oriented distance function in topological spaces in [33, 40]. However, here we recall them for conveniences. If y ∈ cor(Q), then there exists a sequence λn → 0 such that y ∈ vcl(λnQ), thus we get d(y,Q) = 0 and then 4Q(y) < 0. Also, if d(y,Q) = 0, then y ∈ cor(Q). Therefore, we can write y ∈ cor(Q) if and only if 4Q(y) < 0. Moreover, 4Q(y) > 0 if and only if y ∈ vcl(Q). Since bd(Y\Q) = bd(Q) and bd(Q) = vcl(Q)\cor(Q), we have 4Q(y) = 0 if and only if y ∈ bd(Q). Furthermore, it is obvious that 4Y\Q = −4Q . Theorem 4.4. For t ∈ (0, +∞), we have 8 Int. J. Anal. Appl. (2022), 20:11 ∆Q(ty) = t∆Q(y). Proof. Consider y ∈ Y . Thus d(ty,Q) = inf{λ ≥ 0; ty ∈ vcl(λQ)}, from definition of vcl(Q), for all µ ′ > 0, there exist x ∈ Y such that d(ty,Q) = inf{λ ≥ 0; ∃µ ∈ [0,µ ′ ], ty + µx ∈ λQ} = inf{λ ≥ 0; ∃µ ∈ [0,µ ′ ], y + µ t x ∈ λ t Q} = tinf{ λ t ≥ 0; ∃µ ∈ [0,µ ′ ], y + µ t x ∈ λ t Q}. (4.2) Therefore, ∆Q(ty) = t∆Q(y). � In the following theorem, we use the Gerstewitz function to obtain the sufficient condition for Q- GBOV. Theorem 4.5. Let e ∈ cor(K), ∅ 6= Q ⊂ Y be an algebraically open set which is closed under addition, and let vcle(Q) be a cone. If x0 satisfies the following condition heQ(f (x) − f (x0)) ≥ 0 ∀x ∈ S, then x0 is a Q-GBOV for (P). Proof. For x ∈ S we have heQ(f (x) − f (x0)) ≥ 0. Thus, by theorem 4.1, one has f (x) − f (x0) ∈ Y\((−∞, 0)e −vcle(Q)) ∀x ∈ S. Now, let 0 6= y ∈ vcl(cone(f (S) − f (x0))). By definition of vectorial closure, there exist x ∈ Y and a sequence of positive real numbers λn such that λn → 0 and y + λnx ∈ cone(f (S) − f (x0)). (4.3) Therefore, there are sequences αn ≥ 0 and yn ∈ f (S) such that y + λnx = αn(yn − f (x0)). It is obvious that there exist an n ∈ N such that αn > 0. Since heQ(yn − f (x0)) ≥ 0, then it implies that heQ(y + λnx) ≥ 0. Int. J. Anal. Appl. (2022), 20:11 9 and 0 ≤ heQ(y + λnx) ≤ h e Q(y) + λnh e Q(x). By taking limit n →∞, we have heQ(y) ≥ 0. Now, by applying Theorem 4.1, we conclude that y ∈ Y\((−∞, 0)e −vcle(Q)). On the other hand, for q ∈ cor(Q) there exists λ > 0 such that q − [0,λ]e ⊆ Q. It means that q ∈ λe + Q ⊆ (0,∞)e + vcle(Q). Therefore, cor(Q) ⊆ (0,∞)e + vcle(Q). Since Q is an algebraically close set, we have Q ⊆ (0,∞)e + vcle(Q), and −Q ⊆ (−∞, 0)e −vcle(Q), which implies that y ∈ Y\(−Q). Hence, vcl(cone(f (x) − f (x0))) ∩ (−Q\{0}) = ∅ ∀x ∈ S. Thus, x0 is a Q-GBOV for (p). � The following theorems state the necessary condition for a point to become a Q-GBOV for problem (P). In Theorem 4.6, we use the nonconvex separation function to obtain the necessary condition while in Theorem 4.7, the oriented distance function has been used. We would like to point out that the oriented distance function is a simple tool to work, thus optimality conditions can be obtained with simple calculations by using the properties of the oriented distance function without any condition on the set Q. Theorem 4.6. Let ∅ 6= Q ⊂ Y is free disposal with respect to an algebraic solid convex cone K. Suppose that there exists e ∈ cor(K) such that x0 is a Q-GBOV for problem (P), then x0 satisfies the following condition 10 Int. J. Anal. Appl. (2022), 20:11 heQ(f (x) − f (x0)) ≥ 0 ∀x ∈ S. Proof. Let us suppose x0 does not satisfy the condition and we have heQ(f (x) − f (x0)) < 0, for some x ∈ S. Hence, from Theorem 4.1 we get f (x) − f (x0) ∈ (−∞, 0)e −vcle(Q). On the other hand, since Q is free disposal with respect to K, then by Proposition 2.1, one has f (x) − f (x0) ∈ (−∞, 0)e −vcle(Q) = −cor(Q) ⊂−Q\{0}. Therefore, f (x) − f (x0) ∈ (f (S) − f (x0)) ∩ (−Q\{0}). But as one can see, this contradicts the definition of Q-GBOV. � Theorem 4.7. Let e ∈ corK and x0 ∈ S such that x0 is a Q−GBOV for (P), then 4−Q(f (x) − f (x0)) ≥ 0, x ∈ S. (4.4) Proof. By contrary suppose that for e ∈ corK there exists x ∈ S such that 4−Q(f (x) − f (x0)) < 0. Thus, we can write f (x) − f (x0) ∈−cor(Q) ⊂−Q\{0}, which means f (x) − f (x0) ∈ (f (S) − f (x0)) ∩ (−Q\{0}). Furthermore, we have f (x) − f (x0) ∈ vcl(cone(f (S) − f (x0))) ∩ (−Q\{0}) = ∅, ∀x ∈ S, which contradicts the assumption that x0 is Q−GBoV. Therefore, 4−Q(f (x) − f (x0)) ≥ 0. � From Theorems 4.5 and 4.6 we conclude the following corollary. Corollary 4.1. Let e ∈ cor(K), ∅ 6= Q ⊂ Y is free disposal with respect to an algebraic solid convex cone K and be closed under addition such that cor(Q) = Q, and let vcle(Q) be a cone. Then x0 is a Q-GBOV for problem (P) if and only if x0 satisfies the following condition heQ(f (x) − f (x0)) ≥ 0 ∀x ∈ S. Int. J. Anal. Appl. (2022), 20:11 11 In the following theorem, we present necessary and sufficient conditions of Q-WEF for the problem (P). Theorem 4.8. Let us assume x0 ∈ S. The point x0 is a Q-WEF if and only if 4−Q(f (x) − f (x0)) > 0, ∀x ∈ S. (4.5) Proof. Assume that x0 is not Q-WEF. Then one has (f (S) − f (x0)) ∩ (−cor(Q)) 6= ∅. Hence, there exists f (x) ∈ f (S) such that (f (x) − f (x0)) ∈ (−cor(Q)), which implies 4−Q(f (x) − f (x0)) < 0. � Then this proofs the necessary condition. The sufficient condition follows easily. 5. Scalarization and constrained problems Constrained problems in real vector spaces were originally studied in [5,8]. In [8, Theorem 4.3], the authors showed a relation between Hurwicz victorial proper efficient solutions and Benson Vectorial Proper Efficient solutions in unconstrained and constrained problems. Moreover, the relation between ε-Benson Vectorial Proper Efficient solutions in unconstrained and constrained problems studied in [5, Theorem 4.12]. Here, we use the results of previous theorems to obtain optimality conditions for the constrained problems without convexity assumption. In addition, the relation between solutions of constrained and unconstrained problems will be discussed. Definition 5.1. We say that the Slater constraint qualification for constraint problems (CP) holds if there exists x ∈ S such that g(x) ∈ (−cor(M)). Hereafter, the set of all linear operators from Z to Y is denoted by O(Z,Y ), and Γ is denoted by Γ = {T ∈ O(Z,Y ) : T (M) ⊆ cor(K)}, where M and K are as above. It is important to know Lagrangian mapping L : X × Γ −→ Y , corresponding to the constrained vector optimization problem is defined by L(x,T ) = f (x) + T (g(x)), where f and g are defined in section 3 and T ∈ Γ . By the map L, one can convert (CP) to an unconstrained vector optimization problem Min{f (x) + (T ◦g)(x) : x ∈ X}. (5.1) In the following theorem, we discuss Q-GBOV in corresponding problems. 12 Int. J. Anal. Appl. (2022), 20:11 Theorem 5.1. In a constrained vector optimization problem, assume that e ∈ cor(K), ∅ 6= Q ⊂ Y be an algebraically open set which is free disposal with respect to the algebraic solid convex cone K, and closed under the addition that 0 ∈ vcl(Q). Let the convex cones K and M are pointed and the Slater constraint qualification holds, Assume that T (g(x0)) = 0 for x0 ∈ S and T ∈ Γ. If x0 is a Q-GBOV for problem given in 5.1, then x0 is a Q-GBOV for (CP). Proof. As discussed in Theorem 4.6, since x0 is a (Q-GBOV) for problem 5.1, one has heQ(f (x) + (T ◦g)(x) − (f (x0) + (T ◦g)(x0))) ≥ 0. From Theorem 4.3, we have heQ(f (x) − f (x0)) + h e Q((T ◦g)(x) − (T ◦g)(x0))) ≥ heQ(f (x) + T ◦g)(x) − (f (x0) + (T ◦g)(x0))) ≥ 0. Since T (g(x0)) = 0, this implies heQ(f (x) − f (x0)) + h e Q(T ◦g)(x))) ≥ 0. (5.2) On the other hand, 0 ∈ vcl(Q), then by Proposition 2.1, we have (T ◦g)(x) ⊆−cor(K) −vcl(Q) = (−∞, 0)e −vcle(Q) = −cor(Q). From Theorem 4.1, we deduce that heQ((T ◦g)(x)) ≤ 0. (5.3) Therefore 5.2 and 5.3 yield heQ(f (x) − f (x0)) ≥ 0. Thus, by Theorem 4.5 it follows that x0 is a (Q-GBOV) for (CP). � Conflicts of Interest: The author(s) declare that there are no conflicts of interest regarding the publication of this paper. References [1] M. Adán, V. Novo, Efficient and weak efficient points in vector optimization with generalized cone convexity, Appl. Math. Lett. 16 (2003), 221–225. https://doi.org/10.1016/S0893-9659(03)80035-6. [2] M. Adán, V. Novo, Weak efficiency in vector optimization using a closure of algebraic type under cone-convexlikeness, Eur. J. Oper. Res. 149 (2003), 641–653. https://doi.org/10.1016/S0377-2217(02)00444-7. [3] X.-H. Gong, Optimality conditions for Henig and globally proper efficient solutions with ordering cone has empty interior, J. Math. Anal. Appl. 307 (2005), 12–31. https://doi.org/10.1016/j.jmaa.2004.10.001. [4] E. Hernández, B. Jiménez, V. Novo, Benson proper efficiency in set-valued optimization on real linear spaces, Lecture Notes in Economics and Mathematical Systems 563, Springer, Berlin, (2006) 45-59. https://doi.org/10.1016/S0893-9659(03)80035-6 https://doi.org/10.1016/S0377-2217(02)00444-7 https://doi.org/10.1016/j.jmaa.2004.10.001 Int. J. Anal. Appl. (2022), 20:11 13 [5] E. Kiyani, M. Soleimani-damaneh, Approximate proper efficiency on real linear vector spaces, Pac. J. Optim. 10 (2013), 715-734. [6] E. Kiyani, M. Soleimani-damaneh, Algebraic interior and separation on linear vector spaces: some comments, J. Optim. Theory Appl. 161 (2014), 994–998. https://doi.org/10.1007/s10957-013-0416-3. [7] E. Kiyani, S.M. Vaezpour, J. Tavakoli, Optimality conditions for approximate solution of set-valued optimization problems in real linear spaces, TWMS J. Appl. Eng. Math. 11 (2021), 395-407. [8] M. Adán, V. Novo, Proper efficiency in vector optimization on real linear spaces, J. Optim. Theory Appl. 121 (2004), 515–540. https://doi.org/10.1023/B:JOTA.0000037602.13941.ed. [9] Z.-A. Zhou, X.-M. Yang, J.-W. Peng, �-Optimality conditions of vector optimization problems with set-valued maps based on the algebraic interior in real linear spaces, Optim. Lett. 8 (2014), 1047–1061. https://doi.org/ 10.1007/s11590-013-0620-y. [10] Z.-A. Zhou, X.-M. Yang, Scalarization of �-super efficient solutions of set-valued optimization problems in real ordered linear spaces, J. Optim. Theory Appl. 162 (2014), 680–693. https://doi.org/10.1007/ s10957-014-0565-z. [11] E. Hernández, B. Jiménez, V. Novo, Weak and proper efficiency in set-valued optimization on real linear spaces, J. Convex Anal. 14 (2007), 275–296. [12] J. Jahn, Vector optimization, Theory, Applications, and Extensions, Springer, Berlin, 2011. [13] J.H. Qiu, F. He, A general vectorial Ekeland’s variational principle with a P-distance, Acta. Math. Sin.-English Ser. 29 (2013), 1655–1678. https://doi.org/10.1007/s10114-013-2284-z. [14] J.-H. Qiu, A pre-order principle and set-valued Ekeland variational principle, J. Math. Anal. Appl. 419 (2014), 904–937. https://doi.org/10.1016/j.jmaa.2014.05.027. [15] C. Gutiérrez, V. Novo, J.L. Ródenas-Pedregosa, T. Tanaka, Nonconvex separation functional in linear spaces with applications to vector equilibria, SIAM J. Optim. 26 (2016), 2677–2695. https://doi.org/10.1137/16M1063575. [16] A.M. Geoffrion, Proper efficiency and the theory of vector maximization, J. Math. Anal. Appl. 22 (1968), 618–630. https://doi.org/10.1016/0022-247X(68)90201-1. [17] H.W. Kuhn, A.W. Tucker, Nonlinear programming, in Proceedings of the Second Berkeley Symposium on Mathe- matical Statistics and Probability, University of California Press. Berkeley. CA. (1951) 481-492. [18] S. Wang, Z. Li, Scalarization and lagrange duality in multiobjective optimization, Optimization. 26 (1992) 315–324. https://doi.org/10.1080/02331939208843860. [19] S.J. Li, Y.D. Xu, S.K. Zhu, Nonlinear separation approach to constrained extremum problems, J. Optim. Theory Appl. 154 (2012), 842–856. https://doi.org/10.1007/s10957-012-0027-4. [20] J. Borwein, Proper efficient points for maximizations with respect to cones, SIAM J. Control Optim. 15 (1977), 57–63. https://doi.org/10.1137/0315004. [21] T.X.D. Ha, Optimality conditions for several types of efficient solutions of set-valued optimization problems, in: P.M. Pardalos, T.M. Rassias, A.A. Khan (Eds.), Nonlinear Analysis and Variational Problems, Springer New York, New York, NY, 2010: pp. 305–324. https://doi.org/10.1007/978-1-4419-0158-3_21. [22] S. Khoshkhabar-amiranloo, M. Soleimani-damaneh, Scalarization of set-valued optimization problems and variational inequalities in topological vector spaces, Nonlinear Analysis: Theory Meth. Appl. 75 (2012), 1429–1440. https: //doi.org/10.1016/j.na.2011.05.083. [23] Z.A. Zhou, J.W. Peng, Scalarization of set-valued optimization problems with generalized cone subconvexlike- ness in real ordered linear spaces, J. Optim. Theory Appl. 154 (2012), 830–841. https://doi.org/10.1007/ s10957-012-0045-2. [24] C. Gerth, P. Weidner, Nonconvex separation theorems and some applications in vector optimization, J. Optim. Theory Appl. 67 (1990), 297–320. https://doi.org/10.1007/BF00940478. https://doi.org/10.1007/s10957-013-0416-3 https://doi.org/10.1023/B:JOTA.0000037602.13941.ed https://doi.org/10.1007/s11590-013-0620-y https://doi.org/10.1007/s11590-013-0620-y https://doi.org/10.1007/s10957-014-0565-z https://doi.org/10.1007/s10957-014-0565-z https://doi.org/10.1007/s10114-013-2284-z https://doi.org/10.1016/j.jmaa.2014.05.027 https://doi.org/10.1137/16M1063575 https://doi.org/10.1016/0022-247X(68)90201-1 https://doi.org/10.1080/02331939208843860 https://doi.org/10.1007/s10957-012-0027-4 https://doi.org/10.1137/0315004 https://doi.org/10.1007/978-1-4419-0158-3_21 https://doi.org/10.1016/j.na.2011.05.083 https://doi.org/10.1016/j.na.2011.05.083 https://doi.org/10.1007/s10957-012-0045-2 https://doi.org/10.1007/s10957-012-0045-2 https://doi.org/10.1007/BF00940478 14 Int. J. Anal. Appl. (2022), 20:11 [25] D.T. Luc, Theory of vector optimization. Lecture notes in economics and mathematical systems 319, Springer, Berlin, (1989). [26] G.Y. Chen, X.X. Huang, X.G. Yang, Vector optimization: Set-valued and variational analysis, Springer-Verlag, Berlin, (2005). [27] A. Göpfert, Chr. Tammer, C. Zălinescu, On the vectorial Ekeland’s variational principle and minimal points in product spaces, Nonlinear Anal.: Theory Meth. Appl. 39 (2000), 909–922. https://doi.org/10.1016/S0362-546X(98) 00255-7. [28] S.J. Li, X.Q. Yang, G.Y. Chen, Nonconvex vector optimization of set-valued mappings, J. Math. Anal. Appl. 283 (2003), 337–350. https://doi.org/10.1016/S0022-247X(02)00410-9. [29] A.Göpfert, H. Riahi, C. Tammer, et al. Variational methods in Partially ordered spaces, Springer-Verlag, New York, (2003). [30] E. Hernández, L. Rodríguez-Marín, Nonconvex scalarization in set optimization with set-valued maps, J. Math. Anal. Appl. 325 (2007), 1–18. https://doi.org/10.1016/j.jmaa.2006.01.033. [31] D. La Torre, N. Popovici, M. Rocca, Scalar characterizations of weakly cone-convex and weakly cone-quasiconvex functions, Nonlinear Anal.: Theory Meth. Appl. 72 (2010), 1909–1915. https://doi.org/10.1016/j.na.2009. 09.031. [32] D. La Torre, N. Popovici, M. Rocca, A note on explicitly quasiconvex set-valued maps, J. Nonlinear Convex Anal. 12 (2011), 113-118. [33] J. B. Hiriart-Urruty, New concepts in nondifferentiable programming, Bull. Soc. Math. France, 60 (1979) 57-85. [34] J. Jahn, Scalarization in multi objective optimization, in: P. Serafini (Ed.), Mathematics of Multi Objective Opti- mization, Springer Vienna, Vienna, 1985: pp. 45–88. https://doi.org/10.1007/978-3-7091-2822-0_3. [35] G.P. Crespi, I. Ginchev, M. Rocca, First-order optimality conditions in set-valued optimization, Math. Meth. Oper. Res. 63 (2006), 87–106. https://doi.org/10.1007/s00186-005-0023-7. [36] G.P. Crespi, A. Guerraggio, M. Rocca, Well posedness in vector optimization problems and vector variational inequalities, J. Optim. Theory Appl. 132 (2007), 213–226. https://doi.org/10.1007/s10957-006-9144-2. [37] G.P. Crespi, M. Papalia, M. Rocca, Extended well-posedness of quasiconvex vector optimization problems, J. Optim. Theory Appl. 141 (2009), 285–297. https://doi.org/10.1007/s10957-008-9494-z. [38] G.P. Crespi, M. Papalia, M. Rocca, Extended well-posedness of vector optimization problems: the convex case, Taiwan. J. Math. 15 (2011), 1545-1559. https://doi.org/10.11650/twjm/1500406363. [39] Q. Qiu, X. Yang, Some properties of approximate solutions for vector optimization problem with set-valued functions, J Glob Optim. 47 (2010) 1–12. https://doi.org/10.1007/s10898-009-9452-9. [40] J.B. Hiriart-Urruty, Tangent cones, generalized gradients and mathematical programming in Banach spaces, Math. Oper. Res. 4 (1979), 79–97. https://doi.org/10.1287/moor.4.1.79. https://doi.org/10.1016/S0362-546X(98)00255-7 https://doi.org/10.1016/S0362-546X(98)00255-7 https://doi.org/10.1016/S0022-247X(02)00410-9 https://doi.org/10.1016/j.jmaa.2006.01.033 https://doi.org/10.1016/j.na.2009.09.031 https://doi.org/10.1016/j.na.2009.09.031 https://doi.org/10.1007/978-3-7091-2822-0_3 https://doi.org/10.1007/s00186-005-0023-7 https://doi.org/10.1007/s10957-006-9144-2 https://doi.org/10.1007/s10957-008-9494-z https://doi.org/10.11650/twjm/1500406363 https://doi.org/10.1007/s10898-009-9452-9 https://doi.org/10.1287/moor.4.1.79 1. Introduction 2. Preliminaries 3. Proper Efficiency 4. Scalarization 5. Scalarization and constrained problems References