Adv Syst Sci Appl 2023; 03:36–47 Published online at https://ijassa.ipu.ru. Feedback Design in Linear Control Problems as an Optimization Problem Mikhail Khlebnikov* V. A. Trapeznikov Institute of Control Sciences of Russian Academy of Sciences, Moscow, Russia Abstract: We provide and discuss a new approach to the design of linear control systems based on the optimization viewpoint. Three basic classes of control problems are analyzed: a) static state and output feedback for linear quadratic regulator problem; b) rejection of nonrandom bounded exogenous disturbances via static linear feedback; c) the same rejection via dynamic output feedback using an observer. These three problems are considered as optimization ones with feedback gains as matrix variables. The iterative algorithms for its solution are formulated in a uniform way, and the explicit expressions for gradients of the cost functions are provided. The gradient method exhibits its efficiency for test examples, including double pendulum. Keywords: linear systems, output feedback, optimization, gradient method, Lyapunov equation, linear quadratic regulator problem, external disturbances 1. INTRODUCTION Recently, the approach to linear control systems from the point of view of optimization has become very popular. So, in the classical problem of the linear quadratic controller, one can consider the linear feedback matrix as a variable and reduce the problem to minimizing the performance index with respect to this variable; this approach goes back to the works of R. Kalman in the middle of the last century. One of the main objectives of this paper is to discuss the possibility of a unified and systematic approach to a number of problems that arise in linear control theory as matrix optimization problems and trace it on the example of three classical control problems. In addition to the linear quadratic problem, from the same positions, the paper considers the problem of rejecting nonrandom bounded external disturbances by constructing a static linear output feedback, as well as using dynamic output feedback with observer. For each of these problems, we state the corresponding problems of nonconvex matrix optimization, and the iterative algorithms for its solution are formulated in a uniform way. Herewith, the relations for calculating the gradients of the corresponding cost functions are written out in closed form. From now on, | · | is the Euclidean norm of a vector, ∥ · ∥ is the spectral norm of a matrix, ∥ · ∥F is the Frobenius norm of a matrix, and ⟨·, ·⟩ is the Frobenius inner product of matrices. 2. LINEAR QUADRATIC REGULATOR Recall the new approaches to the classical problem of linear quadratic control. It can be considered as an optimization problem, where the variable is the feedback matrix, and the ∗Corresponding author: khlebnik@ipu.ru FEEDBACK DESIGN IN LINEAR CONTROL PROBLEMS AS AN OPTIMIZATION PROBLEM 37 integral quadratic performance index of the transient process is minimized. The gradient of such function (for the state feedback) was written out in the seminal work [5] by Kalman, and for output feedback in the paper [6] by Levin and Athans. Since then, the iterative gradient- type optimization methods have been used repeatedly (see [7]), but the justification for such methods has appeared only recently in [2–4, 8, 12]. Namely, consider the linear stationary continuous-time control system: ẋ = Ax+Bu, y = Cx, with state x(t) ∈ Rn, output y(t) ∈ Rl, and control input u(t) ∈ Rp. The infinite-horizon linear quadratic performance criterion is given by f(K) = E ∞∫ 0 ( xT(t)Qx(t) + uT(t)Ru(t) ) dt, where the expectation is taken over the distribution of an initial condition x(0) with zero mean and covariance matrix Σ, and the quadratic cost is parameterized by positive-definite matrices Q, R. Design of the appropriate static feedback u(t) = −Ky(t) with constant gain matrix K ∈ Rl×p can be reformulated as the optimization problem f(K) = trXΣ subject to the constraint (A−BKC)TX +X(A−BKC) + CTKTRKC +Q = 0 for the matrix variables K ∈ Rl×p and X = XT ∈ Rn×n. The properties of the corresponding function were studied in [3]: it turns out to be smooth but non-convex, see Fig. 2.1. Moreover, it is defined on a possibly disconnected and non- convex domain, see Fig. 2.2. Fig. 2.1 Fig. 2.2 Nevertheless, it was possible to construct a gradient method with a special choice of step length, which converges to the optimal solution in the case of state feedback and to a stationary point in the case of output feedback. Now we formulate the following algorithm to minimize f(K). Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) 38 M. KHLEBNIKOV Algorithm 1. 1. Set the parameters ε > 0, T1 > 0, 0 < τ < 1, and the initial stabilizing approximation K0. 2. At the jth iteration, the value Kj is given. Compute Aj = A−BKjC, solve the Lyapunov equations AT j X +XAj +Q+KT j RKj = 0 and AjY + Y AT j + Σ = 0 and find the matrices X and Y respectively; compute the gradient Hj = ∇f(Kj) from the relation 1 2 ∇f(K) = (RK −BTX)Y. If ∥Hj∥ ≤ ε, then Kj is taken as an approximate solution. 3. Solve the Lyapunov equation AT j X ′ +X ′Aj +MT∇f(Kj) + ( ∇f(Kj) )T M = 0 and find its solution X ′. Compute the value ∇2f(Kj)[Hj, Hj] = 2 〈 R∇f(Kj)Y,∇f(Kj) 〉 − 4 〈 BTX ′Y,∇f(Kj) 〉 and set the trial step as γ = min { T1, ∥∇f(Kj)∥2F ∇2f(Kj)[Hj, Hj] } . 4. Make one step of the gradient method Kj+1 = Kj − γjHj. The step length γj > 0 is selected by splitting γ until the following conditions are met: a. Kj+1 is a stabilizing controller; b. f(Kj+1) ≤ f(Kj)− τγj∥Hj∥2. Go to step 2. 3. SUPPRESSION OF EXTERNAL DISTURBANCES: STATIC FEEDBACK In [9], a similar approach is applied for the first time to the systems subjected to external disturbances. The problem of suppressing bounded external disturbances (peak-to-peak gain minimization) is formulated as follows. Consider the linear time-invariant control system ẋ = Ax+Bu+Dw, x(0) = x0, y = C1x, z = C2x+B1u, (3.1) with state x(t) ∈ Rn, measured output y(t) ∈ Rl, controlled output z(t) ∈ Rr, control u(t) ∈ Rp, and t-measurable exogenous disturbance w(t) ∈ Rm bounded at every moment of time: |w(t)| ≤ 1 for all t ≥ 0. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) FEEDBACK DESIGN IN LINEAR CONTROL PROBLEMS AS AN OPTIMIZATION PROBLEM 39 The problem is to design a stabilizing control in the form of state feedback (if available) u = Kx or output feedback u = Ky to reduce the “peak” output z(t), i.e. the value maxt |z(t)|. The exact solution of such a problem is difficult, but it is possible to minimize the upper bound of this quantity using the concept of invariant ellipsoid. We recall the following Definition 3.1: Ellipsoid E = { x ∈ Rn : xTP−1x ≤ 1 } , P ≻ 0, (3.2) centered at the origin is called invariant for the dynamical system if the condition x(0) ∈ E implies x(t) ∈ E for all times t ≥ 0 and all admissible external disturbances. In other words, any system trajectory starting from a point lying in the invariant ellipsoid, with all admissible external disturbances acting on the system, will be in this ellipsoid at any time. It is easy to see that if E is an invariant ellipsoid with the certain matrix P , then the linear output z = Cx of the system for x0 ∈ E belongs to the so-called bounding ellipsoid Ez = {z ∈ Rn : zT(CPCT)−1z ≤ 1}. This approach based on the technique of invariant ellipsoids was first used in the monograph [1]; a detailed exposition of this technique can be found in the book [11]. In this case, the state feedback design is reduced to a parametric problem of semidefinite programming with the help of special changes of variables; there exist convenient numerical methods for solving such problems. However, for the output feedback design, such reduction is fundamentally impossible. The initial problem of the output feedback design for system (3.1) rejecting the external disturbances can be reduced to the following matrix optimization problem: min f(K,α), f(K,α) = trC2PCT 2 + ρ∥K∥2F , (3.3) subject to the constraint( A+BKC1 + α 2 I ) P + P ( A+BKC1 + α 2 I )T + 1 α DDT = 0 with respect to the matrix variables P = PT ∈ Rn×n, K ∈ Rp×n, and the scalar parameter α > 0. The first component in (3.3) defines the size of the bounding ellipsoid with respect to the trace criterion, and the second one represents the penalty for using large control values (wherein, the coefficient ρ > 0 regulates its importance). The presence of the second component avoids the appearance of large values of the gain matrix. Within the framework of the proposed optimization approach, a gradient method for finding static output feedback is written out and its justification is given, see [9]. We formulate the following algorithm to minimize f(K,α). Algorithm 2. 1. Set the parameters ε > 0, γ > 0, 0 < τ < 1 and the initial stabilizing approximation K0. Compute the value α0 = σ(A+BK0C1). 2. At the jth iteration, the values Kj and αj are given. Compute Aj = A+BKjC1; solve the Lyapunov equations( Aj + αj 2 I ) P + P ( Aj + αj 2 I )T + 1 αj DDT = 0 Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) 40 M. KHLEBNIKOV and ( Aj + αj 2 I )T Y + Y ( Aj + αj 2 I ) + CT 2 C2 = 0, and find the matrices P and Y respectively. Compute the gradient Hj = ∇Kf(Kj, αj) from the relation 1 2 ∇Kf(K,α) = BTY PCT 1 + ρK. If ∥Hj∥ ≤ ε, then Kj is taken as an approximate solution. 3. Make one step of the gradient method Kj+1 = Kj − γjHj. The step length γj > 0 is selected by splitting γ until the following conditions are met: a. Kj+1 is a stabilizing controller; b. f(Kj+1) ≤ f(Kj)− τγj∥Hj∥2. 4. For the resulting Kj+1 compute Aj+1 = A+BKj+1C1 and the matrices P , Y and X as the solution of the Lyapunov equations( Aj+1 + αj 2 I ) P + P ( Aj+1 + αj 2 I )T + 1 αj DDT = 0, ( Aj+1 + αj 2 I )T Y + Y ( Aj+1 + αj 2 I ) + CT 2 C2 = 0, and ( Aj+1 + αj 2 I ) X +X ( Aj+1 + αj 2 I )T + P − 1 α2 j DDT = 0 respectively. 5. Solve the problem of minimizing f(Kj+1, α) with respect to α via the Newton method: αj+1 = αj − ∇αf(Kj+1, αj) ∇2 ααf(Kj+1, αj) where ∇αf(K,α) = trY ( P − 1 α2 DDT ) , ∇2 ααf(K,α) = 2 trY ( X + 1 α3 DDT ) , and obtain αj+1. Go to step 2. The method converges in the following sense. Theorem 3.1: In Algorithm 2, only a finite number of subdivisions γj is realized at each iteration, the function f(Kj) monotonically decreases, and the gradient tends to zero lim j→∞ ∥Hj∥ = 0 at the rate of geometric progression. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) FEEDBACK DESIGN IN LINEAR CONTROL PROBLEMS AS AN OPTIMIZATION PROBLEM 41 4. OUTPUT DYNAMIC FEEDBACK Consider a linear time-invariant control system in continuous time: ẋ = Ax+Bu+Dw, x(0) = x0, y = C1x+D1w, z = C2x, (4.4) with state x(t) ∈ Rn, observed output y(t) ∈ Rl, optimized output z(t) ∈ Rr, control u(t) ∈ Rp and bounded at every moment of time exogenous disturbance w(t) ∈ Rm: |w(t)| ≤ 1 for all t ≥ 0. Assume that the measurements of the state x are not available, and the information about the system is provided by its output y only. The goal is to find the minimal (in a certain sense) ellipsoid containing the optimized output z. Within the framework of the proposed approach, it was possible to write down a gradient method for constructing the feedback using the dynamic controller u = Kx̂, where x̂ is an observer described by the linear differential equation ˙̂x = Ax̂+Bu+ L(y − C1x̂), x̂(0) = 0, including the discrepancy between the output y and its forecast C1x̂ (here L ∈ Rn×l is the observer’s matrix) and provide its justification. Namely, for this setup the initial problem is reduced to the minimization problem f(K,L, α) = tr (C2 0)P (C2 0)T + ρK∥K∥2F + ρL∥L∥2F (4.5) subject to the constraint( A0 +M1KN1 +M2LN2 + α 2 I ) P + P ( A0 +M1KN1 +M2LN2 + α 2 I )T + 1 α ( D D − LD1 )( D D − LD1 )T = 0, where A0 = ( A 0 0 A ) , M1 = ( B 0 ) , N1 = (I −I) , M2 = ( 0 I ) , N2 = (0 −C1) , with respect to the matrix variables P = PT ∈ R2n×2n, K ∈ Rp×n, L ∈ Rn×l, and the scalar parameter α > 0. In (4.5), in addition to the component that determines the size of the bounding ellipsoid with respect to the trace criterion, the penalties ρK and ρL are introduced for the size of the controller and observer matrices. We formulate the following algorithm to minimize f(K,L, α), see the details in [10]. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) 42 M. KHLEBNIKOV Algorithm 3. 1. Set the parameters ε > 0, γK , γL > 0, 0 < τK , τL < 1, and the initial stabilizing approximation (K0, L0). Compute the value α0 = σ(A0 +M1K0N1 +M2L0N2). 2. At the jth iteration, the values Kj , Lj , and αj are given. Compute Aj = A0 + M1KjN1 +M2LjN2 and find the matrices P and Y as the solutions of the Lyapunov equations( Aj + αj 2 I ) P + P ( Aj + αj 2 I )T + 1 αj ( D D − LjD1 )( D D − LjD1 )T = 0 and ( Aj + αj 2 I )T Y + Y ( Aj + αj 2 I ) + (C2 0)T (C2 0) = 0 respectively. 3. Compute the gradient HK j = ∇Kf(Kj, Lj, αj) from the relation 1 2 ∇Kf(K,L, α) = MT 1 Y PNT 1 + ρKK. If ∥HK j ∥ ≤ ε, then Kj is taken as an approximate solution. 4. Make one step of the gradient method for K: Kj+1 = Kj − γK j HK j . The step length γK j > 0 is selected by splitting γK until the following conditions hold: a. Kj+1 stabilizes the matrix A0 +M1KN1 +M2LjN2; b. f(Kj+1) ≤ f(Kj)− τKγ K j ∥HK j ∥2. 5. Having Kj+1, compute the gradient HL j = ∇Lf(Kj+1, Lj, αj) from the relation 1 2 ∇Lf(K,L, α) = MT 2 Y PNT 2 − 1 α (0 I)Y ( D D − LD1 ) DT 1 + ρLL. If ∥HL j ∥ ≤ ε, then Lj is taken as an approximate solution. 6. Make one step of the gradient method for L: Lj+1 = Lj − γL j H L j , The step lengths γL j > 0 is selected by splitting γL until the following conditions hold: a. Lj+1 stabilizes the matrix A0 +M1Kj+1N1 +M2LN2; b. f(Lj+1) ≤ f(Lj)− τLγ L j ∥HL j ∥2. 7. For the resulting Kj+1, Lj+1 compute Aj+1 = A0 +M1Kj+1N1 +M2Lj+1N2 and the matrices P , Y and X as the solutions of the Lyapunov equations( Aj+1 + αj 2 I ) P + P ( Aj+1 + αj 2 I )T + 1 αj ( D D − Lj+1D1 )( D D − Lj+1D1 )T = 0, Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) FEEDBACK DESIGN IN LINEAR CONTROL PROBLEMS AS AN OPTIMIZATION PROBLEM 43 ( Aj+1 + αj 2 I )T Y + Y ( Aj+1 + αj 2 I ) + (C2 0)T (C2 0) = 0, and( Aj+1 + αj 2 I ) X +X ( Aj+1 + αj 2 I )T + P − 1 α2 j ( D D − Lj+1D1 )( D D − Lj+1D1 )T = 0 respectively. 8. Solve the problem of minimizing f(Kj+1, Lj+1, α) with respect to α via the Newton method: αj+1 = αj − ∇αf(Kj+1, Lj+1, αj) ∇2 ααf(Kj+1, Lj+1, αj) , where ∇αf(K,L, α) = trY [ P − 1 α2 ( D D − LD1 )( D D − LD1 )T] , ∇2 ααf(K,L, α) = 2 trY [ X + 1 α3 ( D D − LD1 )( D D − LD1 )T] , and get αj+1. Go to step 2. 5. EXAMPLES Example 1. Consider a double mathematical pendulum consisting of two weightless rods of length l1 and l2, at the ends of which weights of masses m1 and m2 are fixed. The system moves in a viscous medium with a drag coefficient γ, in a vertical plane xy, and the position of the pendulum is determined by the angles φ1 and φ2 of the deviation of the rods from the vertical, see Fig. 5.3. @ @@•A A AA• �u �w m1 ϕ1 m2 ϕ2 -x ?y l1 l2 Fig. 5.3 The lower body is affected by a bounded external disturbance |w| ≤ 1, to compensate for which the control action u is applied to the upper body. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) 44 M. KHLEBNIKOV Introducing auxiliary variables φ3 = φ̇1, φ4 = φ̇2, we arrive at a linearized system φ̇1 = φ3, φ̇2 = φ4, φ̇3 = − ( 1 + m2 m1 ) g l1 φ1 + m2 m1 g l1 φ2 − γ m1 φ3 + 1 m1 u, φ̇4 = ( 1 + m2 m1 ) g l2 φ1 − ( 1 + m2 m1 ) g l2 φ2 − γ m2 φ4 + 1 m2 w. At m1 = m2 = 1, l1 = l2 = g, γ = 0.15, the matrices of system (3.1) have the form A =  0 0 1 0 0 0 0 1 −2 1 −0.15 0 2 −2 0 −0.15  , B = 0 0 1 0  , D = 0 0 0 1  . As the observed output, we choose y = ( φ1 φ2 ) and as a regulated output we take the vector z = ( φ̇1 φ̇2 ) , that is C1 = ( 1 0 0 0 0 1 0 0 ) , C2 = ( 0 0 1 0 0 0 0 1 ) . Let us also ρ = 0.1. Since the open-loop system is stable, we choose K0 = (0 0) as an initial approximation for the controller. Calculations carried out in accordance with Algorithm 2 led to the following results. The dynamics of the criterion f(K) = trC2PCT 2 + ρ∥K∥2 is shown in Fig. 5.4. The process ended at the 9th step by finding the stabilizing static output feedback K∗ = (0.0050 −0.9024) ; wherein f(K∗) = 48.9381. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) FEEDBACK DESIGN IN LINEAR CONTROL PROBLEMS AS AN OPTIMIZATION PROBLEM 45 1 2 3 4 5 6 7 8 9 48 50 52 54 56 58 60 62 64 66 68 Fig. 5.4 At the same time, we obtain the matrix( 2.0663 −0.0516 −0.0516 46.7904 ) of the corresponding bounding ellipse for the regulated output z. Example 2. Consider the system from Example 1 again and construct the dynamic feedback for its observed output. Since the matrix A of the system is stable, it would seem natural to choose as initial approximations zero, but the point K = L = 0 is a saddle point for the minimized function. Therefore, as an initial approximation for the controller, we take the matrix K0 = (0 0 0 0) , and as an initial approximation for the observer, we generate a certain admissible matrix L0 =  0.0826 −0.0346 0.7379 0.6160 0.1141 0.4720 −0.9572 0.1446  . Let ρK = ρL = 0.1. According to Algorithm 3, the optimization procedure led to the controller matrix K∗ = (0.8219 −0.0402 −1.3024 0.4692) , the observer matrix L∗ =  0.7944 −0.0845 0.6222 0.9323 0.8885 −0.0391 −0.4905 0.8054  , Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) 46 M. KHLEBNIKOV and the corresponding bounding ellipse for the regulated output z with the matrix( 1.2117 0.1784 0.1784 2.5120 ) . 0 10 20 30 40 50 60 0 5 10 15 20 25 30 35 40 Fig. 5.5 The dynamics of the criterion f(K,L) = tr (C2 0)P (C2 0)T + ρK∥K∥2F + ρL∥L∥2F is shown in Fig. 5.5. 6. CONCLUSION We consider and discuss a new approach to control design in linear systems. It is based on reducing the original problem to an optimization problem in the feedback matrix variable (by state or by output). Further, the corresponding problems are solved by the gradient method; its convergence is theoretically substantiated for a number of important cases. Numerous examples demonstrate the effectiveness of this approach. In future research, the author plans to extend the proposed approach to the PI/PID controller design and filtering problems. REFERENCES 1. Boyd, S., El Ghaoui, L., Feron, E. & Balakrishnan, V. (1994) Linear Matrix Inequalities in Systems and Control Theory. Philadelphia, PA: SIAM Studies in Applied Mathematics. 2. Bu, J., Mesbahi, A., Fazel, M. & Mesbahi, M. (2019) LQR through the Lens of First Order Methods: Discrete-Time Case, arXiv: 1907.08921, [Online]. Available: https://arxiv.org/abs/1907.08921 Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) FEEDBACK DESIGN IN LINEAR CONTROL PROBLEMS AS AN OPTIMIZATION PROBLEM 47 3. Fatkhullin, I. & Polyak, B. (2021) Optimizing Static Linear Feedback: Gradient Method, SIAM Journal on Control and Optimization, 59(5), 3887–3911. 4. Fazel, M., Ge, R., Kakade, S. & Mesbahi, M. (2018) Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator. Proc. 35th International Conference on Machine Learning, Stockholm, Sweden, 1467–1476. 5. Kalman, R. E. (1960) Contributions to the Theory of Optimal Control, Boletin de la Sociedad Matematica Mexicana, 5(1), 102–119. 6. Levine, W. & Athans, M. (1970) On the Determination of the Optimal Constant Output Feedback Gains for Linear Multivariable Systems, IEEE Transactions on Automatic Control, 15(1), 44–48. 7. Mäkilä, P. M. & Toivonen, H. T. (1987) Computational Methods for Parametric LQ Problems – A Survey, IEEE Transactions on Automatic Control, 32(8), 658–671. 8. Mohammadi, H., Zare, A., Soltanolkotabi, M. & Jovanović, M. R. (2019) Global Exponential Convergence of Gradient Methods Over the Nonconvex Landscape of the Linear Quadratic Regulator, Proc. 2019 IEEE 58th Conference on Decision and Control, Nice, France, 7474–7479. 9. Polyak, B. T. & Khlebnikov, M. V. (2021) Static Controller Synthesis for Peak-to-Peak Gain Minimization as an Optimization Problem, Automation and Remote Control, 82(9), 1530–1553. 10. Polyak, B. T. & Khlebnikov, M. V. (2022) Observer-Aided Output Feedback Synthesis as an Optimization Problem, Automation and Remote Control, 83(3), 303–324. 11. Polyak, B. T., Khlebnikov, M. V. & Shcherbakov, P. S. (2014) Upravlenie lineinymi sistemami pri vneshnikh vozmushcheniyakh: Tekhnika lineinykh matrichnykh neravenstv [Control of Linear Systems under Exogenous Disturbances: Technique of Linear Matrix Inequalities]. Moscow, Russia: LENAND [in Russian]. 12. Zhang, K., Hu, B. & Başar T. (2020) Policy Optimization for H2 Linear Control with H∞ Robustness Guarantee: Implicit Regularization and Global Convergence, arXiv: 1910.09496, [Online]. Available: https://arxiv.org/abs/1910.09496 Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) Introduction Linear Quadratic Regulator Suppression of External Disturbances: Static Feedback Output Dynamic Feedback Examples Conclusion