Differentiable optimization

This well-reasoned synthesis describes the main algorithms for solving differentiable optimization problems, and gives their motivation. These problems arise when the optimal value of a finite number of parameters is to be determined. Optimality here means the minimality of a given criterion. The assumed differentiability of the functions defining the problem immediately rules out combinatorial optimization (the parameters to be optimized take only integer or discrete values, see the "Optimization in integers" dossier [AF 1 251] ) and non-smooth optimization (the functions have irregularities, see the "Optimization and convexity" dossier [AF 1 253] ).

Optimization problems arise in many fields of engineering, as well as in science and economics, often after simulation steps have been completed. These problems are often infinite-dimensional, i.e. we're looking for an optimal function rather than a finite number of optimal parameters. We then have to go through a discretization phase (in space, in time) to get back to our own framework, and thus to a problem that can be solved on a computer. The direct transcription of optimal control problems follows such a discretization procedure. Other examples are described in the "Continuous optimization" section [S 7 210] .

Numerical optimization methods were mainly developed after the Second World War, in parallel with the improvement of computers, and have been constantly enriched ever since. In nonlinear optimization, several waves can be distinguished: penalization methods, the augmented Lagrangian method (1958), quasi-Newton methods (1959), Newtonian or SQP methods (1976), interior point algorithms (1984). One wave doesn't erase the previous one, but provides better answers to certain classes of problem, as was the case with interior point methods in positive semidefinite optimization (PSO). Particular attention is paid to algorithms that can handle large-scale...