This is gsl-ref.info, produced by makeinfo version 4.2 from
gsl-ref.texi.

INFO-DIR-SECTION Scientific software
START-INFO-DIR-ENTRY
* gsl-ref: (gsl-ref).                   GNU Scientific Library - Reference
END-INFO-DIR-ENTRY


File: gsl-ref.info,  Node: Example programs for Multidimensional Root finding,  Next: References and Further Reading for Multidimensional Root Finding,  Prev: Algorithms without Derivatives,  Up: Multidimensional Root-Finding

Examples
========

   The multidimensional solvers are used in a similar way to the
one-dimensional root finding algorithms.  This first example
demonstrates the `hybrids' scaled-hybrid algorithm, which does not
require derivatives. The program solves the Rosenbrock system of
equations,

     f_1 (x, y) = a (1 - x)
     f_2 (x, y) = b (y - x^2)

with a = 1, b = 10. The solution of this system lies at (x,y) = (1,1)
in a narrow valley.

   The first stage of the program is to define the system of equations,

     #include <stdlib.h>
     #include <stdio.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_multiroots.h>
     
     struct rparams
       {
         double a;
         double b;
       };
     
     int
     rosenbrock_f (const gsl_vector * x, void *params,
                   gsl_vector * f)
     {
       double a = ((struct rparams *) params)->a;
       double b = ((struct rparams *) params)->b;
     
       double x0 = gsl_vector_get (x, 0);
       double x1 = gsl_vector_get (x, 1);
     
       double y0 = a * (1 - x0);
       double y1 = b * (x1 - x0 * x0);
     
       gsl_vector_set (f, 0, y0);
       gsl_vector_set (f, 1, y1);
     
       return GSL_SUCCESS;
     }

The main program begins by creating the function object `f', with the
arguments `(x,y)' and parameters `(a,b)'. The solver `s' is initialized
to use this function, with the `hybrids' method.

     int
     main (void)
     {
       const gsl_multiroot_fsolver_type *T;
       gsl_multiroot_fsolver *s;
     
       int status;
       size_t i, iter = 0;
     
       const size_t n = 2;
       struct rparams p = {1.0, 10.0};
       gsl_multiroot_function f = {&rosenbrock_f, n, &p};
     
       double x_init[2] = {-10.0, -5.0};
       gsl_vector *x = gsl_vector_alloc (n);
     
       gsl_vector_set (x, 0, x_init[0]);
       gsl_vector_set (x, 1, x_init[1]);
     
       T = gsl_multiroot_fsolver_hybrids;
       s = gsl_multiroot_fsolver_alloc (T, 2);
       gsl_multiroot_fsolver_set (s, &f, x);
     
       print_state (iter, s);
     
       do
         {
           iter++;
           status = gsl_multiroot_fsolver_iterate (s);
     
           print_state (iter, s);
     
           if (status)   /* check if solver is stuck */
             break;
     
           status =
             gsl_multiroot_test_residual (s->f, 1e-7);
         }
       while (status == GSL_CONTINUE && iter < 1000);
     
       printf ("status = %s\n", gsl_strerror (status));
     
       gsl_multiroot_fsolver_free (s);
       gsl_vector_free (x);
       return 0;
     }

Note that it is important to check the return status of each solver
step, in case the algorithm becomes stuck.  If an error condition is
detected, indicating that the algorithm cannot proceed, then the error
can be reported to the user, a new starting point chosen or a different
algorithm used.

   The intermediate state of the solution is displayed by the following
function.  The solver state contains the vector `s->x' which is the
current position, and the vector `s->f' with corresponding function
values.
     int
     print_state (size_t iter, gsl_multiroot_fsolver * s)
     {
       printf ("iter = %3u x = % .3f % .3f "
               "f(x) = % .3e % .3e\n",
               iter,
               gsl_vector_get (s->x, 0),
               gsl_vector_get (s->x, 1),
               gsl_vector_get (s->f, 0),
               gsl_vector_get (s->f, 1));
     }

Here are the results of running the program. The algorithm is started at
(-10,-5) far from the solution.  Since the solution is hidden in a
narrow valley the earliest steps follow the gradient of the function
downhill, in an attempt to reduce the large value of the residual. Once
the root has been approximately located, on iteration 8, the Newton
behavior takes over and convergence is very rapid.

     iter =  0 x = -10.000  -5.000  f(x) = 1.100e+01 -1.050e+03
     iter =  1 x = -10.000  -5.000  f(x) = 1.100e+01 -1.050e+03
     iter =  2 x =  -3.976  24.827  f(x) = 4.976e+00  9.020e+01
     iter =  3 x =  -3.976  24.827  f(x) = 4.976e+00  9.020e+01
     iter =  4 x =  -3.976  24.827  f(x) = 4.976e+00  9.020e+01
     iter =  5 x =  -1.274  -5.680  f(x) = 2.274e+00 -7.302e+01
     iter =  6 x =  -1.274  -5.680  f(x) = 2.274e+00 -7.302e+01
     iter =  7 x =   0.249   0.298  f(x) = 7.511e-01  2.359e+00
     iter =  8 x =   0.249   0.298  f(x) = 7.511e-01  2.359e+00
     iter =  9 x =   1.000   0.878  f(x) = 1.268e-10 -1.218e+00
     iter = 10 x =   1.000   0.989  f(x) = 1.124e-11 -1.080e-01
     iter = 11 x =   1.000   1.000  f(x) = 0.000e+00  0.000e+00
     status = success

Note that the algorithm does not update the location on every
iteration. Some iterations are used to adjust the trust-region
parameter, after trying a step which was found to be divergent, or to
recompute the Jacobian, when poor convergence behavior is detected.

   The next example program adds derivative information, in order to
accelerate the solution. There are two derivative functions
`rosenbrock_df' and `rosenbrock_fdf'. The latter computes both the
function and its derivative simultaneously. This allows the
optimization of any common terms.  For simplicity we substitute calls to
the separate `f' and `df' functions at this point in the code below.

     int
     rosenbrock_df (const gsl_vector * x, void *params,
                    gsl_matrix * J)
     {
       double a = ((struct rparams *) params)->a;
       double b = ((struct rparams *) params)->b;
     
       double x0 = gsl_vector_get (x, 0);
     
       double df00 = -a;
       double df01 = 0;
       double df10 = -2 * b  * x0;
       double df11 = b;
     
       gsl_matrix_set (J, 0, 0, df00);
       gsl_matrix_set (J, 0, 1, df01);
       gsl_matrix_set (J, 1, 0, df10);
       gsl_matrix_set (J, 1, 1, df11);
     
       return GSL_SUCCESS;
     }
     
     int
     rosenbrock_fdf (const gsl_vector * x, void *params,
                     gsl_vector * f, gsl_matrix * J)
     {
       rosenbrock_f (x, params, f);
       rosenbrock_df (x, params, J);
     
       return GSL_SUCCESS;
     }

The main program now makes calls to the corresponding `fdfsolver'
versions of the functions,

     int
     main (void)
     {
       const gsl_multiroot_fdfsolver_type *T;
       gsl_multiroot_fdfsolver *s;
     
       int status;
       size_t i, iter = 0;
     
       const size_t n = 2;
       struct rparams p = {1.0, 10.0};
       gsl_multiroot_function_fdf f = {&rosenbrock_f,
                                       &rosenbrock_df,
                                       &rosenbrock_fdf,
                                       n, &p};
     
       double x_init[2] = {-10.0, -5.0};
       gsl_vector *x = gsl_vector_alloc (n);
     
       gsl_vector_set (x, 0, x_init[0]);
       gsl_vector_set (x, 1, x_init[1]);
     
       T = gsl_multiroot_fdfsolver_gnewton;
       s = gsl_multiroot_fdfsolver_alloc (T, n);
       gsl_multiroot_fdfsolver_set (s, &f, x);
     
       print_state (iter, s);
     
       do
         {
           iter++;
     
           status = gsl_multiroot_fdfsolver_iterate (s);
     
           print_state (iter, s);
     
           if (status)
             break;
     
           status = gsl_multiroot_test_residual (s->f, 1e-7);
         }
       while (status == GSL_CONTINUE && iter < 1000);
     
       printf ("status = %s\n", gsl_strerror (status));
     
       gsl_multiroot_fdfsolver_free (s);
       gsl_vector_free (x);
       return 0;
     }

The addition of derivative information to the `hybrids' solver does not
make any significant difference to its behavior, since it able to
approximate the Jacobian numerically with sufficient accuracy.  To
illustrate the behavior of a different derivative solver we switch to
`gnewton'. This is a traditional newton solver with the constraint that
it scales back its step if the full step would lead "uphill". Here is
the output for the `gnewton' algorithm,

     iter = 0 x = -10.000  -5.000 f(x) =  1.100e+01 -1.050e+03
     iter = 1 x =  -4.231 -65.317 f(x) =  5.231e+00 -8.321e+02
     iter = 2 x =   1.000 -26.358 f(x) = -8.882e-16 -2.736e+02
     iter = 3 x =   1.000   1.000 f(x) = -2.220e-16 -4.441e-15
     status = success

The convergence is much more rapid, but takes a wide excursion out to
the point (-4.23,-65.3). This could cause the algorithm to go astray in
a realistic application.  The hybrid algorithm follows the downhill
path to the solution more reliably.


File: gsl-ref.info,  Node: References and Further Reading for Multidimensional Root Finding,  Prev: Example programs for Multidimensional Root finding,  Up: Multidimensional Root-Finding

References and Further Reading
==============================

The original version of the Hybrid method is described in the following
articles by Powell,

     M.J.D. Powell, "A Hybrid Method for Nonlinear Equations" (Chap 6, p
     87-114) and "A Fortran Subroutine for Solving systems of Nonlinear
     Algebraic Equations" (Chap 7, p 115-161), in `Numerical Methods for
     Nonlinear Algebraic Equations', P. Rabinowitz, editor.  Gordon and
     Breach, 1970.

The following papers are also relevant to the algorithms described in
this section,

     J.J. More', M.Y. Cosnard, "Numerical Solution of Nonlinear
     Equations", `ACM Transactions on Mathematical Software', Vol 5, No
     1, (1979), p 64-85

     C.G. Broyden, "A Class of Methods for Solving Nonlinear
     Simultaneous Equations", `Mathematics of Computation', Vol 19
     (1965), p 577-593

     J.J. More', B.S. Garbow, K.E. Hillstrom, "Testing Unconstrained
     Optimization Software", ACM Transactions on Mathematical Software,
     Vol 7, No 1 (1981), p 17-41


File: gsl-ref.info,  Node: Multidimensional Minimization,  Next: Least-Squares Fitting,  Prev: Multidimensional Root-Finding,  Up: Top

Multidimensional Minimization
*****************************

   This chapter describes routines for finding minima of arbitrary
multidimensional functions.  The library provides low level components
for a variety of iterative minimizers and convergence tests.  These can
be combined by the user to achieve the desired solution, while providing
full access to the intermediate steps of the algorithms.  Each class of
methods uses the same framework, so that you can switch between
minimizers at runtime without needing to recompile your program.  Each
instance of a minimizer keeps track of its own state, allowing the
minimizers to be used in multi-threaded programs. The minimization
algorithms can be used to maximize a function by inverting its sign.

   The header file `gsl_multimin.h' contains prototypes for the
minimization functions and related declarations.

* Menu:

* Multimin Overview::
* Multimin Caveats::
* Initializing the Multidimensional Minimizer::
* Providing a function to minimize::
* Multimin Iteration::
* Multimin Stopping Criteria::
* Multimin Algorithms::
* Multimin Examples::
* Multimin References and Further Reading::


File: gsl-ref.info,  Node: Multimin Overview,  Next: Multimin Caveats,  Up: Multidimensional Minimization

Overview
========

   The problem of multidimensional minimization requires finding a point
x such that the scalar function,

     f(x_1, ..., x_n)

takes a value which is lower than at any neighboring point. For smooth
functions the gradient g = \nabla f vanishes at the minimum. In general
there are no bracketing methods available for the minimization of
n-dimensional functions.  All algorithms proceed from an initial guess
using a search algorithm which attempts to move in a downhill direction.

   All algorithms making use of the gradient of the fuction perform a
one-dimensional line minimisation along this direction until the lowest
point is found to a suitable tolerance.  The search direction is then
updated with local information from the function and its derivatives,
and the whole process repeated until the true n-dimensional minimum is
found.

   The Nelder-Mead Simplex algorithm applies a different strategy.  It
maintains n+1 trial parameter vectors as the vertices of a
n-dimensional simplex.  In each iteration step it tries to improve the
worst vertex by a simple geometrical transformation until the size of
the simplex falls below a given tolerance.

   Several minimization algorithms are available within a single
framework. The user provides a high-level driver for the algorithms, and
the library provides the individual functions necessary for each of the
steps.  There are three main phases of the iteration.  The steps are,

   * initialize minimizer state, S, for algorithm T

   * update S using the iteration T

   * test S for convergence, and repeat iteration if necessary

Each iteration step consists either of an improvement to the
line-mimisation in the current direction or an update to the search
direction itself.  The state for the minimizers is held in a
`gsl_multimin_fdfminimizer' struct or a `gsl_multimin_fminimizer'
struct.


File: gsl-ref.info,  Node: Multimin Caveats,  Next: Initializing the Multidimensional Minimizer,  Prev: Multimin Overview,  Up: Multidimensional Minimization

Caveats
=======

   Note that the minimization algorithms can only search for one local
minimum at a time.  When there are several local minima in the search
area, the first minimum to be found will be returned; however it is
difficult to predict which of the minima this will be.  In most cases,
no error will be reported if you try to find a local minimum in an area
where there is more than one.

   It is also important to note that the minimization algorithms find
local minima; there is no way to determine whether a minimum is a global
minimum of the function in question.


File: gsl-ref.info,  Node: Initializing the Multidimensional Minimizer,  Next: Providing a function to minimize,  Prev: Multimin Caveats,  Up: Multidimensional Minimization

Initializing the Multidimensional Minimizer
===========================================

   The following function initializes a multidimensional minimizer.  The
minimizer itself depends only on the dimension of the problem and the
algorithm and can be reused for different problems.

 - Function: gsl_multimin_fdfminimizer *
          gsl_multimin_fdfminimizer_alloc (const
          gsl_multimin_fdfminimizer_type *T, size_t N)
 - Function: gsl_multimin_fminimizer * gsl_multimin_fminimizer_alloc
          (const gsl_multimin_fminimizer_type *T, size_t N)
     This function returns a pointer to a newly allocated instance of a
     minimizer of type T for an N-dimension function.  If there is
     insufficient memory to create the minimizer then the function
     returns a null pointer and the error handler is invoked with an
     error code of `GSL_ENOMEM'.

 - Function: int gsl_multimin_fdfminimizer_set
          (gsl_multimin_fdfminimizer * S, gsl_multimin_function_fdf
          *FDF, const gsl_vector * X, double STEP_SIZE, double TOL)
     This function initializes the minimizer S to minimize the function
     FDF starting from the initial point X.  The size of the first
     trial step is given by STEP_SIZE.  The accuracy of the line
     minimization is specified by TOL.  The precise meaning of this
     parameter depends on the method used.  Typically the line
     minimization is considered successful if the gradient of the
     function g is orthogonal to the current search direction p to a
     relative accuracy of TOL, where dot(p,g) < tol |p| |g|.

 - Function: int gsl_multimin_fminimizer_set (gsl_multimin_fminimizer *
          S, gsl_multimin_function *F, const gsl_vector * X, const
          gsl_vector * STEP_SIZE)
     This function initializes the minimizer S to minimize the function
     F, starting from the initial point X. The size of the initial
     trial steps is given in vector STEP_SIZE. The precise meaning of
     this parameter depends on the method used.

 - Function: void gsl_multimin_fdfminimizer_free
          (gsl_multimin_fdfminimizer *S)
 - Function: void gsl_multimin_fminimizer_free (gsl_multimin_fminimizer
          *S)
     This function frees all the memory associated with the minimizer S.

 - Function: const char * gsl_multimin_fdfminimizer_name (const
          gsl_multimin_fdfminimizer * S)
 - Function: const char * gsl_multimin_fminimizer_name (const
          gsl_multimin_fminimizer * S)
     This function returns a pointer to the name of the minimizer.  For
     example,

          printf("s is a '%s' minimizer\n",
                 gsl_multimin_fdfminimizer_name (s));

     would print something like `s is a 'conjugate_pr' minimizer'.


File: gsl-ref.info,  Node: Providing a function to minimize,  Next: Multimin Iteration,  Prev: Initializing the Multidimensional Minimizer,  Up: Multidimensional Minimization

Providing a function to minimize
================================

   You must provide a parametric function of n variables for the
minimizers to operate on.  You may also need to provide a routine which
calculates the gradient of the function and a third routine which
calculates both the function value and the gradient together.  In order
to allow for general parameters the functions are defined by the
following data type:

 - Data Type: gsl_multimin_function_fdf
     This data type defines a general function of n variables with
     parameters and the corresponding gradient vector of derivatives,

    `double (* f) (const gsl_vector * X, void * PARAMS)'
          this function should return the result f(x,params) for
          argument X and parameters PARAMS.

    `int (* df) (const gsl_vector * X, void * PARAMS, gsl_vector * G)'
          this function should store the N-dimensional gradient g_i = d
          f(x,params) / d x_i in the vector G for argument X and
          parameters PARAMS, returning an appropriate error code if the
          function cannot be computed.

    `int (* fdf) (const gsl_vector * X, void * PARAMS, double * f, gsl_vector * G)'
          This function should set the values of the F and G as above,
          for arguments X and parameters PARAMS.  This function provides
          an optimization of the separate functions for f(x) and g(x) -
          it is always faster to compute the function and its
          derivative at the same time.

    `size_t N'
          the dimension of the system, i.e. the number of components of
          the vectors X.

    `void * PARAMS'
          a pointer to the parameters of the function.

 - Data Type: gsl_multimin_function
     This data type defines a general function of n variables with
     parameters,

    `double (* f) (const gsl_vector * X, void * PARAMS)'
          this function should return the result f(x,params) for
          argument X and parameters PARAMS.

    `size_t N'
          the dimension of the system, i.e. the number of components of
          the vectors X.

    `void * PARAMS'
          a pointer to the parameters of the function.

The following example function defines a simple paraboloid with two
parameters,

     /* Paraboloid centered on (dp[0],dp[1]) */
     
     double
     my_f (const gsl_vector *v, void *params)
     {
       double x, y;
       double *dp = (double *)params;
     
       x = gsl_vector_get(v, 0);
       y = gsl_vector_get(v, 1);
     
       return 10.0 * (x - dp[0]) * (x - dp[0]) +
                20.0 * (y - dp[1]) * (y - dp[1]) + 30.0;
     }
     
     /* The gradient of f, df = (df/dx, df/dy). */
     void
     my_df (const gsl_vector *v, void *params,
            gsl_vector *df)
     {
       double x, y;
       double *dp = (double *)params;
     
       x = gsl_vector_get(v, 0);
       y = gsl_vector_get(v, 1);
     
       gsl_vector_set(df, 0, 20.0 * (x - dp[0]));
       gsl_vector_set(df, 1, 40.0 * (y - dp[1]));
     }
     
     /* Compute both f and df together. */
     void
     my_fdf (const gsl_vector *x, void *params,
             double *f, gsl_vector *df)
     {
       *f = my_f(x, params);
       my_df(x, params, df);
     }

The function can be initialized using the following code,

     gsl_multimin_function_fdf my_func;
     
     double p[2] = { 1.0, 2.0 }; /* center at (1,2) */
     
     my_func.f = &my_f;
     my_func.df = &my_df;
     my_func.fdf = &my_fdf;
     my_func.n = 2;
     my_func.params = (void *)p;


File: gsl-ref.info,  Node: Multimin Iteration,  Next: Multimin Stopping Criteria,  Prev: Providing a function to minimize,  Up: Multidimensional Minimization

Iteration
=========

   The following function drives the iteration of each algorithm.  The
function performs one iteration to update the state of the minimizer.
The same function works for all minimizers so that different methods can
be substituted at runtime without modifications to the code.

 - Function: int gsl_multimin_fdfminimizer_iterate
          (gsl_multimin_fdfminimizer *S)
 - Function: int gsl_multimin_fminimizer_iterate
          (gsl_multimin_fminimizer *S)
     These functions perform a single iteration of the minimizer S.  If
     the iteration encounters an unexpected problem then an error code
     will be returned.

The minimizer maintains a current best estimate of the minimum at all
times.  This information can be accessed with the following auxiliary
functions,

 - Function: gsl_vector * gsl_multimin_fdfminimizer_x (const
          gsl_multimin_fdfminimizer * S)
 - Function: gsl_vector * gsl_multimin_fminimizer_x (const
          gsl_multimin_fminimizer * S)
 - Function: double gsl_multimin_fdfminimizer_minimum (const
          gsl_multimin_fdfminimizer * S)
 - Function: double gsl_multimin_fminimizer_minimum (const
          gsl_multimin_fminimizer * S)
 - Function: gsl_vector * gsl_multimin_fdfminimizer_gradient (const
          gsl_multimin_fdfminimizer * S)
 - Function: double gsl_multimin_fminimizer_size (const
          gsl_multimin_fminimizer * S)
     These functions return the current best estimate of the location
     of the minimum, the value of the function at that point, its
     gradient, and minimizer specific characteristic size for the
     minimizer S.

 - Function: int gsl_multimin_fdfminimizer_restart
          (gsl_multimin_fdfminimizer *S)
     This function resets the minimizer S to use the current point as a
     new starting point.


File: gsl-ref.info,  Node: Multimin Stopping Criteria,  Next: Multimin Algorithms,  Prev: Multimin Iteration,  Up: Multidimensional Minimization

Stopping Criteria
=================

   A minimization procedure should stop when one of the following
conditions is true:

   * A minimum has been found to within the user-specified precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The functions
below allow the user to test the precision of the current result.

 - Function: int gsl_multimin_test_gradient (const gsl_vector * G,
          double EPSABS)
     This function tests the norm of the gradient G against the
     absolute tolerance EPSABS. The gradient of a multidimensional
     function goes to zero at a minimum. The test returns `GSL_SUCCESS'
     if the following condition is achieved,

          |g| < epsabs

     and returns `GSL_CONTINUE' otherwise.  A suitable choice of EPSABS
     can be made from the desired accuracy in the function for small
     variations in x.  The relationship between these quantities is
     given by \delta f = g \delta x.

 - Function: int gsl_multimin_test_size (const double SIZE, double
          EPSABS)
     This function tests the minimizer specific characteristic size (if
     applicable to the used minimizer) against absolute tolerance
     EPSABS.  The test returns `GSL_SUCCESS' if the size is smaller
     than tolerance, otherwise `GSL_CONTINUE' is returned.


File: gsl-ref.info,  Node: Multimin Algorithms,  Next: Multimin Examples,  Prev: Multimin Stopping Criteria,  Up: Multidimensional Minimization

Algorithms
==========

   There are several minimization methods available. The best choice of
algorithm depends on the problem.  All of the algorithms uses the value
of the function and most of its gradient at each evaluation point, too.

 - Minimizer: gsl_multimin_fdfminimizer_conjugate_fr
     This is the Fletcher-Reeves conjugate gradient algorithm. The
     conjugate gradient algorithm proceeds as a succession of line
     minimizations. The sequence of search directions is used to build
     up an approximation to the curvature of the function in the
     neighborhood of the minimum.  An initial search direction P is
     chosen using the gradient and line minimization is carried out in
     that direction.  The accuracy of the line minimization is
     specified by the parameter TOL.  At the minimum along this line
     the function gradient G and the search direction P are orthogonal.
     The line minimization terminates when dot(p,g) < tol |p| |g|.  The
     search direction is updated  using the Fletcher-Reeves formula p'
     = g' - \beta g where \beta=-|g'|^2/|g|^2, and the line
     minimization is then repeated for the new search direction.

 - Minimizer: gsl_multimin_fdfminimizer_conjugate_pr
     This is the Polak-Ribiere conjugate gradient algorithm.  It is
     similar to the Fletcher-Reeves method, differing only in the
     choice of the coefficient \beta. Both methods work well when the
     evaluation point is close enough to the minimum of the objective
     function that it is well approximated by a quadratic hypersurface.

 - Minimizer: gsl_multimin_fdfminimizer_vector_bfgs
     This is the vector Broyden-Fletcher-Goldfarb-Shanno (BFGS)
     conjugate gradient algorithm.  It is a quasi-Newton method which
     builds up an approximation to the second derivatives of the
     function f using the difference between successive gradient
     vectors.  By combining the first and second derivatives the
     algorithm is able to take Newton-type steps towards the function
     minimum, assuming quadratic behavior in that region.

 - Minimizer: gsl_multimin_fdfminimizer_steepest_descent
     The steepest descent algorithm follows the downhill gradient of the
     function at each step. When a downhill step is successful the
     step-size is increased by factor of two.  If the downhill step
     leads to a higher function value then the algorithm backtracks and
     the step size is decreased using the parameter TOL.  A suitable
     value of TOL for most applications is 0.1.  The steepest descent
     method is inefficient and is included only for demonstration
     purposes.

 - Minimizer: gsl_multimin_fminimizer_nmsimplex
     This is the Simplex algorithm by Nelder and Mead. It constructs n
     vectors p_i from the starting vector X as follows:

     that form the n+1 vertices of a simplex in n dimensions.  In each
     iteration step the algorithm tries to improve the parameter vector
     p_i corresponding to the highest, i. e.  worst, function value by
     simple geometrical transformations.  These are reflection,
     reflection followed by expansion, contraction and multiple
     contraction. Using these transformations the simplex moves through
     the parameter space towards the minimum, where it contracts itself.

     After each iteration, the best vertex is returned.  Note, that due
     to the nature of the algorithm (getting rid of the worst
     estimate), every iteration doesn't necessarily improve the current
     best parameter vector.  Usually several iterations are required.

     The routine calculates the minimizer specific characteristic size
     as the average distance from the geometrical center of the simplex
     to all its vertices.  This size can be used as a stopping
     criteria, as the simplex contracts itself near the minimum. The
     size is returned by the function `gsl_multimin_fminimizer_size'.


File: gsl-ref.info,  Node: Multimin Examples,  Next: Multimin References and Further Reading,  Prev: Multimin Algorithms,  Up: Multidimensional Minimization

Examples
========

   This example program finds the minimum of the paraboloid function
defined earlier.  The location of the minimum is offset from the origin
in x and y, and the function value at the minimum is non-zero. The main
program is given below, it requires the example function given earlier
in this chapter.

     int
     main (void)
     {
       size_t iter = 0;
       int status;
     
       const gsl_multimin_fdfminimizer_type *T;
       gsl_multimin_fdfminimizer *s;
     
       /* Position of the minimum (1,2). */
       double par[2] = { 1.0, 2.0 };
     
       gsl_vector *x;
       gsl_multimin_function_fdf my_func;
     
       my_func.f = &my_f;
       my_func.df = &my_df;
       my_func.fdf = &my_fdf;
       my_func.n = 2;
       my_func.params = &par;
     
       /* Starting point, x = (5,7) */
     
       x = gsl_vector_alloc (2);
       gsl_vector_set (x, 0, 5.0);
       gsl_vector_set (x, 1, 7.0);
     
       T = gsl_multimin_fdfminimizer_conjugate_fr;
       s = gsl_multimin_fdfminimizer_alloc (T, 2);
     
       gsl_multimin_fdfminimizer_set (s, &my_func, x, 0.01, 1e-4);
     
       do
         {
           iter++;
           status = gsl_multimin_fdfminimizer_iterate (s);
     
           if (status)
             break;
     
           status = gsl_multimin_test_gradient (s->gradient, 1e-3);
     
           if (status == GSL_SUCCESS)
             printf ("Minimum found at:\n");
     
           printf ("%5d %.5f %.5f %10.5f\n", iter,
                   gsl_vector_get (s->x, 0),
                   gsl_vector_get (s->x, 1),
                   s->f);
     
         }
       while (status == GSL_CONTINUE && iter < 100);
     
       gsl_multimin_fdfminimizer_free (s);
       gsl_vector_free (x);
     
       return 0;
     }

The initial step-size is chosen as 0.01, a conservative estimate in this
case, and the line minimization parameter is set at 0.0001.  The program
terminates when the norm of the gradient has been reduced below 0.001.
The output of the program is shown below,

              x       y         f
         1 4.99629 6.99072  687.84780
         2 4.98886 6.97215  683.55456
         3 4.97400 6.93501  675.01278
         4 4.94429 6.86073  658.10798
         5 4.88487 6.71217  625.01340
         6 4.76602 6.41506  561.68440
         7 4.52833 5.82083  446.46694
         8 4.05295 4.63238  261.79422
         9 3.10219 2.25548   75.49762
        10 2.85185 1.62963   67.03704
        11 2.19088 1.76182   45.31640
        12 0.86892 2.02622   30.18555
     Minimum found at:
        13 1.00000 2.00000   30.00000

Note that the algorithm gradually increases the step size as it
successfully moves downhill, as can be seen by plotting the successive
points.

The conjugate gradient algorithm finds the minimum on its second
direction because the function is purely quadratic. Additional
iterations would be needed for a more complicated function.

   Here is another example using the Nelder Mead Simplex algorithm to
minimize the same example object function, as above.

     int
     main(void)
     {
       size_t np = 2;
       double par[2] = {1.0, 2.0};
     
       const gsl_multimin_fminimizer_type *T =
         gsl_multimin_fminimizer_nmsimplex;
       gsl_multimin_fminimizer *s = NULL;
       gsl_vector *ss, *x;
       gsl_multimin_function minex_func;
     
       size_t iter = 0, i;
       int rval = GSL_CONTINUE;
       int status = GSL_SUCCESS;
       double ssval;
     
       /* Initial vertex size vector */
     
       ss = gsl_vector_alloc (np);
     
       if (ss == NULL)
         {
           GSL_ERROR_VAL ("failed to allocate space for ss", GSL_ENOMEM, 0);
         }
     
       gsl_vector_set_all (ss, 1.0);
     
       /* Starting point */
     
       x = gsl_vector_alloc (np);
     
       if (x == NULL)
         {
           gsl_vector_free(ss);
           GSL_ERROR_VAL ("failed to allocate space for x", GSL_ENOMEM, 0);
         }
     
       gsl_vector_set (x, 0, 5.0);
       gsl_vector_set (x, 1, 7.0);
     
       /* Initialize method and iterate */
     
       minex_func.f = &my_f;
       minex_func.n = np;
       minex_func.params = (void *)&par;
     
       s = gsl_multimin_fminimizer_alloc (T, np);
       gsl_multimin_fminimizer_set (s, &minex_func, x, ss);
     
       while (rval == GSL_CONTINUE)
         {
           iter++;
           status = gsl_multimin_fminimizer_iterate(s);
     
           if (status)
             break;
     
           rval = gsl_multimin_test_size (gsl_multimin_fminimizer_size (s),
                                          1e-2);
           ssval = gsl_multimin_fminimizer_size (s);
     
           if (rval == GSL_SUCCESS)
             printf ("converged to minimum at\n");
     
           printf("%5d ", iter);
           for (i = 0; i < np; i++)
             {
               printf ("%10.3e ", gsl_vector_get (s->x, i));
             }
           printf("f() = %-10.3f ssize = %.3f\n", s->fval, ssval);
         }
     
       gsl_vector_free(x);
       gsl_vector_free(ss);
       gsl_multimin_fminimizer_free (s);
     
       return status;
     }

The minimum search stops when the Simplex size drops to 0.01. The
output is shown below.

         1  6.500e+00  5.000e+00 f() = 512.500    ssize = 1.082
         2  5.250e+00  4.000e+00 f() = 290.625    ssize = 1.372
         3  5.250e+00  4.000e+00 f() = 290.625    ssize = 1.372
         4  5.500e+00  1.000e+00 f() = 252.500    ssize = 1.372
         5  2.625e+00  3.500e+00 f() = 101.406    ssize = 1.823
         6  3.469e+00  1.375e+00 f() = 98.760     ssize = 1.526
         7  1.820e+00  3.156e+00 f() = 63.467     ssize = 1.105
         8  1.820e+00  3.156e+00 f() = 63.467     ssize = 1.105
         9  1.016e+00  2.812e+00 f() = 43.206     ssize = 1.105
        10  2.041e+00  2.008e+00 f() = 40.838     ssize = 0.645
        11  1.236e+00  1.664e+00 f() = 32.816     ssize = 0.645
        12  1.236e+00  1.664e+00 f() = 32.816     ssize = 0.447
        13  5.225e-01  1.980e+00 f() = 32.288     ssize = 0.447
        14  1.103e+00  2.073e+00 f() = 30.214     ssize = 0.345
        15  1.103e+00  2.073e+00 f() = 30.214     ssize = 0.264
        16  1.103e+00  2.073e+00 f() = 30.214     ssize = 0.160
        17  9.864e-01  1.934e+00 f() = 30.090     ssize = 0.132
        18  9.190e-01  1.987e+00 f() = 30.069     ssize = 0.092
        19  1.028e+00  2.017e+00 f() = 30.013     ssize = 0.056
        20  1.028e+00  2.017e+00 f() = 30.013     ssize = 0.046
        21  1.028e+00  2.017e+00 f() = 30.013     ssize = 0.033
        22  9.874e-01  1.985e+00 f() = 30.006     ssize = 0.028
        23  9.846e-01  1.995e+00 f() = 30.003     ssize = 0.023
        24  1.007e+00  2.003e+00 f() = 30.001     ssize = 0.012
     converged to minimum at
        25  1.007e+00  2.003e+00 f() = 30.001     ssize = 0.010

The simplex size first increases, while the simplex moves towards the
minimum. After a while the size begins to decrease as the simplex
contracts around the minimum.


File: gsl-ref.info,  Node: Multimin References and Further Reading,  Prev: Multimin Examples,  Up: Multidimensional Minimization

References and Further Reading
==============================

A brief description of multidimensional minimization algorithms and
further references can be found in the following book,

     C.W. Ueberhuber, `Numerical Computation (Volume 2)', Chapter 14,
     Section 4.4 "Minimization Methods", p. 325--335, Springer (1997),
     ISBN 3-540-62057-5.

     J.A. Nelder and R. Mead, `A simplex method for function
     minimization', Computer Journal vol. 7 (1965), 308--315.


File: gsl-ref.info,  Node: Least-Squares Fitting,  Next: Nonlinear Least-Squares Fitting,  Prev: Multidimensional Minimization,  Up: Top

Least-Squares Fitting
*********************

This chapter describes routines for performing least squares fits to
experimental data using linear combinations of functions. The data may
be weighted or unweighted.  For weighted data the functions compute the
best fit parameters and their associated covariance matrix.  For
unweighted data the covariance matrix is estimated from the scatter of
the points, giving a variance-covariance matrix. The functions are
divided into separate versions for simple one- or two-parameter
regression and multiple-parameter fits.  The functions are declared in
the header file `gsl_fit.h'

* Menu:

* Linear regression::
* Linear fitting without a constant term::
* Multi-parameter fitting::
* Fitting Examples::
* Fitting References and Further Reading::


File: gsl-ref.info,  Node: Linear regression,  Next: Linear fitting without a constant term,  Up: Least-Squares Fitting

Linear regression
=================

   The functions described in this section can be used to perform
least-squares fits to a straight line model, Y = c_0 + c_1 X.  For
weighted data the best-fit is found by minimizing the weighted sum of
squared residuals, \chi^2,

     \chi^2 = \sum_i w_i (y_i - (c_0 + c_1 x_i))^2

for the parameters c_0, c_1.  For unweighted data the sum is computed
with w_i = 1.

 - Function: int gsl_fit_linear (const double * X, const size_t
          XSTRIDE, const double * Y, const size_t YSTRIDE, size_t N,
          double * C0, double * C1, double * COV00, double * COV01,
          double * COV11, double * SUMSQ)
     This function computes the best-fit linear regression coefficients
     (C0,C1) of the model Y = c_0 + c_1 X for the datasets (X, Y), two
     vectors of length N with strides XSTRIDE and YSTRIDE.  The
     variance-covariance matrix for the parameters (C0, C1) is
     estimated from the scatter of the points around the best-fit line
     and returned via the parameters (COV00, COV01, COV11).  The sum of
     squares of the residuals from the best-fit line is returned in
     SUMSQ.

 - Function: int gsl_fit_wlinear (const double * X, const size_t
          XSTRIDE, const double * W, const size_t WSTRIDE, const double
          * Y, const size_t YSTRIDE, size_t N, double * C0, double *
          C1, double * COV00, double * COV01, double * COV11, double *
          CHISQ)
     This function computes the best-fit linear regression coefficients
     (C0,C1) of the model Y = c_0 + c_1 X for the weighted datasets (X,
     Y), two vectors of length N with strides XSTRIDE and YSTRIDE.  The
     vector W, of length N and stride WSTRIDE, specifies the weight of
     each datapoint. The weight is the reciprocal of the variance for
     each datapoint in Y.

     The covariance matrix for the parameters (C0, C1) is estimated
     from weighted data and returned via the parameters (COV00, COV01,
     COV11).  The weighted sum of squares of the residuals from the
     best-fit line, \chi^2, is returned in CHISQ.

 - Function: int gsl_fit_linear_est (double X, double C0, double C1,
          double C00, double C01, double C11, double *Y, double *Y_ERR)
     This function uses the best-fit linear regression coefficients
     C0,C1 and their estimated covariance COV00,COV01,COV11 to compute
     the fitted function Y and its standard deviation Y_ERR for the
     model Y = c_0 + c_1 X at the point X.


File: gsl-ref.info,  Node: Linear fitting without a constant term,  Next: Multi-parameter fitting,  Prev: Linear regression,  Up: Least-Squares Fitting

Linear fitting without a constant term
======================================

   The functions described in this section can be used to perform
least-squares fits to a straight line model without a constant term, Y
= c_1 X.  For weighted data the best-fit is found by minimizing the
weighted sum of squared residuals, \chi^2,

     \chi^2 = \sum_i w_i (y_i - c_1 x_i)^2

for the parameter c_1.  For unweighted data the sum is computed with
w_i = 1.

 - Function: int gsl_fit_mul (const double * X, const size_t XSTRIDE,
          const double * Y, const size_t YSTRIDE, size_t N, double *
          C1, double * COV11, double * SUMSQ)
     This function computes the best-fit linear regression coefficient
     C1 of the model Y = c_1 X for the datasets (X, Y), two vectors of
     length N with strides XSTRIDE and YSTRIDE.  The variance of the
     parameter C1 is estimated from the scatter of the points around
     the best-fit line and returned via the parameter COV11.  The sum
     of squares of the residuals from the best-fit line is returned in
     SUMSQ.

 - Function: int gsl_fit_wmul (const double * X, const size_t XSTRIDE,
          const double * W, const size_t WSTRIDE, const double * Y,
          const size_t YSTRIDE, size_t N, double * C1, double * COV11,
          double * SUMSQ)
     This function computes the best-fit linear regression coefficient
     C1 of the model Y = c_1 X for the weighted datasets (X, Y), two
     vectors of length N with strides XSTRIDE and YSTRIDE.  The vector
     W, of length N and stride WSTRIDE, specifies the weight of each
     datapoint. The weight is the reciprocal of the variance for each
     datapoint in Y.

     The variance of the parameter C1 is estimated from the weighted
     data and returned via the parameters COV11.  The weighted sum of
     squares of the residuals from the best-fit line, \chi^2, is
     returned in CHISQ.

 - Function: int gsl_fit_mul_est (double X, double C1, double C11,
          double *Y, double *Y_ERR)
     This function uses the best-fit linear regression coefficient C1
     and its estimated covariance COV11 to compute the fitted function
     Y and its standard deviation Y_ERR for the model Y = c_1 X at the
     point X.


File: gsl-ref.info,  Node: Multi-parameter fitting,  Next: Fitting Examples,  Prev: Linear fitting without a constant term,  Up: Least-Squares Fitting

Multi-parameter fitting
=======================

   The functions described in this section perform least-squares fits
to a general linear model, y = X c where y is a vector of n
observations, X is an n by p matrix of predictor variables, and c are
the p unknown best-fit parameters, which are to be estimated.

   The best-fit is found by minimizing the weighted sums of squared
residuals, \chi^2,

     \chi^2 = (y - X c)^T W (y - X c)

with respect to the parameters c. The weights are specified by the
diagonal elements of the n by n matrix W.  For unweighted data W is
replaced by the identity matrix.

   This formulation can be used for fits to any number of functions
and/or variables by preparing the n-by-p matrix X appropriately.  For
example, to fit to a p-th order polynomial in X, use the following
matrix,

     X_{ij} = x_i^j

where the index i runs over the observations and the index j runs from
0 to p-1.

   To fit to a set of p sinusoidal functions with fixed frequencies
\omega_1, \omega_2, ..., \omega_p, use,

     X_{ij} = sin(\omega_j x_i)

To fit to p independent variables x_1, x_2, ..., x_p, use,

     X_{ij} = x_j(i)

where x_j(i) is the i-th value of the predictor variable x_j.

   The functions described in this section are declared in the header
file `gsl_multifit.h'.

   The solution of the general linear least-squares system requires an
additional working space for intermediate results, such as the singular
value decomposition of the matrix X.

 - Function: gsl_multifit_linear_workspace * gsl_multifit_linear_alloc
          (size_t N, size_t P)
     This function allocates a workspace for fitting a model to N
     observations using P parameters.

 - Function: void gsl_multifit_linear_free
          (gsl_multifit_linear_workspace * WORK)
     This function frees the memory associated with the workspace W.

 - Function: int gsl_multifit_linear (const gsl_matrix * X, const
          gsl_vector * Y, gsl_vector * C, gsl_matrix * COV, double *
          CHISQ, gsl_multifit_linear_workspace * WORK)
     This function computes the best-fit parameters C of the model y =
     X c for the observations Y and the matrix of predictor variables
     X.  The variance-covariance matrix of the model parameters COV is
     estimated from the scatter of the observations about the best-fit.
     The sum of squares of the residuals from the best-fit, \chi^2, is
     returned in CHISQ.

     The best-fit is found by singular value decomposition of the matrix
     X using the preallocated workspace provided in WORK. The modified
     Golub-Reinsch SVD algorithm is used, with column scaling to
     improve the accuracy of the singular values. Any components which
     have zero singular value (to machine precision) are discarded from
     the fit.

 - Function: int gsl_multifit_wlinear (const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, gsl_vector * C,
          gsl_matrix * COV, double * CHISQ,
          gsl_multifit_linear_workspace * WORK)
     This function computes the best-fit parameters C of the model y =
     X c for the observations Y and the matrix of predictor variables
     X.  The covariance matrix of the model parameters COV is estimated
     from the weighted data.  The weighted sum of squares of the
     residuals from the best-fit, \chi^2, is returned in CHISQ.

     The best-fit is found by singular value decomposition of the matrix
     X using the preallocated workspace provided in WORK. Any
     components which have zero singular value (to machine precision)
     are discarded from the fit.

