| Larch: Data Analysis Tools for X-ray Spectroscopy

# 13.8. Advanced Confidence Intervals and Chi-square maps

Having good estimates for uncertainties in fitted parameters is important for any scientific analysis. One of principle attractions to using the Levenberg-Marquardt algorithm that is the default fitting mechanism with the `minimize()` function is that it will automatically calculate estimates of parameter uncertainties and correlations. This is very convenient and generally reliable, but it should be made clear that the basic assumptions made when these uncertainties are estimated are not always perfect. Unfortunately, it is sometimes difficult to tell when this is the case.

It is therefore fairly common to see analyses that include explicit exploration of Parameter values away from their best-fit solution, in order to determine the degree of confidence in the best-fit values. Larch provides two main functions to help explore such cases. To be sure, they are much slower than the automatic estimation of the uncertainties. For many (perhaps most) cases, they do not provide much better insight than the automatic method.

confidence_intervals(minimizer, sigmas=(1, 2, 3), prob_func=None, maxiter=200)

  Calculate confidence intervals for the parameters from a given fit.

  Parameters:

    - minimizer – the minimizer object returned by `minimize()`.

    - sigmas – list of sigma-levels to find parameter values for.

    - prob_func – `None` or callable function to calculate the probability from the optimized chi-square. By default, `f_compare()`, the standard F-test, is used.

  Returns:

    a dictionary of parameter names, with each value containing a list of (sigma, value) pairs.

This function will adjust the value for each parameter, re-optimizing the other parameters until it finds the parameter values that increase sigma by the levels indicated.

confidence_report(conf_values)

  Convert the output of `confidence_intervals()` into a printable report.

  Parameters:

    conf_values – confidence values returned by `confidence_intervals()`.

  Returns:

    a string containing the report, which can be printed or stored.

f_test(ndata, nparams, chisquare, chisquare0, nfix=1)

  Returns the standard F-test value for the probability that one fit is better than another.

## 13.8.1. Confidence interval, Example 1

Let’s begin with a shortened version of the first example from the previous section.

    ## examples/fitting/doc_example_conf1.lar
    # create mock data
    mdat = group()
    mdat.x = linspace(-10, 10, 201)
    mdat.y = 1.0 + gaussian(mdat.x, 12.0, 1.5, 2.0) + \
             random.normal(size=len(mdat.x), scale=0.050)

    params = param_group(off = guess(0),
                         amp = guess(5, min=0),
                         cen = guess(2),
                         wid = guess(1, min=0))

    # define objective function for fit residual
    def myresid(p, data):
        return data.y - (p.off + gaussian(data.x, p.amp, p.cen, p.wid))
    #enddef

    # perform fit

    result = minimize(myresid, params, args=(mdat,))

    # print report of parameters, uncertainties
    print( fit_report(result))

    out = confidence_intervals(result, sigmas=(1, 2, 3))

    print( confidence_report(out))
    ## end of examples/fitting/doc_example_conf1.lar

The printed output from `fit_report(params)` will include this:

    [[Variables]]
       amp            =  12.129867 +/- 0.124722 (init=  5.000000)
       cen            =  1.476822 +/- 0.017266 (init=  2.000000)
       off            =  0.998814 +/- 0.007131 (init=  0.000000)
       wid            =  2.022301 +/- 0.016938 (init=  1.000000)

    [[Correlations]]     (unreported correlations are <  0.100)
       amp, off             = -0.861
       amp, wid             =  0.808
       off, wid             = -0.692
    =======================================================

while the output from the much more explicit search done in `confidence_intervals()` and reported by `confidence_report()` will be:

    # Confidence Interval Report
    # Sigmas:          -3         -2         -1          0          1          2          3
    # Percentiles:  -99.730    -95.450    -68.269      0.000     68.269     95.450     99.730
    #==========================================================================================
     amp             11.755      11.88     12.005      12.13     12.256     12.385     12.517
        -best      -0.37459   -0.24967   -0.12455          0    0.12604    0.25555    0.38759
     wid             1.9707     1.9885     2.0053     2.0223     2.0395     2.0569     2.0749
        -best     -0.051552  -0.033784  -0.016975          0   0.017156   0.034567   0.052626
     off             0.9766    0.98468     0.9912    0.99881     1.0064     1.0128     1.0209
        -best     -0.022212  -0.014138  -0.0076156          0  0.0075667   0.013987   0.022049
     cen             1.4248     1.4423     1.4596     1.4768     1.4941     1.5113     1.5289
        -best     -0.052057   -0.03452  -0.017248          0   0.017249   0.034524   0.052068

The automatic error estimates given from `minimize()` are meant to be 1-\(\sigma\) uncertainties. Comparing the two methods we find:

> Parameter | Best Value | Automatic 1-\(\sigma\) value | Explicit 1-\(\sigma\) value
> --- | --- | --- | ---
> amp | 12.1299 | +/- 0.1247 | +0.1267, -0.1246
> cen | 1.4768 | +/- 0.0173 | +0.0172, -0.0172
> off | 0.9988 | +/- 0.0071 | +0.0076, -0.0076
> cen | 2.0223 | +/- 0.0169 | +0.0172, -0.0170

which seems to justify the use of the automated method. The uncertainties found from the more thorough exploration shows symmetric uncertainties, even out to the 3-\(\sigma\) level, and of the 4 1-\(\sigma\) uncertainties, 3 are within 2%, and the worst agreement, for the smallest uncertainty is within 7%. It also shows that the scaling of uncertainties is fairly linear with \(\sigma\): the 3-\(\sigma\) values are approximately 3 times the 1-\(\sigma\) values.

## 13.8.2. Confidence interval, Example 2

Of course, there are more challenging cases than the one above. A double exponential function is one such example, so we start with a fit to mock data

    ## examples/fitting/doc_example_conf2.lar
    #  Shows usage of fitting with non-normal correlation of variables,
    #  and confidence intervals.

    random.seed(1)
    dat = group(x = linspace(0, 10, 101))
    dat.y = 3*exp(-dat.x/2) - 5*exp(-dat.x/9) + random.normal(size=len(dat.x), scale=0.05)

    fitparams = param_group(a1 = guess(3.5),
                            a2 = guess(-9.5),
                            t1 = guess(3),
                            t2 = guess(15))

    def fit_exp(pars, dat):
        model = pars.a1 * exp(-dat.x / pars.t1) + pars.a2 * exp(-dat.x / pars.t2)
        return model - dat.y
    enddef

    minout = minimize(fit_exp, fitparams, args=(dat,))
    print( fit_report(minout, min_correl=0))

    final = fit_exp(fitparams, dat) + dat.y
    newplot (dat.x, dat.y, label='data', marker='o', linewidth=0)
       plot (dat.x, final, label='fit')

    ## end of examples/fitting/doc_example_conf2.lar

The resulting statistics report with the automated uncertainties is:

    ===================== FIT RESULTS =====================
    [[Statistics]]    Fit succeeded,  method = 'leastsq'.
       Message from fit    = Fit succeeded.
       npts, nvarys, nfree = 101, 4, 97
       nfev (func calls)   = 36
       chi_square          = 0.191322
       reduced chi_square  = 0.001972

    [[Variables]]
       a1             =  2.828857 +/- 0.149776 (init=  3.500000)
       a2             = -4.819553 +/- 0.159495 (init= -9.500000)
       t1             =  1.878519 +/- 0.100212 (init=  3.000000)
       t2             =  9.270866 +/- 0.309035 (init=  15.000000)

    [[Correlations]]     (unreported correlations are <  0.100)
       a2, t2               =  0.991
       a1, a2               = -0.991
       a1, t2               = -0.988
       a2, t1               = -0.968
       t1, t2               = -0.937
       a1, t1               =  0.935
    =======================================================

You can see that the correlations between all 6 pairs of variables is above 90%. The resulting plot of the best-fit looks fairly reasonable:

Figure 13.8.2.1 Fit to double exponential function.

But if we we ask for the more thorough investigation of the confidence intervals in these parameters with:

    conf_int = confidence_intervals(minout)
    print confidence_report(conf_int)

the resulting report is:

    # Confidence Interval Report
    # Sigmas:          -3         -2         -1          0          1          2          3
    # Percentiles:  -99.730    -95.450    -68.269      0.000     68.269     95.450     99.730
    #==========================================================================================
     a1              2.4622     2.5704      2.691     2.8289     2.9936     3.1985      3.467
        -best      -0.36665   -0.25842   -0.13785          0    0.16472    0.36968    0.63812
     a2             -5.4926    -5.2111    -4.9945    -4.8196    -4.6726     -4.544     -4.429
        -best      -0.67309   -0.39152   -0.17499          0    0.14698    0.27553    0.39058
     t2              8.2604     8.6221     8.9556     9.2709     9.5761     9.8776     10.182
        -best       -1.0105   -0.64882   -0.31531          0    0.30526    0.60676    0.91132
     t1              1.6107     1.6942     1.7827     1.8785     1.9841     2.1049     2.2456
        -best      -0.26785   -0.18436   -0.09583          0    0.10553    0.22637    0.36709

Now can see more asymmetric uncertainty values, specifically that the -n-\(\sigma\) and +n-\(\sigma\) are different, and don’t seem to be linear in n. Comparing the 1-\(\sigma\) levels between the automated and explicit methods as we did above, we now have

> Parameter | Best Value | Automatic 1-\(\sigma\) value | Explicit 1-\(\sigma\) value
> --- | --- | --- | ---
> a1 | 2.8289 | +/- 0.1498 | +0.1647, -0.1379
> a2 | -4.8196 | +/- 0.1595 | +0.1470, -0.1750
> t1 | 9.2709 | +/- 0.1002 | +0.1055, -0.0958
> t2 | 1.8785 | +/- 0.3090 | +0.3053, -0.3153

In fairness, the automated values don’t look too bad, given that they cannot reflect asymmetric uncertainties. But, like the reported correlations, the full report above hints at a less than ideal case.

## 13.8.3. Chi-square Maps

We can further explore the correlation between pairs of variables by making and visualizing a map of the chi-square (\(\chi^2\)) statistic. This can be helpful to determine if the automatically calculated uncertainties and correlation are reasonable, and to look for pathological cases. The `chi2_map()` function will calculate a map of \(\chi^2\) for a pair of variable parameters by brute force.

chi2_map(minimizer, xname, yname, nx=11, ny=11, sigmas=5, xrange=None, yrange=None)

  Return a chi-square map for two parameters in a fit

  Parameters:

    - minimizer – the minimizer object returned by `minimize()`.

    - xname – name of variable for x-axis

    - yname – name of variable for y-axis

    - nx – number of steps in x [11]

    - ny – number of steps in y [11]

    - sigmas – extent of x, y values to calculate, in \(\sigma\)

    - xrange – explicit x range of calculations [x.best +/- `sigmas` * x.stderr]

    - yrange – explicit y range of calculations [y.best +/- `sigmas` * y.stderr]

  Returns:

    a tuple of (x_vals, y_vals, chi2_map)

  The `xrange` and `yrange` arguments can be used to fully dictate the x and y values to use. By default, the x and y values are automatically determined from `nx`, `ny`, and `sigmas`, with the `sigmas` argument sets how far from the best value to extend the ranges.

As an example usage, we return to the first example of the “well-behaved” fit, and run `chi2_map()` on a pair of variables with low correlation and a pair with high correlation:

    x1, y1, chi2_ampcen = chi2_map(mout, 'amp', 'cen', nx=21, ny=21)
    x2, y2, chi2_ampwid = chi2_map(mout, 'amp', 'wid', nx=21, ny=21)
    contour(chi2_ampcen, x=x1, y=y1, title='Correlation(amp, cen)')
    contour(chi2_ampwid, x=x2, y=y2, win=2, title='Correlation(amp, wid)')

with the resulting Chi-square maps looking like this:

The circular map for the uncorrelated parameters amp and cen and the elliptical map for the highly correlated parameters amp and wid are exactly as would be expected, and what the automated estimate of uncertainties and correlations assumes.

But now, if we turn to the more pathological case of the double exponential, we calculate the chi-square maps as:

    x1, y1, c1 = chi2_map(minout, 'a1', 'a2', nx=25, ny=25)
    x2, y2, c2 = chi2_map(minout, 'a1', 't1', nx=25, ny=25)
    contour(c1, x=x1, y=y1,        title='Correlation(a1, a2)')
    contour(c2, x=x2, y=y2, win=2, title='Correlation(a1, t1)')

with the resulting contour plots:

Here, the values of chi-square quickly grow very large away from the ideal fit. More importantly, the maps are not remotely elliptical.

Finally, it should be emphasized that while all the tests in this section are informative, they are also fairly slow, re-running the fits many times. It is probably safe to rely on the automatic calculations of uncertainties and correlations, and use these methods on occasions of extremely high correlation, or when nearing a final analysis.