exponential family

A probability (densityPlanetmathPlanetmath) functionMathworldPlanetmath fX(xθ) given a parameter θ is said to belong to the (one parameter) exponential family of distributionsDlmfPlanetmathPlanetmath if it can be written in one of the following two equivalentMathworldPlanetmathPlanetmathPlanetmathPlanetmathPlanetmath forms:

  1. 1.


  2. 2.


where a,b,c,d are known functions. If c(x)=x, then the distribution is said to be in canonical form. When the distribution is in canonical form, the function d(θ) is called a natural parameter. Other parameters present in the distribution that are not of any interest, or that are already calculated in advance, are called nuisance parameters.


  • The normal distributionMathworldPlanetmath, N(μ,σ2), treating σ2 as a nuisance parameter, belongs to the exponential family. To see this, take the natural logarithmMathworldPlanetmath of N(μ,σ2) to get


    Rearrange the above expression and we have


    Set c(x)=x, d(μ)=μ/σ2, b(μ)=-μ2/(2σ2), and a(x)=-1/2[x2/σ2+ln(2πσ2)]. Then we see that N(μ,σ2) does indeed belong to the exponential family. Furthermore, it is in canonical form. The natural parameter is d(μ)=μ/σ2.

  • Similarly, the Poisson, binomial, Gamma, and inverse Gaussian distributions all belong to the exponential family and they are all in canonical form.

  • Lognormal and Weibull distributionsMathworldPlanetmath also belong to the exponential family but they are not in canonical form.


  • If the p.d.f of a random variableMathworldPlanetmath X belongs to an exponential family, and it is expressed in the second of the two above forms, then

    E[c(X)]=-b(θ)d(θ), (1)


    Var[c(X)]=d′′(θ)b(θ)-d(θ)b′′(θ)d(θ)3, (2)

    provided that functions b and d are appropriately conditioned.

  • Given a member from the exponential family of distributions, we have E[U]=0 and I=-E[U], where U is the score functionMathworldPlanetmath and I the Fisher informationMathworldPlanetmath. To see this, first observe that the log-likelihood functionMathworldPlanetmath from a member of the exponential family of distributions is given by


    and hence the score function is


    From (1), E[U]=0. Next, we obtain the Fisher information I. By definition, we have

    I = E[U2]-E[U]2
    = E[U2]
    = d(θ)2Var[c(X)]
    = d′′(θ)b(θ)-d(θ)b′′(θ)d(θ)

    On the other hand,



    E[Uθ] = b′′(θ)+E[c(X)]d′′(θ)
    = b′′(θ)-b(θ)d(θ)d′′(θ)
    = b′′(θ)d(θ)-b(θ)d′′(θ)d(θ)
    = -I
  • For example, for a Poisson distributionMathworldPlanetmath


    the natural parameter d(θ) is lnθ and b(θ)=-θ. c(x)=x since Poisson is in canonical form. Then

    U(θ)=-1+Xθ and I=-E[-Xθ2]=1θ

    as expected.

Title exponential family
Canonical name ExponentialFamily
Date of creation 2013-03-22 14:30:08
Last modified on 2013-03-22 14:30:08
Owner CWoo (3771)
Last modified by CWoo (3771)
Numerical id 7
Author CWoo (3771)
Entry type Definition
Classification msc 62J12
Defines canonical exponential family
Defines nuisance parameter
Defines natural parameter