Sigmoid function



A sigmoid function is a mathematical function that produces a sigmoid curve &mdash; a curve having an "S" shape. Often, sigmoid function refers to the special case of the logistic function shown at right and defined by the formula
 * $$P(t) = \frac{1}{1 + e^{-t}}.$$

Derivative of the sigmoid function
The derivative of the sigmoid function can be written

$$\frac{dP}{dt}=P(1-P).$$

Members of the sigmoid family
In general, a sigmoid function is real-valued and differentiable, having either a non-negative or non-positive first derivative and exactly one inflection point.

Besides the logistic function, sigmoid functions include the ordinary arc-tangent, the hyperbolic tangent, and the error function, but also algebraic functions like $$f(x)=\tfrac x\sqrt{1+x^2}$$. The integral of any smooth, positive, "bump-shaped" function will be sigmoidal, thus the cumulative distribution functions for many common probability distributions are sigmoidal.

The logistic sigmoid function is related to the hyperbolic tangent, e.g. by
 * $$2 \, P(t) = 1 + \tanh \left( \frac{t}{2} \right).$$

Sigmoid functions in neural networks
Sigmoid functions are often used in neural networks to introduce nonlinearity in the model and/or to clamp signals to within a specified range. A popular neural net element computes a linear combination of its input signals, and applies a bounded sigmoid function to the result; this model can be seen as a "smoothed" variant of the classical threshold neuron.

A reason for its popularity in neural networks is because the sigmoid function satisfies the differential equation
 * $$y' = y(1-y).$$

The right hand side is a low order polynomial. Furthermore, the polynomial has factors $$y$$ and $$1 - y$$, both of which are simple to compute. Given $$y = {\rm sig}(t)$$ at a particular t, the derivative of the sigmoid function at that t can be obtained by multiplying the two factors together. These relationships result in simplified implementations of artificial neural networks with artificial neurons.

Double sigmoid function
The double sigmoid is a function similar to the sigmoid function with numerous applications. Its general formula is:
 * $$ y = \mbox{sign}(x-d) \, \Bigg(1-\exp\bigg(-\bigg(\frac{x-d}{s}\bigg)^2\bigg)\Bigg), $$

where d is its centre and s is the steepness factor.

It is based on the Gaussian curve and graphically it is similar to two identical sigmoids bonded together at the point x = d.

One of its applications is non-linear normalization of a sample, as it has the property of eliminating outliers.