Conditional random field

A conditional random field (CRF) is a type of discriminative probabilistic model most often used for the labeling or parsing of sequential data, such as natural language text or biological sequences.

Description
Much like a Markov random field, a CRF is an undirected graphical model in which each vertex represents a random variable whose distribution is to be inferred, and each edge represents a dependency between two random variables. In a CRF, the distribution of each discrete random variable $$Y$$ in the graph is conditioned on an input sequence $$X$$.

In principle, the layout of the graph of random variables $$Y$$ can be arbitrary; most often, however, the $$Y_{i}$$ are structured to form a chain, with an edge between each $$Y_{i-1}$$ and $$Y_{i}$$. As well as having a simple interpretation of the $$Y_{i}$$ as "labels" for each element in the input sequence, this layout admits efficient algorithms for model training, learning the conditional distributions between the $$Y_{i}$$ and feature functions from some corpus of training data, inference, determining the probability of a given label sequence $$Y$$ given $$X$$, and decoding, determining the most likely label sequence $$Y$$ given $$X$$.

The conditional dependency of each $$Y_{i}$$ on $$X$$ is defined through a fixed set of feature functions of the form $$f(i, Y_{i-1}, Y_{i}, X)$$, which can informally be thought of as measurements on the input sequence that partially determine the likelihood of each possible value for $$Y_{i}$$. The model assigns each feature a numerical weight and combines them to determine the probability of a certain value for $$Y_{i}$$.

Relationship to hidden Markov models
CRFs have many of the same applications as conceptually simpler hidden Markov models (HMMs), but relax certain assumptions about the input and output sequence distributions. An HMM can loosely be understood as a CRF with very specific feature functions that use constant probabilities to model state transitions and emissions. Conversely, a CRF can loosely be understood as a generalization of an HMM that makes the constant transition probabilities into arbitrary functions that vary across the positions in the sequence of hidden states, depending on the input sequence.

Notably in contrast to HMMs, CRFs can contain any number of feature functions, the feature functions can inspect the entire input sequence $$X$$ at any point during inference, and the range of the feature functions need not have a probabilistic interpretation.

The well-known forward-backward and Viterbi algorithms for HMMs have direct analogues for CRFs, with the same asymptotic running times. The training step, which determines a weight for each feature function, is somewhat more complex; generally, there is no closed-form solution for the optimal assignment of weights, so it must be found using numerical optimization techniques. Common techniques for this include gradient descent algorithms and quasi-Newton methods, such as the L-BFGS algorithm.

Higher-order CRFs and semi-Markov CRFs
CRFs can be extended into higher order models by making each $$Y_{i}$$ dependent on a fixed number $$o$$ of previous variables $$Y_{i-(o-1)}, ..., Y_{i}$$. Training and inference are only practical for small values of $$o$$ (such as $$o \leq 5$$), since their computational cost increases exponentially with $$o$$.

There exists another generalization of CRFs, the semi-Markov conditional random field (semi-CRF), which model variable-length segmentations of the label sequence $$Y$$. This provides much of the power of higher-order CRFs to model long-range dependencies of the $$Y_{i}$$, at a reasonable computational cost.

Software
This is partial list of software that implement CRF related tools.


 * MALLET ( Java)
 * ABNER ( Java)
 * MinorThird ( Java)
 * Kevin Murphy's MATLAB CRF code ( Matlab)
 * Sunita Sarawagi's CRF package ( Java)
 * HCRF library (including CRF and LDCRF) ( C++, Matlab)
 * Xcrf for Xml data ( Java)
 * CRF++ ( C++)