Dimensionality reduction

You don't need to be Editor-In-Chief to add or edit content to WikiDoc. You can begin to add to or edit text on this WikiDoc page by clicking on the edit button at the top of this page. Next enter or edit the information that you would like to appear here. Once you are done editing, scroll down and click the Save page button at the bottom of the page.

Please Take Over This Page and Apply to be Editor-In-Chief for this topic: There can be one or more than one Editor-In-Chief. You may also apply to be an Associate Editor-In-Chief of one of the subtopics below. Please mail us [1] to indicate your interest in serving either as an Editor-In-Chief of the entire topic or as an Associate Editor-In-Chief for a subtopic. Please be sure to attach your CV and or biographical sketch.

In statistics, dimension reduction is the process of reducing the number of random variables under consideration, and can be divided into feature selection and feature extraction. In physics, dimension reduction is a widely discussed phenomenon, whereby a physical system exists in three dimensions, but its properties behave like those of a lower-dimensional system. It has been experimentally realised at the quantum critical point in an insulating magnet made of the pigment Han Purple^[1][1].

Feature selection

Main article: Feature selection

Feature selection approaches try to find a subset of the original variables (also called features or attributes). Two strategies are filter (e.g. information gain) and wrapper (e.g. genetic algorithm) approaches. See also combinatorial optimization problems.

It is sometimes the case that data analysis such as regression or classification can be done in the reduced space more accurately than in the original space.

Feature extraction

Main article: Feature extraction

Feature extraction is applying a mapping of the multidimensional space into a space of fewer dimensions. This means that the original feature space is transformed by applying e.g. a linear transformation via a principal components analysis.

Consider a string of beads, first 100 black and then 100 white. If the string is wadded up, a classification boundary between black and white beads will be very complicated in three dimensions. However, there is a mapping from three dimensions to one dimension, namely distance along the string, which makes the classification trivial. Unfortunately, a simplification as dramatic as that is rarely possible in practice.

The main linear technique for dimensionality reduction, principal components analysis (PCA), performs a linear mapping of the data to a lower dimensional space in such a way, that the variance of the data in the low-dimensional representation is maximized. In practice, the correlation matrix of the data is constructed and the eigenvectors on this matrix are computed. The eigenvectors that correspond to the largest eigenvalues (the principal components) can now be used to reconstruct a large fraction of the variance of the original data. Moreover, the first few eigenvectors can often be interpreted in terms of the large-scale physical behaviour of the system. The original space (with dimension of the number of points) has been reduced (with data loss, but hopefully retaining the most important variance) to the space spanned by a few eigenvectors.

Principal component analysis can be employed in a nonlinear way by means of the kernel trick. The resulting techniques is capable of constructing nonlinear mappings that maximize the variance in the data. The resulting technique is entitled Kernel PCA. Other nonlinear techniques include techniques for locally linear embedding (such as locally linear embedding (LLE), Hessian LLE, Laplacian eigenmaps, and LTSA). These techniques construct a low-dimensional data representation using a cost function that retains local properties of the data (actually these techniques can be viewed upon as defining a graph-based kernel for Kernel PCA). In this way, the techniques are capable of unfolding datasets such as the Swiss roll. Techniques that employ neighborhood graphs in order to retain global properties of the data include Isomap and maximum variance unfolding.The neighbourhood preservation can also be achieved through the minimisation of the weighted difference between distances in the input and output spaces (i.e. curvilinear component analysis (CCA) and data-driven high-dimensional scaling (DD-HDS)).

Principal curves and manifolds^[1] give another framework for PCA generalization and extend the geometric interpretation of PCA by explicitly constructing a low-dimensional embedded manifold for data approximation.

A completely different approach to nonlinear dimensionality reduction is through the use of autoencoders, a special kind of feed-forward neural networks. Although the idea of autoencoders is quite old, training of the encoders has only recently become possible through the use of Restricted Boltzmann machines.

References

See also

• Feature space
• Data mining
• Feature selection
  • Information gain
  • Chi-square distribution
  • Recursive feature elimination (SVM based)

• Feature extraction
  • Principal components analysis
  • Semidefinite embedding
  • Multifactor dimensionality reduction
  • Nonlinear dimensionality reduction
  • Isomap
  • Kernel PCA
  • Latent semantic analysis
  • Wavelet compression

• Topological data analysis

• Physics
  • magnetism

External links

• A survey of dimension reduction techniques (US DOE Office of Scientific and Technical Information, 2002)
• JMLR Special Issue on Variable and Feature Selection
• Unsupervised Learning of Image Manifolds by Semidefinite Programming
• ELastic MAPs
• Locally Linear Embedding
• Relational Perspective Map
• A Global Geometric Framework for Nonlinear Dimensionality Reduction
• Dimensional reduction at a quantum critical point (realisation of dimensional reduction in a magnet)
• Matlab Toolbox for Dimensionality Reduction
• kernlab - R package for kernel-based machine learning (includes kernel PCA, and SVM)
• DD-HDS homepageaf:Dimensionaliteitvermindering

fr:Réduction dimensionnelle

WikiDoc Help Menu Quick Start.. Get Started Here! How to Login How to Search Pages How to Start a New Page How to Edit a Page Cheat Sheet Editing basics Learning Basic Formatting How to make Links in WikiDoc & to External Sites How to Make Lists How to Ignoring Formatting Adding References or Footnotes to Articles Tracking Changes You Have Made How to Put an Article Into a Category When Multiple Topics Should Go To The Same Page (MI, Acute MI, AMI) Using Redirection Advanced editing How to Make Tables How to Insert Images and Other Media How to Add Video Help:How to Make Columns Web Colors How To Make Templates How to Move Pages Inserting Dynamic Dates and Times Inserting WikiDoc Statistics Editing the Table of Contents Communicating your edits Let others know what you changed: The importance of Edit Summaries Minor Edits Edit Conflicts Help Videos You Can Watch Getting Started Video