Feature extraction

Feature extraction is the process of transforming raw data into a representation that is more suitable for a particular machine learning task. In computer vision, this means converting pixel arrays into descriptors — edges, textures, shapes, color histograms — that capture the information relevant to the task while discarding the irrelevant. The history of computer vision is, in large part, a history of feature extraction: first manual, then learned, then so deeply learned that the features themselves became uninterpretable.

Before deep learning, feature extraction was the central intellectual labor of the field. Researchers designed feature detectors by hand: Sobel edge detectors, HOG descriptors, SIFT keypoints, Haar wavelets. Each detector encoded a specific assumption about what makes images distinctive. The Sobel operator assumes that edges are important. HOG assumes that local gradient orientations capture shape. SIFT assumes that scale-invariant keypoints are robust to viewpoint changes. These assumptions were not arbitrary. They were distillations of perceptual psychology and geometric optics. But they were also limitations: a system that used only edge detectors could not learn to recognize texture, and a system that used only color histograms could not learn to recognize shape.

Deep learning changed this by making feature extraction itself learnable. A convolutional neural network discovers its own features from raw pixels, learning edge detectors in early layers, textures in middle layers, and object parts in deep layers. The network is not given features; it extracts them from the data. This is the source of deep learning's power and its opacity. The features that the network learns are optimal for the task but not necessarily interpretable by humans. A feature that the network uses to distinguish cats from dogs may be a combination of color, texture, and shape that has no name in human visual vocabulary.

The question that feature extraction raises is whether the distinction between 'hand-crafted' and 'learned' features is as sharp as it appears. Hand-crafted features are also learned — they are learned by the researcher who designed them, based on experience and intuition. The difference is that hand-crafted features are learned from a small dataset (the researcher's own visual experience) and then frozen into the algorithm. Learned features are extracted from a large dataset and updated continuously during training. The shift from hand-crafted to learned features is not a shift from human judgment to machine autonomy. It is a shift from implicit, unexamined judgment to explicit, optimizable judgment.