Tensor Product

Tensor-product representations are a technique for encoding symbolic structures — trees, lists, predicates — as high-dimensional vectors by combining vector embeddings through tensor (outer) products and their inverses. Developed in cognitive science by Paul Smolensky and others, the approach allows distributed neural networks to represent compositional structure without abandoning their continuous, gradient-friendly substrate. A predicate-argument structure like "ABOVE(cup, table)" can be encoded as the tensor product of role vectors and filler vectors, then compressed into a single vector that retains recoverable constituent structure. Tensor-product methods are one of the foundational techniques in the neural-symbolic integration toolkit.

The practical obstacle is dimensionality: tensor products of high-dimensional vectors live in spaces that grow exponentially, demanding compression techniques that trade off representational fidelity for tractability.