ArticlePDF Available

Semantic memory as a computationally free side-effect of sparse distributed generative episodic memory (GEM 2023 abstract)

Authors:
  • Neurithmic Systems

Abstract

Semantic memory as a computationally free side-effect of sparse distributed generative episodic memory By generative model, we mean a model with sufficient parameters to represent the deep statistical structure (not just pairwise, but ideally, statistics of all orders present) of an input domain (in contrast to a discriminative model whose goal is to learn only enough information to classify inputs). These higher-order statistics include not just class information, but more generally, the full similarity structure, over the inputs, and constitute the basis for what we call semantic memory (SM). A generative model can be run "in reverse" to produce (in general, novel) plausible (likely) exemplars. By episodic memory (EM), we mean (typically, rich detailed) memories of specific experiences, which, by definition, are formed with single trials in the flow of the experience, apparently with no concurrent goal of learning the class of the experience, or even its similarity relations with other experiences. In a classical storage model of EM, where all inputs (experiences) are stored in full detail, all statistics of the input set are (at least implicitly) retained. This allows retrieval of the precise inputs, but also computations over the stored EM traces, in principle, producing any higher-order statistic of the input set, i.e., any output viewable as the operation of SM. A key question is: how are the EM traces stored? If they are stored in localist fashion, i.e., wherein the traces of the individual inputs are disjoint, then any higher-order statistic must be computed either at retrieval time or sometime after storage and before retrieval. This "pre-computational" view is essentially consistent with the still-preponderant batch learning paradigm of machine learning. In either case, explicit computational work must be done to produce SM from EM, i.e., additional work beyond the work of storing the EM traces themselves. However, suppose instead that EM traces are stored in distributed fashion, and more specifically, as sparse distributed representations (SDRs), i.e., each individual input is stored as a relatively small subset of coactive neurons [a kind of cell assembly (CA)], chosen from a much larger field of such. And suppose further that there exists an on-line, single-trial learning mechanism (algorithm) able to cause more similar inputs to be assigned to more highly intersecting SDRs [my prior work (1996, 2010, 2014) describes one, which is moreover not optimization based]. In this case, the (in principle, full) intersection structure over all the inputs is embedded in the intersection structure of the CAs in the act of storing each EM trace. In other words, no additional work beyond the work of simply storing the EM traces themselves is needed in order to produce the physical representations of the statistics that constitute SM. Thus, SM is physically superposed with EM and, crucially vis-à-vis explaining the efficiency of biological learning and cognition, SM is produced as a computationally free side-effect of the operation of EM. Depending on the specificity of subsequent cues, such a model can output verbatim memories (i.e., episodic recall) but allows for the type of semantic (similarity-based) substitutions (e.g., confabulations) that GEM wants to explain.
Semantic memory as a computationally free side-effect
of sparse distributed generative episodic memory
By generative model, we mean a model with sufficient parameters to represent the deep
statistical structure (not just pairwise, but ideally, statistics of all orders present) of an
input domain (in contrast to a discriminative model whose goal is to learn only enough
information to classify inputs). These higher-order statistics include not just class
information, but more generally, the full similarity structure, over the inputs, and
constitute the basis for what we call semantic memory (SM). A generative model can be
run in reverse to produce (in general, novel) plausible (likely) exemplars. By episodic
memory (EM), we mean (typically, rich detailed) memories of specific experiences,
which, by definition, are formed with single trials in the flow of the experience,
apparently with no concurrent goal of learning the class of the experience, or even its
similarity relations with other experiences. In a classical storage model of EM, where all
inputs (experiences) are stored in full detail, all statistics of the input set are (at least
implicitly) retained. This allows retrieval of the precise inputs, but also computations
over the stored EM traces, in principle, producing any higher-order statistic of the input
set, i.e., any output viewable as the operation of SM.
A key question is: how are the EM traces stored? If they are stored in localist fashion,
i.e., wherein the traces of the individual inputs are disjoint, then any higher-order statistic
must be computed either at retrieval time or sometime after storage and before retrieval.
This pre-computational view is essentially consistent with the still-preponderant batch
learning paradigm of machine learning. In either case, explicit computational work must
be done to produce SM from EM, i.e., additional work beyond the work of storing the
EM traces themselves. However, suppose instead that EM traces are stored in distributed
fashion, and more specifically, as sparse distributed representations (SDRs), i.e., each
individual input is stored as a relatively small subset of coactive neurons [a kind of cell
assembly (CA)], chosen from a much larger field of such. And suppose further that there
exists an on-line, single-trial learning mechanism (algorithm) able to cause more similar
inputs to be assigned to more highly intersecting SDRs [my prior work (1996, 2010,
2014) describes one, which is moreover not optimization based]. In this case, the (in
principle, full) intersection structure over all the inputs is embedded in the intersection
structure of the CAs in the act of storing each EM trace. In other words, no additional
work beyond the work of simply storing the EM traces themselves is needed in order to
produce the physical representations of the statistics that constitute SM. Thus, SM is
physically superposed with EM and, crucially vis-à-vis explaining the efficiency of
biological learning and cognition, SM is produced as a computationally free side-effect of
the operation of EM. Depending on the specificity of subsequent cues, such a model can
output verbatim memories (i.e., episodic recall) but allows for the type of semantic
(similarity-based) substitutions (e.g., confabulations) that GEM wants to explain.
ResearchGate has not been able to resolve any citations for this publication.
ResearchGate has not been able to resolve any references for this publication.