I've been doing a bit of research into formal models for AGI, searching for fertile ground for developing new ideas. One area that didn't seem too thoroughly explored was in designing agents that “reason scientifically”, by which I mean agents that extract underlying representations of their environments from brute percept sequences that enhance their predictive and causal powers in the world. Most formal AGI models that I'm familiar with use RL paradigms that don't build explicit representations analogous to natural laws or else don't establish a tight enough connection between those representations and their causal abilities. The closest current candidate seems to be causal representation learning (which seems promising), but based on my cursory investigation it seems like it might be a bit too rigid and less adaptable to novel data.
My own proposal here is inspired by compression-theoretical models of information and detection of patterns over string streams. A lot of work has already been done in problem domains like binary stream prediction models, which extract patterns or regularities from an input stream that enable prediction of subsequent values. My idea would be to apply a similar approach to data streams that include both agent percept and action states, where such a sequence of states represents a cybernetic feedback loop between the agent and its environment. Such an agent while in “exploration mode” would choose essentially random actions, or else in a manner so as to maximize exploration of the search space. As this is done they would track the action-percept stream for patterns or regularities across subsegments. When a pattern or compression of a particular segment is discovered, that segment’s entropy decreases (a consequence of the formal equivocation of entropy-theoretic and compression-theoretic definitions of information), indicating that the agent’s uncertainty in outcomes over the sequence of the action-perception states decreases. Conceptually, this elegantly unifies representation, information, prediction, and causality. A particular such pattern could be termed a “mechanism”, and detected mechanisms would be stored by the agent in some representation format. When the agent model is augmented with a utility function that prioritizes the attainment of certain agent states, the agent could make use of its stored mechanisms to more efficiently attain that goal, since each of these represents a tool or augmented action over its base actions. A comprehensive agent would also be capable of some degree of meta-learning, wherein it synthesizes known mechanisms or expands them into something more general when presented with additional data.
As a simple illustrating case, consider an agent operating in an environment that consists of a grid of two-dimensional cells. The agent’s perceptual faculties allow it to detect the status of cells within some radius, as well as the brightness and hue of the ambient light in the environment. This environment is bounded by impassable walls and has one cell which contains a distinctive pressure plate which, when depressed by the agent, causes the ambient light to become a bright red for 10 time steps. Every time the agent enters this cell, the current duration is extended by 10, so the agent could add time by moving off the switch and back on it repeatedly. If we were to unroll the sequence of action-percept states that results from the agent exploring this environment randomly over an arbitrary time period, we would find that the verbatim representation can be compressed through an understanding of the mechanism described above. In plain language, one could say that for every depression of the pressure plate there will follow a 10 unit segment of red light, otherwise the light will be white. This would clearly be a more efficient representation than storing the light percept at every time step, and at the same time some knowledge seems to have been gained about how the agent’s actions, in this case depressing the plate, affect their environment. An agent may also combine known mechanisms into more complex plans; for example, one might imagine that nourishing spheres randomly spawn in unoccupied tiles, and that the utility the agent gains from consuming the sphere is increased when they are consumed under a red light. Thus a maximally informed agent will know to trigger the plate before consuming available spheres.
I think that is a fairly well-motivated base for this kind of model, but it is still quite conceptual; actually nailing down a concrete model would be more difficult. I would have to do some more analysis myself to see how efficiently one could track an incoming stream for patterns across arbitrarily-sized sub-segments while simultaneously scoring existing patterns. The exact mechanics of incorporating goal-setting would have to be ironed out as well, as well as the precise format of the stream-predictor sub-models themselves. Some obvious related ideas include Solomonoff Induction, Algorithmic Compression Theory, and the AIXI framework, though to my knowledge nothing exactly like what I’ve proposed here exists.
Please comment and direct me towards any similar research - thanks.
