Learning as an Automaton

Sequential training feeds one labelled example, updates, then repeats, which is exactly the shape of a deterministic automaton reading a stream. Casting it that way makes one identification unavoidable and one gap glaring.

Model a training run as an automaton . The alphabet is , a single labelled example. The state space is , the parameter space of the model. The transition is the update rule , which reads a datum and moves to a new parameter, and is the initialisation. A whole training set is then a word over , and training is running the automaton to the end of that word.

Three spaces that turn out to be one

The reading forces an identification across three vocabularies. The parameter space of a function family, the state space of the automaton, and the configuration space of a dynamical system are the same set wearing three hats. A model’s weights are where the machine currently is, and learning is a trajectory through that space driven by the data word.

No final state, so no clean composition

A recogniser has accepting states, whereas this automaton has none. Without a final state there is nothing to compose against in the time direction, since you cannot glue the end of one training run to the start of another as a typed composite when “done” is not a state the machine can be in. Composition along time is therefore not free here. Some extra device is needed to mark and carry termination, and supplying it is the open design point.