NN Blocks as an Operad

A neural network is usually pictured as a stack, but the thing actually being composed is a wiring of blocks, outputs plugged into inputs, in parallel and in series. That composition law is not a category’s but an operad’s, with many inputs feeding one output, glued like discs inside a disc.

Each block is an operation with a list of typed input ports and one output port. Composing blocks is substitution, feeding the outputs of several blocks into the input slots of another, and this is exactly the multi-input, single-output composition of an operad. Because ports carry types such as tensor shapes and modalities, the right object is a coloured operad, equivalently a multicategory. The colours are the port types, and an operation is a typed block. Pre- and post-composition of blocks, which append a layer before or after, are the special unary substitutions. The general nested substitution is the operadic analogue of little-disc composition, where each argument block occupies a slot the way a small disc sits inside the big one, associatively and equivariantly under reordering of the slots.

Most blocks break the single-output restriction, since attention, residual splits, and multi-head fan-out all have several outputs. A block with inputs and outputs is a poly-map , and these compose along shared wires as a properad, a polycategory with an interchange law, where composition glues one output of one block to one input of another. Dropping to recovers a plain category; keeping single output recovers the operad; the full properad is what a general dataflow diagram lives in.

Remark on wiring versus stacking

Sequential depth is only the unary fragment of this structure. What the operad/properad adds is the bookkeeping of parallel composition and of typed ports, so that a wiring diagram is a well-formed term rather than a picture. This places block composition beside string diagrams. The diagram is the syntax, the (pr)operad is the algebra it denotes, and a trained network is one algebra of that theory.

Remark on equivariance and sharing, deferred

A bare operad has no way to duplicate or discard a wire, yet weight-tying and skip connections need that. Supplying each colour with a comonoid (copy/delete) upgrades the properad toward a hypergraph-category discipline, a move worth flagging but not made here.