Mixture of Domain Experts

The seed idea of the whole programme is that a dataset is not one population but a mixture of domains, and that learning should model the domains explicitly rather than average over them. What counts as a domain, how to discover them, and how to fold that structure back into the loss is the question this note sets up.

What a domain is

Data lives over , and one can carve it along either axis. A Y-domain groups by label, an X-domain groups by input, an XY-domain by both. The essence is the x-domain, defined by a preimage. Fix a map and a value, and the domain is , the inputs that a chosen feature sends to the same place. Domains are then preimages, not arbitrary partitions, which is what makes them composable and inspectable.

MAML as removing rows

Read meta-learning in this frame. MAML’s inner loop, adapting to a task, acts like removing rows from the design matrix, restricting to one domain’s slice before fitting. Seen so, “learn to adapt fast” becomes “learn a representation in which each domain’s preimage is an easy sub-problem”.

Building domains, and using them

Domains need not be given. Biclustering, which clusters rows and columns of the table jointly, proposes candidate domains directly from data. The design move is then to inject a heterogeneous domain into a homogeneous-mixture loss. A mixture model assumes exchangeable, homogeneous components, so the point is to let one component be deliberately different, making the loss see the heterogeneity instead of smoothing it away.

Domain selection itself is reflected in the loss. Which domain a point belongs to becomes a latent variable, not a preprocessing choice. This is naturally a graphical-model Bayesian picture: latent domain assignments, per-domain experts, and a prior over the partition, with the whole thing fit by MCMC rather than a point estimate. The exponential-family gives the tractable component likelihoods that keep such a sampler workable.

Modelling domain selection inside the loss enlarges the latent space, and MCMC over partitions scales poorly. Whether the domain structure pays for its inference cost is exactly what the programme has to demonstrate.

References

• C. Finn, P. Abbeel, S. Levine, Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (ICML 2017, arXiv:1703.03400)