There are several theories, all supported by some pretty sophisticated mathematics, that can explain the generation of coat patterns. As far as I know, the one that is generally the most accepted is the one developed by Murray (1989), which is based on the principle of Turing bifurcation and the positional informational theory by developmental biologist Lewis Wolpert (Wolpert, 1981).
Without going into the mathematics, I’ll summarise this theory. The first step is the spreading of a morphogen – afaik, this chemical hasn’t been characterised yet – over several patches of the embryo. These patches are where colour pattern will arise. When the melanocyte precursors develop and start wandering into the outer layers of the embryo, they can detect this morphogen and its concentration. In response to it, they either produce melanin (or whatever pigment is used by the organism) or they wander into a different area.
In other words, the melanocytes’ position is determined by the presence of this morphogen. Of course, I simplified the matter quite a lot here, since going into details would require whipping out LaTEX and going through tons of equations. Boring shit.
Murray’s theory is referred to as a prepattern theory, where a genetically-determined pattern is produced: the morphogen’s position is allocated genetically (according to Wolpert’s positional theory), and Turing bifurcation is involved in the sensing of the morphogen concentration by the melanocyte precursors.
There are several assumptions here, and as I said, there are competing theories. But this one is quite successful because it explains a lot, for example why the smallest mammals and largest mammals have no coat colours. If you solve the maths behind the theory, you’ll find that the Turing instability cannot be satisfied in small mammals (rat-sized) – not enough space for diffusion, hence the melanocytes can’t sense the different concentrations of the morphogen. On the other hand, in large mammals (we’re talking elephants here), the diffusion is so great that too many patterns are produced on top of each other and you have to zoom in to actually tell them apart – from a macroscopic perspective, they just blend together.
Note that this only applies to mammal coats. Patterns on shells or butterfly wings are an entirely different matter. Butterfly wings deserve a post of their own, since they’re part of some quite exciting and somewhat groundbreaking research these days.
Murray JD. 1989. Mathematical Biology.
Wolpert L. 1981. Positional information and pattern formation. Philosophical Transactions of the Royal Society of London B 295, 441-450.