I would recommend checking _Museum News_ , produced by AAM.
There are certainly articles, if not entire issues, devoted to museum
publications.  AAM also does an annual award with categories for
different kinds of exhibit publications.  Talking to recent winners could be
helpful.  Finally, every year there is an museum publishing conference.
In fact, I just threw out the mailing on it.  I'm not sure who the sponsor is,
but someone else on Museum-l might know.

Riva Feshbach
Curator
DePaul University Art Gallery