Thanks to you who have responded.
I am still on a pen quest.

I am looking for museums and historic sites that sell souvenir pens - the
pens that when tilted have a boat or object that slides back and forth.  If
you have seen these pens somewhere - please tell me where!  If you have them
in your museum gift shop - please let me know!

I have found pens from the museum list so far at:
Fort Steele Heritage Museum
Seattle Art Museum
Cliff House in San Francisco
Parthenon in Nashville
Weisman Art Museum at the University of Minnesota
among other sites
What a great variety! Thank you!