Get yourself a Marine Corps "Wooly Pully" sweater. It'll be warmer then any bulky down vest or several sweatshirts, but only as thick as a normal sweater. But just don't hike with it on.
http://www.marineshop.net/browse.cfm/wooly-pully-sweater-%282-colors%29/4,7425.html
http://www.marineshop.net/browse.cfm/wooly-pully-sweater-%282-colors%29/4,7425.html