You really can't beat backpacking guide-wear in my experience for UE. It's light, tear resistant, looks reasonably dressy, and is designed to wick moisture and keep you cool.
Every time I've ever successfully talked my way onto a site I've dressed this way. In combination with my large brown expedition camera bag (which holds two to three cameras, a gas mask, first aid kit, snickers bar, and gloves; I look like a construction foreman that just came from a meeting.
I've had people approach me while I was scouting a site and ask what we were planning to do with the building.