Customer Self-Service Portal

Alerts on build servers that start up and shut down on demand

I have Cloud-hosted build servers that I don't want to be running all the time (to save costs).  So, the instances are started when a build has been triggered, and then 10 minutes after the processing completes, the instance is shut down (but not terminated).  I don't want alerts going out when the host is "down".  How can I do that?  I can't find a way to "not care" when a host is "down".  That, or what is the intended way to monitor this kind of host?