The colonists in this game do a LOT of walking. Allowing them to seek warmth in ANY house seems like a good idea. 
Also Taverns should  be a source of food and warmth. Course that means the tavern keeper would have to make food runs but that seems reasonable.