One of my builders just died because he got stuck while building a house (the 10 ppl version). I could get him out again after i finished the building in 1st person view and the support logs around the building site vanished but it was already too late for him.
Imho a simple solution would be when the fence around the building wouldn't appear until it's finished.