Since we already have a day/night cycle, how about improving simulation? The villagers could walk like 20 kilometres in a day, however to build a house, they would need 5-6 months. They would also need to visit home 2-3 times a day (for meals etc.) and get back home before nightfall.
Sickness will be an issue, as people will have less time in a day to work, also older guys will work less in a day, but they will make less mistakes.
When it comes to time speed, we would need more than what there is now - 1x 2x 5x and 10x time. We could use speeds like 1s real time = 1s in game, 1s = 5s, 1s = 1m, 1s = 5m, 1s = 1h, 1s = 4h. This would mean that on high speed only key stages of the simulation will be rendered (like crop growth, building stages finishing etc.), and actual positions of villagers would only be visible on lower speeds.