When seasons change i tend to 'clear the cache' not sure if it's really necessary or if the distribution system is that soffisticated that i can just slap people around if enough laborers are available and they will always have the closest home. Maybe that's what's claculating when changing around alot?
Maybe i am imagining things, but for me it helps to reset the workers every now and then, so they get a clean distribution, switching them around usually ends up in a mess.
I auggest a button to reset all workers and then buttons for all the individual workers (Those would fit between the number and the arrow pointing to the right)