I've noticed a similar issue when villagers are doing any type of work at all. For example, I gave a villager the task of building a warehouse. He went out to the building location, spent about 5 seconds on it, then went back to his house to eat and made about 5 trips carrying food into his house. Then he went back to the warehouse, spent 5 seconds on it, and again went back to his house to carry food into it. All of that walking back and forth between the building location and his house wasted a LOT of time.
Similarly, I had an entire village die off in the winter. They were running low on food and firewood, so I tasked all of the villagers to fish or make firewood (had over 500 wood in the warehouse). The villagers would go to the lumberjack building or the fishing building, then immediately go back to their house to warm up without having done any work. Once there, they would immediately leave because there was no firewood to warm up with, and head to work. Once they got to the work building, they immediately went back to their house to warm up. They just kept going back and forth like that, rather than actually get any work done to be able to survive!