Here's a couple of new thoughts on how to make it more difficult:


-age
Right now villagers are not affected by age (I think?). Their productivity should decrease with age, and how easily they get sick should also increase with age.

-Conflict
Villagers are very peaceful, even in the tavern. Perhaps they could have disagreements every now and then, and the consequences that might follow (for example refusing to do a specific job, or refusing to do a job in a specific way which could decrease efficiency for that villager in that profession, or a fight where one ends up needing medical attention and is unable to work for a day or two).

-Stess
Villagers work all day, and change professions all the time, at the same time they may be lacking greatly in stuff they need; such as clothes in the winter, or have only one source of food and still work the whole day just fine. Giving villagers a stress level that could affect their happiness, productivity and health could be interesting, as planning, or the lack of it, could have more long term impact.

-Diplomacy (+Mayor?)
There are many ways to run a society, and there are always somebody to disagree. Each policy could have specific bonuses, and specific downsides, and combine that with public opinion to affect happiness, then you've got a challenge, filled with long-term consequences that really requires planning. A mayors office could be a fun addition, where policies can be made and developed, aswell as enabeled or discarded.

-Currency + Crime
introducing currency would make it more challenging in the sense that unemployed surplus population would lack access to basic goods for survival (just like in real life!) and the more jobless/unemployed people you have, the more crimes they can commit in order to sate their needs. You could elaborate even further on that by itroducing new jobs such as shreiff or police.

-Resource Quality
There could be differences in how, for example, food, affects people. Maybe some people should have lactose intolerance, or some could be allergic to certain fruits or fish. The food itself could be more eventful, for example if villagers have a balanced diet they could get positive bonus stats, and more negative consequences of a one-sided diet. There could also be a greater variety in the quality between the other resources; for example some types of trees may be better suited for building, while others are better for firewood.

-Accidents
Accidents could have more nuance. For example, if villagers break an arm or a leg and thereby have to rest for some days/weeks. Instead of always dying, have accidents with different degrees of severity and effects. Perhaps one of the lumberjacks walks into a hunters trap and twists his foot. Silly lumberjack.

-Disasters
Another thing that'd be fun would be more variety of disasters, for example if larger portions of the food in a storage goes rotten, or a well goes dry, or a portion of the stored wood is infested with termites, or a crop could get infested.

-Major disasters
Having one devestating disaster perhaps every 5, 10 or 20 years could be very entertaining. As the plot is located on an island, a tsunami, or a major tide/flooding could happen. Other suggestions; forest fire, earthquake, plague, meteor shower, volcano eruption, locust swarm, extreme drought, winter extended to a year long. This could make better use of walls, also a disaster shelter would inevitably follow.


In conclusion I make the suggestions just 'cause I think there's a lot of solo players like me who wish for a greater variety of challenges in the late-game that won't easily solve themselves unattended once a village is up and running.