Please please please don't let the weather make farming any harder. If winters and droughts are going to be a thing for the sake of realism, the yields of good harvests need to be more realistic than a max of twelve per seed planted, and there needs to be some sort of almanac skill to help predict weather so you're not sunk if you picked the wrong day to plant.
For instance, real life expected yields for wheat are 20-30 grains per grain of wheat planted (reference
http://amarillo.tamu.edu/files/2010/11/estimating-wheat-yield-osu.pdf) and an average of 27 pounds of green peas per pound of seed planted (reference
http://www.ipmcenters.org/cropprofiles/docs/idpeas-green.pdf)
And that's for realism--not to make it fun, just to make it a real time farming sim.
The difficulty vs the reward of farming is no where near balanced to the same productivity as hunting or mining or forestry. It makes apples the only appealing food to grow, since you get 15 apples per day from the single apple sprout you planted months ago. Unless the crop yields are better than apple trees, there's not much incentive to bother with farming.