Second Nephi 2:1

And now, Jacob, I speak unto you: Thou art my firstborn in the days of my tribulation in the wilderness. And behold, in thy childhood thou hast suffered afflictions and much sorrow, because of the rudeness of thy brethren.

Lehi's time in the wilderness was no picnic. It was a trial for every member of the family. It must have been hard for Lehi to watch his family struggle through it with him. He could help and encourage only so much, and then they each had to work it out for themselves. That would have been the hardest part.