A teacher of mine named Gulner actually conducted an experiment on his twin boys that relates to this subject.
When feeding his children he and his wife decided that with one child they would look intently into his eyes while talking to him during feeding. With the other son they still spoke to him while feeding him, but they made only brief eye contact and allowed their eyes to wander.
Gulner claims that to this day one son looks people straight in the eye when talking to them, and his other son's eyes tend to flit from place to place only stopping to look into your eyes directly on occasion. I have never met his children, so I can't verify his story, but next time you find yourself about to smack someone because they can't keep their gaze focused on you, remember that it might not be their fault.