Scared cat was not always so scared?

We got the kitten when he was a kitten. We saved his life because he had a severe upper respiratory infection, and the shelter was not medicating him. Once he started getting a little better, he started sleeping on my mom's head the entire night. He was very loving and trusting of everyone. My ex-boyfriend about 4 years ago (a few months after we'd had him) threw a bouncy ball to him to play with, but it hit him. The cat (Kitty) was terrified and ran under the bed. My boyfriend felt bad, but wouldn't calm down or "apologize to the cat" (I mean encouraging the cat to come back out and regain his trust). He is about 5 years old now, maybe a little younger, but he is terrified of everyone, just over the past couple of months will he actually go up to my mom. He has, since the incident, only let me be the one he trusts. He is so loving with me and craves attention. He won't let others see him, friends don't know we have three cats until we tell (to be cont'd)