Afraid of relationships, how to deal?

I have wanted to be in a relationship since I was like 12 years old. At the time it seemed like everyone my age was getting into a relationship, obviously not serious relationships but a relationship nonetheless. I remember asking my parents if I could have a boyfriend and at the time they said I was too young.
I had "boyfriends" in elementary school, again nothing serious. It was more of a title at the time, just to say that you had a boyfriend/girlfriend. Through my teens more and more of my peers were getting into relationships and I remained single. Now I am 21 and have never been in a serious relationship.

I say I want a relationship, but I am starting to realize that I am afraid.
It's not that I haven't been in a relationship because no one is interested in me, I've had some opportunity over the years but I turn them down. I get so nervous that I just shut them down before giving them a chance.

- Maybe it's because I'm inexperienced.
- Maybe it's because I'm afraid of change.
- Maybe it's because I'm afraid to open myself to somebody. I've opened myself up to many people only to later be abandoned by them.
- Maybe it's because I believe I am asexual. I crave the kissing, cuddling, hand holding aspects of a relationship along with the friendship, but I have no desire to be intimate with anyone. Sex doesn't interest me. Yet I feel like it's so expected in a relationship and I don't know how to deal with that.
- Maybe it's because of my parents. I'm an only child and was cyber schooled through middle and high school. I don't want to say I'm immature but I do feel that I may have been babied. Talking about things like being a relationship, having kids, moving out feels strange to talk about with them. Probably because we never really talked about these things until more recently.
- Maybe it's because I have lingering feelings for someone else. When I started college in 2014 I met a guy who I really clicked with, I could envision a future with him. Looking back I think he liked me too but neither of us did anything about it. He transferred schools at the end of the year and we've lost contact since. Despite this, part of me still hopes to end up together.

Loneliness is safer. It's easier to lie or blow off guys so they leave me alone and I don't have to deal with it all.
But I don't want to be alone, certainly not forever.
And now it's getting to the point where people are asking when I'm going to get a boyfriend or try to set me up with people.

This is all stemming from a guy asking me out. He's the son of my parent's friend. We were at the same party at the end of the summer. I could tell he was checking me out, and he immediately added me on Facebook and Instagram. Even though we've never really spoken to each other, he sent me a message asking me out a couple days ago.
He's cute, probably out of my league, and seems very nice. I want to tell myself to give him a shot.
But instead I started panicking. I started thinking of what excuse I could give. Not wanting to lie, I deleted the message and plan to act like I never received it.

What is wrong with me? How do I get over my fear? I can't continue to do this or I'm going to wind up alone forever.

I'm sorry this is so long. If you read it all thank you.