Don't trust him? Why??

I have a friend who I have seen maybe 4 times in as many years. We also talk very infrequently, despite having each other's contact info. I know this is not totally my fault, because I respond when he does initiate. But I never initiate. Everytime we meet it is mostly for him to ask for advice and for us to vent about his ex gf (who was a terrible person and used to be my best friend and I made a mistake setting them up). I don't really have a specific reason to dislike him. We have similar hobbies, my other friends like him, he doesn't want to cross the boundary of friendship, he asks very little of me, and I know at heart he's a decent person.

And yet.

There is just something about him that makes me feel uneasy. He has these comments he makes sometimes, these jokes that are almost inoffensive but not quite. He is sometimes very blunt in how he speaks to the point that he sounds rude...but what he's saying isn't necessarily that bad. He also peer pressures people sometimes. I just feel like he is using me for advice and judging me to an extent. I also feel immensely guilty for setting him up with a girl who ruined his life and then bailing on that because I couldn't deal with her. I guess I don't see how he could genuinely like me and I'm afraid of why he might actually want to be friends. There is juI don't know what to do but I know I am not being a very good friend right now because my heart isn't in it. And I want to know how to fix this while still feeling safe.