Should I ask him or...?

Go ahead and do the hanging out with him thing, and if you wind up with a crowd of people every time, then ask him why he never wants to hang out with just you. And that is how you you get him to tell you if he really likes you. He will say either "I didn't know what you would say" or "I didn't think you would want to" or "I think of you like one of my friends" if he gives you something like the first two answers then you just tell him that of course you would say yes. If it is something like the last answer then you say well that is how you feel too, but you thought he might be thinking something else and then you stop with the hugging and kissing. If you to do wind up hanging out alone together, and he doesn't ask you to be his girlfriend then you just ask him if that means he wants to be your boyfriend. Sometimes you have to take the lead even when you don't know what your doing.