Can One Transition From Family Zone to Partner?

Hi there.
I met this guy some 15 years ago through my sister when I had just started varsity. She’s older than me and was a club dj at the time so she pretty much knew all the “cool” people around. This promoter who taught her the ropes happened to be one of them. Their relationship automatically made me view him as nothing but a big brother because that’s what he basically was to my sister. He treated and myself as such too.
Fast forward some years later, we both have been in serious relationships and we both have children. We’ve never lost contact coz we’ve pretty much remained in the same circles.
This one night after an event he was hosting he sat me down to basically tell me that he has had the biggest crush on me all these years. He said he has never done anything about it because he didn’t want to break the trust between him and my older sister and he also didn’t want to scare me off. He’s a sweet, responsible and respectful guy. I think any woman who’d have him would be lucky. Some 2 weeks ago (this is after he confessed his feelings) we were at his house for a bbq he was hosting. Everybody got wasted and left. I don’t think we were drunk, maybe tipsy but on that night we had sex. Now, that for me was the problem. We are completely incompatible sexually (atleast based off that one night.) It left me so confused and possibly traumatised that I’ve been actively avoiding being in the same spaces as him because it’s just awkward for me. I know I like him but how do you begin to pursue anything with someone whom you’re not sexually compatible with?
The sex in itself wasn’t bad, he just said something that threw me off completely. Mid-sex this man confessed that he once masturbated to a bikini pic on my instagram. In his he as whispering this to me mid-sex on our first time made sense. How!?! I just went blank. I didn’t know what to feel. Flattered? Violated? Weirded out??? I have about 3 bikini pics online so I even pondered if it was possibly not just once. Anyway, he’s a good man I just don’t know if a relationship can work with someone who I firstly once considered my family and who obviously was secretly attracted to me from the get go.