Am I wrong for hanging out with my boss sometimes?

Hey guys, so I had a bit of an argument with my boyfriend and would like some opinions on it. I’m a senior in college, about to graduate and I’ve been working at this restaurant for about a year. I have a manager who I’ve gotten close to, but he’s twice my age with a kid, and is not with his ex wife anymore. I am not close with my family at all and being around him feels like almost a fatherly figure if that makes sense. He listens to me when I have troubles at school and gives me life advice. We had been spending time together outside of work where we’d talk or smoke for a while in his car and we had gone to eat together before. He told me his child doesn’t really speak to him, so I felt like maybe I remind him of her in a way so that’s why he’s so nice to me. My boyfriend however, brought up to me that he has a big issue with this and that we shouldn’t be spending so much time together. I don’t know if he’s overreacting, but he is really upset and thinks my manager is into me. It just doesn’t seem plausible, but maybe I am in the wrong? I care for my manager a lot and he really comes off as a what a good dad is…I guess like a dad I’ve never had. Please give me your insight on this, my boyfriend is so upset and I don’t know if I should apologize or defend myself.