FWB turned feelings turned confusion.

I am recently divorced almost a year ago. Ive been spending time with someone for 4 months that initially started as a D/s (BDSM) relationship. Feelings were not supposed to get involved and I pursued this relationship for the sex. Now him and I believe both have feelings, to the point where he doesn’t want to lose me. From the beginning I knew he slept with his best friend (which has been happening monthly for about 6 years). It started hurting me when it happened and because he didn’t want to lose me he said he’d be willing to cut off the sexual part of their relationship. We had a big conversation yesterday about where this was going and about how he has things he needs to work on, but he is willing to try for me. He always said he kept partners to this BDSM lifestyle because he is not yet in a place to offer someone what they need and considers me to be “out of his league”. Anyways – I have caught feelings, but there are things I am not sure about. Currently he does not have a job, does not drive (both things that he is trying to work on and wants to have in place). He also smokes a lot of weed which somewhat bothers me, but not really. When I got divorced I told myself I would not settle for someone and they had to be absolutely perfect for me to commit to another relationship. I feel he is perfect for me in so many ways, sexually, emotionally, intellectually. I’ve never felt more myself around someone, more validated and reassured. But there are things that worry me about what people will think. Because of this, today I told him not to stop sleeping with his friend because unless I am 100% sure on where i want this to go, I dont want his lifestyle to change. He respected that and appreciated that but now I can tell he feels hurt because he said “I already knew i wasn’t good enough for you. It just hurts hearing it.” We plan on keeping to our arrangement which is D/s. I am technically allowed to date, but I don’t even know if I should when I have these feelings for him. I don’t want to hurt him. Also we had rules not to do sleepovers and “datey” things, but we do all of those things.. not sure if I should pull back on that either until I get more clarity. I don’t want to jump into a decision or place expectations when I’m not sure.
Is money, job, upbringing, mental health, etc. Deal breakers? Can those things be overlooked?