My family don’t like my ex but we want to try again

My ex and I were together 6 years it was a toxic relationship everytime I got upset and hurt my sons who live with me saw it ( they are adults) they don’t like him for it the way he disrespected me and my friends and mum witnessed this too. We broke up but I stoll loved the guy I tried to move on but some reason I kept in touch with him and we met up and it was great we talked about the relationship and our faults and he said he would like us to try again but I cant because my family especially my sons can’t stand him they don’t even want him coming to the house and want nothing to do with him. They also told me they are disappointed that I’m entertaining him after how he’s treated me. I feel torn and slightly depressed I know exactly why no one likes him but I love him I don’t know what to do. Either way om never gonna be fully happy with any decision can someone give me some advice.