how to get him back?

In a relationship since October. We were on and off and then became serious. Long story short, it was a lot of drama (too much to write here) and back in early March he pissed me off so I just blocked him and stopped responding to him because I was just over it. I definitely had strong feelings for him and loved him but it was just nonstop overwhelming drama. So its been over a month since no contact and my birthday was this week. When I woke up on my Bday, I had 3 missed calls from him. That put a smile on my face because I did not expect to hear from him. So I ended up calling him back and we ended up meeting up and having sex.
He vented to me and told me that he was really upset about the whole situation etc and that we should try this again but I kept telling him no. I told him, if you want to meet up and have sex occasionally then fine but a relationship is to much drama so we both agreed that it would be casual sex.

Well since then, I’ve been thinking of him a lot and now I feel like since we both had over a month to think about things and realize what we truly wanted, when I saw him on my bday he was looking so good and after he left I didn’t realize how much I actually had missed him.

So randomly he called me today because he was in my area and wanted to meet up for a quickie. so I met up with him and it lasted literally 20 minutes and then he was just like thanks see you soon. So the rest of the conversation went as below:

Me: damn can we talk at least
Him: About what?
Me: in regards to what you said on my bday about trying this again, do you still want to?
him: honestly I thought about it and you were right, it was to much drama and its just best if we keep like this just having casual sex, you messed everything up, I was so crazy and in love with you and you just completely stopped talking to me. That hurt me a lot and I just feel like next time you get at me you are just going to disappear on me again.
Me: But we can work on how to communicate when he have problems and I swear I won’t ever do that again.
Him: no it will never work but lets just stay friends.

So now I’m confused. I know I told him on my bday that it wouldn’t work but when we met up it was only for like 1 hour, I didn’t expect him to contact me that day, so I didn’t have time to even process and think about everything until after I got back home on my bday. So I just wanted some advice, if you think I should pursue to get him back, should I just disappear again because I don’t think just having casual sex will work and not end up catching all those feelings again even though its amazing.

Should I keep attempting to be with him and if so any tips/steps I should do to get him back? I don’t want to nag him and then end up losing him completely. I’m really upset about the whole situation and want him back so bad.