Is my EX BF threatening to sue me as revenge?

I dated a man for 10 months. I was just getting out of my own financial hardship after divorce. I told him I was not in a financial situation where I could monetarily contribute. He knew this before we even started dating. He was very generous. He always paid when we went out and occasionally he would give me money to pay my bills because “he wanted me to be comfortable”, bought me groceries, voluntarily bought me a part of my car, let me store my holiday stuff in his basement and even paid for a few weekend getaways. I helped him out the best I could with what I had so I watched his two daughters on the Saturdays he worked, in addition to minor housework and doing his laundry at my house. When I broke up with him he said I owed him money. I didn’t respond to him. So in a desperate attempt to contact me he came pounding on my door one day and even stopped by my sisters work. I still didn’t respond so he pretended to have an STD then accused me of intentionally giving it to him so he could “get my attention”. I was feeling threatened so I replied I would set up automatic payments for the $475. A few days went by, I started thinking clearly again and realized I had babysat a lot for him. So I itemized every single day I babysat and then deducted what I “owed” him and any other unpaid wages I deducted as misc expenses. I sent him the babysitting and deductions ledger, along with the diamond necklace and earrings he gave me. His response was I see you are refusing to pay me. I’ll see you in court for the groceries, trips, storage fees, car parts, etc. There wasn’t ever any kind of verbal or written agreement to deem this a loan. Should I be worried he might actually sue me or is this just his way of “punishing” me or getting revenge? Or am I breaching a contract by saying I would pay the $475? Thank you for your thoughts. :)