Do casual relationships all have a time limit?

I guess I’m just looking for some insight from others on experiences with casual sex with a consistent partner...

I’ve been hooking up with someone for two or so years, we’re chummy but we don’t hang out or date- just have sex. At one point I did like him and expressed that to him, but despite showing some mutual interest he didn’t do anything to move forward and I let it go and continued a casual relationship. I think I’m looking for someone to relate to on this because my friends all think it’s insane that I’m in a long term casual hook up situation that probably isn’t going anywhere. It sort of feels like a dead end and part of me is convinced I have to stop, but i do like having sex with him.

to make things a bit weirder, I haven’t seen anyone else in the past year so I feel sexually monogamous to someone I’m not even close to! It’s just weird!