Moving Away From HCOL Surf

I currently live in a very high cost of living area about 30 minutes from ok surf. I have some family inland (6 hours from the nearest not great surf) that I'd like to live near. I recently realized that if I sell and leave my current location for inland near family, I’d have enough money to retire. Not lavishly, but comfortably. 

Surfing is a significant part of my life. I surf a couple days every week, and most of my non-work social interactions are related to surfing. 

I should add that I neither like my job, nor my profession, so I am very ready to be done with that part of my life.

Have any of you done this? I know I have seen some posts about folks moving inland for the quality of life. How has that gone? Would you retire early—say 15 years early—if it meant you were six hours from the beach? I could do some surf travel, probably a month a year, plus some long weekend trips but nothing crazy, and not for months at a time. Would you keep working into your 70’s so that you could stay in a high cost of living area with OK surf nearby, or move inland and retire early but give up regular surf?