Withdrawing or turning down offers

When I worked in county government, if you declined three hiring interviews, you dropped to the bottom of the list. State is a little different because you pick which jobs to apply for. But what if you go to the interview and find out it's not a good fit? Like maybe they advertised it as hybrid but it's actually 100% in-office. Is there a penalty for withdrawing or turning down an offer?