Is it a hot take to hate passive perception / insight?

I feel like it's game breaking. Even players at very low levels can have huge passive perception numbers. So basically they walk into a room and see all the secret doors, traps, etc. without even trying. A player with high perception is usually going to say they look around and will succeed most of the time, anyway. But it at least makes the dice part of the game and the player actually gets to play instead of having the game served up to them.

How is this fun for anyone? What am I missing?

(Not rhetorical, I'm actually asking DMs who like and use it to explain how it works in their game without spoiling stuff).

EDIT: Lots of good advice on how to use it without making it OP. Thanks, all.