Is it worth to use underleveld tower cards if they fit better?

My King and Princess tower are level 11.
Is it worth to, for example, use a lvl 10 Chef tower for a Golem deck because it generally fits better? Or is a one level difference in tower already a good reason to generally not use it?