Doctor prescribed a medication I'm allergic to and told me to come back if I react

I'm 30 years old, female, 160 lbs

I was recently diagnosed with an infection from a c section. I am allergic to penicillin, had it when I was a baby and had to go to the ER with hives/swelling/shortness of breath. I haven't had it since then. My doctor prescribed amoxicillin for me and notified me that it is theoretically similar to penicillin. He said take the meds and stop taking them if I start reacting. I am a bit concerned to take medication with the possibility of an intense allergic reaction.

Am I being overly cautious here?

UPDATE: Had a chat with my pharmacist and they seem to be aligned with a lot of the comments here! Also in my history I apparently did take some antibiotics in the past from the penicillin family without any issue so there's even less of a chance I'll have an allergic reaction. Going to take the medicine and hope for the best 🙂