Diversity in Vermont?

My husband travels for work and I work remotely. We’re African American (from Louisiana) we’ll be here for awhile….and we can’t help but notice the strange looks we get in coffee shops/grocery stores (especially Healthy Living) What’s the deal? Is there an unspoken rule about where to and where NOT to go as a minority? We understand the statistics. However, the locals seem to …be a little shocked to see us?

EDIT:

Thank you all so much for the great insight. I really appreciate it and feel a lot better about our experience here, so far. Needless to say, I’m going to continue my shopping in HL and the local coffee shops. And if I get SUPER googly eye looks over the next couple of days, I’ll know you read this Reddit post and it’s probably me you’re looking at!