Why is castling out of check disallowed?

I've been playing for decades and not considered this rule until today. What is the reasoning behind it?

I understand and can see why castling through check isn't allowed but why not out of check, given that it takes the king out danger.

Is the rule a bit arbitrary? Would it make that much difference to gameplay generally, or am I missing something?