I must be honest and say I struggle to empathise with anxiety because I am lucky enough not to get it. However, I am off work with depression, and I get judgment by people for socialising whilst off. But for me I tell them I do what I have to do...
... to get better. You should to, do what you need to do. What ever the cost, your health is more important.