We all have those doubts and anxieties. Try to remind yourself that fears are not reality. If you're really struggling with this, talk to him and be honest. The more you keep it in, the more it festers and becomes real!
Don't worie ask him if there is anything going on. If he says no then just go about daily life it happens to all of us