He is more than a little absurd (not that he realises it).

Considering her statement for a moment or so, Mitchell replies, "We don't have to." Which means ideally he would like to ignore it, but he's not going to push away her enquiries. He's a little too mentally exhausted from the self-denial and it shows.