The shorter (...much...shorter) man's reaction comes after a slight delay, like dragging his attention away from that door presents a physical difficulty. When he does finally look, though, it's evident that he is at least somewhat easy to distract, if only superficially. "Oh, yeah? Sure, okay."

Just one more glance back thataway (it's pretty much a glare, as if rooms can feel threatened), and then he retraces his steps... literally, even, backing up for several paces before turning to exit properly. He left the door open, at least, so there's no need to touch it again on the way out. Not that the door is offensive, or anything. It's an okay door, he doesn't mind it. Anyway, he's going outside, where once again he will be able to communicate in multi-syllabic words, presumably.