She answers the question herself, so he doesn't feel the need to find a way to gloss over it - then again, it is a pretty general observation, albeit a keen one applied to him personally. It's not the loss, not really; the Penelope he knows isn't dead or hurt and certainly better off not having her life torn open again by abduction. It's more how protective he gets of the few people in his life who tolerate having relationships with him.

So he'll concede that, and not mention how he catches how she's holding herself right now.

"Not a lot of people put up with me."

Something they had - have - in common.