A non-native speaker wrote the following sentence. I'm puzzled by "had they ever been." I always thought that "had they + past participle" is an inversion of "if they had + past participle" and that it is used in counterfactual sentences. For example, "Had he seen Jane, he'd have fallen in love with her." But there seems to be no counterfactual meaning in the following sentence. How do I make sense of it? Would it better to rewrite it as "if they had ever been"?
We retreated to the car while Marco kept apologizing for the rudeness of the twins to our hosts, whose gazes were now anything but friendly, had they ever been.
EDIT: I thought it was obviously wrong, but somehow it passed unnoticed among some natives.
I'd appreciate your help.