I see a clear difference in meaning.
In (1) the verb "saw" refers to a real possibility: if they actually saw you, now you are in trouble.
In (2) "had seen" is counterfactual or past hypothetical: what would/could have happened if they had seen you? (fortunately, they did not see you, but if they had, the consequences could have been serious).
Hello, Navi and Gustavo,
I agree with both of Gustavo's interpretations. "What if they had seen you?" is clearly counterfactual, presupposing that they did not in fact see John, and "What if they saw you?" can easily mean that it is still possible that they did in fact see John and that he may still face real consequences.
I think that one other interpretation may be possible for "That was stupid. What if they saw you?," and that is that the reply may be addressing itself to the point in time at which John made the decision to attempt to slip into the house and steal their book, cf.: "Did you consider what would happen if they saw you?" Compare:
John: I put on the first parachute pack I saw before jumping out of the plane.
Tom: That was stupid. What if it wasn't packed right?
John: We were going down. I didn't have time to be choosy.
Now, Tom could have asked "What if it hadn't been packed right?" instead, since John's being alive and having a conversation with Tom is evidence that John's parachute opened and therefore was packed right (or right enough to do its job). The version with "wasn't" seems to address itself to the moment of decision.