1. Who was that book written by?
2. By whom was that book written?
I agree with Sarah that both are correct. Although it is possible to say, "Whom was that book written by?," most people would not say that, because it sounds stiltedly formal. So it would be silly to say that the version with "who," which almost everyone would use instead, is incorrect. On the contrary, it's fine.
Why not use the active voice? You could simply say: "Who wrote that book?"