1 If you hadn't helped me, I would have been in big trouble.
2 If you didn't help me, I would have been in big trouble.
Are those sentences different in meaning?
I guess the second one seems so not right, though.
Hi, Dude—Past counterfactual conditionals (sometimes called Type 3 conditionals), in which the "if"-clause refers to an event that did not happen, should have the past perfect in the "if"-clause, as in (1) but not in (2).
3 It was turned out that he hadn't completed the project
passive is wrong?
Yes, it is ungrammatical to use the passive voice there. "Turn out" is a special type of verb (a raising verb), which functions like "happen." You can say:
(3a) It turned out that he hadn't completed the project.
(3b) He turned out not to have completed the project.