Both "harder" and "more" are correct, but they have slightly different meanings.
If I had studied harder, means "If I had studied with more intensity."
"If I had studied more means "If I had spend more time studying."
One entry for the adverb "hard" in the American Heritage Dictionary*:
"¢ With strenuous effort; intently: worked hard all day; stared hard at the accused criminal.
"Harder" is the comparative form of the adverb "hard."
One entry for the adverb "more" in the American Heritage Dictionary*:
"¢ Comparative of much. To or in a greater extent or degree: loved him even more.
*The American Heritage Dictionary of the English Language, Fourth Edition. Houghton Mifflin Company. 2003