I think the relative clause modifies both "a letter" and "a group of letters". The reason that it takes the singular form is also because of the word "or". Am I right?
No, the reason for the singular form does not have anything to do with "or." However, the relative clause does implicitly apply to the first disjunct. The sentence may be regarded as a reduction of the following:
- A prefix is a letter that comes at the beginning of a word or a group of letters that comes at the beginning of a word.