Joining of to all is necessary in a few constructions: 1. before certain personal pronouns : all of them, all of us, all of it 2. some idiomatic expressions : all of a sudden. However, in most constructions, it is optional: all of the people, all the people.
Theodore M. Bernstein states in his book* that "except with pronouns, the of is superfluous and the careful writer may wish to omit it on that ground, but its use is well based and cannot be objected to on any other ground."
Burchfield** also suggests that "all of is often used as a means of separating all from a proximate pronoun. It is also used in partitive of-phrase."
In my opinion, omitting of usually improves the construction, making it tighter, but the small unstressed word may be desirable in the cadence of a specific sentence. It's wise for a writer to have consistent style. An optional of in one sentence and the omission of optional of a sentence or two later is the kind of inconsistency that can make writing seem characterless.
* The careful writer: A modern guide to English usage (1965), Free Press.
**The New Fowler's Modern English Usage (1998), Oxford University Press.