"By a brick" would be correct here. With the passive voice, "by" indicates the agent doing the action, in this case the brick which was thrown by a hooligan. "By" goes with many verbs in the passive voice, including "break." In this case, the brick breaks the window by accident; the brick was not meant to be used as a tool.
"With a brick" would indicate that the brick is a tool. The tool would be something normally used on purpose for a particular task. We'd say that someone eats ice cream with a spoon, opens the door with a key, and cooled the drink quickly with ice:
"¢ Ice cream is usually eaten with a spoon.
"¢ The door can be opened with a special key.
"¢ The coffee can be cooled quickly with a lot of ice, but the coffee will be diluted.
You might say this:
"¢ The window was broken with a special tool used by a professional locksmith.
"¢ The window was broken with an ax that the firefighters used.
On the other hand, when the window is broken not by a special instrument -- by accident or by nature -- you would say "broken by":
"¢ The window was broken by a large hailstone.
"¢ The window was accidentally broken by a baseball that the kids were playing with.
"¢ The window was broken by the branch of the tree that fell down in the storm.
"¢ The window was broken by a brick thrown by a hooligan.
"With" is normally used in the passive voice when you use a special instrument with a purpose: with an ax, with a fork, with chopsticks, with my fingers. "By" is used for many things: run over by a truck, written by Hemingway, attended by thousands of people, broken by a badly aimed football.
The sentence in question -- the window was broken with /by a brick thrown by a hooligan -- seems to need "by." Probably the window was broken when the hooligan hurled it from a place not too near the window. The hooligan didn't think beforehand about using the brick as a tool.
If the hooligan had brought the brick with him and intended to use it as a tool, then we could use "with." The sentence would be something like this:
The window was broken with a brick that the hooligan had brought with him especially for the purpose of breaking windows.