In (1), only is an adjective, as it precedes the noun "t-shirt" and refers directly to it, not to the existence of only one large t-shirt. Compare with:
3. There was only one large t-shirt left in that color. (Here, "only" is an adverb, as it modifies the numeral "one.")
Likewise, in (2) only is an adverb as it refers to the presence of four fans ("only" modifies "four").
The presence of an article before "only" helps define it as an adjective:
- an only child / the only large t-shirt / the only large t-shirts
In the plural, the presence or absence of an article will determine whether "only" is an adjective or an adverb.
4. Those were the only large t-shirts in that color (adjective).
5. There were only large t-shirts (no small or medium ones) in that color (adverb).