Most of the perception of bloat is caused by how Linux reports memory. An X server maps the display card into its address space, and on current graphics cards this can easily be 8, 16, 32 or even 64 megabytes of address space (for the frame buffer and registers of the display). Naive people look at "ps" or "top" and draw the wrong conclusion.
(via Slashdot)