What is your use case? For me, something like a fileserver which I am mainly SSH-ing into anyway, I may not install a DE at all, but if this is going to be a general-use desktop, I see no reason not to install the DE right from the beginning, and selecting a DE will be part of the install process of most Linux distros, or some distros have different install disk images that you can download with any given DE which it supports.

If you are very concerned about keeping your system lean and want full control of what gets installed, you might want to look up guides for installing Arch Linux. The initial setup process is more involved than other distros, but once you have it installed, I think its reputation for being difficult is overblown.