Check out the Store!
If you like to download your music, you can get individual songs or whole albums, either in 224K MP3 format, open OGG format or in full quality FLAC. You can also purchase downloads from the iTunes music store and other online music stores.
As for CDs, don’t feel bad, I still buy them too sometimes. You can get them on Amazon.