Absolutely does not have to be carbon. Oftentimes carbon shouldn’t be the choice. Nothing wrong per se with aluminum - some manufacturers like Cannondale do great things with it. And hey, no rust! Titanium is a great higher-end alternative to carbon as well. Heck, some small, “craft” shops are reviving steel for their custom bikes. Many, many factors involved.
The old adage is you want build quality, stiffness, and low price in your bike - but you can only have two out of the three. Obviously frame is the biggest factor in weight, but wheelsets and components matter too (as you noted). And of course, as coach used to say, you can spend thousands to shave a couple kilos’ weight off your bike, or next-to-nothing to shave a couple kilos’ weight off your backside.
My personal take on carbon is that - for non-elites - many people get carbon because it’s the “in” thing, and what they see online and at big events. Great material, and clearly a lot of benefit to both the manufacturer and the buyer, but there’s a cost/benefit with everything. As an example, build quality and long-term endurance re: carbon long-term is an open question. Issues cropping up in older, or less expensive carbon bikes with lower build quality, fiber delamination, and the difficulty and expense of detecting, testing, and repairing damage to those frames. Just recently read an article in Outside Magazine along these lines from a couple years’ back (not to scare you!):
(FWIW, I happily raced an aluminum bike for a number of years - beats you up a little on rough roads, but endured a long time).
Happy riding with whatever you’re on.