Milk is not recommended for its calories. It is recommended because it has a great amino acid profile, a complementary quantity of proteins to carbs to fats and because it is readily available. The game is not all about calories. We haven't even ascertained whether the OP should be on the GOMAD - or similar diet in this thread. I can't remember what his stats are. The GOMAD diet has been used and abused by individuals who believe it is the required diet for Starting Strength. This is entirely false. The overwhelming majority of people on Starting Strength should not be on the GOMAD diet.
But calories are necessary even if they aren't sufficient. When adjusting the diet for both calories and nutient balance, it's wise to keep a check on the macros, total weight, body fat as a rough reference and general performance in the gym. There is a correlation between calories consumed and all those criteria. You are obviously quite correct to state that no one should arbitrarily drink a gallon of milk loaded with a couple of pints of heavy cream without knowing what they are trying to achieve against their current condition.
I remember years ago having a body building friend tell me the answer to getting muscular was to consume pounds and pounds of cheese. I tried it. I got fat. My weight training consisted of a couple of days in his garage gym with weights I couldn't handle and muscles so sore that I couldn't pick up a pencil for an entire week. So, I know exactly why you are giving sound advice, but that's not to say someone who has done the research properly shouldn't try something like GOMAD, or adding heavy cream to milk.