No it is not baloney, and this is where you fail to get it.
You example is a perfect one, there are no constitutional issue with drug use, that is not a federal government issue, it is a states issue outside of the controlling it interstate. There was a court case, a bizarre one at that...