Interesting hypothesis. I went down the rabbit hole on it but couldn’t find supporting evidence (not to discredit your anecdotal experience of course). Seems if anything slightly the opposite on average. See for example “Sexual dimorphism in body size occurs when … living standards are high, and health correlate positively with male/female height ratio” here: https://pubmed.ncbi.nlm.nih.gov/32793524/