Exposure to childhood adversities (CA) is associated with subsequent alterations in regional brain grey matter volume (GMV). Prior studies have focused mainly on severe neglect and maltreatment. The aim of this study was to determine in currently healthy adolescents if exposure to more common forms of CA results in reduced GMV. Effects on brain structure were investigated using voxel-based morphometry in a cross-sectional study of youth recruited from a population-based longitudinal cohort. 58 participants (mean age = 18.4) with (n = 27) or without (n = 31) CA exposure measured retrospectively from maternal interview were included in the study. Measures of recent negative life events (RNLE) recorded at 14 and 17 years, current depressive symptoms, gender, participant/parental psychiatric history, current family functioning perception and 5-HTTLPR genotype were covariates in analyses. A multivariate analysis of adversities demonstrated a general association with a widespread distributed neural network consisting of cortical midline, lateral frontal, temporal, limbic, and cerebellar regions. Univariate analyses showed more specific associations between adversity measures and regional GMV: CA specifically demonstrated reduced vermis GMV and past psychiatric history with reduced medial temporal lobe volume. In contrast RNLE aged 14 was associated with increased lateral cerebellar and anterior cingulate GMV. We conclude that exposure to moderate levels of childhood adversities occurring during childhood and early adolescence exerts effects on the developing adolescent brain. Reducing exposure to adverse social environments during early life may optimize typical brain development and reduce subsequent mental health risks in adult life.