Emerging evidence suggests that psychiatric disorders are associated with disturbances in structural brain networks. Little is known, however, about brain networks in those at high risk (HR) of bipolar disorder (BD), with such disturbances carrying substantial predictive and etiological value. Whole-brain tractography was performed on diffusion-weighted images acquired from 84 unaffected HR individuals with at least one first-degree relative with BD, 38 young patients with BD and 96 matched controls (CNs) with no family history of mental illness. We studied structural connectivity differences between these groups, with a focus on highly connected hubs and networks involving emotional centres. HR participants showed lower structural connectivity in two lateralised sub-networks centred on bilateral inferior frontal gyri and left insular cortex, as well as increased connectivity in a right lateralised limbic sub-network compared with CN subjects. BD was associated with weaker connectivity in a small right-sided sub-network involving connections between fronto-temporal and temporal areas. Although these sub-networks preferentially involved structural hubs, the integrity of the highly connected structural backbone was preserved in both groups. Weaker structural brain networks involving key emotional centres occur in young people at genetic risk of BD and those with established BD. In contrast to other psychiatric disorders such as schizophrenia, the structural core of the brain remains intact, despite the local involvement of network hubs. These results add to our understanding of the neurobiological correlates of BD and provide predictions for outcomes in young people at high genetic risk for BD.