Heritability of brain anatomical connectivity has been studied with diffusion-weighted imaging (DWI) mainly by modeling each voxel's diffusion pattern as a tensor (e.g., to compute fractional anisotropy), but this method cannot accurately represent the many crossing connections present in the brain. We hypothesized that different brain networks (i.e., their component fibers) might have different heritability and we investigated brain connectivity using High Angular Resolution Diffusion Imaging (HARDI) in a cohort of twins comprising 328 subjects that included 70 pairs of monozygotic and 91 pairs of dizygotic twins. Water diffusion was modeled in each voxel with a Fiber Orientation Distribution (FOD) function to study heritability for multiple fiber orientations in each voxel. Precision was estimated in a test-retest experiment on a sub-cohort of 39 subjects. This was taken into account when computing heritability of FOD peaks using an ACE model on the monozygotic and dizygotic twins. Our results confirmed the overall heritability of the major white matter tracts but also identified differences in heritability between connectivity networks. Inter-hemispheric connections tended to be more heritable than intra-hemispheric and cortico-spinal connections. The highly heritable tracts were found to connect particular cortical regions, such as medial frontal cortices, postcentral, paracentral gyri, and the right hippocampus.