The search for genes involved in autism spectrum conditions (ASC) may have been hindered by the assumption that the different symptoms that define the condition can be attributed to the same causal mechanism. Instead the social and nonsocial aspects of ASC may have distinct causes at genetic, cognitive, and neural levels. It has been posited that the core features of ASC can be explained by a deficit in empathizing alongside intact or superior systemizing; the drive to understand and derive rules about a system. First-degree relatives also show some mild manifestations that parallel the defining features of ASC, termed the broader autism phenotype. Factor analyses were conducted to assess whether the latent structure of empathizing, systemizing, and autistic traits differs across samples with a high (individuals on the spectrum), medium (first-degree relatives) or low (general population controls) genetic vulnerability to autism. Results highlighted a two-factor model, confirming an empathizing and a systemizing factor. The relationship between these two factors was significantly stronger in first-degree relatives and the autism group compared with controls. The same model provided the best fit among the three groups, suggesting a similar latent structure irrespective of genetic vulnerability. However, results also suggest that although these traits are relatively independent in the general population, they are substantially correlated in individuals with ASC and their parents. This implies that there is substantially more overlap between systemizing and empathizing among individuals with an increased genetic liability to autism. This has potential implications for the genetic, environmental, and cognitive explanations of autism spectrum conditions.