Estimating barriers to gene flow from distorted isolation-by-distance patterns

In continuous populations with local migration, nearby pairs of individuals have on average more similar genotypes than geographically well-separated pairs. A barrier to gene flow distorts this classical pattern of isolation by distance. Genetic similarity is decreased for sample pairs on different sides of the barrier and increased for pairs on the same side near the barrier. Here, we introduce an inference scheme that uses this signal to detect and estimate the strength of a linear barrier to gene flow in two dimensions. We use a diffusion approximation to model the effects of a barrier on the geographic spread of ancestry backward in time. This approach allows us to calculate the chance of recent coalescence and probability of identity by descent. We introduce an inference scheme that fits these theoretical results to the geographic covariance structure of bialleleic genetic markers. It can estimate the strength of the barrier as well as several demographic parameters. We investigate the power of our inference scheme to detect barriers by applying it to a wide range of simulated data. We also showcase an example application to an Antirrhinum majus (snapdragon) flower-color hybrid zone, where we do not detect any signal of a strong genome-wide barrier to gene flow.