Geophysical granular flows, such as landslides, pyroclastic flows and snow avalanches, consist of particles with varying surface roughnesses or shapes that have a tendency to segregate during flow due to size differences. Such segregation leads to the formation of regions with different frictional properties, which in turn can feed back on the bulk flow. This paper introduces a well-posed depth-averaged model for these segregation mobility feedback effects. The full segregation equation for dense granular flows is integrated through the avalanche thickness by assuming inversely graded layers with large particles above fines, and a Bagnold shear profile. The resulting large particle transport equation is then coupled to depth-averaged equations for conservation of mass and momentum, with the feedback arising through a basal friction law that is composition dependent, implying greater friction where there are more large particles. The new system of equations includes viscous terms in the momentum balance, which are derived from the mu(I)-rheology for dense granular flows and represent a singular perturbation to previous models. Linear stability calculations of the steady uniform base state demonstrate the significance of these higher-order terms, which ensure that, unlike the inviscid equations, the growth rates remain bounded everywhere. The new system is therefore mathematically well posed. Two-dimensional simulations of bidisperse material propagating down an inclined plane show the development of an unstable large-rich flow front, which subsequently breaks into a series of finger-like structures, each bounded by coarse-grained lateral levees. The key properties of the fingers are independent of the grid resolution and are controlled by the physical viscosity. This process of segregation-induced finger formation is observed in in laboratory experiments, and numerical computations are in qualitative agreement.