The promise posed by half-Heusler compounds as thermoelectric materials depends on their thermal conductivity, which is strongly affected by doping. Here we elucidate the effect of p dopants on the lattice thermal conductivity (κ$_{ph}$) of seven selected half-Heusler compounds and for twelve different substitutional defects. We unveil a strong reduction in κ$_{ph}$ even for low concentrations of transition-metal substitutional atoms. Furthermore, we quantify the strength of the bond perturbation induced by substitutional impurities and interpret it in terms of the changes in the local electronic density of states. In several cases we find a significant destructive interference between the mass difference and bond perturbations which reduces the phonon scattering rates below the value expected if the two effects were treated independently. We compare our first-principles calculations with the available experimental measurements on the thermal conductivity of (Zr, Hf)$_{Nb}$-doped NbFeSb and Sn$_{Sb}$-doped ZrCoSb. For the latter, including the effect of independent Co vacancies and interstitials yields an almost perfect agreement with experiment.