Context. Empirical models of galaxy-halo connection such as the halo occupation distribution (HOD) model have been widely used over the past decades to intensively test perturbative models on quasi-linear scales. However, these models fail to reproduce the galaxygalaxy lensing signal on non-linear scales, over-predicting the observed signal by up to 40%.Aims. With ongoing Stage-IV galaxy surveys such as DESI and Euclid that will measure cosmological parameters at sub-percent precision, it is now crucial to precisely model the galaxy-halo connection in order to accurately estimate the theoretical uncertainties of perturbative models.Methods. This paper compares a standard HOD (based on halo mass only) to an extended HOD that incorporates as additional features galaxy assembly bias and local environmental dependencies on halo occupation. These models were calibrated against the observed clustering and galaxy-galaxy lensing signal of eBOSS luminous red galaxies and emission line galaxies in the range 0.6 < z < 1.1. We performed a combined clustering-lensing cosmological analysis on the simulated galaxy samples of both HODs to quantify the systematic budget of perturbative models.Results. By considering not only the mass of the dark matter halos but also these secondary properties, the extended HOD offers a more comprehensive understanding of the connection between galaxies and their surroundings. In particular, we found that the luminous red galaxies preferentially occupy denser and more anisotropic environments. Our results highlight the importance of considering environmental factors in empirical models with an extended HOD that reproduces the observed signal within 20% on scales below 10 h−1 Mpc. Our cosmological analysis reveals that our perturbative model yields similar constraints regardless of the galaxy population, with a better goodness of fit for the extended HOD. These results suggest that the extended HOD should be used to quantify modelling systematics. This extended framework should also prove useful for forward modelling techniques.