A waveguide mode solver based on boundary integral equation (BIE) method and matrix compression is developed in this study. Using an accurate discretization based on a Nystrom method and a kernel-splitting technique, the BIE method gives rise to three different formulations of a nonlinear eigenvalue problem. H-matrices are used in order to accelerate and increase the precision of the subsequent computations. Results from these investigations on a canonical photonic crystal fiber (PCF) chosen as an example demonstrate that the data sparse representation of the BIE discretization reduces the memory storage, as well as the assembly and solution times.