UV-visible absorption spectroscopy and Raman spectroscopy were used to study damage production in cerium dioxide epitaxial films and polycrystalline sintered samples after irradiation with electrons for three energies to span the threshold displacement energies of cerium and oxygen atoms, and 2.4-MeV Cu ions. Neither amorphization nor specific colour-centre bands were detected. Evolutions of the refractive index were derived from the interference fringes in the optical transmission spectra of epilayers after irradiation. No significant change of the refractive index occurred for the 1.0-MeV electron irradiation, whereas a maximum decrease by 28 $\pm$ 8% was deduced for the 1.4-MeV and 2.5-MeV energies. These modifications are consistent with ballistic damage on the cerium sublattice for high electron energies producing Ce$^{3+}$ ions. However, no significant change of refractive index was found for the Cu ion irradiation. This likely stems from the high rate of Frenkel pair recombination in the collision cascades induced by more energetic recoils than for the electron irradiations, combined with electronic excitations and hole capture on Ce$^{3+}$ ions. This study reveals modifications of the electronic structure upon irradiation that could take place in other non-amorphizable oxide systems.