The basic properties of uranium oxide are essential to understanding and predicting the behavior of fuels under all conditions. UO$_{2+x}$ exists over a broad range of oxygen concentrations. In the hyperstoichiometric region, oxygen defects form clusters the behavior of which determines to a large extent a number of very important engineering properties diffusion coefficients, phase transitions, etc.In this study we aim to characterize the local crystalline structure that develops and the oxygen rearrangement that occurs in UO$_{2+x}$ (0 < x< 0.25) in order to improve our description of the nature of oxygen defects as a function of temperature and oxygen content. Neutron diffraction is one of the rare analytical techniques which enable the bulk properties of the anion sub-lattice to be studied.We present here a neutron diffraction experiment carried out at ILL Grenoble on several samples with different deviations-from-stoichiometry in a temperature range covering the phase transition between the two-phase (UO$_{2+x}$ and U$_4$O$_9$) and single-phase (UO$_{2+x}$) zones. The PDF (Partial Distribution Function) analysis and the Rietveld refinement on both UO$_{2+x}$ and stoichiometric UO$_2$ fuel specimens were performed to understand how and to what extent the temperature distorts the original fluorite structure. We focus particularly on the dependence upon temperature and deviation-from-stoichiometry of lattice parameter and the Debye-Waller factors for both the uranium and oxygen sub-lattices between 296 K and 1273 K. The temperature dependence observed in UO$_2$ compares favorably with previous research. The knowledge of oxygen thermal motion is taken into account to investigate defect clusters in hyperstoichiometric samples. A Rietveld refinement of the data for a UO$_{2+x}$ sample shows that a Willis type di-interstitial defect cluster is consistent with the average structure in high temperature UO$_{2.03}$ This conclusion is also consistent with the only other available data reported at higher deviations-from-stoichiometry.