REBCO coated conductors are hoped to be the next enabling technology for fusion and many other applications. Despite their superior current densities, Jc, and upper critical field Bc2, their comparably low radiation robustness at their expected operation conditions poses a problem specifically for smaller fusion reactor designs namely SPARC and STEP, where high fluences are expected at the magnets. While smaller fusion designs target cost efficiency, magnets will have to be massively overdesigned to withstand the expected high neutron fluences over a life cycle, contradicting the original goal. We studied the influence of fast and slow neutron irradiation on the superconducting properties and how to mitigate their negative effects by defect annealing. The irradiation with fast neutrons from a fission spectrum leads to a broad distribution of differently sized defects. To specifically study the effects of small versus large defects, coated conductors containing gadolinium were irradiated with the full reactor spectrum. Gadolinium, exhibits a large absorption cross section for thermal neutrons, which subsequently leads to the introduction of high densities of point-like Frenkel defects. These small defects do not significantly contribute to flux pinning, however they degrade Tc and Jc 12 -13 x faster than samples irradiated solely with high energy neutrons. By assessing the contribution of small and large defects individually we aim at providing a more reliable prediction of the performance degradation of fusion magnets. Furthermore, we will show how annealing the previously irradiated samples at elevated temperatures might lead to a potential lifetime gain of 100 %.