In this paper we propose an intuitive way to improve out-of-sample performance for yield curve forecasting. We show that when the yield curve is modelled using an affine factor model, residuals still contain relevant information and do not adhere to the familiar white noise assumption. We conduct a pseudo out-of-sample forecasting exercise implementing the widely used Dynamic Nelson Siegel model. Signicant improvement in forecasting performance is achieved throughout the curve and for different forecasting horizons. Results are robust with respect to different timeperiods, as well as to different model specification.