Think of it like nested time loops. Yes, the duke is going back in time constantly but maybe he only met and adopted Creed in the current life he's living now. Navia and the duke aren't going back in time together. She's going back in time only within his one life.
Duke ethelred looks like he has memories intact whenever he goes back in time but why doesnt he recognize creed