guys why does the 'erasing memory' thingy doesn't work on grandpa?

Probably due to him being a holy knight or something? It will be likely explained later as we progress how the relics work.

Based on the novel, the reasons might be:
1. Mc doesn't know how to use the relic properly coz the relic required an exchange to erase one's memory.
2. Because Duke Luminel is stronger than typical holy knights and he is blessed by his deceased wife.
3. Duke Luminel was cursed when he's protecting his 4th child and wife.