He blames him for an wrong assumption that was never clarify. I don't defence him for who he is treating the MC in the present, but from his point of view (by the clues he has) he can't do better honestly. Plus if he was so bad as some comment paint him, he would have ever agreed by inviting MC over? He loves he nephew and he is doing his best.
He left his nephew alone in a stinky room and thought it was good enough because it had a fridge. He literally expected his nephew to babysit himself and I guess yeah you can say he's dumb because he was young but at present he should realize that he himself is in no position to blame some kid who he has absolutely ZERO evidence against

Why people are insulting the uncle now?!
He was worried that he was bullied (because Jun didn't explain the situation), left him in a safe place without knowing that Jun would have mived to another spot and then got injured.
In his character I see only a protective father figure who really tried it best considering his age