This is only my speculation but I think he did visit Taemin but just looking at him from afar to check that he's okay considering Siwoon's position (Hansung's group heir and all) and then when the accident happened to Taemin, Siwoon became lifeless (just like Minjun said "A guy who lives just because he's alive) and that's why he was so dead set on the fact that Siwon's Taemin, even though the clues are very vague (he needs a reason to be alive again. He's just "conveniently" right) and now that he could see Taemin again, he probably doesn't want to make that same mistake again (of keeping distance) and just outright does anything to keep him around
Can someone tell us in spoilers what Joohoon said or asked to Taemin in ch 31 ?