Confused af – try to chase him or nah?

The crush I am dealing with rn is quite a tricky one. I am crushing hard on this one friend of mine. Of course, I was thinking about confessing to him. But... I dont know.
You see, around one month ago, he realised his crush, who he liked for over three years, is actually kinda bitchy. However, he still talks about her and chats with her. That wouldnt be such a big problem, but I have a feeling he actually hates me(?). Sometimes, when I talk about other people I am close with or I am actually touching them, he looks at me with a cold expression.
But the other time I asked him to move next to me so I could talk with him about games and he was so happy to do so. He looked like he enjoyed our little chit-chat. And yesterday... well, I patted him. I just put my hand on his head and ruffled through his hair. He got all shy, hiding is face for a second. Later on, I had a pillow with me, because I knew I will get sleepy eventually. So I took it and once he was next to me, I lied on him with the pillow under my head. He got suddenly nervous. And dont let me start on the moment when our hands were touching for like five minutes straight.
However, after a while he moved away, sitting differently. When I asked him to sit the same way, he pretty much declined the offer and eventually stood up to go away. Later on, when he came back, he sat on a different chair. And until the moment he left, he hadnt talked with me, he pretty much ignored me. Except for this one moment, when I was describing a funny story about me and my classmate to a different friend, while we were laughing. At that moment, he looked at me again with those cold eyes.

So, to sum it up – we have this one big lunch break every day at school. And I know for a fact that he is free around that time every Tuesday. Should I ask him to go somewhere to chat, or just forget it?