it's a bit hard to explain, but the technique they're describing has an approach to music that is really atypical to how human's naturally engage in music composition. it's much more mathematical and rigid, and is at risk for coming across as lacking emotion. my interpretation is that compton is using it to illustrate that the professor is hard to understand for "normal" people like he and garam, with the implication that he may be lacking typical human emotion and connection.

I don’t know about music and piano so what did compton mean when he said the professor is SCHOENBERG'S TWELVE-TONE TECHNIQUE