Really,
If it's this bad - where note taking won't suffice - then when he calls say, "I'm recording this call, if you don't consent, hang up." The other alternative is to simply tell him you need everything in writing - i.e. if he has anything to say, send a letter.
In reality, even if he...