Sounds like both parties are telling what they believe to be the truth.
What has been stated:
The IFR guy was in the clouds at 4,000'
The OP was above them at 4,500',
The cloud layer was broken.
If all of these statements are true it is HIGHLY unlikely the OP had 1000' of cloud clearance below.
If the IFR pilot is factually correct, and the OP is factually correct about the cloud distances, then the OP couldn't have been flying at 4,500 MSL. The altimeter is either REALLY off, or the altimeter setting was wrong and the OP was cruising at a wrong altitude assuming terrain elevation was less than 1500MSL.
If the IFR pilot was correct about being in the clouds at 4,000 and the OP was correct about the altimeter, then the OP needs some more experience in determining cloud distances.
It's possible the IFR pilot wasn't at 4,000' and the OP was both at 4,500' and had 1000 feet of clearance, but if the IFR pilot wasn't at 4000 ATC would have probably been constsntly yelling at him to get at the correct altitude.
So I wonder what the real story is. Post the flight aware tracks.