I think your instructor is taking a wise track, but maybe for a little different reason that people have yet said here:
Using conventional nav aids and using pilotage requires that you develop a good sense of urgency for situational awareness. You will develop the habit of always knowing where you are, especially relative to important things like airports, airspace, tall towers, and cities. If something bad happens, you will not have to fiddle with the GPS or a tablet to figure out where you are. You will also not be clueless about the airport codes that pop up on the NRST screen. You will also not risk bumbling into restricted or controlled airspace or crashing into a tower because you didn't notice the warning flashing on the GPS.
If you train with a GPS, I don't think this kind of sensitivity to situational awareness develops. I often fly with guys who just punch the Direct-To and motor along completely clueless as to where they are and what is around them. All they know is ETE and the destination airport code. IMO this is A Bad Thing.
All the other reasons cited here for starting with conventional nav tools are sound, but I think that developing an urgency for situational awareness during your training is the best one.