Ok, so pretty substantial and something that meets FAR 21.191. If a plane gets a new engine that is not the original then it could be reclassified as experimental. Bear with me because I don't use the exact aviation language. The new engine could also fall under the other option to remain certified if the engine has an STC? Is that sorta close?
In a 172, what would be the least expensive way to convert the certified plane to an experimental (camera hole thoughts not included)? I'm just curious why more old planes don't leave the certified category.
The simple answer is that the FAA doesn't allow the airplane into Experimental category merely on owner whim. There must be a purpose for the conversion... as others have posted, it must be performed as part of a Reseach and Development effort, or as part of a marketing survey, etc.
There isn't a one-size-fits all "Experimental" category. You're probably thinking of homebuilt aircraft, where the owner gets substantial freedom to make changes and maintain the aircraft themselves. However, these aircraft are awarded their Airworthiness Certificates "For the purpose of operating an amateur-built aircraft."
Your modified 172 would not qualify as an amateur-built aircraft, so you'd have to fit one of the other purposes. You could certainly claim that the modified 172 would be exhibited, or used for Market Research, or act as a test bed for Research and Development, or even raced. These (and several others) are all approved operations under Experimental.
However...and here's the bugaboo... you have to prove to the FAA that this is the purpose the aircraft will be used for. In your application, you will have to describe, quite specifically, the modifications to the plane and how they tie into your stated purpose.
Finally, if you manage to leap all these hurdles and convince the FAA, there are two other things to overcome. The first is the Operating Limitations. These are given to you by the FAA, and they describe what you are allowed to do with your Experimental aircraft. For amateur-built aircraft they're pretty simple...but they can be quite complex for other types of Experimental aircraft. The Operating Limitations may prohibit the carriage of passengers, and will probably restrict the operating area of the aircraft. Your experimental 172 may not be much fun if you can't fly your family and can't fly more than 100 miles from your base.
The operating limitations will probably specify who is allowed to work on the aircraft...and in in the case of your Experimental 172, they'll probably require you to use a licensed A&P.
The other thing that'll bite you is the fact that the Experimental airworthiness certificate is not permanent. It has to be renewed at intervals, typically yearly. "Renew," in most cases, means "Repeat the original application." The fun thing is, the FAA person who gets it may not be the one you used last time. He may look at it and say, "This is nuts, I'm not approving this." And then you've got a worthless airplane until you find an inspector who WILL sign it off.
Experimental Amateur-Built and Experimental Exhibition both are generally easier, and don't have as much problems. I have a friend with a Fourier RF-5B motorglider with an Experimental Exhibition certificate, and his life is about as simple as mine with an Experimental Amateur-Built certificate.
But the process is designed to PREVENT folks from converting certified airplanes to Experimental for personal use. I knew a guy who developed a Chevy V-6 conversion and got a 172 licensed Experimental for a test bed. He had yearly hassles with the FAA come renewal time, and, like someone else mentioned, he was required to convert the plane back into standard configuration when he was done.
Ron Wanttaja