It's hard to go past FRAPS for in-game recording, but there are other programs that do the same thing. You could opt for one of those, I guess.
Outside of that - update your drivers, try a different FRAPS version or tinker with resolution and see what happens. If it's a CPU problem (your invariably pushing your system harder when you record with FRAPS), then it might be an idea to record on a secondary system, if you have access to one.
EDIT: Saw that you mentioned your game crashes - not your system. Disregard the CPU thingo. Try recording a different game (if you have one installed) and see if it still crashes. At least then you might get an idea of which program is causing the crash - FRAPS or GTA.
EDIT 2: Be sure that whatever key you have assigned to record is not conflicting with something else you have running that may cause a crash.