Exhausting magic

I think one of the most intriguing mechanical elements of 5e is the "levels of exhaustion". I have yet to use it in game but I think it would make a very interesting story driver/consequence engine. It adds granularity to damage in a way that hit points don't. Something else I have been pondering is how one might add some grittiness to the magic system and I came up with a way to showcase levels of exhaustion while adding some risk to using magic. taking inspiration from the wild magic of the sorcerer class, what if whenever a magic user had a crit fail he or she gained a level of exhaustion? Now a lot of magic doesn't use a direct attack roll and is just save versus spell, in which case the exhausting magic might not apply. alternatively if the target of the spell Crits their save maybe the spell bounces back up on the caster? These house rules might go nicely in a higher danger setting with the permanent disability rule in effect. The key would be finding a balance where the added risks don't make the lethality unplayable. Perhaps it would be fun to try in a higher level one-shot where the power balance is purported to swing in the players favour. My literary proof text for this idea, btw, is in A Wizard of Earthsea when Ged over exerts his ability and is blinded for several weeks.