At my conceal/carry class, the instructor told us to memorize this phrase: "I was afraid for my safety and the safety of those around me." You bet I would draw and use my weapon if I felt this way. The key word is "safety". Doesn't just apply to the threat of dying. Sure the nut job with the knife/gun/tire iron etc. might not kill you or your family, he might just rape and beat your wife, kidnap your kids and do God knows what.
This is why my Dad always told me and my sister if we were being kidnapped to kick, scream and fight even if they were threatening to kill us, cuz they probably would anyway after doing worse.
As far as backing away, I'm clumsy enough just trying to walk forward. Might buy you a little time and a little distance, might just give the real aggressor the extra seconds they need to have the upper hand.