If Kaoruko was seriously a badass, she probably gave Avshar a headache or two during her career. It's not like he wasn't around, after all. He is not above seeking revenge on her granddaughter to satisfy his own feelings, should he realize who she is.

Obviously he won't kill her outright, for plot reasons, but if anyone was to show her what harshness the world has to offer, it could certainly be him.