Nucleo F411RE How read encoder with 0.005mm step?

Yes, the allocation won't be a big issue - in fact the current solution is probably best.

Since you're only actually doing stuff if a is high, why not do:

var step = 0;
function encFun(){
   if (!digitalRead(C8)){
     step ++;
     print(step);
  } else {
     step --;
     print(step);
  }
}
setWatch(encFun, C6, {edge:"rising",repeat:true}); 

or for superfast:

function setup() {
  var s = 0;
  var r = C8.read.bind(C8);  
  setWatch(function () {s+=r()?1:-1;}, C6, {edge:"rising",repeat:true}); 
  return function(){ return s; };
}

var getStep = setup();

As has been said, at anything over 100/sec the prints will probably cause problems and personally I probably wouldn't trust even the above much above 4000/sec. For 200kHz you're better off seeing if you can set up the STM32's built in hardware timers to do it for you - I'm pretty sure I've read that they can be set to work with encoders.