JS library for animation of text on matrix display?

I just hoped it would be fast enough

In pretty much all cases it is! There's no reason why there couldn't be a general purpose all JavaScript library to do text animations.

One or two displays (like the full RGB ones with SPI interfaces) are a bit slow and require big hacks to get decent speed animations, but I don't think they are anything you should be worried about catering for.

As you say, using the Graphics library would allow you to use it on all displays, and also allows you to take advantage of speed improvements made for specific display types.

As @allObjects notes, some graphics drivers don't allow for reading data back from the display (or are slow at it) so using g.scroll(x,y) can be difficult - but usually simply drawing the text you want to scroll fresh for each frame is more than fast enough.

The only two examples on the Espruino site I'm aware of are:

http://www.espruino.com/rgb123_reaction_timer
http://www.espruino.com/Pico+LCD+Hello+World

but there may be one or two others and I have some things I've done here that aren't published. There are also some games which do animation that could be worth looking at: http://www.espruino.com/Search?kw=game