How to go about tracking down this error?

as recv only gets to run every ~500ms while serial data is arriving

I'm extremely surprised at that - if so then there's some very big, deep-seated problem.

As I'd said above, the idle loop in Espruino will only process 128 items from the input queue (4 chars in each item, so 512 chars total) before carrying on - and then it will call into the socket handling code.

So yes, it should run more often than once every 500ms, and the max stored data should be around 512 bytes.

Honestly, I think running custom firmware on the ESP8266 is overkill :) This really should be pretty simple to fix.

Of course if the ESP8266 firmware could implement flow control, that'd be pretty cool - but it really shouldn't be needed.