WiFi power issues

After a fairly extensive debugging session, I've finally produced the following error message:

ets Jan 8 2013,rst cause:1, boot mode:(3,6)
load 0x40100000, len 23620, room 16
tail 4

which I believe means the ESP8266 has browned out and reset itself due to lack of power.

Cut down version of the code:

http.request(httpOptions, function(requestResult) {
  requestResult.on('data', function(data) {
    digitalWrite(A15,1); // FC Close

    console.log(data);
    var written = f.write(data);
    f.skip(written);

    digitalWrite(A15,0); // FC Open
  });
});

This happens fairly regularly if the file I'm fetching is big enough. I've hooked my SD card power up to an external voltage regulator so the only thing being powered by the Espruino is itself.

Gordon can you give this a go at your end and see if you can replicate?

Hardware FC is enabled using

'AT+UART_CUR=115200,8,1,0,2\r\n'

file is fetched over HTTP, about 46kb. SD card is at default baud rate. The problem occurs at different stages of the file transfer, but it happens almost every time for me.

Can you think of a board trace I could cut and supply voltage to the ESP8266 myself from another external voltage regulator? Problem with that is the ESP8266 needs a capacitor across VCC and GND otherwise you get these power issues...

Oh I've actually just tried it with the writing to the SD card commented out and I still get the issue.

Hmm - rst cause:1 sure looks like a power issue as you say.

You should be able to supply power straight to the 3.3v pin on the board - but perhaps you could stick a scope on it (and the VUSB line) first and see if it really does fluctuate?

I guess you're powering the board itself straight over USB? Could you try a different USB cable?

Sounds dumb, but the regulator should be able to handle 400mA peak, with the biggest issue being heat dissipation. I've never personally noticed any problems like this (and I've used the flow control before).

I've tried several different cables and I'm now running from an externally powered USB hub to make sure I've got enough juice going to the Espruino.

I've hooked up a USB-TTL directly to the ESP8266 so I can try and sniff things without having the Espruino log them, and something is definitely up:

+IPD,0,1412:f4b2<snip>1f6ff00f0
ERROR

+IPD,0,5840:fc<snip>001

and then I can also see the resets happen as well.

I tried looking through the schematics, but its not my strong suit. If I supply 3v3 to the board I'm guessing I have to unplug the usb beforehand otherwise I let out the magic smoke?

Also do you have a capacitor between VCC and GND on the ESP8266? I can't see one on the schematic but I might just be missing it.

When you use the external USB-TTL, do you ever see that ,rst cause:1 appearing?

Also, what if you disable the flow control you'd added? If you're not writing to the SD card you won't need it I guess, and I'd be interested to see if that is making it worse somehow.

It'll be fine to power by USB and use an external 3.3v regulator - the one on there will still do its job, but if another regulator is on there too then it'll take up the slack if the volts on the 3.3v line drops.

There's a 47uF capacitor (and a 0.1uF as well) on the 3.3v line next to the regulator in the circuit diagram - they're physically very close to the ESP8266 so should do pretty well. 47uF should do a pretty good job at soaking up any short-term power requirements.

Also, the ESP8266 itself only needs 1.7v (less than the STM32 I think?) so if that's browning out then I'd be very surprised if the STM32 is still working fine. I guess it could be your board in particular has a bad connection to the ESP8266?

Yes I see the resets on the USB-TTL.

I've gone back to using the pipe() we were discussing in the other thread that lead to memory issues.

I thought the ESP was minimum 3v? http://www.electrodragon.com/w/ESP-12F_E­SP8266_Wifi_Board

I'm going to try hooking my components to an entirely seperate power supply, currently my voltage regulator is powered by the 5v output of the espruino (which I assume is just the 5v line from USB), so perhaps it is drawing too much from there and browning out the onboard vreg.

I will note, that quite often I experience an issue where I can't flash to the Espruino once something has gone wrong with the WiFi. I usually have to disconnect power and reflash, I wonder if this is a symptom of the STM32 browning out?

Ahh, I just googled and took the top hit, which was wrong :) Still, https://forum.makehackvoid.com/t/esp8266­-operating-voltage-range-and-sleep-curre­nt/286 implies that they tested and 2.3v seems to work ok. Perhaps not under load, but there's a lot of leeway.

On the Wifi that VUSB pin is connected straight to USB 5v - so it should be fine - however I guess it depends how it's connected.

I will note, that quite often I experience an issue where I can't flash to the Espruino once something has gone wrong with the WiFi. I wonder if this is a symptom of the STM32 browning out?

It could well be, yes... Since the ESP8266 driver's all JS at the moment you should be able to Ctrl-C out of pretty much any execution regardless of what's happened.

Do you have something like an oscilloscope that you could connect to the 5v and 3.3v lines? It might help to narrow this down quite a lot.

Unfortunately no oscilloscope here as far as I know, anything this low level is kind of our of our realm, it took me hours yesterday to realise I was getting SD card failures because that was browning out, I kept just thinking my code was wrong.

Did you have any luck replicating at your end?

No luck replicating power issues here. I'm using this code:

var wifi;

function onInit() {
  wifi = require("EspruinoWiFi");
  wifi.connect(WIFI_NAME, { password : WIFI_KEY }, function(err) {
    if (err) {
      console.log("Connection error: "+err);
      return;
    }
    console.log("Connected!");
    wifi.getIP(function(err,ip) {
      console.log("IP address is http://"+ip.ip);
      createServer();
    });
  });
}

function createServer() {
  var http = require("http");
  http.createServer(pageHandler).listen(80­);
}

var fileNo = 0;

function pageHandler(req, res) {
  var d = "";
  var boundary;
  req.on('close', function() { 
    console.log("File uploaded");
  });
  req.on('data', function(data) { 
    console.log("Got "+data.length+" bytes");
  });
  res.writeHead(200, {'Content-Type': 'text/html'});
  res.write('<html><body><form method="post" enctype="multipart/form-data">');
  res.write('<input type="file" name="file1" id="file"><input type="submit">');
  res.end('</form></body></html>');
}

and uploading a big file. The voltage is rock solid at 3.32v

Having said that, I could reproduce your out of memory issues at least - and hopefully I found something that'll fix them. It seemed to be related to what happens when we're trying to send data to a socket at the same time as ESP8266 is sending it out.

If you upload now the out of memory issues should be gone (or at the very least, better).

While I imagine it probably won't make a difference, could you try it going the other way? Download rather than upload. Doesn't need to be a huge file, 40-50kb should do the trick... maybe just try and download https://www.espruino.com/press/logo_squa­re_blur.png ?

Just tried this, and it works fine here:

require("http").get("http://www.espruino­.com/press/logo_square_blur.png", function(res) {
  res.on('data', function(data) { console.log("Got "+data.length+" bytes"); });
  res.on('close', function() { console.log("Closed"); });
});

Even setting a few of them off at once. Using https still works, but I can't do more than one because of memory.

Please can you try checking E.getAnalogVRef() and seeing what it says? It should be pretty close to 3.3 (well within 0.1v). Could you also check E.getTemperature()?

Perhaps you could do something like:

v=0
setInterval(function() {
  v = (v+E.getAnalogVRef())/2;
  digitalWrite(LED1, (v<3) || (v>3.6));
},1);

It'll then light the red LED if the voltage goes too far out of range.

I've hooked it up to the oscilloscope with the WiFi router about as far away as I can sensibly get it, so it'll be using as much power as it can, and I do see some little troughs in the voltage - it goes down to 2.75v for about 250uS - which seems to be about what you'd expect based on the voltage regulator's response time (it's not overloaded, it just takes 200uS or so to respond), but that would appear to be fine.

Do you have more than one Espruino WiFi device? Does it happen on all of them?

Oh now this is interesting.

Here's what I get when I use your code with all the rest of my stuff deleted:

8 bytes
125 bytes
73 bytes
70 bytes
69 bytes
69 bytes
69 bytes
69 bytes
69 bytes
69 bytes
68 bytes
69 bytes
69 bytes
69 bytes
69 bytes
69 bytes
69 bytes
68 bytes
69 bytes
69 bytes
55 bytes
109 bytes
72 bytes
69 bytes

(fairly consistent 69byte chunks).

With my code:

9 bytes
383 bytes
26 bytes
188 bytes
302 bytes
383 bytes
26 bytes
369 bytes
25 bytes
142 bytes
209 bytes
333 bytes
383 bytes
26 bytes

(fairly consistent 300+ byte chunks).

If I add some extra logging:

383 bytes
20042 so far
30 bytes
20425 so far
383 bytes
20455 so far
30 bytes
20838 so far
383 bytes
20868 so far
31 bytes
21251 so far
383 bytes
21282 so far
31 bytes
21665 so far
383 bytes
21696 so far
31 bytes
22079 so far
383 bytes
22110 so far
271 bytes
22493 so far
9 bytes
22764 so far
4 bytes
22773 so far

(mine crashes around here)

yours happily cruises along:

74 bytes
25212 so far
74 bytes
25286 so far
60 bytes
25360 so far
115 bytes
25420 so far
77 bytes
25535 so far
75 bytes
25612 so far
74 bytes
25687 so far
74 bytes
25761 so far
74 bytes
25835 so far
74 bytes
25909 so far
74 bytes
25983 so far
74 bytes
26057 so far
74 bytes
26131 so far

So the question becomes, why does my code fetch larger chunks? I don't think I've configured anything to specifically ask for larger chunks...

I've just added the flow control code to your example and I still get 69byte chunks, so its not the flow control.

Got it. I'll have another code sample for you in a few minutes.

So this whole debacle is at least mostly my fault. A while ago I had to reflash the ESP8266 firmware and ended up with 0.50 AT command set, which as we discussed here: http://forum.espruino.com/conversations/­304736/?offset=25#13630588 has a longer boot preamble than the 0.40 command set, which seems to confuse the FIFO buffer and therefore the standard EspruinoWiFi module.

To work around it I've been doing my own hack to reset the ESP8266 and then hand over control to the EspruinoWiFi module.

What I didn't realise is that this causes two .on('data') event handlers to be registered (my one is never freed), and this causes many, many, many issues further down the stack. It also explains why everything WiFi has been so slow, the data event handler is being run twice for everything!

@Gordon as I am stuck with having to use my workaround to boot the 0.50 firmware, could you possibly provide a disconnect function in the AT library to release the event handlers?

var wifi;

function onInit() {
  var doPreambleReset = true;
  
  wifi = require("EspruinoWiFi");
  
  var connectGetIPAndFetchData = function() {
    wifi.connect(WIFI_NAME, { password : WIFI_KEY }, function(err) {
      if (err) {
        console.log("Connection error: "+err);
        return;
      }

      console.log("Connected!");

      wifi.getIP(function(err,ip) {
        console.log("IP address is: "+ip.ip);

        var http = require("http");
        var fetched = 0;

        http.get("http://www.espruino.com/press/­logo_square_blur.png", function(res) {
          res.on('data', function(data) {
            fetched += data.length;
            console.log("Got "+data.length+" bytes");
            console.log(fetched + ' bytes so far');
          });

          res.on('close', function() {
            console.log("Closed");
            console.log(fetched + ' bytes total');
          });
        });
      });
    });
  };
  
  if (doPreambleReset) {
    initWifi(connectGetIPAndFetchData);
  } else {
    connectGetIPAndFetchData();
  }
}

/* Perform a manual reset of the WiFi chip, sometimes the preamble from a hardware reboot
 * exceeds the FIFO and confuses EspruinoWiFi module. If we do a software reboot,
 * the preamble is smaller.
 */
function initWifi(callback) {
  digitalWrite(A14, 0);

  Serial2.setup(115200, { rx: A3, tx : A2 });
  
  /* Herein lies the problem. The AT module has a 'connect' function, but no 'disconnect'
   * In the 'connect' function it registers an event handler on the 'data' event of the
   * Serial port, and this never gets released. When we use EspruinoWiFi.connect later,
   * this registers a second event handler, and this causes something to screw up.
   */
  at = require('AT').connect(Serial2);

  at.cmd('\r\nAT+RST\r\n', 10000, function(data) {
    setTimeout(callback, 10);
  });

  digitalWrite(A13, 1);
  digitalWrite(A14, 1);
}

Aha! Glad it's sorted! Really odd about the low power reset error though?

You should be able to do Serial2.removeAllListeners() to remove them? You could also check with process.memory to make sure it's actually freed your old AT instance?

About the packet sizes: the data comes into Espruino pretty much whenever, and Espruino handles it as and when it can. If the handling takes longer then the handler gets called less often - but with more data... So as you make more complex projects I'd expect the packet sizes you get to increase, or some to be larger than others (if a function took a long time to execute while data was being received).

That seems to fix it!

Its worth noting that this drops the maximum baud rate that seems reliable.

I used to get 3000000 as a workable baud, but now with that I'm getting

82 bytes
87ffc3ffff00c0f9000001fdff0003ffff0003fe­ff00f0fc0067005Af500027c78f0e500ffff00
+I
16892 so far

in the received data, I guess the +IPD isn't complete so Espruino gets confused.

I'll try and find a slower (but higher than 115200!) baud that is stable.

1382400 is the highest baud I can get that is reliable.

Wow, that's still pretty impressive! That's with the flow control, or without?

Just to say that I've just committed some changes that turn framing/parity errors off by default, with an option to turn them back on. If you get the latest builds now I imagine you may not need your hack around reset.

Thats with flow control on.

Just to add to this, flow control is now on by default, and there's a new method for the Espruino WiFi called turbo that lets you increase the baud rate to WiFi