Custom or additional baud rates for Espruino Web IDE? (HelTec WB32 communicates @ 74880 baud)

Hi guys,

So I recently bought one of these very gorgeous and feature packed boards, based on their LoRa module (this one has no LoRa radio).

http://www.heltec.cn/project/wifi_kit_32/

It incorporates: a LiPo battery charger, USB interface, what appears to be a custom made OLED screen (128x64), which looks very cool (brighter and sharper than some I've used), its extremely compact for an ESP32 board...and finally...its WHITE(!).

So long story short, I flashed it couldn't communicate with it. I would type something and get garbage back.

rst:0x1 (POWERON_RESET),boot:0x17 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:2364
load:0x40078000,len:0
load:0x40078000,len:10880
entry 0x40078c4c
ž›#$„|óBnàän΃oãäûooNrƒNNŒooŸlnî‚N“oâ|‚“‚œl’ƒ’`rlïoÿäŽp’`䛲’ìd ð䓒’ì’’첒äòû’’ì²’$`ðìÛòlb„"äŒB“`ðò“›²ò›Ûì2òû’’òó2ò’› ódœ|€c²Ÿä|óocp~’nd$Œ¾ãäbpƒbpc„lä|ì’äpƒoì’N|sdrl{"’ooNln|äl“ãoä“nò#r’œbìdû„ì¾BrNlo|ä’òN|›‚œl~ãoNüŒpãäƒoãŽìoœänܜänàp„óoà„‚slâNÜœNNà2oNn|ìl `ƒ‚žbìccãŽcÛNnocp~’ncNnà²œlrdü$~ócocBnbcNldBnc$$ll#lll# 

I suspected a baud rate problem but none of the baud rates in the Web IDE worked. I eventually brought up ESPlorer which is a JAR file (can be found online), and had success connecting at 74880.

After I had to go through the Espruino Web IDE code to figure out to change this line:

  function init() {
    Espruino.Core.Config.add("BAUD_RATE", {
      section : "Communications",
      name : "Baud Rate",
      description : "When connecting over serial, this is the baud rate that is used. 9600 is the default for Espruino",
      type : {9600:9600,14400:14400,19200:19200,28800:28800,38400:38400,57600:57600,74880:74880,115200:115200},
      defaultValue : 9600,
    });

...and it works now.

Connected to port /dev/tty.SLAB_USBtoUART
>*«PªV¡¡ET¡Q«UU«T¢Ô-«QR©š«Tš«ªRUJªªV¥Z¹j¡ªTÕJ¨R®J¢•®Ô5TªT©µUµ­Å-QBUª«j”ªªEJªR­Z¹QÊWÕ­Q(©uT«Z¹QJ-]ªZ‹,QRªR­Z¹QÚ¬R­Z¹QQT®UZª¦B)µ©…ªÕ­%µ(Õªª(¥©UVK%µ(Õªª(¥J©jZU•%µ(ՔTQeQRªÕTTªT‹,EÔYQ©UV+QYEj´«P±EÔYJ¥QüWARNING: has simple connection not implemented yet
 ____                 _
|  __|___ ___ ___ _ _|_|___ ___
|  __|_ -| . |  _| | | |   | . |
|____|___|  _|_| |___|_|_|_|___|
         |_| espruino.com
 1v97.114 (c) 2018 G.Williams
Espruino is Open Source. Our work is supported
only by sales of official boards and donations:
http://espruino.com/Donate
>
=undefined
>
=undefined
> 

I've flashed quite a few ESP32s (dozens?), and this is the first time I've come across this situation.

I am curious how Espruino sets its console communications baud rate, and as we can see above when I had the wrong baud rate, the module would boot at 115200 baud, showing me power on information, then I assume switch to 74880 and produce garbage.

Additionally this brings up the question of if we should add "custom" baud rates to the Web IDE console. I know the ESP32 can communicate at up to 921600.

NOTE: This issue occurred through 1v97 to 1v99. I'm flashing to 1v97 here because of the neopixel problem in subsequent releases.

-hfc

When Espruino is running on ESP32, it should set the baud rate to 115200 regardless of the board it runs on - if it isn't something is going extremely wrong indeed.

Rather than trying to add a non-standard baud rate to the IDE, I think it'd be better to figure out why this board isn't working correctly and fix that (since presumably it means that setting the Serial port's baud rate to any value will be broken).

If anyone else has this issue you can actually just type http://www.espruino.com/webide?settings={"BAUD_RATE":74880} into your browser's address bar and it should make the IDE display a prompt asking you if you want to change the rate. It won't show up in settings, but it should have the desired effect.

Hi,

I did a bit of goggling - like at the 2nd post here - looks like your board might have a different frequency crystal -

https://forum.sparkfun.com/viewtopic.php?t=46628

@Gordon, thanks, thats very helpful -- I'll try that in the future, and make a note of it.

@Wilberforce -- thats funny you mention that -- the WiFi doesn't return any values on this board it turns out. There are no errors, but everything like wifi.scan() etc returns null, and its unable to find any APs. Subsequently it can't connect to networking. This is in Espruino. However I flashed it with some test programs on Arduino, and the wifi is working fine there. It can scan and find base stations. I had originally thought that it was because the board is running at 240 mhz, rather than 160mhz which is set in the ESP32.py board definition, but after reading your post...

Is there anything in the ESP32 code that is timing dependent with regard to the wifi?

There's a parameter in the Espressif IDF that can set the crystal frequency, and apparently has an "autodetect"setting. Its mentioned that Arduino also "autodetects" the crystal frequency. I have no idea how thats done.

https://github.com/espressif/esp-idf/issues/1351

The above thread actually deals exactly with this Heltec board. Unfortunately my Espruino SDK is built using a downloaded "prebuilt" IDF for OS X. In theory I could recompile it, with the right flag ("esp32-xtal-freq-sel") from scratch, but I'd have to look at what that would involve, so I cannot test this at the moment.

@Wilberforce So just to wrap up this topic for posterity, after spending an evening investigating, I found a couple of things.

First of all the board does work with 1v97 and earlier. The reason it did not work before on 1v97 was I was using a later build (non release), that had Bluetooth. The non Bluetooth version works fine.

When investigating the difference between Arduino and Espruino, I saw that in sdkconfig.h (located in "app/build/include" in the Espruino SDK), the config is set to:

"#define CONFIG_ESP32_XTAL_FREQ 40"
"#define CONFIG_ESP32_XTAL_FREQ_40 1"

Where as in the Arduino sdkconfig.h the settings are:

"#define CONFIG_ESP32_XTAL_FREQ 0"
"#define CONFIG_ESP32_XTAL_FREQ_AUTO 1"

The official docs from Espressif say you can have 40mzh, 26mhz, 24mhz, or "auto".

I changed the settings actually and did a few recompiles and flashings, none of these made any difference. The baud rate on my board was stil 74880, and the radios didn't work. Wifi returned nothing, and Bluetooth worked maybe 10% of the time. I can't confirm that Bluetooth worked on 1v97, obviously, as the non Bluetooth build was the only one which seemed to function "normally".

So adding the Bluetooth code definitely is the marker in the build progression where this started. While for sure it is just one board, as we saw there is also at least a Sparkfun ESP32 module that has a 26mhz crystal. Maybe something to look at when everyone has time, I know there are alot of other priorities in this great effort you and JumJum are doing. I am curious what the change was with adding the Bluetooth code.

I'll see if I can do a build with the setting at auto at some stage for you to test... not sure when.

You can use ESP32.enableBLE(false) and see if that make a difference on the recent builds. Possible it's a using both radios at once issue.

I changed the settings actually and did a few recompiles and flashings, none of these made any difference

Changing the include in the sub folder and compiling won't make a diffierence - it's the sdk libs that would be picking up this setting - so in our case it's the espruino build tools project that needs to get built and then linked too.

Found a few minutes so did a pull request here:

https://github.com/espruino/Espruino/pull/1493

Sdk is now at 3.03 and the XTAL is set to auto. When the travis build is done you'll be able to download and try the new firmware.

The build failed - not had a chance to look at it - sorry.

Wow thats amazing -- I'm ready to try it out -- too bad it failed. :/ What was the problem I'm just curious (when you get around to it).

Yeah -- I did suspect that changing the settings in sdkconfig.h wouldn't do anything, but it was worth a shot right? :P And I learned about the ESP32 SDK which caused me no harm ;)

Yes I did try the ESP32.enableBLE(false) -- it didn't change anything. The problem is really deeper I believe.

Ah, I see see it seemed to have failed somewhere at the linker phase. Anyways if you get it going I'll for test it out also on a few different ESP32 boards, so we can see if it works universally.

Only FYI,
I would like to automate ESP-IDF make to Espruino Make
In a first step all .mk files in template folder are searched and converted to an additional make file
This is a first step only, but may be, its of interest
Sorry, but I'm in vacation on an island right now, and don't have time to work on this for about a month.

@hungryforcodes

Please try this build:

https://www.espruino.com/binaries/travis/wilberforcepatch11/espruino_esp32.bin

You should see v3.0.3:

>ESP32.getState();
={
  sdkVersion: "v3.0.3",
  freeHeap: 37848, BLE: false, Wifi: true, minHeap: 34612 }

You might also need to load the partition table.bin as this was changed going to v3.0x

I know there are alot of other priorities in this great effort you and JumJum are doing. I am curious what the change was with adding the Bluetooth code.

We moved from the 2.x esp-idf to the 3.0x esp-idf. This changed a lot around Makefiles and things. We have no clue to why the neopixel code that was working with 2.x is not working with the 3.0x libs.

I would like to automate ESP-IDF make to Espruino Make

This is exactly what this does:
https://github.com/espruino/EspruinoBuildTools/tree/master/esp32

This is a series of scripts, that build the app and esp-idf folder that espurino uses for the ESP32 build.

source script/provision ESP32

This is used by Travis, or by anyone wanting to build from source. This downloads the toolchain, and adds the two folders above.

I suggest that you clone the EspruinoBuildTools repository and do the build to see how it works.

In the EspruinoBuildTools/esp32/build/build-idf.sh script, there are variables for changing what version you want to build:

esp_idf_branch=${1:-v3.0.3}
espruino_branch=${2:-master}

OK this is super interesting, especially the updates to the provision script (I really love this script, btw). I manually built the IDF the other day on OS X by "hand", but kind of stopped after I got a basic app template. There was alot to configure from scratch - and actually looked like "work" as opposed to "fun" :P

Let me take sometime this weekend to review, and try the code on the new board. I'll report back :)

The provision script assumes Linux- you might want to adapt to detect OS X and get the correct gcc for that - it would not be too hard to do.

OK that was a good tip. There were a couple of strange things such as an "apt-get: command not found", which is normal on OS X, but I didn't see where this was coming from -- I could not find it in the build-idf.sh

For the rest it compiled fine! I guess its because I had already installed alot of the requirements from this page when I built it last time:

https://dl.espressif.com/doc/esp-idf/latest/get-started/macos-setup-scratch.html

The weirdness I got was:

WARNING: Toolchain version is not supported: 1.22.0-73-ge28a011
Expected to see version: 1.22.0-80-g6c4433a
Please check ESP-IDF setup instructions and update the toolchain, or proceed at your own risk.
rm: app: No such file or directory
rm: esp-idf: No such file or directory
make: *** [app.tgz] Error 1
../esp-idf/components//app_trace/include/esp_app_trace.h: Path contains '..'
../esp-idf/components//app_trace/include/esp_app_trace_util.h: Path contains '..'
../esp-idf/components//app_trace/sys_view/Config/Global.h: Path contains '..'
../esp-idf/components//app_trace/sys_view/Config/SEGGER_RTT_Conf.h: Path contains '..'
../esp-idf/components//app_trace/sys_view/Config/SEGGER_SYSVIEW_Conf.h: Path contains '..'
../esp-idf/components//app_trace/sys_view/Sample/OS/SEGGER_SYSVIEW_FreeRTOS.h: Path contains '..'
../esp-idf/components//app_trace/sys_view/SEGGER/SEGGER.h: Path contains '..'
../esp-idf/components//app_trace/sys_view/SEGGER/SEGGER_RTT.h: Path contains '..'
../esp-idf/components//app_trace/sys_view/SEGGER/SEGGER_SYSVIEW.h: Path contains '..'
../esp-idf/components//app_trace/sys_view/SEGGER/SEGGER_SYSVIEW_ConfDefaults.h: Path contains '..'

pages and pages of that.

Anyways so the results! Good and bad!

Basically the build you made for me didn't work. I assume it had the auto detect frequency settings like you had said (and I saw this in the IDF config). It had the same problem as all the other builds.

So now with my handy handy build-idf.sh script, I was able to compile it with the crystal set to 26MHZ -- and it worked! Wifi radio is fine, Bluetooth can find things easily. Baudrate is 115200 :P

So thats good and bad, because its good for me, but not good for putting out one build for all crystal types. I have to run out now, but I will try a few different things later to see if I can find out why the autodetect didn't work. I can just try building it again with those settings actually, I didn't try it yet, just having used the one you offered me.

I also have contacts here at Espressif (I'm based in Asia at the moment), and could try and ask them if I want to go deeper I guess.

So for me -- great! Thank you so much.

As for generally building the OS X version, I'll review the info on the on Espruino about building, and see if there is anything that should be changed. It would be ideal to run it on a "virgin" Mac, that hadn't had this stuff built on it before to see where it could go really wrong.

It would be interesting with your build to set it to auto to see if that sticks and it works of the board.

I need to check the sdkconfig and ensure that auto is set.

https://github.com/espruino/EspruinoBuildTools/blob/master/esp32/build/app/sdkconfig#L210

Yes - it is.

I'm wondering if the change would actually be in the bootloader.bin - did you update that when you flashed?

Actually thats funny, because the link you gave me above was a direct download to espruino_esp32.bin. Just the one file. There was no bootloader.bin. It could well be thats the case. If you can provide me a link to your bootloader.bin I can try it out. I didn't recompile mine yet.

So good news! I finally recompiled it on my side with "auto" and....da, da...it worked!

So you may well be right about the "bootloader.bin" situation. It would still be good to get a compile from you as mine was on OS X, but I'd be surprised if they weren't the same.

I also flashed this on a few other boards (that all worked before, so I assume 40mhz), and they're fine, radios etc.

So I think we nailed this one :)

I'll test a few more boards over the week as I do some development work.

That's great news - I might need to order this board.
I'll need to check that the boot loader.bin that gets saved into the tar file in espruinobuildtools is the latest built version. The files are the same size so might have to run a checksum to check it is the latest - or check the file date.

@Wilberforce
to be open, I don't understand your point and need some advice.
What I would like to automate is one part, which always happens to me after we get a new version.

• structure of components is different to older versions.
  

Therefore we need to include some more headerfiles, libraries and binarys. And remove some others. All of these changes are done in espruino/make/familiy/esp32.make.
Usually I've to search a lot to figure out these changes.
My idea now is, to use all .mk files which are available after compiling components.
Please see attached file, which is a first step to replace a lot assignments to INCLUDE,LIBS and LDFLAGS
If my understanding of your tool is totally wrong, and you are already doing this, please help me out.

1 Attachment

• addComponents.mk

Ahh - I understand what you mean now.

This sounds like a great idea. If you can work out what changes ares required to the espruino esp32 build part, I can add the file to the build tools so that it will be available - is it only addComponents.mk I need to add to the tar? Which path is it in?