Status of Bluetooth for ESP32

I spend lot of days to get Bluetooth up and running in a very first simple step
So is time to give a first status.
To be open, I'm not sure, we will ever get it running in a nice way.
Memory consumption in binary file and in RAM is huuuuge

This is what I did in some short steps:

1. got the example from esp-idf for UART-Server running
   
2. got connected from my mobile, using nRF Toolbox
   
3. rebuilt source and added to Espruino/targets/ESP32
   
4. got huge binary, up to 1400KB
   
5. created a new partition table to support this big application size
   
6. flashed and got "no heap" and reset, reset, ....
   
7. changed config and removed classic Bluetooth and some other memory eater
   
8. tons of problems, sometimes even without any error message
   
9. had to reduce jsvars from 5000 down to 2000 and now its working
   
   

There is still no functionality available in Javascript, and UART is receiving data only.
Question is, should I spend more time on this, or wait for availibility of breadboards with ESP32 wrover which has additional PSRAM.

Hi @JumJum
Well at least it is progress of sorts. Which ever way you go, it would be great to capture what you have learnt, so we can get back to this point later...

So, perhaps start a new branch esp32-ble and put the espruino changes under that, and also a new branch under espruino build tools with the same name, that will capture the partition changes and sdkconfig changes.

I can get this all working with Travis if you like - then if we want to return to this point, or someone else want to pick up the gauntlet, we have a starting point.

I think the bluedroid libraries might be quite large - but at the end of the day there is plenty of flash, it's the ram there the issue.

It would be great to get it working, as a ble to wifi bridge would be awesome.

Contact me on gitter if you would like some help.

Actual status of my work for Bluetooth is here https://github.com/espruino/Espruino/tree/ESP32/targets/esp32/docs/Bluetooth_first_attempt
@wilberforce, tried to contact you on gitter, without success.

Just tested mem release function for BLE and got about 30k back.
Will do some more testing, but looks like we will get up to 2000 vars back :)

BTW, after some chatting with Gordon, I'll try to come closer to already existing Bluetooth in Espruino (for Puck.js). World is different between nrf and esp-idf, so this will take some time.

I was able to get the ESP32 branch of Espruino to build with functional BT but I had to modify line 45 of targets/esp32/bluetooth.c

if(ret) {jsWarn("mem release failed:%x\n",ret); /*return;*/}

It always fails trying to release ESP_BT_MODE_BTDM and if it returns none of the following initialization gets executed.

Is there any newer version of this? It seems like a work in progress...

First of all, BLE is work in progress. From time to time changes are uploaded.
At same time, esp-idf is changed very often.
Since this is my hobby only, please don't expect a full blown up implementation soon.
Your problem seems to be one of those changes in esp-idf. Sequence of calls during setup (and params for release ??) changed.
You could try this

void jsble_init(){
	esp_err_t ret;
    
	ret = esp_bt_controller_mem_release(ESP_BT_MODE_CLASSIC_BT);
	if(ret) {jsWarn("mem release failed:%x\n",ret); return;}
	
	esp_bt_controller_config_t bt_cfg = BT_CONTROLLER_INIT_CONFIG_DEFAULT();
    ret = esp_bt_controller_init(&bt_cfg);if(ret) {jsWarn("initialize controller failed:%x\n",ret); return;}

	ret = esp_bt_controller_enable(ESP_BT_MODE_BLE);if(ret) {jsWarn("enable controller failed:%x\n",ret); return;}
	
	ret = esp_bluedroid_init();if (ret) {jsWarn("init bluetooth failed:%x\n",ret);return;}
    
	ret = esp_bluedroid_enable();if (ret) {jsWarn("enable bluetooth failed:%x\n",ret);return;}
	
	ret = esp_ble_gap_register_callback(gap_event_handler);if (ret){jsWarn("gap register error:%x\n", ret);return;}

	ret = esp_ble_gatts_register_callback(gatts_event_handler);if(ret){jsWarn("gatts register error:%x\n", ret);return;}

ret = esp_bt_controller_mem_release(ESP_BT_MOD­E_CLASSIC_BT);
 if(ret) {jsWarn("mem release failed:%x\n",ret); return;}

This version does work. I left the ret = esp_ble_gatt_set_local_mtu(500); section in place at the end.

Thanks--
Dave

I've abandoned Espruino, its performance is just insufficient for my application. I've switched to a duktape javascript engine.

https://github.com/nkolban/duktape-esp32

Their bluetooth implementation hasn't been fleshed out. I think your work on the Espruino would translate well.

Has been a long time since last push to github, but today its done.
I'm a lazy guy in documentation :-(
Let me copy commit message here:

First of all, thanks to Gordons help in SW, discussions, hints, .... …
…and last not least 2 PUCK's
Without Gordons support, this port would have never happened.
General:

• Status from my point of view is still early.
  
• Server, Client, Advertising can be used, and are running at least with a Puck or NRF-connect on the other side
  
• waiting for ESP-IDf V3, and waiting and....
  
• Bluetooth needs much memory, size of binary is 1172 right now
  
• vars need to be reduced to 2000, for boards with wrover module this can be extended to 20000 or more
  
• some minor but helpful changes in core sources of Espruino
  
  

Added:

• GATTC support
  
• Advertising
  
• support malloc for vars, instead of static allocation
  
• use of additional 4mb PSRAM (if available) to have 20000 vars
  
• support lists in ifdef for Javasript wrapper, so "ifdef" : "NRF52,ESP32" are possible
  
• some more documents, should be more, but I'm a lazy guy :-(
  
• new manual for install
  
• lot of internal changes to make it more C-style
  Open items
  
• much more testing and testers
  
• no support for BlueTooth UART
  
• several NRF commands are not supported
  
• reset of Bluetooth and UART in jshreset crashes from time to time, therefore UART reset is switched off
  
• discuss how changes in core sources, mainly lib/bluetooth/* can be reduced
  
• some people offered help in forums, good time to start now ;-)
  
  

Changes to be committed:

modified:   boards/ESP32.py
modified:   libs/bluetooth/bluetooth.h
modified:   libs/bluetooth/bluetooth_utils.c
modified:   libs/bluetooth/jswrap_bluetooth.c
modified:   libs/bluetooth/jswrap_bluetooth.h
modified:   make/family/ESP32.make
modified:   make/targets/ESP32.make
modified:   scripts/build_platform_config.py
modified:   scripts/common.py
modified:   src/jsvar.c
modified:   src/jsvar.h
modified:   targets/esp32/BLE/esp32_bluetooth_utils.c
modified:   targets/esp32/BLE/esp32_bluetooth_utils.h
modified:   targets/esp32/BLE/esp32_gap_func.c
modified:   targets/esp32/BLE/esp32_gap_func.h
modified:   targets/esp32/BLE/esp32_gattc_func.c
modified:   targets/esp32/BLE/esp32_gattc_func.h
modified:   targets/esp32/BLE/esp32_gatts_func.c
modified:   targets/esp32/BLE/esp32_gatts_func.h
modified:   targets/esp32/bluetooth.c
modified:   targets/esp32/docs/Bluetooth/sdkconfig
modified:   targets/esp32/jshardware.c
modified:   targets/esp32/jswrap_esp32.c
modified:   targets/esp32/jswrap_esp32.h
modified:   targets/esp32/main.c

gut Ding will Weile haben... congratulations! - auf die Schnelle lebt nicht lange!

That's great news!

Just for anyone wondering, the firmware is here: https://github.com/espruino/Espruino/tree/ESP32

This looks really good - and it'll build with the current ESP-IDF?

I'm less sure about adding a totally ESP32-specific variables_psram to the board.py file, but that's extremely minor :)

@Wilberforce obviously the available memory for vars on normal boards has really dropped, but is there any reason why it'd be a bad idea to merge this?

Firstly, well done @JumJum, I know that is has been a mammoth task to get this far!

At the moment there are dependencies that would also need to be added to the espruino build tools... for example there is a new partition.bin, however I think we can make one partition that could be used for both, as it just makes the firmware partition bigger, and we could get rid of the ota partition as that is not implemented (it's a good idea in principle but not sure about the practicality- that is a different issue). The ble libs might also need to be added, I have not yet tried to build this. If I get a chance over Easter I might be able to do a pull and see what happens.

I'm not too keen on dropping the number of vars in the esp32 normal build. In a conversation I had on gitter with @jumjum I don't think it has yet been tested with the Bluetooth turned off in the Makefile.
However, @jumjum I understand has made this work using Malloc for the jsvars, so perhaps it can detect which build and allocate for:
Wifi only
Wifi and ble
Wifi and ble and psram.

For example at the moment there are changes to main.c that could have conditional includes and some of the intialisation code could be in conditional includes.

I'm not sure how much more memory the ble stack uses, and if it only uses the memory when it is turned on. If that was the case we could have a run time switch to turn in on just one build.

Gordon, what is the best way to handle this - have two board files - one with wifi and one with wifi and ble? Is there another board that handles things like this that way? I know for the esp8266 there are two builds, one for the 4mb flash, the other for 512.

It doesn't look like Travis has run against the build - I think that happens when a pull request occurs. I think it will break because the dependencies will need to be updated in the build tools. Might need to do an esp32-ble branch there to check against.

@jumjum - I see the sdk config changes - is this for ESP-idf 2.1 still?

I've pulled the ESP32 and attempted a compile. It errors out with:

xtensa-esp32-elf-gcc: error: unrecognized command line option '-mfix-esp32-psram-cache-issue'

make clean && BOARD=ESP32 RELEASE=1 make
Cleaning targets
Generating platform configs
Generating pin info
Generating JS wrappers
WRAPPERSOURCES = src/jswrap_array.c src/jswrap_arraybuffer.c src/jswrap_dataview.c src/jswrap_date.c src/jswrap_error.c src/jswrap_espruino.c src/jswrap_flash.c src/jswrap_functions.c src/jswrap_interactive.c src/jswrap_io.c src/jswrap_json.c src/jswrap_modules.c src/jswrap_pin.c src/jswrap_number.c src/jswrap_object.c src/jswrap_onewire.c src/jswrap_pipe.c src/jswrap_process.c src/jswrap_promise.c src/jswrap_regexp.c src/jswrap_serial.c src/jswrap_storage.c src/jswrap_spi_i2c.c src/jswrap_stream.c src/jswrap_string.c src/jswrap_waveform.c libs/filesystem/jswrap_fs.c libs/filesystem/jswrap_file.c libs/math/jswrap_math.c libs/graphics/jswrap_graphics.c libs/network/jswrap_net.c libs/network/http/jswrap_http.c libs/network/js/jswrap_jsnetwork.c libs/network/jswrap_wifi.c libs/network/esp32/jswrap_esp32_network.c targets/esp32/jswrap_esp32.c libs/network/telnet/jswrap_telnet.c libs/crypto/jswrap_crypto.c libs/neopixel/jswrap_neopixel.c
DEFINES = -DGIT_COMMIT=46d6a12 -DNO_ASSERT -DRELEASE -DBUILDNUMBER="40" -DESP_PLATFORM -DESP32=1 -DUSE_DEBUGGER -DUSE_TAB_COMPLETE -DUSE_HEATSHRINK -DUSE_FILESYSTEM -DUSE_FLASHFS -DUSE_MATH -DUSE_GRAPHICS -DUSE_NET -DUSE_ESP32 -DUSE_TELNET -DUSE_CRYPTO -DUSE_TLS -DUSE_AES -DUSE_NEOPIXEL -DESP32 -DEMBEDDED
CC libs/compression/heatshrink/heatshrink_encoder.o
xtensa-esp32-elf-gcc: error: unrecognized command line option '-mfix-esp32-psram-cache-issue'
make: *** [libs/compression/heatshrink/heatshrink_encoder.o] Error 1

So it looks like a new tool chain is required.

@jumjum Are you using ESP-IDF Pre-release 3.0-rc1 ?
https://github.com/espressif/esp-idf/releases

Shall I start an issue on git hub so we can continue this conversation there?

Yes, it's probably better to chat about this on GitHub.

If we're going to have a build with Bluetooth and one without it's probably better with 2 board files. I do have 2 builds for some Espruino boards (eg. the Original with CC3000 or WIZnet support) but I'd rather not repeat it since it involves hacks in the script that creates the distribution zip. We could modify the build to include the extra parameters in the BOARD.PY file for all boards that did this, but it's not great.

I'd still really advise against splitting the build up though. 2000 vars is still quite a lot (and I'm sure that can be increased again as time goes on), and I doubt many people will hit it as a limit especially with the save on send functionality that most seem to use now.

@wilberforce, I'm on latest on github, not on pre release.
There are a lot of changes since that release.
Latest download on my side is some days ago.

Yes, lets use github for more

Most of sdk config changes are not available in ESP 2.1

I will not fight for OTA, its a big idea, but at the end, there are a lot of limitations.
For example, I would not like to load a new binary over Bluetooth ;-)

Having a lot of jsvar sizes based on used hardware is something which needs to be checked.
ESP-IDF is complex in that point. In actual version, BLE is always started, as WIFI is.

Following some discussions around ESP8266, I would prefer to have one binary only. Otherwise a lot of confusion could appear, "which binary is running on which board ?".

Checked compiling with out Bluetooth and ran into problems.
A lot of ifdef need to be added to main.c, jshardware.c, .....
Working on that now

Compiling without Bluetooth is working.
Heap size with BLE : 76500
Heap size without BLE: 54500
Means we could have 1350 jsVars more in a version without Bluetooth

Next tested with default of 3000 jsVars. Starting and some first small tests work fine.
I would be happy with 3000 jsVars, anyway others could require more.
My advice in that case would be to use a WROVER board which could have much more

That looks great - in fact you might be able to push the vars up to 3300? 55k heap is still a lot.

I'd say given the choice most people would happily drop those vars to have Bluetooth. People manage pretty well with the available memory on normal Espruino boards, which is around that.

Oh, sorry, there was a mistake on my side. Free heap is swapped.
I would stay with 3000 jsVars.
ESP32 uses a lot of heap during startup, and I would prefer the safe side.

HI Team

I am working on project which need on board wifi and Ble support
i am using ESP32 for the same .

do we have espruino support for the both on ESP32?

We are (still) waiting for next release of ESP-IDF.
You could use branch ESP32 of Espruino on github, which already includes a lot of BLE support.

Hello, I was looking into how to test BLE on the ESP32, but it looks like the firmware is bigger than the allowed space on the ESP32. If any of you could direct me on how to properly apply a custom partition table with the Espruino firmware I'd appreciate it. I would love to test this and update the documentation, as there is none currently.