Multiple ESP32 targets - the future for the Espruino Build structure

I think it would be usefull to share ideas between us on how we see the ideal shape of the Espruino codebase and build structure for the multiple ESP chips (targets) in the future. Ive got a few thoughts to start.

I will cut to the chase and suppose that a good end position is to have all ESP targets on one version of the ESP-IDF and that all build variances of the targets configured within the ESP-IDF sdkConfig (The SDK project configuration) mechanism.
(I suppose this may be obvious ... but worth stateing :)

With a common code base for all ESP targets (eg jshardwareESP.c ) using minimal #idfefs. And implementing a method in the boards.py file to specify the config file to be used in the IDF.py build.

See this readme On the espressif mechanism for building for multiple configurations.

Ive also used Chat GBT to answer this question.

when building for multile targets how can the idf.py parameter called -SDKCONFIG_DEFAULTS simplify the application structure

The idf.py parameter --sdkconfig-defaults can simplify the application structure when building for multiple targets by allowing you to maintain a base configuration that's shared across all targets, while still accommodating target-specific configurations. This approach reduces redundancy and keeps your project organized.

How --sdkconfig-defaults Works

The --sdkconfig-defaults option in idf.py allows you to specify one or more files that contain default configuration options. These files are used as a base when generating the final sdkconfig file for the build. You can layer multiple configuration files on top of each other, enabling you to define a base configuration that's common across all targets and then overlay target-specific settings.

Simplified Application Structure

Here's how you can use --sdkconfig-defaults to simplify your project structure:

1. Base Configuration for Common Settings

Create a common configuration file, say sdkconfig.defaults, that includes all the settings that are common across all targets.

Example sdkconfig.defaults:

   CONFIG_FREERTOS_UNICORE=n
   CONFIG_PARTITION_TABLE_SINGLE_APP=y
   CONFIG_EXAMPLE_WIFI_SSID="MySSID"
   CONFIG_EXAMPLE_WIFI_PASSWORD="MyPassword­"

2. Target-Specific Configuration Files

For each target, create a separate configuration file that includes only the settings specific to that target.

Example sdkconfig.esp32:

   CONFIG_IDF_TARGET_ESP32=y
   CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM=1­0

Example sdkconfig.esp32s2:

   CONFIG_IDF_TARGET_ESP32S2=y
   CONFIG_ESP32S2_WIFI_STATIC_RX_BUFFER_NUM­=12

Example sdkconfig.esp32c3:

   CONFIG_IDF_TARGET_ESP32C3=y
   CONFIG_ESP32C3_WIFI_STATIC_RX_BUFFER_NUM­=8

3. Building with --sdkconfig-defaults

When you build for a specific target, you specify both the common defaults and the target-specific configuration file. This allows the build system to generate an sdkconfig that combines both configurations.

Example for ESP32:

   idf.py set-target esp32
   idf.py build --sdkconfig-defaults sdkconfig.defaults --sdkconfig-defaults sdkconfig.esp32

Example for ESP32-S2:

   idf.py set-target esp32s2
   idf.py build --sdkconfig-defaults sdkconfig.defaults --sdkconfig-defaults sdkconfig.esp32s2

Example for ESP32-C3:

   idf.py set-target esp32c3
   idf.py build --sdkconfig-defaults sdkconfig.defaults --sdkconfig-defaults sdkconfig.esp32c3

Benefits

• Reduced Redundancy: Common settings are centralized in sdkconfig.defaults, reducing the need to duplicate them across multiple target-specific sdkconfig files.
  
• Cleaner Project Structure: Your project remains organized with clearly defined configuration files for common and target-specific settings.
  
• Easier Maintenance: When you need to change a common setting, you only need to update sdkconfig.defaults, and the change will automatically apply to all targets.
  
• Flexibility: You can easily add new targets by creating a new target-specific configuration file and using it with the existing base configuration.
  
  

Example Directory Structure with --sdkconfig-defaults

my_project/
├── CMakeLists.txt
├── main/
│   ├── CMakeLists.txt
│   ├── main.c
├── components/
│   ├── component1/
│   │   ├── CMakeLists.txt
│   │   ├── component_code.c
│   └── component2/
│       ├── CMakeLists.txt
│       ├── ...
├── sdkconfig.defaults        # Common settings
├── sdkconfig.esp32           # ESP32-specific settings
├── sdkconfig.esp32s2         # ESP32-S2-specific settings
├── sdkconfig.esp32c3         # ESP32-C3-specific settings
└── CMakeLists.txt

By using --sdkconfig-defaults, you streamline the process of managing configurations across multiple targets while keeping your project tidy and maintainable.

However the current state is that the newer ESP targets ( eg C3 and S3 ) which are being slowly worked into Espruino. Are each needing more recent versions of the ESP IDF and hence the current development of the builds are relying on a different version of the SDK and the common code to use #ifdefs to id the corrrect sdk functions for the version.

I suggest that this is contributing to a very messy and difficult to test code base for all the devices and will frustrate the move to a common version of the ESP IDF.

So I suggest with the above approach (and end game) in mind (and openning the discussion on that) we refocus on moving all devices to an agreed latest version of the IDF. And we do that with a new development branch to Espruino, a common code base for all the ESP targets, and start using the config files to manage the build variances.
??? so ideas and builds please , thanks

@Gordon, @rgomezwap , @MaBe , @JumJum , @fanoush, @Jean-Philippe_Rey, @tve
Just bringing the above to your attention, and would appreciate the bennefit of your experience on an approach (Even if your not able to help moving forward)
Regards, Simon

Ps @rgomezwap I’m not suggesting you abandon your current efforts with the #ifdef approach. I suggest we build on your work in a new IDF v5 development branch and take out the old code with the #ifdefs. We would need to become masters of the espesssif config file system.

Hello,

that's a lot of information, and let's wait to hear what the others have to say about it. As for me, I wasn't aware of --sdkconfig-defaults. It seems like a good idea.

Currently, the important configurations for Espruino in sdkconfig are not well managed with this type of file. We have three files, and it's not easy to see what's important in them.

The --sdkconfig-defaults could make things easier.

Not to mention that with ESP-IDF v5.x and the new P4 chip (which I'm waiting for), more sdkconfig files will be generated.

So, as I understand it right now, we have some old IDF for the basic ESP32 build as we did before, and now we have 4.4.8 for the C3 and S3?

I'm all for attempting to move the original ESP32 build over to 4.4.8 too

However I have very limited time right now, and the choice to allow the _C3 build with IDF4 while keeping the standard ESP32C3 as before was done because I didn't have time to test ESP32 and ensure that swapping over to IDF4 didn't break something (because it probably will!).

We still seem to be in a position where I think the C3 build isn't connecting over WiFi, so ideally before we start messing around and breaking things even more, I'd ideally like to be in a state where everything's kind of working ok.

I feel like once we have that, building a sdkconfig.esp32c3/etc by diffing what we have with the default file should be pretty straightforward if we then want to go that route.

ok , I agree there is stuff broken in the ESP 32 targets at the moment (although @rgomezwap work is moving it forward greatly). I suppose what im saying is how to fix it. Saying it a different way , my proposal is to Invest now, start a fresh and , build a new 5.1 (say) idf sdk espruino port to drive all the current esp targets. A clean slate, without the #idefs for each of the current idf variants and test it once for each target. Using @rgomezwap work, the Arduino core and the micropython /circuit pythons esp32 library as guides for this build.

This approach is as opposed to moving the original esp32 to 4.x (persisting with the complication of TARGET/IDF VERSION ifdefs) , testing it and then later moving , it and other variants to 5.x , testing again etc.

My understanding (and i could be wrong and im learning from scratch here) is that one version of the application and using the config files for the target differences is how the ESP IDF is intended to be used , using macros and #ifdefs defined from the config files for the variants.
(anyone know otherwise ??) see here how circuit python (running on idf v5.3) sets up the config files. see attached also.

but im still not sure, if this is correct, or if i can make an impact (and there does not seem to be many others with the time/inclination to work on this. @rgomezwap excepted. )

So I am currently trying to prove my understanding above AND understand what a compilable vanilla version (eg all peripherial jswraps pointing to empty functions) of Espruino looks like for the basis of the clean sheet* (eg I got yesterday a cflow report of the function calls from app_main)
and then looking at other applications like micro-python , circuit python and esp32 Arduino core to try to understand their approach. And also a good espruino/esp32 test set is needed !?

• if anyone has a vanilla espruino version , I would be great full to see it ( i do see the headers in jshardware.h as a clue. )
  
  

1 Attachment

• cflowReport.txt

sorry mixed up my attachements , above is my cflow report . Attached here is example from circuit python on using config file parameters in the build (I will try to understand how pervasive this is in Circuit Python)

1 Attachment

I see that more or less we are all on the same path.

However, it's true that for the rest of the community, it's important to have a version of ESP32 IDF 3.x with the latest version of Espruino available. This way, specific apps can continue to progress without new errors.

I believe that once we have a fairly functional ESP32XX with ESP-IDF 4.4.8, we can start cleaning up and trying to rebuild the build process in a more modular and simpler way.

But we still have many things to test. I’m focused on the ESP32S3, and I’ve put the ESP32 aside for now. I don’t use the C3. We need users for each of the three versions to compare the results.

I think there's a post on this earlier, but the big issue with moving to IDF5 is the network connection handling - IDF4 introduced a new API but still had a backwards compatibility layer, but IDF5 gets rid of that layer so you'd have to completely rewrite Espruino's handling. I took a look initially but it appears to be a decent chunk of work, which is why I stuck with IDF4 for now.

start a fresh and , build a new 5.1 (say) idf sdk espruino port

So you're suggesting basically copying the entirety of https://github.com/espruino/Espruino/tre­e/master/targets/esp32 and starting again?

without the #idefs for each of the current idf variants

I'm pretty sure there will always be IFDEFs needed for each hardware variant - certain hardware peripherals will be present in some but not others, or in different places. So as an example:

• You'd be able to get rid of https://github.com/espruino/Espruino/blo­b/5f3f6782adc4fe4535c76a802b6b3b0fa26bbe­9b/targets/esp32/jshardwareAnalog.c#L63-­L69
  
• But https://github.com/espruino/Espruino/blo­b/5f3f6782adc4fe4535c76a802b6b3b0fa26bbe­9b/targets/esp32/jshardwareAnalog.c#L45-­L49 would always have to be there
  
  

But personally I feel like given the thousands of lines of code in the esp32 port, there are actually very few ifdefs needed to move to IDF4. And the ones that are could be improved greatly.

For example, if we added a idf3_compat.h file with:

[#define](https://forum.espruino.com/sea­rch/?q=%23define) ADC_ATTEN_DB_0 ADC_ATTEN_0db
...
[#define](https://forum.espruino.com/sea­rch/?q=%23define) ADC_WIDTH_BIT_12 ADC_WIDTH_12Bit
[#define](https://forum.espruino.com/sea­rch/?q=%23define) dac_output_voltage dac_out_voltage
[#define](https://forum.espruino.com/sea­rch/?q=%23define) adc1_get_raw adc1_get_voltage

You could basically remove every #if ESP_IDF_VERSION_MAJOR>=4 line from that file. And the same kind of thing for other files too - I'd be up for doing something like that, and then at some point if we do want to drop support for older IDFs it's just a matter of deleting that file.

Yes, I think at the moment the use of #ifdefs is appropriate and balanced due to the different hardware boards.

But with 'sdkconfig' files, is something I believe we should clean up and clarify a bit more. It should be clear which options are important for Espruino to function correctly. The rest of the options can remain as defaults since they don't impact proper functionality.

This is all good input, thanks very much. It does seem we are converging ( slowly but surely) on an eventual esp32 combined idf v5 solution and def more progress in the last few months. Which is great.
I get it now that some #ifdefs are needed , I suppose for the purposes of accommodating target differences and not for IDF version differences. (This is evident in the esp idf examples - but to a relatively limited extent - more to follow on this analysis).

I also take the advice that networking will need a big effort in the move to idf v5. So im continuing to spend my time understanding how the SDKconfigs are used and how the networking is setup.

Ive had some joy using chatGBT to code a node.js tool to compare sdkconfig files. (as an aside I find this quite incredible). I attache a current version. I found it usefull for learning. Let me know if any one wants details on this. Simple usage as follows:

node compareConfigs.js <file1> <file2> <outputFile>

I also note that when using the esp IDF examples , in a given example top directory, the build command

idf.py set-target xxxxxx

will build an sdkconfig for that target. And so I suppose these form, a basis for a config in the development of an espruino build of that target in the given idf. With this and a compare tool the work of @MaBe , @rgomezwap and others could be morphed into the sdkconfig.defaults model.

I will contribute to the ongoing testing of the new developments also. But im away from my dev station for a couple of weeks now.

1 Attachment

• compareConfigs.js

Here is a edited / condensed version of the output , from the above tool (cut and pasted from vscode) comparing the current c3 and s3 versions of idfconfig.
Ive manually edited/added ......................... where lines were condensed.
Note the four sections in the output file: common, different, unique in file1, unique in file2

Common key-value pairs:

# Automatically generated file. DO NOT EDIT.
# Espressif IoT Development Framework (ESP-IDF) Project Configuration
CONFIG_IDF_CMAKE=y
# SDK tool configuration
# CONFIG_SDK_TOOLCHAIN_SUPPORTS_TIME_WIDE_­64_BITS is not set
# end of Compatibility options
# Build type
CONFIG_APP_BUILD_TYPE_APP_2NDBOOT=y
# CONFIG_APP_BUILD_TYPE_ELF_RAM is not set
CONFIG_APP_BUILD_GENERATE_BINARIES=y
CONFIG_APP_BUILD_BOOTLOADER=y
CONFIG_APP_BUILD_USE_FLASH_SECTIONS=y
..................
..................


Different key-value pairs:

Key: CONFIG_IDF_TARGET
    sdkconfig_c3: CONFIG_IDF_TARGET="esp32c3"
    sdkconfig_s3: CONFIG_IDF_TARGET="esp32s3"

Key: CONFIG_IDF_FIRMWARE_CHIP_ID
    sdkconfig_c3: CONFIG_IDF_FIRMWARE_CHIP_ID=0x0005
    sdkconfig_s3: CONFIG_IDF_FIRMWARE_CHIP_ID=0x0009

Key: CONFIG_SDK_TOOLPREFIX
    sdkconfig_c3: CONFIG_SDK_TOOLPREFIX="riscv32-esp-elf-"­
    sdkconfig_s3: CONFIG_SDK_TOOLPREFIX="xtensa-esp32s3-el­f-"

...................
...................

Key: CONFIG_ESPTOOLPY_FLASHSIZE
    sdkconfig_c3: CONFIG_ESPTOOLPY_FLASHSIZE="4MB"
    sdkconfig_s3: CONFIG_ESPTOOLPY_FLASHSIZE="8MB"

Key: CONFIG_PARTITION_TABLE_SINGLE_APP
    sdkconfig_c3: CONFIG_PARTITION_TABLE_SINGLE_APP=y
    sdkconfig_s3: # CONFIG_PARTITION_TABLE_SINGLE_APP is not set

Key: CONFIG_PARTITION_TABLE_FILENAME
    sdkconfig_c3: CONFIG_PARTITION_TABLE_FILENAME="partiti­ons_singleapp.csv"
    sdkconfig_s3: CONFIG_PARTITION_TABLE_FILENAME="partiti­ons.csv"

.........................
.........................

Unique key-value pairs in sdkconfig_c3:

CONFIG_IDF_TARGET_ARCH_RISCV=y
CONFIG_IDF_TARGET_ESP32C3=y
# CONFIG_ESPTOOLPY_FLASHFREQ_26M is not set
# CONFIG_COMPILER_SAVE_RESTORE_LIBCALLS is not set
# CONFIG_ADC_ONESHOT_FORCE_USE_ADC2_ON_C3 is not set
# ESP32C3-Specific
# CONFIG_ESP32C3_DEFAULT_CPU_FREQ_80 is not set
CONFIG_ESP32C3_DEFAULT_CPU_FREQ_160=y
CONFIG_ESP32C3_DEFAULT_CPU_FREQ_MHZ=160

.........................
.........................

Unique key-value pairs in sdkconfig_s3:

CONFIG_IDF_TARGET_ARCH_XTENSA=y
CONFIG_IDF_TARGET_ESP32S3=y
# CONFIG_ESPTOOLPY_OCT_FLASH is not set
CONFIG_ESPTOOLPY_S3_STR=y
# CONFIG_ESPTOOLPY_FLASHFREQ_120M is not set
CONFIG_BT_CTRL_PINNED_TO_CORE_0=y
# CONFIG_BT_CTRL_PINNED_TO_CORE_1 is not set
CONFIG_BT_CTRL_CHAN_ASS_EN=y
...

@Gordon , hi what's your thinking on making the IDF available for any production builds. Will you be re cloning the IDF from expressif github each build. If so Recloning at a given version number ??
I've seen one suggestion to tag to a given commit ( not 100% sure what that means) but suggestion was config variables change at the commit level.

I suppose from the experience to date that we don't have eyes on all the changes in the IDF continually and we will need control rather than keeping up to date with the latest IDF ???

pS I'm not suggesting G we go back to the Espruino tools approach.

pS I'm not suggesting G we go back to the Espruino tools approach.

You mean what we do for ESP32 by pulling a single file out of https://github.com/espruino/EspruinoBuil­dTools and decompressing it?

Personally I feel like that might still be a good approach - it makes the build time a lot faster and does allow us complete control of the IDF.

As an example, on the current master:

• ESP32 (https://github.com/espruino/Espruino/act­ions/runs/10768636543/job/29858166005) - 27 sec
  
• ESP32C3 (https://github.com/espruino/Espruino/act­ions/runs/10768636543/job/29858169688) - 3m42s
  
  

If GitHub ever start making Open Source projects pay for build time that's going to start to be an issue.

(I know that build time of the C3 would never get down to 30s as the ESP32 build seems to be partially built already, but I imagine the GitHub download is a nontrivial amount of that)

Thanks for the input.

You mean what we do for ESP32 by pulling a single file out of https://github.com/espruino/EspruinoBuil­­dTools and decompressing it?

yes , ill call that the 'buildtools' approach. Now I see the motivation for this approach. More grist for the mill.

For future reference this link appears to be the Micropython GitHub commit for upgrading micropython on the ESP 32 board family from ESP IDF v4.? to ESP IDF 5.2. This difference analysis should help as a reference for any future Espruino migration to the same.

The scope of the single commit change is across 68 source files including the SDKconfig files. The file structure seems intuitive with peripherial/functionality centric file contents.
The commit contains the following comment:

This commit updates the esp32 port to work exclusively with ESP-IDF
v5. IDF v5 is needed for some of the newer ESP32 SoCs to work, and it
also cleans up a lot of the inconsistencies between existing SoCs (eg
S2, S3, and C3).

Support for IDF v4 is dropped because it's a lot of effort to maintain
both versions at the same time.

The following components have been verified to work on the various
SoCs:

            ESP32     ESP32-S2  ESP32-S3  ESP32-C3
build       pass      pass      pass      pass
SPIRAM      pass      pass      pass      N/A
REPL (UART) pass      pass      pass      pass
REPL (USB)  N/A       pass      pass      N/A
filesystem  pass      pass      pass      pass
GPIO        pass      pass      pass      pass
SPI         pass      pass      pass      pass
I2C         pass      pass      pass      pass
PWM         pass      pass      pass      pass
ADC         pass      pass      pass      pass
WiFi STA    pass      pass      pass      pass
WiFi AP     pass      pass      pass      pass
BLE         pass      N/A       pass      pass
ETH         pass      --        --        --
PPP         pass      pass      pass      --
sockets     pass      pass      pass      pass
SSL         pass      ENOMEM    pass      pass
RMT         pass      pass      pass      pass
NeoPixel    pass      pass      pass      pass
I2S         pass      pass      pass      N/A
ESPNow      pass      pass      pass      pass
ULP-FSM     pass      pass      pass      N/A
SDCard      pass      N/A       N/A       pass
WDT         pass      pass      pass      pass

And also see the official espressif ESP IDF migration guides found here.