use RESIZABLE_JSVARS for other boards than linux ?

AFAIK, RESIZABLE_JSVARS are set for linux only.
Would it be possible to use this for other boards, like ESP32, too ?
I've some boards with additional psram, so memory is available like crazy.
Since psram needs a driver, its not available from start, means cannot be used for statically allocated block

Idea would be to check available heap during init and based on that malloc, memory for JsVars.
There would be a definitions in board.py for large heap and for small heap.

If something sounds as easy as this, usually there are some hidden problems,....
Any comment ?

Yes, I think that could work. It's worth a try...

However:

• Resizable JsVars may default to using 32 bits for JsVarRef when actually 16 bit would do fine - it'd be worth checking
  
• It's a bit slower for JS execution, but maybe that doesn't matter too much?
  

hmm, we have 2 versions_

• esp32 wroom, here we fight for each byte
  
• esp32 wrover, here we have plenty of memory
  I would support one solution which supports all.
  
  

Would it be possible to replace static allocated block with malloc allocated block ?
In that case it wouldn't be resizable, between us, I would pay this price :-).
My understanding of JsVars is poor, would 32 bit reference be a problem ?

My expectations for speed is the same. Espruino supports an option to move all WIFI and BLE memory to psram. I'm pretty sure, they would not do this, if its much slower.

Ahh, I think really there are 2 options...

• malloc the jsVars data at boot time. This should be totally fine to do with everything basically as-is - but it's not the same as RESIZABLE_JSVARS
  
• RESIZABLE_JSVARS is where extra jsVars are allocated as needed, and it's what is done on Linux. The issue here is that in order to do that there's not just one jsVars array, but different 'blocks' of jsVars - since you can't just move a JsVar around in memory if some bit of code has its address in a pointer. Personally, I'd avoid doing that since having the different blocks slows down everything since it's no longer a simple lookup to find a variable's address from its reference.
  
  

About 32 bits - yes, it's bad :) See http://www.espruino.com/Internals for the layout - 32 bits means vars are no longer 16 bytes each, but are 24 (IIRC) - so you now only get 2/3 of the amount of variables in the same RAM :(

Hmm, for ESP32 I would prefer simple malloc solution.
I checked this in a first simple test(see code at the end) and in a first test it works fine.
But doing board specific changes in jsvar.c is not my favourite solution.

Anyway I'm still confused about the impact of 32bits.
In internals link and in source of jsvar I only find 12 byte or 16 byte.
There is a note JsVars for 32 bit refs are similar, but what does this mean ?
Same for build_platform_config.py, which checks for less than 1024 only.
In which case would we need 24 bytes instead of 16 ?

BTW is there a limit for variables ? I can see ESPRUINOWIFI has 7135.
What would happen with 10000 or 50000 jsvars ?
It's a kind of theoretical question, but having 4MB heap, .....

#ifdef RESIZABLE_JSVARS JsVar
**jsVarBlocks = 0;
unsigned int jsVarsSize = 0;
#define JSVAR_BLOCK_SIZE 4096
#define JSVAR_BLOCK_SHIFT 12
#else
#ifdef ESP32
JsVar *jsVars = NULL;
unsigned int jsVarsSize = 0;
#else
JsVar jsVars[JSVAR_CACHE_SIZE];
unsigned int jsVarsSize = JSVAR_CACHE_SIZE;
#endif
#endif

and this:

void jsvInit() {
#ifdef RESIZABLE_JSVARS
jsVarsSize = JSVAR_BLOCK_SIZE;
jsVarBlocks = malloc(sizeof(JsVar*)); // just 1
jsVarBlocks[0] = malloc(sizeof(JsVar) * JSVAR_BLOCK_SIZE);
#endif
#ifdef ESP32
jsVars = (JsVar *)malloc(33600);
jsVarsSize = 2100;
#endif
jsVarFirstEmpty = jsvInitJsVars(1/first/, jsVarsSize);
jsvSoftInit();
}

Normal JsVarRefs for most boards are 16 bits, so can store up to 65535 variables. But the Resizable vars build can do 4 billion because of 32 bit references (I believe). But adding 32 bit references really pushes the size of a JsVar up, so you don't want to do that.

Basically, what you've done is fine - although ultimately I guess you'll want to set jsVarsSize to something else before calling jsvInit?

I would do this instead in the second bit though - no point hard-coding numbers where it's not needed:

#ifdef ESP32
jsVarsSize = 2100;
jsVars = (JsVar *)malloc(sizeof(JsVar)*jsVarsSize);
#endif

Hardcoded value was for easy testing only :)

Hmmm, I could move the functionality to assign jsVarSize into function main of jshardware.c.
In there would be a switch based on heap size (or psram recognition) to assign "small memory VarSize" or "big memory VarSize"
Both values should be in ESP32.py.
Hmmm, have to check how these values will make it to platform_config.h. This should be no problem.
AFAIK Everything could be done without changes in jsvInit(jsvar.c), but I would prefer to make it more general. Means adding one more definition in ESP32.py, something like "board defined jsVarSize". This way we would add a more general functionality for other boards in the future.

What would you prefer ? Or is there another(better) option, which did not come to my mind ?
Anyway, I've a starting point now for more testing.

My be a bit hacky.... and need a suitable comment.

Set the jsvars to 0 in the board py to mean allocated at runtime? And then fix anything that depends on that?

If you've got 4mb of PSRAM to play with, why wouldn't one make the whole 64K jsvars permitted by the 16-bit addressing available?

If you've got 4mb of PSRAM to play with, why wouldn't one make the whole 64K jsvars permitted by the 16-bit addressing available?

Sure - for the psram case - @JumJum was looking to have one firmware that could be used on both psram and the non-psram boards as I understand it.

@wilberforce,
thats correct, I'm looking for one firmware running on all (known) ESP32 boards.
Right now, I need a change in ESP-IDF.
In actual version there is an option to use psram, but if no psram is found during startup, it aborts.
Better would be an option (choice) with abort or log missing psram only.
Just tested this, but its a change in ESP-IDF, so they need to accept.

Set the jsvars to 0 in the board py to mean allocated at runtime?

That sounds like the best idea, and wouldn't require any board-specific defines in jsvar.c, which is a massive bonus.

Could we move definition for jsVarBlocks, jsVarsSize and jsVars into jsvars.h ?
Other option would be to add a function to jsvar.c to set jsVarsSize from main.c

As always, if you have any better idea...

I got it running this way.
Simple board gives me 2000 vars
Board with additional PSRAM gives me 20000 vars.
@DrAzzy, we could use 65500 here too

ESP32.py

 'variables'                : 2000,
 'variables_psram'          : 20000,
 'variables_mode'           : "malloc",             ESP32 uses malloc

jsvar.h

extern unsigned int jsVarsSize;

jsvar.c

[#ifdef](https://forum.espruino.com/sear­ch/?q=%23ifdef) RESIZABLE_JSVARS
JsVar **jsVarBlocks = 0;
unsigned int jsVarsSize = 0;
[#define](https://forum.espruino.com/sea­rch/?q=%23define) JSVAR_BLOCK_SIZE 4096
[#define](https://forum.espruino.com/sea­rch/?q=%23define) JSVAR_BLOCK_SHIFT 12
[#else](https://forum.espruino.com/searc­h/?q=%23else)
[#ifdef](https://forum.espruino.com/sear­ch/?q=%23ifdef) VARIABLES_MODE_MALLOC
unsigned int jsVarsSize = 0;
JsVar *jsVars = NULL;
[#else](https://forum.espruino.com/searc­h/?q=%23else)
JsVar jsVars[JSVAR_CACHE_SIZE];
unsigned int jsVarsSize = JSVAR_CACHE_SIZE;
[#endif](https://forum.espruino.com/sear­ch/?q=%23endif) //end VARIABLES_MODE_ALLOC
[#endif](https://forum.espruino.com/sear­ch/?q=%23endif)
......
void jsvInit() {
[#ifdef](https://forum.espruino.com/sear­ch/?q=%23ifdef) RESIZABLE_JSVARS
  jsVarsSize = JSVAR_BLOCK_SIZE;
  jsVarBlocks = malloc(sizeof(JsVar*)); // just 1
  jsVarBlocks[0] = malloc(sizeof(JsVar) * JSVAR_BLOCK_SIZE);
[#else](https://forum.espruino.com/searc­h/?q=%23else)
[#ifdef](https://forum.espruino.com/sear­ch/?q=%23ifdef) VARIABLES_MODE_MALLOC
  jsVars = (JsVar *)malloc(sizeof(JsVar) * jsVarsSize);
[#endif](https://forum.espruino.com/sear­ch/?q=%23endif)
[#endif](https://forum.espruino.com/sear­ch/?q=%23endif) 
  jsVarFirstEmpty = jsvInitJsVars(1/*first*/, jsVarsSize);
  jsvSoftInit();
}

build_platform_config.py

if board.info["variables_mode"]:
  codeOut('#define VARIABLES_MODE_'+board.info["variables_m­ode"].upper() + ' // mode for allocating memory, standard is statically assigned');
  codeOut('');
...
  if board.chip["class"]=="ESP32":
    codeOut("#define JSVAR_CACHE_SIZE_PSRAM          "+str(board.info['variables_psram'])+" // Number of Javascript variables in RAM for boards with additional RAM (like ESP32 WROVER)");

last not least in main.c

  if(esp_get_free_heap_size() > 0x300000) jsVarsSize = JSVAR_CACHE_SIZE_PSRAM;  // looks like 4MB SPI_RAM is enabled for heap, pretty sure there is a better way, but for now ....
  else jsVarsSize = JSVAR_CACHE_SIZE;   // number of variables for boards without additional SPI RAM
  jsvInit();     // Initialize the variables

That looks good - I guess you could change jsvInit to take the size as a parameter?

If someone's interested in it, there's the possibility of using this on some STM32 devices - for instance the original Espruino board has a chip that's only supposed to have 48kB of RAM, but which actually has 64kB (I believe). I just can't be sure every chip has it so I can't enable it in firmware.

Potentially some code could check for the RAM at boot and rearrange the stack to use what was available - although it's more difficult as Espruino for STM32 doesn't use malloc :)

So a build that would be like the bigram builds I used to do (at some point, the toolchain stopped working, and then I was forced to rebuild the system by amazon and forgot how to set it up anyway), only it would auto-detect whether that RAM was actually there? That would be really cool

An esp32 build that had 64K jsvars would be pretty cool. Would finally be hard to run out of ram on espruino.

@Gordon,
changing jsvInit to use size as a parameter would be easy.
But there are some other boards, that also use jsvInit, they would need to be changed too. Something that I would not like to do.
AFAIK, C does not support optional parameters. If somebody knows better, I'm still on first part of learning curve, please give me a hint.
@DrAzzy, I'm pretty sure, sooner or later there will be someone to run out of memory, even with 64K :-)

@JumJum I can do that, but I've got a bunch of other stuff on my list at the moment. Maybe let me know when it makes it into ESP-IDF and I can do the changes then?

So a build that would be like the bigram builds I used to do

Yep, that's the idea.

@DrAzzy maybe you know something about this: https://github.com/espressif/ESP8266_MP3­_DECODER

For that they needed more RAM on an ESP8266 so they piggy-backed an SPI RAM chip onto the SPI flash. Do you think that would end up getting exposed in the ESP8266's address space as well? Could be a really interesting way of boosting cheap ESP8266s as well?