A graphic Web-IDE plugin to use modules on your flash (using tve FlashString.js)

Hi all,
I made this small WebIDE plug-in to use the plenty of flash pages of the ESP8266 and upload then execute easily modules there, and have more memory free for other use. I use it every day, as my programs always need more memory and getting bigger than the default space of 10kb. You can minify and transfer modules into pages, in one click, and then link them to be executed from flash, in another. As extra you can also load/unload modules direct to memory, etc. It mainly allows to use tve FlashString.js, without the hassle of manual commands and minification (mentioned in http://forum.espruino.com/conversations/290975/). Be careful, the execution from flash and use of E.memoryArea is restricted to the first mb, so you can use it only on the 44kb(11 pages) under 1mb. I use Wilberforce FlashStore module for the 3mb left (http://forum.espruino.com/conversations/283045/).

My code is based on the Project plug-in, here is it (messily programmed I am afraid but works well).
https://github.com/popok75/Espruinuity/blob/master/Espruino-WebIDE/js/plugins/flashView.js

It uses a modified version of tve FlashString.js . Here is this version : https://github.com/popok75/Espruinuity/blob/master/EspCode/project/modules/FlashString.js
also you may want the stand-alone loader : https://github.com/popok75/Espruinuity/blob/master/EspCode/project/modules/FlashStringLoader.js
You want FlashString.js to be in modules directory when you start, so that you can transfer it to memory and start using your flash.

For installation, you can get my version of WebIDE from the github (fetched 2 month ago) or you just need the source of WebIDE, add the source of espruino-tools in the right directory, flashview.js in plugin directory, and insert the following statements at the right place (not sure they are all needed) : https://github.com/popok75/Espruinuity/search?utf8=%E2%9C%93&q=flashview&type=

Here is how I load them in the final program :

function loadFlashModules(mods,loader) { // load module and show mem usage
	var sm=0;
	for(var i=0;i<mods.length;i++){var mod=mods[i];
	if(Modules.getCached().indexOf(mod)>-1) Modules.removeCached(mod);
	var m=require(loader).load(mod+".js");
	if(!m)  {print("Module not found in flash : "+mod);return false;}
	else {
		var s;
		s=process.memory().usage;	
		Modules.addCached(mod,m);
		sm+=(process.memory().usage-s);print("Module in flash loaded: "+mod+" : "+(process.memory().usage-s)+" block mem usage");}
	}
	print("All Modules : "+sm+" block mem usage");
	return true;
}


function init(){
	l=loadFlashModules(["WifiSync","DHT22non","FlashStore","IOPlug","IOLog","IOData","IOServer"],"FlashStringLoader");
}

I hope you'll try it and tell me what you think.

1 Attachment

Cool... tried it out - on Espruino-Wifi... tried to try it out... :(

Screenshots attached...

What I did:

1. pulled your project from git
   
2. added ide as chrome extension in developer mode to chrome
   
3. opened ide - see the new puzzle icon
   
4. clicked on puzzle - get obviously the "no folder defined" option
   
5. created folder ~/_/esp/flashViewTest, placed SWBtn.min.js module in there (in os)
   
6. opened Settings, selected ~/_/esp/flashViewTest folder --- a sandbox dir structure is created
   
7. copied FlashString.js from https://github.com/popok75/Espruinuity/b­lob/master/EspCode/project/modules/Flash­String.js to modules folder of sandbox (in os)
   
8. clicked on puzzle icon - get Flash atring module not loaded
   
9. clicked Projects folder - see - as expctect - all my projects ....js files
   
10. clicked Projects folder and Modules tab: see empty list (should see at least FlashString.js
    
11. clicked Projects folder and Flash tab: see empty list...
    
    

Looks like I'm missing something...

And there are some unexpected things happening:

• sandbox structure is created in flashView folder
  
• cannot see modules anymore
  

2 Attachments

Thanks very much allObjects for trying, obviously I forgot to mention about the options, but you found it, yes you have to set a folder.
There is a bug that prevents the popup from opening when binary folder is empty, so put any file inside, until I fix it. The puzzle button is the one you want, the folder button is an old version that may not work.

Once you have the window from puzzle button to open, it will still say "no flashstring found", that mean you need to transfer to the board with the button top right from flash or module tab.
Here's a snapshot of the application newly installed:

1 Attachment

@Polypod, thanks... sometimes it is so frustrating that a little, non-essential details just crashes a success experience and prevents the great work from showing!

Here my last results of setup:

• selection of files to show in modules tab has issues with xyz.min.js files (adding plain xyz.js file makes module to show...). I assume this is because of how modules then get picked later on from flash...
  

2 Attachments

...usage is though still a bit unclear... as you can see from the logs... was unable to upload to a Pico. Copied then the FlashString.js as well to the FlashViewTest/modules folder. Upload to memory worked, but not to flash... but I'm not sure about the sequence and what would have to go when and where.

Please provide a few lines of mode d'emploi...

Flash:[]
>MODULES:[]
MEMORY:{"free":5043,"usage":57,"total":5100,"history":104,"gc":6,"gctime":4.32682037353,"stackEndAddress":536958144,"flash_start":134217728,"flash_binary_end":379832,"flash_code_start":134234112,"flash_length":393216}
=undefined
>Uncaught Error: Module "FlashString" not found
 at line 1 col 23
;require('FlashString').save('SWBtn.js',"function SWBtn(a,b,...
                      ^
>Flash:[]
>MODULES:[]
MEMORY:{"free":5043,"usage":57,"total":5100,"history":104,"gc":5,"gctime":4.34589385986,"stackEndAddress":536958144,"flash_start":134217728,"flash_binary_end":379832,"flash_code_start":134234112,"flash_length":393216}
=undefined
>Uncaught SyntaxError: Got ?[27] expected EOF
 at line 1 col 1
dfunction s2a(a){while(a.length&3)a+=' ';return E.toUint8...
^
={ "free": 5046, "usage": 54, "total": 5100, "history": 261,
  "gc": 13, "gctime": 4.48131561279, "stackEndAddress": 536958144, "flash_start": 134217728, "flash_binary_end": 379832,
  "flash_code_start": 134234112, "flash_length": 393216 }
=[
  {
    "name": "FlashString",
    "size": 3 }
 ]
>Uncaught Error: Function "list" not found!
 at line 1 col 90
...h:'+require('FlashString').list()); else print('Flash:'+'[]'...
                              ^
>MODULES:[{"name":"FlashString","size":3}]
MEMORY:{"free":5032,"usage":68,"total":5100,"history":269,"gc":5,"gctime":4.40979003906,"stackEndAddress":536958144,"flash_start":134217728,"flash_binary_end":379832,"flash_code_start":134234112,"flash_length":393216}
=undefined
>Uncaught Error: Function "save" not found!
 at line 1 col 25
;require('FlashString').save('SWBtn.js',"function SWBtn(a,b,...
                        ^
>Uncaught Error: Function "list" not found!
 at line 1 col 90
...h:'+require('FlashString').list()); else print('Flash:'+'[]'...
                              ^
>MODULES:[{"name":"FlashString","size":3}]
MEMORY:{"free":5032,"usage":68,"total":5100,"history":269,"gc":5,"gctime":4.40979003906,"stackEndAddress":536958144,"flash_start":134217728,"flash_binary_end":379832,"flash_code_start":134234112,"flash_length":393216}
=undefined
>Uncaught SyntaxError: Got ?[27] expected EOF
 at line 1 col 1
dfunction s2a(a){while(a.length&3)a+=' ';return E.toUint8...
^
={ "free": 5046, "usage": 54, "total": 5100, "history": 269,
  "gc": 12, "gctime": 4.43553924560, "stackEndAddress": 536958144, "flash_start": 134217728, "flash_binary_end": 379832,
  "flash_code_start": 134234112, "flash_length": 393216 }
=[
  {
    "name": "FlashString",
    "size": 3 }
 ]
>Uncaught Error: Function "list" not found!
 at line 1 col 90
...h:'+require('FlashString').list()); else print('Flash:'+'[]'...
                              ^
>MODULES:[{"name":"FlashString","size":3}]
MEMORY:{"free":5032,"usage":68,"total":5100,"history":269,"gc":5,"gctime":4.39643859863,"stackEndAddress":536958144,"flash_start":134217728,"flash_binary_end":379832,"flash_code_start":134234112,"flash_length":393216}
=undefined
>Uncaught SyntaxError: Got ?[27] expected EOF
 at line 1 col 1
dfunction SWBtn(a,b,c){this.f=a?a:function(){},this.b=b?b...
^
={ "free": 5043, "usage": 57, "total": 5100, "history": 346,
  "gc": 10, "gctime": 4.54521179199, "stackEndAddress": 536958144, "flash_start": 134217728, "flash_binary_end": 379832,
  "flash_code_start": 134234112, "flash_length": 393216 }
>E.getSizeOf(global["ÿ"].modules,1);
=[
  {
    "name": "FlashString",
    "size": 3 },
  {
    "name": "SWBtn",
    "size": 3 }
 ]
>Uncaught Error: Function "list" not found!
 at line 1 col 90
...h:'+require('FlashString').list()); else print('Flash:'+'[]'...
                              ^
>MODULES:[{"name":"FlashString","size":3},{"name":"SWBtn","size":3}]
MEMORY:{"free":5029,"usage":71,"total":5100,"history":346,"gc":5,"gctime":4.47368621826,"stackEndAddress":536958144,"flash_start":134217728,"flash_binary_end":379832,"flash_code_start":134234112,"flash_length":393216}
=undefined
>Uncaught Error: Function "save" not found!
 at line 1 col 25
;require('FlashString').save('SWBtn.js',"function SWBtn(a,b,...
                        ^
> 

You are almost there. I never noticed that it ignores files ending with .min.js, it must be the way the project plugin was made. I'll have a look at that as well. If you try plain .js files, you have a 2 step left, to place the FlashString.js file in the modules directory, and to upload it to the board with the button with two arrows.

I had FlashString.js in place...

BUT: NEVER MIND...

looking closer into the code, I noticed that it is ESP8266 oriented... So my Espruino-Wifi and Pico would anyway not be able to do... last but not least what @tve mentioned: page size (and total memory) of STMs are prohibitive to this simple, straight forward approach - post #6 in origin conversation.

Thanks again allObjects for all your tries, I appreciate.

I think it is not working because you didn't upload the FlashString.js module to the board yet. I can see it is not there in your log. The buttons to upload to flash are not supposed to show yet until he finds this module on the board, but for some reason they show up on yours, just a glitch. So when you use them, he complain about the absence of this module. (Uncaught Error: Module "FlashString" not found)

If it is not an ESP8266, you may also need to change the default address/size in FlashString.js to match your flash setup (line 28-31 and remove the 2 extra pages use line 48).

The sequence of action is :
1- have your module and FlashString.js in modules directory (what you did) - picture 1
2- upload FlashString.js to the board memory (it must show off as module in global[0xFF].modules and not with 3blocks of memory only). It seems you did not do this step. - picture 2, it takes 146block of memory
3- now you can see your flash pages (pict 3) and upload your module to flash (pict4)

Here is the log :
step1 :

 _____                 _
|   __|___ ___ ___ _ _|_|___ ___
|   __|_ -| . |  _| | | |   | . |
|_____|___|  _|_| |___|_|_|_|___|
          |_| http://espruino.com
 1v93.162 Copyright 2016 G.Williams
Espruino is Open Source. Our work is supported
only by sales of official boards and donations:
http://espruino.com/Donate
Flash map 4MB:512/512, manuf 0xef chip 0x4016
>Flash:[]
>MODULES:[]
MEMORY:{"free":1651,"usage":49,"total":1700,"history":13}
=undefined

step2 :

>process.memory();
={ "free": 1517, "usage": 183, "total": 1700, "history": 176 }
>E.getSizeOf(global["ÿ"].modules,1);
=[
  {
    "name": "Flash",
    "size": 3 },
  {
    "name": "FlashString",
    "size": 146 }
 ]

step3 :

>Flash:[{"name":"DHT22non.js","size":858,"address":"f8010"},{"name":"IOData.js","size":3433,"address":"f9010"},{"name":"FlashStore.js","size":4028,"address":"fa014"},{"name":"WifiSync.js","size":1044,"address":"fc010"},{"name":"IOPlug.js","size":3936,"address":"fd010"},{"name":"IOServer.js","size":2475,"address":"fe010"},{"name":"IOLog.js","size":3777,"address":"ff00c"},{"free":4}]
>MODULES:[{"name":"Flash","size":3},{"name":"FlashString","size":146}]
MEMORY:{"free":1497,"usage":203,"total":1700,"history":184}
=undefined
>File uploaded to flash : Add.js
>Flash:[{"name":"DHT22non.js","size":858,"address":"f8010"},{"name":"IOData.js","size":3433,"address":"f9010"},{"name":"FlashStore.js","size":4028,"address":"fa014"},{"name":"Add.js","size":639,"address":"fb00c"},{"name":"WifiSync.js","size":1044,"address":"fc010"},{"name":"IOPlug.js","size":3936,"address":"fd010"},{"name":"IOServer.js","size":2475,"address":"fe010"},{"name":"IOLog.js","size":3777,"address":"ff00c"},{"free":3}]
>MODULES:[{"name":"Flash","size":3},{"name":"FlashString","size":146}]
MEMORY:{"free":1497,"usage":203,"total":1700,"history":246}
=undefined

In case it was not clear, to communicate with the flash, the plugin uses the FlashString.js module he find ON the board memory, not the one on the hard drive. Therefore, you need to upload it there first by placing it in the module folder of the hard drive and use the button. The plugin just minify and transfer the file to FlashString.js on board, that take care of the rest.

Yes, the plugin was made using tve module that is esp8266 oriented as you mention. That's why I posted in this section of the forum. But the plugin can work with any flash module that works the same way with save/load/(json)list, I was planning to extend it with another tab using FlashStore.js (I mentioned in first post), but this is still Esp8266. Anyone made a similar module yet for other platforms flash ? For me, it is worth have a unified plugin for flash, SD and so on, but, I have no SD, and truth is, I never got to understand completely the Project plugin code I made my plugin from, hence the little bugs here and there.

Honestly, I am quite surprised by the difficulty you meet to make it run. I realize it is one thing to use it on my setup, and another to make it run in other people's setup. I would sure appreciate to see it run somewhere else, but if that doesn't help you forget it. If it is not an ESP8266, the benefit is quite limited, I understand if you give up :) I can fix the bugs that you already found, if that really helps. Some days, I don't understand how people can program on esp8266 without using this plugin, memory is so small and javascript is so gluttonous. Other days, I realize I am the crazy one for wanting to use 44kb of modules, plus the rest 3mb for storing files and logs, and have free memory.

4 Attachments

@Polypod, thanks for the details... and also the obvious enhancements that show in your new screenshots. I attribute these to the differences I notice. I will return some time later to 'play again'. May be I can then also pull a new version of the IDE with the enhanced *Flash Viewer / Flash Manager * plug-in. For now I would like to talk about the folder structure to the point I understand.

The attachment shows the directory structure and files I was working with yesterday.

The sandbox folder - light blue box - is _ecslsbx. The flash folder - purple box - created for testing/using FlashView / FlashManager plugin is flashViewTest. From the conversations I conclude that when you are talking about modules folder (and binary folder), you are talking about the flash modules folder (and flash binary folder) established by Flash View / Flash Manager plugin when selecting the flash folder (flashViewTest in my case) in Espruino IDE SETTING-FLASHVIEW. The relevant files are the files in flash modules folder (and flash binary folder):

• FlashString.js is required as part of the mechanism
  - SWBtn.js is a module to be loaded into flash (your's: Add.js and others)
  - dummy.bin is required (for now?) that folder is not empty and FlashView pop-up shows
  
  Said that, some files are just in too many places... ;). But I assume - when loadFlashModules(mods,loader) will be part of a (local) module, it would be found in the modules folder of the sandbox.
  

1 Attachment