create specific firmware online

As mentioned in another chat, I'm just restarting a project to create specific firmware for Espruino.
It's in early state and will take some time.
Create specific firmware means

• you only need a browser, no development tools on your computer
  
• select board
  
• set/unset definitions like USE_BLUETOOTH or USE_GRAPHICS
  
• remove functions, you don't need. Like eval to give an example
  
• add JS modules to firmware, Gordon already does this for some boards
  
• add your own source files (c-filesand h-files)
  
  

@Gordon already mentioned, that support of c-files opens a door for security issues.
AFAIK, these are mainly security issues for the server. One example is the use of
#include "/usr/passwd" in C-source
Ideas to bypass this problems are

• build my own scanner to recognize
  
• search the internet for scanners
  
• load sources only from "secure github sites"
  
• remove the option for adding c sources, in my eyes the worst option
  
  

Please give comments/feedback/experience
Hopefully somebody has some knowledge around this.

I think this is a great idea - especially as on some boards like the Original or ESP8266 it's now getting to be a squeeze to fit everything in.

To work around the security issues I believe you can either use a chroot to stop access to any sensitive files, or could spin up a Docker container (which I think is more or less the same thing, but maybe a bit more heavyweight?). For http://www.espruino.com/InlineC I disabled the preprocessor, but that's not really sensible for entire C files.

I do wonder how many people would actually supply their own C code? It would be really hard to write new code by repeatedly uploading source code and checking for compile errors - so I guess for development it makes a lot more sense to build locally?

I imagine it might be more use if there was a registry of pre-tested C code that could be compiled into Espruino? I guess if we had a separate GitHub repo for that we could at least sanity check PRs before merging them - although I really haven't seen much interest in it. If I was getting lots of contributions for new Espruino C libraries it might be different, but the only one I can remember was you wanting to add support for the P3 matrices to ESP32.

Maybe in a way it's more useful to have the other features working than it is to allow custom C files and then have to worry about all the security implications?

Hmm, your feedback to userdefined C code sounds like "there is no market"
People with knowledge to do that will already have their own development.
Thats a good point.

So lets go back one step. The idea of predefined C code sounds good to me.
I can imagine some reasons why you did not get lots of contributions

• lack of knowledge how to built
  
• bottleneck for additional functions due to actual restrictions in size
  
• functions are not helpful for majority of users, for example who needs a Schmitt Trigger. On the other hand I would like to have lowpass, highpass, pid control, etc. etc.
  
• restriction to "must run on all boards", for example I've a full color P3 driver, which works on ESP32 only. some guys could be interested
  
• no "how to proceed" for plugins
  
  

Summary, I would like to start a first version of a registry. About 3 years ago I've created a github with my extensions https://github.com/jumjum123/EspruinoExtensions c files.
And never worked on that again :-( Lets give it a try.
I will create an interface to that into my firmware build tool, and remove upload for C files

I would like to start a first version of a registry.

That sounds like a great plan - and then some instructions on how to turn an extension as you documented in http://www.espruino.com/Extending+Espruino+1 into a PR for that repository?

As mentioned some days ago, I've created a simple dictionary for Espruino Extensions in github.
Some extension are there, mainly for testing. They are left over from some hacking months/years ago.
An interface to this is added in actual version of firmware tool.

In a next step I would like to do something similiar for modules.
Options comin to my mind are

• upload from local computer, that would support self written modules. This is an easy part and is already implemented in a not fully tested version
  
• use modules list from https://github.com/espruino/EspruinoDocs/tree/master/modules. This is a short list without minified files but with md files
  
• use modules from https://www.espruino.com/modules/ Thats a very long list with minified files but without md files. Would have to scan this list with regex, which is not my favourite tool. Anyway, would be my problem
  
  

Is there an better option ?

As a start, just being able to click and select Espruino modules would be a huge bonus.

If you did https://github.com/espruino/EspruinoDocs­/tree/master/modules you'd want https://github.com/espruino/EspruinoDocs­/tree/master/devices as well...

espruino.com/modules/ - Thats a very long list with minified files but without md files.

That is tricky - there's usually a 1:1 mapping between espruino.com/modules/filename.js and espruino.com/filename - but not always.

However, I guess realistically anyone using this is going to have a good idea what they're doing? I guess in most cases you'll develop code normally, figure out what modules you want, and then build your own firmware including those modules - so not having direct links to the docs isn't a huge problem.

I guess just pulling out the first comment from the head of the .js file would give a pretty good into to the module. If it didn't, we should probably update the modules to have a description :)

quickly checked and the result is:

• master/modules s a subset of master/devices
  
• master/devices is a subset of espruino.com/modules ( for example AT.js, ATSMS.js. BMPLoader.js, etc., etc.)
  
• espruino.com/modules has *.min.js for each module, github repos don't
  
  

For now I would start with master/devices and create a *.min.js on the fly
If no doc is available, we could show source instead.

Update:Sorry, misunderstanding from my side
Looks like I've to combine both github directorys to get most correct list

Looks like I've to combine both github directorys to get most correct list

Yes - although I wonder whether in future that might change. Is it possible to just use the list of files that are in http://www.espruino.com/modules/ ?

Hmmm,
on one hand I don't like to scan html files to get information,
and oppsite to you I like to give information, if it already exists in md files

On the other hand, german wording "sometimes you have to jump over your own shadow", means my dislike should not stop a better solution
And would it be possible to add md files to http://www.espruino.com/modules ?

Just one more idea, how do you create modules directory?
If there is a script, similiar to common.getJsonData, I could use this directly.

One more problem, if PROJECTNAME is set, make returns
Nothing to be done for `all'.
So PROJECTNAME=hugo BOARD=ESPRUINOBOARD make does not work
but BOARD=ESPRUINOBOARD make works well

On a quick test for different boards, looks like only ESP32 and ESP8266 work fine with PROJECTNAME=something

PROJECTNAME

That's odd - looking at the git commits the only mentions of PROJECTNAME for the Makefile are in lines you added, so I don't know why it wouldn't work.

would it be possible to add md files to http://www.espruino.com/modules

Not really - there isn't a 1:1 mapping (eg. fonts - there are lots of fonts, but only one font page).

The modules dir is filled using https://github.com/espruino/EspruinoDocs/blob/master/buildmodules.sh - so you could just scan devices modules boards for js files, it's just not a very robust solution.

But you'll still find that for certain .js files there is no .md file to go along with them - if you're going to do that, why not check whether espruino.com/MODULENAME returns a 404 or not and then link to it if it's fine.

The .md files (as mentioned in the header in each one) contain extra tags (like APPEND_KEYWORDS/etc) that doesn't render properly if you just view the file on its own, so realistically you're better off pointing direct to espruino.com.

I would like to automate upload for latest release to EFEU and checked this
https://api.github.com/repos/espruino/Espruino/releases
Problem is, that github API returns an empty array
Next checked
https://api.github.com/repos/espruino/Espruino/tags
Now I get a long list, now problem is sorting of this list.

Found a (strange?) description, why I don't find anything for releases
https://stackoverflow.com/questions/18506508/whats-the-difference-between-tag-and-release
Any idea what I can do to get most actual release of Espruino ?

I thought I could offer help here as I often use this https://api.github.com/repos/:owner/:repo/releases/latest myself to get the latest release of a repo.

But it fails also.

Maybe this is to do with it being an organisation? But then tags work fine!?!

The tags JSON isn't easily sorted, I can see that. Maybe a good old fashioned web scrape of the releases page, and take the first element in the list? Not ideal.

My Q&U solution (Quick and Ugly) is to read tags, search for names starting with RELEASE_2V and sort by name.
As mentioned above its Q&U, but better than nothing.

Sorry for the delay with this...

Espruino doesn't use the 'releases' feature - which I believe is a way to attach binaries to GitHub projects (could be wrong here).

Instead, each release is tagged (which is merely a pointer to a specific Git commit).

In actual fact your quick and ugly solution is basically what I do. If you look in the Makefile you'll see:

git tag | grep RELEASE_ | sort | tail -1

So I get the list of tags, use any starting with RELEASE_, then sort them and take the last one :)