Espruino on ESP8266

I just had a session with @Kolban. We were able to get Espruino running on one of my ESP01 boards. I could run all kinds of JS including connecting to the network and listing the access points in my vicinity. Wow. Congratulations to @Gordon for making the really cool Espruino and @Kolban for porting it to the ESP01. This is really a cool addition to the Espruino family. Certainly not as easy to use and setup as the Pico but still very useful!

Great news! I just ordered a ESP-12E board for myself this morning. Can't wait to put Espruino on it when I receive it :) (Usually takes several weeks as it's shipped from China)

That's great - is it at the stage where HTTP requests work yet?

Just to add, @aplikatika deserves a lot of credit for this too - he did the first bit of porting work, and had Espruino running quite well by the look of it - albeit without any actual socket communications.

I have 4 or 5 different models of ESP8266 sitting on the corner of my work bench ready to test Espruino.

Seconding what @Gordon said, @aplikatika is the father and originator of this porting project. When I got involved, the port was already functioning at the GPIO level and at the JavaScript level just fine. We consider the first work a successful proof of concept and like any proof of concept, we will do anything to get it working. This second phase is the slow, methodical drudgery of detailed architecture mapping Espruino models of the world to what is meaningful on ESP8266 and adding as much polish and support as possible before the 1st inclusion in Espruino proper.

This is an open source port and once done will be fully merged with the master repository (assuming it passes all the necessary quality tests). At that time, the Espruino port to ESP8266 project as a separate entity will come to an end and the GitHub project deleted. What will remain is a single project (Espruino) where the good folks who worked on the original will still be available for any ESP8266 specific issues.

@cwilt

VERY early testing of a binary distribution has started. See the following:

https://gitter.im/aplikatika/Espruino-on-ESP8266

For the last couple of days (2015-09-21 - 2015-09-22). This is NOT a release. Again ... it is NOT stable and should only be attempted by those who are super interested. However, we do need testers who will validate that the basics are present and also so that we can start making fix lists.

When I get back to my work bench I will give it a go.

Absolutely, congratulations also to @aplikatika! Sorry for forgetting you in the excitement.

Hip hip hurray, I've just got Espruino running on ESP8266 ESP-01.
I used the nodemcu-flasher
Only change I had was to flash ESP8266_ox00000.bin and ESP8266_0x10000.bin seperately. It didn'twork with both in one session.
First (minor problem) was baudrate, it had to be switched to 115200 and now WebIDE works.
First feedback is below, I will start testing immediately.

>process.memory();
={ "free": 984, "usage": 39, "total": 1023, "history": 28 }
>

Thanks a lot for this beautiful job to Gordon, Kolban, aplikatika and all others that have been involved !!!!

Howdy @JumJum

Great news that you got it running. But lets make sure that we realize that the coding is no where near finished. I loosely target November for a more functional and stable proper release. Things are known to be broken all over the place including things like network connections claiming formed before they are ready. GPIO testing has also been minimal and there are likely to be memory leaks to be fixed and optimizations to be made.

With these words of caution ... I don't want folks to NOT play with it ... but to tire kick it and report bugs ... but don't take its current state as a statement of its quality into the future ... it will get better :-)

Neil

Hello Kolban,
following your description, it is somewhere between alpha and beta.
Thats not a problem for me, I would like help testing.
For wifi functions I will wait for your "ready for testing".
Some minutes ago, flashing of my ESP201 ran fine. So I could do some testing with I/O too. For that I will wait until some more information about testworthy functions is available.

Up to now I got these simple javascript running in my first test:

var i = 7;
i++;
console.log(i);
for(i = 0; i < 5; i++){
  console.log(i);
}
function hello(t){
  console.log("hello " + t);
}
hello("Hugo");
function hello2(n){
  var me = this;
  me.n = n;
  me.show = function(t){
    console.log(n + " " + t);
  };
}
hello2.prototype.showNumber = function(){console.log(this.n + 3);};
var h = new hello2(42);
h.show(" is the answer");
h.showNumber();
console.log(process.memory());

var  si = setInterval(function(){
  hello(i++);
  if(i > 8) clearInterval(si);
},1000);

I just got it running too, looks great! It looks like there's a watchdog timer that only gets called from Idle and that resets after 1 sec though? It's probably not a big problem, but worth bearing in mind :)

I notice you've made loads of comments on functions in the Wiki, and have done some comments like this.

If you did them with the comment after (in Markdown) like this instead (like the other functions in Espruino):

/*JSON{
  "type"     : "staticmethod",
  "class"    : "ESP8266WiFi",
  "name"     : "connect",
  "generate" : "jswrap_ESP8266WiFi_connect",
  "params"   : [
    ["ssid","JsVar","The network SSID"],
    ["password","JsVar","The password to the access point"],
    ["gotIpCallback", "JsVar", "An optional callback invoked when we have an IP"]
  ]
}
Connect the station to an access point.
 * ssid - The network id of the access point.
 * password - The password to use to connect to the access point.
*/
void jswrap_ESP8266WiFi_connect(JsVar *jsv_ssid, JsVar *jsv_password, JsVar *gotIpCallback) {

Then when build_docs.py is run over Espruino, it'll be able to automatically document everything - and crucially all the functions and their documentation are automatically kept in sync, and they come up as tooltips in the Web IDE.

Just some other things I noticed:

• Any chance you could develop with your fork of Espruino here? rather that the misspelt esp8266-espurino that has the directories moved around? I think I mentioned this before, but it's actually a huge bonus having the commit history when you're developing with Git, and you lose all that if you do it in a completely different repo. It'll also be a whole load easier to do pull requests for individual changes, and also for you to keep track of improvements I make.
  
• You seem to have put loads of effort into documentation on the Wiki, but stuff like this looks like a copy/paste of the reference pages? I can't tell if anything has changed at all. If you documented the functions in the code - like above - then everyone (not just the esp8266 project) could benefit from any additions you make.
  
• Same with the other stuff - if you put your documentation in a doc folder in the Espruino project (or even in some 'Espruino Internals' pages in EspruinoDocs), and kept issuing pull requests then we'd be able to keep everything in sync between your project and the main one. As-is it's going to be a nightmare to pull your documentation back in.
  
• ... also, when you write pages like this one I'd see in the pull request, and I could say 'it's documented in this source file' :)
  
• I would really suggest turning ESP8266WiFi into a library, so you use it like var wifi = require("esp8266"); wifi.xyz. It makes things more in sync with the other WiFi modules, and means people could stand a fighting chance of copy/pasting network-related code from other modules - all they'd need to change is the module name.
  
• You did ask on here how things should be named and I tried to offer some suggestions, but it looks like you've renamed everything anyway? For example the existing module has .getAPs that returns [ { ssid, enc, signal_strength, mac_address } ]. But you've added .getAccessPoints that returns [ {rssi, channel, authMode, isHidden, ssid} ]. Instead of .getIP it's now .getIPInfo - is there a reason for those changes? It seems crazy to me to make everything different - and it's just going to be far more difficult to change at a later date (especially as people are now starting to use it), and means that nobody can use the existing example code.
  
  

I don't mean to be negative - it just seems like you're doing a lot of extra work that is going to be hard to put back into normal Espruino... Like rather than contributing to the existing project, you're building your own one.

Howdy @Gordon

Great post. The primary goal is to ensure that you are 110% happy with the final inclusions that will be in the Espruino code base that you own. When the ESP8266 port is finished, as I say, the parallel Github repository will be destroyed (after harvesting things like the docs).

Let me try and provide thoughts on your comments and questions.

At a high level, a PRIMARY goal of the project is "do no harm" (just like a doctor). There is a snapshot of the Espruino project here:

https://github.com/esp8266-espruino/esp8266-espurino

This is not a fork but a "sand box" against which all development is progressing. This provides a rock steady base for ESP8266 specific work. Anything you or others may have "in-flight" will not impact ESP8266 specific work. But (I hear you say) ... what about changes that are made by the ESP8266 port to the base Espruino code? The good news is that so far ... those have been minimal AND ... each of those I have submitted to you as changes in a "master" fork which you have accepted.

So ... when the time comes to "merge" the ESP8266 port with the master Espruino code base, my hope is that the ONLY final changes will be the inclusion ESP8266 specific project files that will be delta NEW to the master code base with ... ideally ... ZERO changes to the rest of the code base that is not ESP8266 specific.

There are a couple of MAJOR exceptions to that ... which are to do with the build system. The ESP8266 port will introduce new "board files" and also significant additions to the Makefile.

The thinking is that on merge day ... there will be a "pull request" awaiting you which will add the new ESP8266 source files AND changes to the Makefile ... and once you approve ... we will be done. The changes that I find that are necessary in the core Espruino base ... these have been filtering in to you already for your review and approval.

So what will we loose following this mechanism? The answer is the file change history for the new ESP8266 files that don't yet exist in the master repository ... but since those are net new anyway ... that shouldn't be a problem. The history just now is ... well ... useless to you and me.

For the Wiki documentation ... I am huge on documentation. I don't do anything without writing a story behind it when done :-) ... For me right now, the documentation is easiest for me to capture in Wiki format. The new ESP8266 source files are being liberally commented internally. As for the generated end user docs ... you are 100% correct. When the files are provided for merging, they will be updated to include the docs necessary for "build_docs.py" (is the hope and intent).

I fully agree that a staticmodule of ESP8266WiFi is wrong and will indeed change to making it a library. That is the kind of polish that we need to capture before completion. In the original Proof of Concept, we used a staticmodule and ... so far ... haven't gotten to the stage of reworking to a library.

As for the naming of functions ... I fully agree with you that before completion, we should all get together and review for discrepancies and improvements. Today, I am using the names that are the ones used in the ESP8266 SDK as opposed to ones which have heritage in Espruino ... but by merge day, we should all be consistent.

As for the project as a whole ... once the Proof of Concept proved that Espruino running on ESP8266 was feasible, the next task was to approach it formally with a primary goal of building something that could "slot into" the primary Espruino project. While the naming and the docs are still not where either of us want them to be, the focus just now remains on getting the function working. As you know, the ESP8266 port is still currently broken on the connection callbacks and your post earlier today gave us the knowledge to attempt to rectify that. It is things like that that have my highest priority. I break tasks down into:

1. Tasks that we know how to accomplish and merely need to expend time to do them.
   
2. Tasks that we have unknowns upon which include designs, bug fixes and functional testing.
   
   

It is the items in category 2 that need most of our attention. The items in 1 can be accommodated before merge day and are merely chores. Items in category 2 are the things that keep me up at night.

The Espruino port to running on the ESP8266 is not complex ... where folks like me are having problems is in understanding how to augment the fantastic Espruino architecture. The core knowledge is locked up in your head and it is only by forum thread exchanges and deep study of the existing code that folks like me can begin to make progress. You have so many irons in the fire, I realize that the ESP8266 port can't commandeer too much of your time so your prompt response on the forum are awesome. If you do have time however, please please don't hesitate to contact me directory (kolban1@kolban.com or skype: neil.kolban) should you have spare time to give us architectural direction and guidance in a more direct fashion.

And ... less there be any doubt. The Espruino code running on the ESP8266 is 99% your current and existing code with the remaining 1% being the ESP8266 specific code ... and that speaks VOLUMES to the quality of what you have created.

Neil

Thanks!

I reckon what you're trying to do with the Sandbox repository is actually something that you could do with branches on your own repository. It's pretty much like having a separate repo, only you can pull in changes from other places, push your changes back, or delete them.

It would really be worth having a play around with them - it'll save you a lot of time and heartache!

Didn't realise about the naming - it definitely makes sense to use esp8266 naming for now (its surprisingly similar to Espruino's, just not quite there!)

I reckon it'd be worth pulling the ESP8266 stuff into the main Espruino repo as soon as possible though - it doesn't matter if it's finished or not. It's not going to hurt anything else as pretty much all your changes will be in libs/network/esp8266 by the look of it?

I think I should probably just start some pages on EspruinoDocs (that will end up on the main Espruino site) about the Espruino code itself - and maybe you could contribute to those? Because it's GitHub as well, the UI is almost identical to the Wiki.

Is it ok if I copy/pasted some of your existing documentation?

I'm doing a lot of answering questions on Espruino internals lately, and as @mjdietz is doing work on porting to NRF52 chips it makes sense to have all this stuff in one place rather than pasted all over the forum.

I've also noticed I tended to write documentation inside source files - which is actually pretty hard to find sometimes. It'd be well worth me referencing it from some main human-readable place :)

@Gordon - a map of the directory structure, and what functionality was in which files would be a big help for people starting to get into modifying the Espruino code, imo.

@Gordon,
You are absolutely welcome to lift any and all of the docs in the Wiki. I was thinking that at the end of the ESP8266 port project, they would be offered to you for assets for your disposal but if you want them now ... GREAT!! ... however, if you lift them now, tell me which files you lift and I'll delete them from the ESP8266 repository ... don't want to have duplicates. I saw the posts from Mr @mjdietz and have reached out to him in case we can collaborate on experiences building boards and share common experiences and questions in case you are not available.

My work day is nearly over and I'm excited to test out the connection callback designs :-)

a map of the directory structure, and what functionality was in which files

There is: https://github.com/espruino/Espruino/blob/master/README.md#directories-and-files

Which at least provides an overview...

The problem I think is even now, there is quite a lot of documentation around - it's just very hard to find everything. README.md is too huge at the moment and I think it's hard to even find stuff in there at a first glance.

Ok, there's now a lot more info here

I decided to document in the Espruino repo itself, because then I can link directly to the source files. Contributions would be great, even if you just post a GitHub issue asking for clarification.

... also pull requests that add code comments line See ../README_BuildProcess.md#wrapperfiles are great too ;)

I think we have just achieved a milestone. We now have an ESP8266 running a simple HTTP server where the JavaScript to do that should be the same generic JavaScript as would be used for other Espruino boards.

The sample JavaScript used is:

var http = require("http");
var httpSrv = http.createServer(function(request, response) {
    response.write("<html><body>");
    if (request.url == "/hello") {
        response.write("<b>Welcome</b> to the ESP8266 test.");
    } else if (request.url == "/goodbye") {
        response.write("<b>Please</b> come back again soon.");
    } else {
        response.write("Sorry ... I didn't understand!");
    }
    response.end("</body></html>");
});
httpSrv.listen(80);

The latest binaries of the firmware as of 9/25 onwards should be used for testing. See here for URL information:

https://github.com/esp8266-espruino/esp8266-espurino/wiki/07-Samples

Wow, that's fantastic news!

Does the HTTP client work as well?

Howdy @Gordon, we actually haven't tested the HTTP client stuff yet ... this is incremental bits and we want to be as methodical as we can be. I thought you might like to see this ... it is based on the design pattern you gave us over the last week ... it is the state diagram of a socket at the ESP8266 level ... (see attachments).

Which begs a new question ... where within the source tree of the Espruino project should we put document files (images) like this?

1 Attachment

That's great!

Hmm... tricky. I guess you could add the diagram inside 'doxygen/images' and could then reference it from a comment in the code? https://www.stack.nl/~dimitri/doxygen/manual/commands.html#cmdimage

But if you're doing a specific .md file on the network implementation, I guess we might need to start a docs folder?

The HTTP client (at least GET) has now been tested and is included in the 2015-09-26 binaries.

I reckon it'd be worth pulling the ESP8266 stuff into the main Espruino repo as soon as possible though - it doesn't matter if it's finished or not.

IMHO that would be really, really good to do asap. I'd hate to work on the port and then have to rename and rework a zillion things in order to integrate. Also, it would be really good to normalize the names now and not later.

And congratulations @Kolban, you just got on hackaday!

... but IMO the increased interest from this means it's probably even more important to get things in the main GitHub repo ASAP

Ok ... I'm sold ... TODAY is merge day. I have a full 8 hours to devote to this.

@Gordon I'm going to do the merge ... and then lock down the separate repository. What is your availability today to work with me on this task?

@tve If you are available today too ... let me know.

The chatter will be on the gitter stream ... see:

https://gitter.im/aplikatika/Espruino-on-ESP8266

@Gordon ... can you create a new Gitter stream for Espruino? Once the merge takes place and the separate project is locked down, I'd like to move everyone interested in Espruino chit chat (including ESP8266 support) onto that formal chat stream and since a chat stream associated with a Github project has to be created by the Github owner, that would be you.