Espruino modules packaged as NodeJS modules

Hey, I'm working on a little project to integrate the Espruino a bit more tightly into common NodeJS toolchains.

First, I'm repackaging the modules into something NodeJS can understand. That's underway here (master branch only).

I have another project, Spire, which is, at first glance, another one of those "command-line" tools to upload scripts. But, beneath the surface, there's a Duplex stream--theoretically you should be able to consume it as you would any other writable or readable stream. Its current implementation communicates via a serial port and STDOUT, but I would like to create an abstraction layer over this, so you can simply stream data to/from the Espruino via NodeJS. For example, you could set the Espruino to serve real-time data over a socket, then use NodeJS to pipe it into a database, file, or web app. There are other ways to accomplish this sort of thing, of course, but when dealing with large amounts of data, streams allow you to avoid memory consumption--this is especially important if you wish to send large amounts of data to the Espruino.

Anyway, I'm mostly just playing with ideas at this point, but I'm wondering if anyone has any input or would be interested in the outcome.

I think the Web IDE could benefit from using a communication interface such as this--and as-of-yet unwritten apps that people may wish to build on top of it.

I know you've come across https://github.com/espruino/espruino-tools and the code it uses - I think some of this would be really good if it was integrated into that?

Specifically I'd love to get browserify (or something like it) working inside the Web IDE - ideally with some offline minification too. If it were able to drop functions that weren't used, and inline/rename ones that were (I guess this is the idea?) then it'd save a load of Espruino's RAM.

I wonder whether a better way to accomplish that (than re-packaging all of Espruino's modules) is to override the 'module getter' in browserify to just look online for the Espruino modules? As chrome doesn't have access to the local filesystem, creating a copy of all the modules there won't really help - and as the library of modules grows over time it'll become slower and slower.

Or were you doing this so that the editor you use can find the modules and follow function definitions into them?

As far as Spire, will it use EspruinoTools? I ask because a lot of work has gone into (and will continue to go into) making EspruinoTools format code in the right way for Espruino in order to avoid subtle problems. Re-implementing that code really doesn't help very much, but if you could actually make EspruinoTools provide that nice stream interface (which the WebIDE Terminal could then connect to) it would mean that people could pull in that library, or in fact use the command-line tool and pipe to stdin/out.

In the module you ask about the binary notation. It's because when accessing registers in hardware it's often a lot easier to read. It's part of ES6 so you could probably just run node --harmony and it'd work fine?

Just wondering about https://www.npmjs.com/package/espruino too - would it me possible for you to rename this to espruino-modules or similar? It's my fault for not grabbing the module name as soon as I could I guess - but personally I think it might be confusing for people having a main module called espruino that doesn't really involve communicating with Espruino at all. I think it probably makes more sense to stick the espruino-tools command-line tool under that name.

Let me think this stuff over and get back to you this evening. I'm thinking an espruino module should include everything you need to write your Espruino code in NodeJS, and a programmatic interface into the board. So basically official modules + communication layer, but no CLI tool. espruino-cli for that.

(FWIW, I grabbed the name so someone wouldn't do something dumb with it. Simply "squatting" a module name is not allowed and can get your publishing privs revoked, so needed to have something there anyway.)

Hmm, it's a tricky one... 'write Espruino code in NodeJS' is a big vague - I don't think you actually want to develop Espruino code in NodeJS. Instead, you want tools that help you to write code for (and to interface with) a device that is running Espruino?

To me, that kind of means the command-line tools. You don't need the modules on your PC to develop with, because they're automatically uploaded by the command-line tools as they are needed.

I guess he's thinking in the line of writing server side code that controls espruinos in an API-like fashion, rather than simply sending it serial commands?

Something like

var espruino1 = new Espruino("/dev/tty...");
espruino1.LED1.set();
//and maybe a simple way to wrap into custom code
espruino1.myFunctionOnEspruinoBoard();

//EDIT: a very simple implementation could do
espruino1(function(){ LED1.set(); }); //then simply toString and send over serial

Hi, sorry didn't have time to reply until now.

No, it's neither of these things really. :D

Before I go into more detail, please understand that come from a software engineering background. I work developing a large suite of SaaS web apps, as well as OSS projects in my free time. I've encountered many issues centered around dependency management, code integration, and deployment. I'm not trying to tell you guys, "I know what I'm talking about". I just have, perhaps, different a experience here, and I'd like to use it to propose a few things.

So, what I'm about to suggest may not have a 1:1 relationship to the hardware world. If I were to have originally designed how the Espruino modules were packaged, and toolchain structured, it would be different than what exists now. Maybe you agree with my points, but haven't had the resources to take action. Or, if you disagree, hopefully you can tell me why I'm wrong, because my experiences with software are not applicable.

At the very least, it should be interesting to think about things this way!

Espruino Modules in npm

1. npm is not just for JavaScript packages that run on NodeJS. We're seeing an influx of packages which run in the browser's context. npm is simply a package management tool and registry for JavaScript; the context doesn't matter.
   
2. If NodeJS is a context and the browser is a context, then Espruino is also a context. More importantly, it's just JavaScript.
   
3. Given the above, Espruino modules:
   
   1. Albeit small in size, should each have their own repository.
      
   2. Should be published to npm, and individually installable.
      
   3. Should be versioned using Semantic Versioning and tagged appropriately.
      
4. If tools exist to translate one context to another (they do; Browserify being a prime example), a tool should exist to translate Espruino code to another context, and vice-versa.
   
   1. For example, I've rewritten example code for the DHT11 (temp/humidity sensor) in a NodeJS context. It's almost identical, of course, since require() is used in both contexts.
      
   2. A trivial Browserify "transform" can convert this source into code to run in an Espruino context. Of course, it will need some work, as Gordon mentioned there are some edge cases the Web IDE handles.
      
      

Dependency management is a difficult problem. npm represents many thousands of hours of development time working to solve this problem. Out of the many package management systems that I've used, npm is likely the best.

The Espruino Web IDE has all but ignored dependency management by opting for automatic installation instead. Not everyone wants this!

Automatic Installation

Automatic installation of required modules seems very strange to me. I understand that this can help people get up-and-running, here's why I find it harmful:

1. I write some code to be run on an Espruino
   
2. I "deploy" it to an Espruino
   
3. Time passes, and one or more of the modules I'm using has been upgraded.
   
4. I make a small change to my code.
   
5. I redeploy to an Espruino, and my code fails, because the remote module has breaking changes.
   
6. Alternatively, I redeploy and my code fails because my code had a workaround for a bug which was fixed in the meantime.
   
7. Caching of modules which are automatically installed increases any issues by an order of magnitude, because now you must consider the cached state of any given module for any given problem you're having.
   
   

The only practical drawback I can think of to using explicit versioning is that code must be written to support it. Mitigating this is that 98% of the work will done by npm, and not the Espruino toolchain.

The "extra step" of declaring your dependencies can be avoided (see below).

A caveat is that most Espruino users probably don't care about this stuff, which means it must be implemented in such a way that it's transparent.

An Alternate Workflow

To support those that do care about versioning, like myself, I'm envisioning an alternate workflow like this (it could be an "advanced" mode or something):

1. Create an Espruino project. A package.json manifest is created for you. Note: a main file must be present in the manifest--probably the .js file you are working in.
   
2. Decide you want to use a DHT11 in your project.
   
3. Select the DHT11 Espruino module via command-line tool or Web IDE. Specify "latest" version, "fuzzy" version, an exact version--up to you--but "latest" would be the default.
   
   1. Respectively:
      
      1. npm install --save dht11 (latest)
         
      2. npm install --save dht11@^0.1.0 (anything up to but not including version 1.0.0)
         
      3. npm install --save-exact dht11@0.1.2
         
   2. Each of these commands writes information to the manifest.
      
4. Because your module is written to the manifest, installation of the module becomes effectively "automatic" upon a subsequent execution of npm install.
   
5. The final step before deployment would be an "npm install" before repackaging into an Espruino context.
   
6. A few common commands should be supported via Web IDE (not necessary on command line), such as npm remove, npm upgrade and npm info.
   
   

Keeping What We Have

To retain the current workflow--and make these changes transparent--but also be compatible with the alternate workflow:

1. Dependencies are determined in nearly the same manner the Web IDE does it now (scan for require(...) or whatever)--except probably with a 3p lib instead, because there's really no need to reinvent the wheel.
   
2. Upon deployment, package.json is written with the dependencies if it does not yet exist. The "version" written to package.json will be *, which lets npm choose. Typically this is the "latest" version of the dependency.
   
3. Before repackaging, npm install is simply run, and the modules are installed "automatically".
   
4. Because a package.json is written, users can switch between the workflows without any incompatibilities.
   
   

"Deployment"

Browserify can do basically everything necessary here, coupled with a custom transform.

One thing I don't understand is this: what is the point of not minifying your files before uploading? You can't really debug it anyway, right, so why do you care what your code looks like?

Regardless, I've identified the following steps:

1. As mentioned above, the first step here would be to npm install and ensure all modules are present and accounted for.
   
2. browserify is run at the entry point specified by the main property of package.json
   
3. Browserify then will "bundle" the dependencies. This includes official Espruino modules and any random npm or NodeJS core modules you wanted. The result is one .js file with everything in it. It just works.
   
   1. By default, Browserify will ignore or otherwise suppress modules which cannot run in the browser context. For example, if you need the fs module, you're SOL.
      
   2. The same goes for the Espruino context.
      
4. Several "transforms" occur, in this order:
   
   1. The code is altered or otherwise munged to be Espruino compatible. For example, atob is not present in NodeJS, but it is on the Espruino, so we declare atob as a global, and not a 3p package we need to bundle. I believe the code must be wrapped in an onInit(), but that may need to be the last step.
      
   2. The code is minified
      
   3. The code is gzipped (??? not sure about this; I saw some sort of gzip implementation in the web IDE code, but wasn't sure why it was there).
      
5. The final output is piped to the Espruino at an open serial port.
   
   

Where and How

We should spend a bit more time thinking about the "minimum viable product". It's clear to me that we don't have to do this all at once, but what's the bare minimum we can start with?

• The espruino package:
  
  • Two core features:
    
  • A serial communication layer. This would be written in NodeJS proper, hopefully using streams. To get it into the Web IDE, we could just browserify it (note that you can tell browserify to allow things like filesystem access). The current code to handle communication, which lives in the Web IDE, would be extracted and reused as much as possible.
    
  • An adapter that would handle interfacing with npm. This may have to be in the form of issuing shell commands instead of using its programmatic API, because obviously the Web IDE doesn't run under NodeJS, and I doubt Browserify will want to place nice with it.
    
  • We should think about this particular module very carefully, because its API becomes the platform. If someone is going to write an Espruino plugin for WebStorm, Sublime Text, Atom or even vim, it's going to have to leverage this module.
    
  • I would write unit tests for this, since we're building on top of it.
    
  • Anything else here?
    
• The espruino-cli package:
  
  • A thin wrapper around the espruino package.
    
  • To be "preferred" globally--typical use case is the shell, but if you were going to say, publish your own Espruino project on npm, you could add espruino-cli as a dependency, so whoever has your project would be able to deploy without extra steps.
    
  • Effectively replaces both espruino-tools and my spire tool.
    
• Official Espruino modules could be given their own repos and versioning and all that stuff, and perhaps we could just hack something together for the Web IDE to use it in the interim.
  
• The Web IDE should have its own package. It will likely be reduced vastly in size.
  
  

Why?

This buys Espruino a lot.

• If a user is familiar with the NodeJS toolchain, it's there.
  
• If the user doesn't need that, and just wants to quickly deploy simple scripts--that doesn't change.
  
• I can't think of a drawback to versioning the modules, if it's behind-the-scenes for those who do not care about it.
  
• An API built on NodeJS (or iojs) opens up the Espruino development environment to a greater audience.
  
• Perhaps close the feature gap a bit with the T*ssel. On Espruino, you get to write NodeJS code too. It just doesn't run NodeJS--but it doesn't have to.
  
• Potential integration with popular packages such as johnny-five.
  
• Task runners, unit tests, linters--make it natural to use tools like these if the user wants to.
  
• It makes me giddy
  
  

If I have any grumbles about the Web IDE, it seems that it's gone the same path the Arduino IDE has--it does not provide an API. To use an Arduino, you basically need to be a human interacting with a window. There is a CLI tool, ino, which is pretty unstable, but also seems the best alternative. It's unstable because the Arduino IDE team will freely introduce breaking changes--why shouldn't they? It's not an API!

Likewise, the Web IDE is not an API. Anything package trying to leverage it will face the same instability, unless you happen to develop both the Web IDE and a CLI-tool--because you know what's going to break. But the Web IDE shouldn't be an API, it should be a layer on top of one. Granted, it's much nicer than the Arduino IDE, but still should just be an interface.

In conclusion...

Ideally, if the resources and interest is there, I'd like to work with Espruino instead of in parallel. I don't want to develop and maintain an entire separate toolchain on my own. I would probably just continue to patch things together so it works for me, but I have no interest in maintaining "competing" tools.

Let me know what you think. These are just a bunch of ideas. I'd be happy to discuss all of them. And I apologize if they've been suggested before and shot down--I just got this board a couple weeks ago. :D

Some thoughts in random order:

Espruino IDE vs 'API':
You may get what you need from the espruino-tools project. I think it's unfair to compare the Web IDE to the Arduino IDE, which derives from Wiring which in turn is a fork of Processing. The Web IDE is much easier to extend and based on more modern and well documented technologies. I for one like it (even though I've ended up rarely using it, it's neat for demos and when trying out interfacing with new devices).
With the espruino-tools it is very easy to use practically any editor. I use sublime, brackets and atom with success.

NPM:
One concern with using npm (especially if npm install is not done each upload, which would then probably add some overhead in the time it takes to upload) would be that it kind of encourages quick fixes per user. With the current system, a user is more inclined to file a report, or fix the problem immediately.

Also, making npm available could quickly lead to a lot of questions about why things does not work etc. The Espruino is an embedded device with low memory and the interpreter does have some 'quirks' which a module author should know about and leverage/work around. Pulling in something 'simple' as _underscore will not probably be very successful.

"I make a small change to my code":
There's no such thing ;)
You should and would test the thing. So far I've had one such instance with the RFID module, it broke on my 1v72. The first thing I did was to upgrade to 1v74 (it was a module that had been running for a while) - and you guessed it. It worked!

In conclusion:
I think both approaches make sense. With the npm approach you put more responsibility on the user.
I imagine scenarios like this:
User reads a blog with a project that uses explicit versions (that's what bloggers are used to, because often the runtime itself can be upgraded with npm - so you can 'require' an identical setup). Then user copies code and config. Uploads but it does not work. Why? Because there's been an API change since the release and the Espruino running on the user's device is newer and incompatible.

Maybe the Espruino could be an npm module?
Maybe modules should alert the user if it's using a well known incompatible module/espruino version pair?

I don't mean to seem overly negative, I would probably like this feature - but I'm not convinced it's the right approach.
Npm shines for web because there are so many permutations of the javascript environment, but for the Espruino there's just one. Furthermore, the code that runs on the device should be written specifically for the device, so it's really not much argument about being able to pull in other npm modules.

My all too many bytes on the matter.

Ok, I'm a bit busy right now, so forgive me if I'm missing something...

You can't really debug it anyway, right, so why do you care what your code looks like?

You can debug by checking and setting variables, and changing code on the fly. A lot of Espruino users do this routinely using the left-hand side of the IDE.

It's actually one of the main ideas behind Espruino. If it wasn't I'd have just got the Web IDE to compile code on the PC, and Espruino would have been much faster than it currently is. However as it is, you can come back to a project, plug it in to your PC, and tweak it without even having the original source code available.

You're coming at this from the point of view of a node.js developer and assuming it's way behind Tessel because it's not what you're used to. But the reality is that they are aimed at very different areas... Tessel is going after node.js devs and that's great - but Espruino is about creating things quickly and easily without having to learn a bunch of extra tools.

Version control

Libraries changing under you is a tricky one, and yes, it is extremely frustrating.

NPM does solve that, however any solution to the problem needs to run completely in the Web Browser with extremely limited permissions. Right now you can actually program an Espruino board from your iPhone's headphone jack, from a normal website, without any special apps or software.

I'm not convinced that what you're actually suggesting will do that? If you're suggesting that everyone needs to have filesystem access, install node, npm, and a bunch of modules first then it's not an improvement at all.

I don't want to develop and maintain an entire separate toolchain on my own

Please don't do that, at least try and leverage EspruinoTools. I can pretty much guarantee that you'll lose interest in a few months, and then it'll just be sitting there on NPM along with node-espruino, grunt-espruino and espruino-cli as yet another tool that uploads 'hello world' but fails when faced with anything more complex.

Sorry, I don't want to seem negative - but it seems to me that maybe you haven't used Espruino that much yet, and you're trying to apply the lessons you learned on million-line enterprise projects to a device that can handle maybe 10k lines maximum?

It's not to say they're not good ideas - but the reality is that NPM for Node.js is a necessity because of the amount of different libraries people tend to pull into their projects. On Espruino we haven't even hit the stage where version control is a minor annoyance yet.

At the moment, I think <10 people have contributed modules to Espruino. Keeping the modules together means I can be a lot more sure that they're of good quality and all work with Espruino, and I think that's a real bonus when compared to something like Arduino and Node.js where you have 5 different libraries which all do the same thing.

At some point we'll reach the stage where I can't keep track of everything, and then yes, NPM will make a huge amount of sense. But until then I think switching over to NPM may not actually be in everyone's best interests.

You can't really debug it anyway, right, so why do you care what your
code looks like?

There's this http://www.espruino.com/Reference#l__global_edit :)

Hello,
I am a 20+ years developer working on Telco internal software system. After a hard day, I prefer a simple way of developing, testing and running my controllers at home. So far Espruino is the easiest board I have used. True 'Plug and Play' board.

I can see both worlds... and kind-of 'going-back' to more 'free-flying' after or while still working with one leg+ in a 'perfectly well', very heavy (prison-gang)-tool-chained world is a challenge... to say the least... as for me.

That does not mean that we have to be stupid and throw out all what 'we' have learned since the mythical man month and while growing with all the more recent code dev, build, test, deploy, and management 'techniques'. But applying them in their spirit by living out a more self-disciplined approach than in a system-over-policed way is the way in this resource-constrained (Espruino) micro controller world.

I feel like being allowed to take the trainings-wheels off when switching to the Espruino platform. ...may be it is because I learned (and still learning) in the well guarded context. On the other hand I have to confess that even with Espruino I still take advantage of the guarded world:

• To figure out and get to a working algorithm or formula I resorted to the browser world and made the development there to then 'move' the code (slimmed down) into the Espruino context (see reply #13 to this post about GPS code/module/library leaking memory before 'satellite lock', where I wondered what a more Espruino suitable way of (coding) life would be for a piece of (functionally) perfectly correct working code for decoding the latitude zones. How your code looks matters a lot.... Any whatso-ever tool-chain can do only that much 'good' to the code what is already implemented in the spirit of the code. Since the example is pretty small - and almost trivial - I just followed the spirit of what has been discovered in last decades rather than pulling in all the tool arsenals...
  
• Another example is the memory manager for external memory - EEPROM/FRAM/MRAM - which is still in the works and has not moved yet to the Espruino wonder land ;-). A Clickable working example in this post. Just click the link of the uploaded - self contained - mm.html file. For more details how works and some use cases see posts #14 through #16 in the conversation about Running code off of an EEPROM.
  
• Working lately heavily on the existing testing plug-in of the Espruino WebIDE I'm thinking to go beyond just jQuery with some plug-ins to 'hack' the DOM. I got triggered to work on the plug-in because I was looking for how I can do testing of my javascript code in the Espruino context (last but not least code that I 'developed outside and moved in onto' Espruino). I wanted to have only the runtime code in Espruino and 'all other' code 'outside', such as the regression testing and also monitoring. For the 'outside' the WebIDE is a good choice. It has already a certain amount of infrastructure and most important a good and easy to use connectivity in close vicinity to the board (over USB). Currently I'm tackling UI and functionality enhancements, but in the end I'd would like also to add (Espruino) event driven test data flow on individual test data point vs. current polling of all defined and enabled data points at once on interval . To do that I'm considering aspect oriented programming techniques to have the least - but dynamic - intrusion into the runtime code. Furthermore, I'd like to add some regression function, because current testing is all manual triggered and by 'manually' evaluating the test results (no assertions). - For now I still learning more of the WebIDE framework and setup and jQuery (...I'm more from the dojo / dijit and backbone / underscore / bootstrap wold - and miss some of that contexts...).
  
  

As you - @boneskull - can see, there are things going on into your direction, but it should - as you say - be transparent to the user - if the user does 'not care' explicitly - and should happen in the spirit of Espruino... without compromising its resource frugalness.

For sure, more is to be conversed about and figured out how to make available for the Espruino community.

Btw, @alexanderbrevig, thanks to point out function edit. It does not only allow to modify (particular) code in an instance on the living object (without loosing the data) but also capture the final version to put it into the (edited) file... ;-)

@alexanderbrevig

One concern with using npm (especially if npm install is not done each upload, which would then probably add some overhead in the time it takes to upload) would be that it kind of encourages quick fixes per user. With the current system, a user is more inclined to file a report, or fix the problem immediately.

I'm not sure I understand what you're getting at here.

@Gordon

I'm not convinced that what you're actually suggesting will do that? If you're suggesting that everyone needs to have filesystem access, install node, npm, and a bunch of modules first then it's not an improvement at all.

No, this will not work on an iPhone or whatever. You would need NodeJS installed.

You're coming at this from the point of view of a node.js developer and assuming it's way behind Tessel because it's not what you're used to. But the reality is that they are aimed at very different areas... Tessel is going after node.js devs and that's great - but Espruino is about creating things quickly and easily without having to learn a bunch of extra tools.

I want to first point out I'm not saying Espruino is "behind" Tessel. Please forgive me if I gave that impression. I understand they are different tools.

Perhaps what I didn't understand is that "ease of use" is paramount to the project.

With the "alternate" workflow I detailed, you should probably know a little more about what you're doing, but regardless, the interaction with npm would probably be obscured behind the IDE. Maybe you missed the bit about "keeping what we have" which would not require the user to even know npm was behind the scenes.

But, it sounds like, at the current time, using npm is a no-go because we don't want to rely upon a filesystem? If the likelihood of a user programming an Espruino from an iPad is greater than the likelihood of a user wanting npm integration, so be it. I have no data in either direction.

Please don't do that, at least try and leverage EspruinoTools. I can pretty much guarantee that you'll lose interest in a few months, and then it'll just be sitting there on NPM along with node-espruino, grunt-espruino and espruino-cli as yet another tool that uploads 'hello world' but fails when faced with anything more complex.

Please consider this constructive criticism: I began writing my own CLI tool because leveraging EspruinoTools is extremely awkward. It's not an npm module, so I can't add it as a dependency. It's not versioned, so even if I did, it'd be a crapshoot. It doesn't seem to expose an isolated API layer, which means you need packages like jsdom to even get at them. It exports global variables and doesn't use something like AMD, so I'm not sure whether I need to load the whole thing at once or not. My hope was to actually develop an isolated API layer, wherein I could lift code from EspruinoTools, and build a CLI on top of it (cause that's easy), but publish it as a standalone module which could be extended in any which-way.

Ideally, it would become the "first-class" API for communication with the Espruino, and the Web IDE itself would simply consume it. But without some sort of buy-in, I don't see the project as having much future. I think I mentioned ino, the cmd-line tool for Arduino. They don't work closely with the IDE team--don't have "buy-in"--and thusly have continuous, never-ending issues of the exact nature you're talking about.

(In a perfect world, the Arduino IDE and ino would be built upon a common codebase, like what I'm proposing here)

I will relinquish the espruino module if/when you want to do something with it, but we probably shouldn't leave it empty or in a "placeholder" state to avoid squatting. You can email me about this if you wish; I can't seem to figure out how to "subscribe" to a thread here.

Gordon, I would like to apologize for coming out of left-field on this, and trying to jam the Espruino into a hole which it clearly does not fit. As I wrote, my point of view is as a software engineer. Hardware is a hobby of mine. I'm interested in increasing accessibility of hardware to software developers, so they can more readily apply their skills to the platform(s). Hardware which supports high-level languages is a great start; I suppose that's what drew me to Espruino originally. Another way is to support common toolchains and methodologies. There are a lot of things.

(I do sense a general unease about the influx of software developers to hardware, perhaps spurred by the popularity of the Arduino. I am concerned I won't get much support in my efforts. Some of this venom comes from garden-variety elitists who don't like the "dumbing down" of hardware, but others arguments include a lot of "ain't-broke-don't-fix-it".

For example, I'd smile if the official Espruino modules each had unit tests. Funny, right? Maybe to some it seems like extra complexity, or simply unnecessary. When you're hacking on a breadboard, you don't have a second breadboard that asserts the first breadboard works--you just look at it, and if it works, it works. I don't feel that the same strategy works well for software libraries, even if those libraries run on a microcontroller.)

Anyway, I don't need to spend another 2 hours writing this post... :D

@boneskull
RE npm leads to per-user fixes:
When a library is pulled down to local file system, and the user finds a bug - and the fix - it's very convenient to simply fix the problem locally. Of course this is a completely wrong approach but users tend to bend the rules. With the current way things work, the only way to fix a bug is to fix it for everyone. Which is good.

What scenario do you want to realize with an API?
What are you trying to do with the espruino-tools that you're having problems with?
I'm pulling it in with npm as such: dependencies: { "espruino-tools": "git+https://<token-from-github>:x-oauth-basic@github.com/espruino/espruino-tools.git" /*...*/ }

Then

var spawn = require('child_process').spawn;
var child = spawn('node', ['/path/to/espruino-tools/index.js', '-p', '/dev/ttyACM1', 'mycode.js']);

child.stdout.on('data', function(stream) {
  //look at result of espruino-tools
});

*This is written from memory on my windows disc, the code is on linux so may be something wrong. Let me know if you want me to post the real stuff...

One of the reasons for Espruino's creation was the initial 'ease of use' - putting all the tools I could on the chip itself meant that you need very little in the way of software to communicate - in fact just a serial terminal for simple coding. It's unfortunate that with modules, Assembler, minification, etc. the software on the host has actually become quite a bit more complicated - but at least it can still be installed with a few clicks in Chrome...

As a node dev you'll already have node.js installed and will be used to it, so it's easy to just assume it'll be there. I think it'd surprise you how much difficulty people would get in installing it though - especially somewhere like a school where the IT department is very strict about what can and cannot go onto the system.

There's also the issue of platform support - while communication from iPad/etc isn't really used at the moment (with blockly it might get some use in education soon though), a surprising amount of people do use the Web IDE with Chrome OS.

That's not to say we can't use NPM though - there's already some simple support in the Web IDE for NPM modules containing a single file (you mentioned gzip being included, and it's there for unpacking those modules). It could be expanded relatively easily, although obviously it's a shame to reimplement something that already exists (it doesn't have any of the version control stuff at the moment).

As I'd said before I don't think we should move all the modules over to NPM just yet, but at least it doesn't rule it out. The tests are definitely a good idea, and then having a separate repo per module starts to make a lot of sense. While modules that just interface to hardware can't have many meaningful tests, we're starting to get non-hardware libraries like MQTT that would benefit hugely from testing.

I totally agree about EspruinoTools - it really needs to be a module that can easily be used. The way I see it, there are a few separate parts:

Communications

This is really bad at the moment, with several JS files implementing the same interface. You just have to load the one you want.

Ideally we'd have something that could load multiple communication interfaces. You'd call .getPorts() and it'd return a list of all available interfaces - TTYs, network sockets, and audio jacks. You could then connect() to one of them and get the ability to read/write data.

Now here, streams would seem like a good bet. I guess you know more about this than me, but are they something that is implemented on the browser, or are they Node-specific? If they're node-specific I'd guess it's actually a no-go again, but we can just implement write and the data event when running under Chrome and leave it at that.

Code transformation

As I guess you'd noticed, there's sort of a framework where each bit of code places itself inside a global 'Espruino' variable, uses a 'Config' object, and calls addProcessor to register functions that get called on the code at various points.

Basically I had to come up with something that'd run in the browser with no filesystem access, and I wanted to use the code directly (rather than shoving it through browserify) so it could be debugged and changed from within Chrome's debug tools.

UI stuff

The UI is handled the same way - the Web IDE just adds to what EspruinoTools provides.

It's not fantastic, but it's there now, and it works on node and in the browser.

My hope was to actually develop an isolated API layer, wherein I could lift code from EspruinoTools, and build a CLI on top of it (cause that's easy), but publish it as a standalone module which could be extended in any which-way.

This sounds like a good plan. Personally, rather than lifting code out of EspruinoTools, could we not just add an API to the existing EspruinoTools repo and stick it in NPM? It wouldn't be an epic refactor, but would mean that EspruinoTools became something truly useful.

The existing command-line tool could then be modified to use that API and added to NPM too, and everyone's happy?

At some point later on we could refactor what's there, but as you say, at least there would be a consistent API so even if the internals were changed, it didn't matter.

@Gordon Actually it seems NodeJS and ChromeOS are not mutually exclusive.

Yes, you can - but it looks like you have to put it into developer mode first (which IIRC wipes the hard disk). Sadly it's not as easy as going to the chrome web store and clicking 'install' :)

Webpack's shiny new feature is tree shaking (when writing with ES6 Module format rather than CommonJS, or using a webpack loader that takes current CommonJS and outputs ES6). We can set Babel as a transpiler, and make a babel-preset-espruino which compiles only the language features that Espruino doesn't have natively yet. After that, running the "webpack" command can compile a minified bundle from an entry point (unused code not included, and duplicate code consolidated), then one more step with the espruino CLI will upload the file.

I'm sure that can then be easily automated in the IDE.

Will webpack run fully in-browser? Definitely having some tree-shaking, plus the ability to minify using some of the ES6 that Espruino supports, would be awesome.