Espruino modules packaged as NodeJS modules

Hi, sorry didn't have time to reply until now.

No, it's neither of these things really. :D

Before I go into more detail, please understand that come from a software engineering background. I work developing a large suite of SaaS web apps, as well as OSS projects in my free time. I've encountered many issues centered around dependency management, code integration, and deployment. I'm not trying to tell you guys, "I know what I'm talking about". I just have, perhaps, different a experience here, and I'd like to use it to propose a few things.

So, what I'm about to suggest may not have a 1:1 relationship to the hardware world. If I were to have originally designed how the Espruino modules were packaged, and toolchain structured, it would be different than what exists now. Maybe you agree with my points, but haven't had the resources to take action. Or, if you disagree, hopefully you can tell me why I'm wrong, because my experiences with software are not applicable.

At the very least, it should be interesting to think about things this way!

Espruino Modules in npm

1. npm is not just for JavaScript packages that run on NodeJS. We're seeing an influx of packages which run in the browser's context. npm is simply a package management tool and registry for JavaScript; the context doesn't matter.
   
2. If NodeJS is a context and the browser is a context, then Espruino is also a context. More importantly, it's just JavaScript.
   
3. Given the above, Espruino modules:
   
   1. Albeit small in size, should each have their own repository.
      
   2. Should be published to npm, and individually installable.
      
   3. Should be versioned using Semantic Versioning and tagged appropriately.
      
4. If tools exist to translate one context to another (they do; Browserify being a prime example), a tool should exist to translate Espruino code to another context, and vice-versa.
   
   1. For example, I've rewritten example code for the DHT11 (temp/humidity sensor) in a NodeJS context. It's almost identical, of course, since require() is used in both contexts.
      
   2. A trivial Browserify "transform" can convert this source into code to run in an Espruino context. Of course, it will need some work, as Gordon mentioned there are some edge cases the Web IDE handles.
      
      

Dependency management is a difficult problem. npm represents many thousands of hours of development time working to solve this problem. Out of the many package management systems that I've used, npm is likely the best.

The Espruino Web IDE has all but ignored dependency management by opting for automatic installation instead. Not everyone wants this!

Automatic Installation

Automatic installation of required modules seems very strange to me. I understand that this can help people get up-and-running, here's why I find it harmful:

1. I write some code to be run on an Espruino
   
2. I "deploy" it to an Espruino
   
3. Time passes, and one or more of the modules I'm using has been upgraded.
   
4. I make a small change to my code.
   
5. I redeploy to an Espruino, and my code fails, because the remote module has breaking changes.
   
6. Alternatively, I redeploy and my code fails because my code had a workaround for a bug which was fixed in the meantime.
   
7. Caching of modules which are automatically installed increases any issues by an order of magnitude, because now you must consider the cached state of any given module for any given problem you're having.
   
   

The only practical drawback I can think of to using explicit versioning is that code must be written to support it. Mitigating this is that 98% of the work will done by npm, and not the Espruino toolchain.

The "extra step" of declaring your dependencies can be avoided (see below).

A caveat is that most Espruino users probably don't care about this stuff, which means it must be implemented in such a way that it's transparent.

An Alternate Workflow

To support those that do care about versioning, like myself, I'm envisioning an alternate workflow like this (it could be an "advanced" mode or something):

1. Create an Espruino project. A package.json manifest is created for you. Note: a main file must be present in the manifest--probably the .js file you are working in.
   
2. Decide you want to use a DHT11 in your project.
   
3. Select the DHT11 Espruino module via command-line tool or Web IDE. Specify "latest" version, "fuzzy" version, an exact version--up to you--but "latest" would be the default.
   
   1. Respectively:
      
      1. npm install --save dht11 (latest)
         
      2. npm install --save dht11@^0.1.0 (anything up to but not including version 1.0.0)
         
      3. npm install --save-exact dht11@0.1.2
         
   2. Each of these commands writes information to the manifest.
      
4. Because your module is written to the manifest, installation of the module becomes effectively "automatic" upon a subsequent execution of npm install.
   
5. The final step before deployment would be an "npm install" before repackaging into an Espruino context.
   
6. A few common commands should be supported via Web IDE (not necessary on command line), such as npm remove, npm upgrade and npm info.
   
   

Keeping What We Have

To retain the current workflow--and make these changes transparent--but also be compatible with the alternate workflow:

1. Dependencies are determined in nearly the same manner the Web IDE does it now (scan for require(...) or whatever)--except probably with a 3p lib instead, because there's really no need to reinvent the wheel.
   
2. Upon deployment, package.json is written with the dependencies if it does not yet exist. The "version" written to package.json will be *, which lets npm choose. Typically this is the "latest" version of the dependency.
   
3. Before repackaging, npm install is simply run, and the modules are installed "automatically".
   
4. Because a package.json is written, users can switch between the workflows without any incompatibilities.
   
   

"Deployment"

Browserify can do basically everything necessary here, coupled with a custom transform.

One thing I don't understand is this: what is the point of not minifying your files before uploading? You can't really debug it anyway, right, so why do you care what your code looks like?

Regardless, I've identified the following steps:

1. As mentioned above, the first step here would be to npm install and ensure all modules are present and accounted for.
   
2. browserify is run at the entry point specified by the main property of package.json
   
3. Browserify then will "bundle" the dependencies. This includes official Espruino modules and any random npm or NodeJS core modules you wanted. The result is one .js file with everything in it. It just works.
   
   1. By default, Browserify will ignore or otherwise suppress modules which cannot run in the browser context. For example, if you need the fs module, you're SOL.
      
   2. The same goes for the Espruino context.
      
4. Several "transforms" occur, in this order:
   
   1. The code is altered or otherwise munged to be Espruino compatible. For example, atob is not present in NodeJS, but it is on the Espruino, so we declare atob as a global, and not a 3p package we need to bundle. I believe the code must be wrapped in an onInit(), but that may need to be the last step.
      
   2. The code is minified
      
   3. The code is gzipped (??? not sure about this; I saw some sort of gzip implementation in the web IDE code, but wasn't sure why it was there).
      
5. The final output is piped to the Espruino at an open serial port.
   
   

Where and How

We should spend a bit more time thinking about the "minimum viable product". It's clear to me that we don't have to do this all at once, but what's the bare minimum we can start with?

• The espruino package:
  
  • Two core features:
    
  • A serial communication layer. This would be written in NodeJS proper, hopefully using streams. To get it into the Web IDE, we could just browserify it (note that you can tell browserify to allow things like filesystem access). The current code to handle communication, which lives in the Web IDE, would be extracted and reused as much as possible.
    
  • An adapter that would handle interfacing with npm. This may have to be in the form of issuing shell commands instead of using its programmatic API, because obviously the Web IDE doesn't run under NodeJS, and I doubt Browserify will want to place nice with it.
    
  • We should think about this particular module very carefully, because its API becomes the platform. If someone is going to write an Espruino plugin for WebStorm, Sublime Text, Atom or even vim, it's going to have to leverage this module.
    
  • I would write unit tests for this, since we're building on top of it.
    
  • Anything else here?
    
• The espruino-cli package:
  
  • A thin wrapper around the espruino package.
    
  • To be "preferred" globally--typical use case is the shell, but if you were going to say, publish your own Espruino project on npm, you could add espruino-cli as a dependency, so whoever has your project would be able to deploy without extra steps.
    
  • Effectively replaces both espruino-tools and my spire tool.
    
• Official Espruino modules could be given their own repos and versioning and all that stuff, and perhaps we could just hack something together for the Web IDE to use it in the interim.
  
• The Web IDE should have its own package. It will likely be reduced vastly in size.
  
  

Why?

This buys Espruino a lot.

• If a user is familiar with the NodeJS toolchain, it's there.
  
• If the user doesn't need that, and just wants to quickly deploy simple scripts--that doesn't change.
  
• I can't think of a drawback to versioning the modules, if it's behind-the-scenes for those who do not care about it.
  
• An API built on NodeJS (or iojs) opens up the Espruino development environment to a greater audience.
  
• Perhaps close the feature gap a bit with the T*ssel. On Espruino, you get to write NodeJS code too. It just doesn't run NodeJS--but it doesn't have to.
  
• Potential integration with popular packages such as johnny-five.
  
• Task runners, unit tests, linters--make it natural to use tools like these if the user wants to.
  
• It makes me giddy
  
  

If I have any grumbles about the Web IDE, it seems that it's gone the same path the Arduino IDE has--it does not provide an API. To use an Arduino, you basically need to be a human interacting with a window. There is a CLI tool, ino, which is pretty unstable, but also seems the best alternative. It's unstable because the Arduino IDE team will freely introduce breaking changes--why shouldn't they? It's not an API!

Likewise, the Web IDE is not an API. Anything package trying to leverage it will face the same instability, unless you happen to develop both the Web IDE and a CLI-tool--because you know what's going to break. But the Web IDE shouldn't be an API, it should be a layer on top of one. Granted, it's much nicer than the Arduino IDE, but still should just be an interface.

In conclusion...

Ideally, if the resources and interest is there, I'd like to work with Espruino instead of in parallel. I don't want to develop and maintain an entire separate toolchain on my own. I would probably just continue to patch things together so it works for me, but I have no interest in maintaining "competing" tools.

Let me know what you think. These are just a bunch of ideas. I'd be happy to discuss all of them. And I apologize if they've been suggested before and shot down--I just got this board a couple weeks ago. :D