Search results for:

@Gordon Actually it seems NodeJS and ChromeOS are not mutually exclusive.

@alexanderbrevig

One concern with using npm (especially if npm install is not done each upload, which would then probably add some overhead in the time it takes to upload) would be that it kind of encourages quick fixes per user. With the current system, a user is more inclined to file a report, or fix the problem immediately.

I'm not sure I understand what you're getting at here.

@Gordon

I'm not convinced that what you're actually suggesting will do that? If you're suggesting that everyone needs to have filesystem access, install node, npm, and a bunch of modules first then it's not an improvement at all.

No, this will not work on an iPhone or whatever. You would need NodeJS installed.

You're coming at this from the point of view of a node.js developer and assuming it's way behind Tessel because it's not what you're used to. But the reality is that they are aimed at very different areas... Tessel is going after node.js devs and that's great - but Espruino is about creating things quickly and easily without having to learn a bunch of extra tools.

I want to first point out I'm not saying Espruino is "behind" Tessel. Please forgive me if I gave that impression. I understand they are different tools.

Perhaps what I didn't understand is that "ease of use" is paramount to the project.

With the "alternate" workflow I detailed, you should probably know a little more about what you're doing, but regardless, the interaction with npm would probably be obscured behind the IDE. Maybe you missed the bit about "keeping what we have" which would not require the user to even know npm was behind the scenes.

But, it sounds like, at the current time, using npm is a no-go because we don't want to rely upon a filesystem? If the likelihood of a user programming an Espruino from an iPad is greater than the likelihood of a user wanting npm integration, so be it. I have no data in either direction.

Please don't do that, at least try and leverage EspruinoTools. I can pretty much guarantee that you'll lose interest in a few months, and then it'll just be sitting there on NPM along with node-espruino, grunt-espruino and espruino-cli as yet another tool that uploads 'hello world' but fails when faced with anything more complex.

Please consider this constructive criticism: I began writing my own CLI tool because leveraging EspruinoTools is extremely awkward. It's not an npm module, so I can't add it as a dependency. It's not versioned, so even if I did, it'd be a crapshoot. It doesn't seem to expose an isolated API layer, which means you need packages like jsdom to even get at them. It exports global variables and doesn't use something like AMD, so I'm not sure whether I need to load the whole thing at once or not. My hope was to actually develop an isolated API layer, wherein I could lift code from EspruinoTools, and build a CLI on top of it (cause that's easy), but publish it as a standalone module which could be extended in any which-way.

Ideally, it would become the "first-class" API for communication with the Espruino, and the Web IDE itself would simply consume it. But without some sort of buy-in, I don't see the project as having much future. I think I mentioned ino, the cmd-line tool for Arduino. They don't work closely with the IDE team--don't have "buy-in"--and thusly have continuous, never-ending issues of the exact nature you're talking about.

(In a perfect world, the Arduino IDE and ino would be built upon a common codebase, like what I'm proposing here)

I will relinquish the espruino module if/when you want to do something with it, but we probably shouldn't leave it empty or in a "placeholder" state to avoid squatting. You can email me about this if you wish; I can't seem to figure out how to "subscribe" to a thread here.

Gordon, I would like to apologize for coming out of left-field on this, and trying to jam the Espruino into a hole which it clearly does not fit. As I wrote, my point of view is as a software engineer. Hardware is a hobby of mine. I'm interested in increasing accessibility of hardware to software developers, so they can more readily apply their skills to the platform(s). Hardware which supports high-level languages is a great start; I suppose that's what drew me to Espruino originally. Another way is to support common toolchains and methodologies. There are a lot of things.

(I do sense a general unease about the influx of software developers to hardware, perhaps spurred by the popularity of the Arduino. I am concerned I won't get much support in my efforts. Some of this venom comes from garden-variety elitists who don't like the "dumbing down" of hardware, but others arguments include a lot of "ain't-broke-don't-fix-it".

For example, I'd smile if the official Espruino modules each had unit tests. Funny, right? Maybe to some it seems like extra complexity, or simply unnecessary. When you're hacking on a breadboard, you don't have a second breadboard that asserts the first breadboard works--you just look at it, and if it works, it works. I don't feel that the same strategy works well for software libraries, even if those libraries run on a microcontroller.)

Anyway, I don't need to spend another 2 hours writing this post... :D

Hi, sorry didn't have time to reply until now.

No, it's neither of these things really. :D

Before I go into more detail, please understand that come from a software engineering background. I work developing a large suite of SaaS web apps, as well as OSS projects in my free time. I've encountered many issues centered around dependency management, code integration, and deployment. I'm not trying to tell you guys, "I know what I'm talking about". I just have, perhaps, different a experience here, and I'd like to use it to propose a few things.

So, what I'm about to suggest may not have a 1:1 relationship to the hardware world. If I were to have originally designed how the Espruino modules were packaged, and toolchain structured, it would be different than what exists now. Maybe you agree with my points, but haven't had the resources to take action. Or, if you disagree, hopefully you can tell me why I'm wrong, because my experiences with software are not applicable.

At the very least, it should be interesting to think about things this way!

Espruino Modules in npm

1. npm is not just for JavaScript packages that run on NodeJS. We're seeing an influx of packages which run in the browser's context. npm is simply a package management tool and registry for JavaScript; the context doesn't matter.
   
2. If NodeJS is a context and the browser is a context, then Espruino is also a context. More importantly, it's just JavaScript.
   
3. Given the above, Espruino modules:
   
   1. Albeit small in size, should each have their own repository.
      
   2. Should be published to npm, and individually installable.
      
   3. Should be versioned using Semantic Versioning and tagged appropriately.
      
4. If tools exist to translate one context to another (they do; Browserify being a prime example), a tool should exist to translate Espruino code to another context, and vice-versa.
   
   1. For example, I've rewritten example code for the DHT11 (temp/humidity sensor) in a NodeJS context. It's almost identical, of course, since require() is used in both contexts.
      
   2. A trivial Browserify "transform" can convert this source into code to run in an Espruino context. Of course, it will need some work, as Gordon mentioned there are some edge cases the Web IDE handles.
      
      

Dependency management is a difficult problem. npm represents many thousands of hours of development time working to solve this problem. Out of the many package management systems that I've used, npm is likely the best.

The Espruino Web IDE has all but ignored dependency management by opting for automatic installation instead. Not everyone wants this!

Automatic Installation

Automatic installation of required modules seems very strange to me. I understand that this can help people get up-and-running, here's why I find it harmful:

1. I write some code to be run on an Espruino
   
2. I "deploy" it to an Espruino
   
3. Time passes, and one or more of the modules I'm using has been upgraded.
   
4. I make a small change to my code.
   
5. I redeploy to an Espruino, and my code fails, because the remote module has breaking changes.
   
6. Alternatively, I redeploy and my code fails because my code had a workaround for a bug which was fixed in the meantime.
   
7. Caching of modules which are automatically installed increases any issues by an order of magnitude, because now you must consider the cached state of any given module for any given problem you're having.
   
   

The only practical drawback I can think of to using explicit versioning is that code must be written to support it. Mitigating this is that 98% of the work will done by npm, and not the Espruino toolchain.

The "extra step" of declaring your dependencies can be avoided (see below).

A caveat is that most Espruino users probably don't care about this stuff, which means it must be implemented in such a way that it's transparent.

An Alternate Workflow

To support those that do care about versioning, like myself, I'm envisioning an alternate workflow like this (it could be an "advanced" mode or something):

1. Create an Espruino project. A package.json manifest is created for you. Note: a main file must be present in the manifest--probably the .js file you are working in.
   
2. Decide you want to use a DHT11 in your project.
   
3. Select the DHT11 Espruino module via command-line tool or Web IDE. Specify "latest" version, "fuzzy" version, an exact version--up to you--but "latest" would be the default.
   
   1. Respectively:
      
      1. npm install --save dht11 (latest)
         
      2. npm install --save dht11@^0.1.0 (anything up to but not including version 1.0.0)
         
      3. npm install --save-exact dht11@0.1.2
         
   2. Each of these commands writes information to the manifest.
      
4. Because your module is written to the manifest, installation of the module becomes effectively "automatic" upon a subsequent execution of npm install.
   
5. The final step before deployment would be an "npm install" before repackaging into an Espruino context.
   
6. A few common commands should be supported via Web IDE (not necessary on command line), such as npm remove, npm upgrade and npm info.
   
   

Keeping What We Have

To retain the current workflow--and make these changes transparent--but also be compatible with the alternate workflow:

1. Dependencies are determined in nearly the same manner the Web IDE does it now (scan for require(...) or whatever)--except probably with a 3p lib instead, because there's really no need to reinvent the wheel.
   
2. Upon deployment, package.json is written with the dependencies if it does not yet exist. The "version" written to package.json will be *, which lets npm choose. Typically this is the "latest" version of the dependency.
   
3. Before repackaging, npm install is simply run, and the modules are installed "automatically".
   
4. Because a package.json is written, users can switch between the workflows without any incompatibilities.
   
   

"Deployment"

Browserify can do basically everything necessary here, coupled with a custom transform.

One thing I don't understand is this: what is the point of not minifying your files before uploading? You can't really debug it anyway, right, so why do you care what your code looks like?

Regardless, I've identified the following steps:

1. As mentioned above, the first step here would be to npm install and ensure all modules are present and accounted for.
   
2. browserify is run at the entry point specified by the main property of package.json
   
3. Browserify then will "bundle" the dependencies. This includes official Espruino modules and any random npm or NodeJS core modules you wanted. The result is one .js file with everything in it. It just works.
   
   1. By default, Browserify will ignore or otherwise suppress modules which cannot run in the browser context. For example, if you need the fs module, you're SOL.
      
   2. The same goes for the Espruino context.
      
4. Several "transforms" occur, in this order:
   
   1. The code is altered or otherwise munged to be Espruino compatible. For example, atob is not present in NodeJS, but it is on the Espruino, so we declare atob as a global, and not a 3p package we need to bundle. I believe the code must be wrapped in an onInit(), but that may need to be the last step.
      
   2. The code is minified
      
   3. The code is gzipped (??? not sure about this; I saw some sort of gzip implementation in the web IDE code, but wasn't sure why it was there).
      
5. The final output is piped to the Espruino at an open serial port.
   
   

Where and How

We should spend a bit more time thinking about the "minimum viable product". It's clear to me that we don't have to do this all at once, but what's the bare minimum we can start with?

• The espruino package:
  
  • Two core features:
    
  • A serial communication layer. This would be written in NodeJS proper, hopefully using streams. To get it into the Web IDE, we could just browserify it (note that you can tell browserify to allow things like filesystem access). The current code to handle communication, which lives in the Web IDE, would be extracted and reused as much as possible.
    
  • An adapter that would handle interfacing with npm. This may have to be in the form of issuing shell commands instead of using its programmatic API, because obviously the Web IDE doesn't run under NodeJS, and I doubt Browserify will want to place nice with it.
    
  • We should think about this particular module very carefully, because its API becomes the platform. If someone is going to write an Espruino plugin for WebStorm, Sublime Text, Atom or even vim, it's going to have to leverage this module.
    
  • I would write unit tests for this, since we're building on top of it.
    
  • Anything else here?
    
• The espruino-cli package:
  
  • A thin wrapper around the espruino package.
    
  • To be "preferred" globally--typical use case is the shell, but if you were going to say, publish your own Espruino project on npm, you could add espruino-cli as a dependency, so whoever has your project would be able to deploy without extra steps.
    
  • Effectively replaces both espruino-tools and my spire tool.
    
• Official Espruino modules could be given their own repos and versioning and all that stuff, and perhaps we could just hack something together for the Web IDE to use it in the interim.
  
• The Web IDE should have its own package. It will likely be reduced vastly in size.
  
  

Why?

This buys Espruino a lot.

• If a user is familiar with the NodeJS toolchain, it's there.
  
• If the user doesn't need that, and just wants to quickly deploy simple scripts--that doesn't change.
  
• I can't think of a drawback to versioning the modules, if it's behind-the-scenes for those who do not care about it.
  
• An API built on NodeJS (or iojs) opens up the Espruino development environment to a greater audience.
  
• Perhaps close the feature gap a bit with the T*ssel. On Espruino, you get to write NodeJS code too. It just doesn't run NodeJS--but it doesn't have to.
  
• Potential integration with popular packages such as johnny-five.
  
• Task runners, unit tests, linters--make it natural to use tools like these if the user wants to.
  
• It makes me giddy
  
  

If I have any grumbles about the Web IDE, it seems that it's gone the same path the Arduino IDE has--it does not provide an API. To use an Arduino, you basically need to be a human interacting with a window. There is a CLI tool, ino, which is pretty unstable, but also seems the best alternative. It's unstable because the Arduino IDE team will freely introduce breaking changes--why shouldn't they? It's not an API!

Likewise, the Web IDE is not an API. Anything package trying to leverage it will face the same instability, unless you happen to develop both the Web IDE and a CLI-tool--because you know what's going to break. But the Web IDE shouldn't be an API, it should be a layer on top of one. Granted, it's much nicer than the Arduino IDE, but still should just be an interface.

In conclusion...

Ideally, if the resources and interest is there, I'd like to work with Espruino instead of in parallel. I don't want to develop and maintain an entire separate toolchain on my own. I would probably just continue to patch things together so it works for me, but I have no interest in maintaining "competing" tools.

Let me know what you think. These are just a bunch of ideas. I'd be happy to discuss all of them. And I apologize if they've been suggested before and shot down--I just got this board a couple weeks ago. :D

(FWIW, I grabbed the name so someone wouldn't do something dumb with it. Simply "squatting" a module name is not allowed and can get your publishing privs revoked, so needed to have something there anyway.)

Let me think this stuff over and get back to you this evening. I'm thinking an espruino module should include everything you need to write your Espruino code in NodeJS, and a programmatic interface into the board. So basically official modules + communication layer, but no CLI tool. espruino-cli for that.

Hey, I'm working on a little project to integrate the Espruino a bit more tightly into common NodeJS toolchains.

First, I'm repackaging the modules into something NodeJS can understand. That's underway here (master branch only).

I have another project, Spire, which is, at first glance, another one of those "command-line" tools to upload scripts. But, beneath the surface, there's a Duplex stream--theoretically you should be able to consume it as you would any other writable or readable stream. Its current implementation communicates via a serial port and STDOUT, but I would like to create an abstraction layer over this, so you can simply stream data to/from the Espruino via NodeJS. For example, you could set the Espruino to serve real-time data over a socket, then use NodeJS to pipe it into a database, file, or web app. There are other ways to accomplish this sort of thing, of course, but when dealing with large amounts of data, streams allow you to avoid memory consumption--this is especially important if you wish to send large amounts of data to the Espruino.

Anyway, I'm mostly just playing with ideas at this point, but I'm wondering if anyone has any input or would be interested in the outcome.

I think the Web IDE could benefit from using a communication interface such as this--and as-of-yet unwritten apps that people may wish to build on top of it.

http://boneskull.com

Excellent. Atom plugin in 3...2...1