Espruino plugin for Rollup module bundler

Posted on
Page
of 2
/ 2
Next
  • I've made a plugin for Rollup that lets you push the resulting code straight to Espruino devices.

    Rollup is a module bundler, which takes JS modules (preferably ES modules) and bundles them into one file. While doing so, it employs tree-shaking and static analysis to remove unused code, resulting in a file that contains only the bare minimum. Perfect for Espruino :)

    It supports plugins, so you can easily add minification of code, linting, support for CoffeScript – and thankfully, support for CommonJS modules, which is the format of Espruino's modules.

    Basically, this allows you to structure your Espruino projects using modern ES modules, and have Rollup bundle it all up and send the resulting code to Espruino in one command.

    I've only tested it on Puck.js so far, but it's based on code from EspruinoTools' CLI, so hopefully it works for all supported devices. There's also one more feature I'd like to add, the ability to save after sending.

    If this sounds useful, try it out and let me know how it works (or doesn't work) :)

    https://www.npmjs.com/package/rollup-plu­gin-espruino

  • In order to use Espruino modules (and other CommonJS modules) a few extra steps are required, especially if you also want to uglify/minify the code. This assumes you already have Node, npm and Rollup installed.

    First, install the plugins…

    npm install rollup-plugin-espruino
    npm install rollup-plugin-commonjs
    npm install rollup-plugin-uglify
    

    You need to use the "harmony" branch of the uglify-js library for the uglify plugin to support ES modules. This will fetch it from GitHub:

    npm install mishoo/UglifyJS2#harmony
    

    Finally, add the following configuration to your Rollup config (rollup.config.js if using the CLI):

    import espruino from 'rollup-plugin-espruino'
    import commonjs from 'rollup-plugin-commonjs'
    import uglify from 'rollup-plugin-uglify'
    import { minify } from 'uglify-js'
    
    export default {
      entry: 'src/main.js',
      dest: 'dist/bundle.js',
      format: 'es',
      plugins: [
        commonjs(),
        uglify({}, minify),
        espruino(),
      ],
    }
    

    Here I'm passing the "harmony" branch of uglify-js as minify to the uglify plugin.

    And with that, it should work as expected. In harmony.

    Luckily, other plugins are more straight forward.

  • And with that configuration, here's the DS18B20 module's example rewritten in modern JS:

    import { connect } from './modules/DS18B20.js'
    
    let ow = new OneWire(A1)
    let sensor = connect(ow)
    
    setInterval(() => {
      sensor.getTemp(temp => console.log('Temp is ' + temp + '°C'))
    }, 1000)
    

    The same code bundled with the module, uglified and sent to my Espruino after running rollup -c:

    function DS18B20(t,e){if(this.bus=t,void 0===e?this.sCode=this.bus.search()[0]:pa­rseInt(e).toString()==e&&e>=0&&e<=126?th­is.sCode=this.bus.search()[e]:this.sCode­=e,!this.sCode)throw new Error("No DS18B20 found");this.type=parseInt(this.sCode[0]­+this.sCode[1])}var C={CONVERT_T:68,COPY:72,READ:190,WRITE:7­8};DS18B20.prototype._r=function(){var t=this.bus;return t.select(this.sCode),t.write(C.READ),t.r­ead(9)},DS18B20.prototype._w=function(t,­e,s){var r=this.bus;r.select(this.sCode),r.write(­[C.WRITE,t,e,s]),r.select(this.sCode),r.­write(C.COPY),r.reset()},DS18B20.prototy­pe.setRes=function(t){var e=this._r();t=[31,63,95,127][E.clip(t,9,­12)-9],this._w(e[2],e[3],t)},DS18B20.pro­totype.getRes=function(){return[31,63,95­,127].indexOf(this._r()[4])+9},DS18B20.p­rototype.isPresent=function(){return this.bus.search().indexOf(this.sCode)!==­-1},DS18B20.prototype.getTemp=function(t­){function e(e){for(var s=e._r(),r=0,i=0;i<8;i++){r^=s[i];for(va­r o=0;o<8;o++)r=r>>1^140*(1&r)}var n=null;return r==s[8]&&(n=s[0]+(s[1]<<8),32768&n&&(n-=­65536),n/=10==e.type?2:16),t&&t(n),n}if(­this.bus.select(this.sCode),this.bus.wri­te(C.CONVERT_T,!0),!t)return e(this);var s={9:94,10:188,11:375,12:750};setTimeout­(e,s[this.getRes()],this)},DS18B20.proto­type.searchAlarm=function(){return this.bus.search(236)},DS18B20.prototype.­setAlarm=function(t,e){t--,t<0&&(t+=256)­,e<0&&(e+=256);var s=this._r();this._w(e,t,s[4])};var connect=function(t,e){return new DS18B20(t,e)};let t=new OneWire(A1),e=connect(t);setInterval(()=­>{e.getTemp(t=>console.log("Temp is "+t+"°C"))},1e3);
    

    Not so pretty anymore..

  • Wow, thanks - this looks great! It might well be able to minify it even further as well (changing the name of DS18B20 for instance).

    So is it also able to fetch the modules from the Espruino website as well?

    If you want to do save, you might just be able to set Espruino.Config.SAVE_ON_SEND to 1 before sending.

  • Yea, I noticed that it could be minified even further. That's up to the Uglify plugin to do, I'll see whether it can be configured to be more ugly.

    At the moment you have to manually download the modules, but I think it might be possible to automatically download Espruino modules like the Web IDE does, I'll see what I can do! Would be very nice if this could work just like the Web IDE, that's a good benchmark :)

    I don't think setting SAVE_ON_SEND would work, as I'm skipping the part of EspruinoTools that does minification, which is what calls the plugin for saving, if I recall correctly. Can I call that plugin directly somehow?

  • If you want to skip minification (which shouldn't be on by default anyway?) you can set the config variables for it - you probably shouldn't try and cut bits of the pipeline as there might well be some other stuff in there that's required (like the newline fiddling)...

  • You're right of course! I think I skipped the transformForEspruino processor call when I was trying to get things to work, and honestly hadn't really understood how the plugin system works until now. Now that the processor call is back my code follows the same pipeline as sendCode, and it seems to be doing just the same (I'm not calling it directly because I want more control over output).

    Some more changes:

    • Better options
      • Now supports reset, save, setTime, BLE, audio, baudRate and throttle (all with Espruino's defaults)
      • Any other Espruino config options may be set using their uppercase names directly
      • Added output option for showing the output from Espruino when sending code
    • Detects and reports any uncaught JS errors after code has been sent (very useful)
    • Prettier CLI with colours and all

    I also figured out how to minify it even further, and added a tip for how to configure Uglify to be even uglier on the project page.

  • I've looked into loading Espruino modules straight from espruino.com/modules, and while that seems possible, I think It will have to wait untill I actually need is, I have enough on my plate as it is.

    If someone else wants to give it a shot, the relevant Rollup hooks appear to be resolveId and load.

  • I made a noob mistake when publishing the package to NPM, it probably didn't work for anyone else but me :/ It should be fixed now, hopefully it works for everyone. It does require Node v6 or greater.

  • Couldn't you automate detection of the Espruino so that the port doesn't have to be entered manually? Even just changing to a different USB port will cause this value to change. In the script that I wrote for this, I copied the code that the web IDE uses to identify Espruino boards and connect to the corresponding port:
    https://github.com/stokebrain/morra-buil­d/blob/master/main.js#L23

    Webpack also has tree-shaking. When I first tried rollup, I concluded it wasn't a good choice because I saw that it was placing top-level variables from all modules, all together in the same scope. I was too concerned about name collision and went back to webpack.

    Also, I don't see the smaller footprint that rollup is claiming.
    https://vkbansal.me/blog/rollup-first-im­pressions/

    I continually see people writing that rollup configuration is easier, but that has not been true for me. Webpack just worked, and rollup continues to require debugging.

    I still have some complaints and issues to solve with webpack. For my first project it all worked well, but in my current MQTT test, I'm getting errors when using my build-and-save script. One thing that concerns me is the use of arrays in the build, because I don't know if these will cause performance issues for Espruino.

  • Hi @Joakim, I'm getting this:

    root@office-pc:~/builder/rollup# rollup -c
    [!] Error: Cannot find module 'lodash.forEach'
    Error: Cannot find module 'lodash.forEach'
        at Function.Module._resolveFilename (internal/modules/cjs/loader.js:571:15)
        at Function.Module._load (internal/modules/cjs/loader.js:497:25)
        at Module.require (internal/modules/cjs/loader.js:626:17)
        at require (internal/modules/cjs/helpers.js:20:18)
        at Object.<anonymous> (/root/node_modules/rollup-plugin-esprui­no/dist/index.cjs.js:8:31)
        at Module._compile (internal/modules/cjs/loader.js:678:30)
        at Module._extensions..js (internal/modules/cjs/loader.js:689:10)
        at Object.require.extensions..js (/root/.nvm/versions/node/v10.1.0/lib/no­de_modules/rollup/bin/rollup:2834:17)
        at Module.load (internal/modules/cjs/loader.js:589:32)
        at tryModuleLoad (internal/modules/cjs/loader.js:528:12)
    
    

    Any ideas?
    Thanks

  • Oops, I merged a pull request too quickly.

    I haven't released the latest changes on NPM though, so if you fetch the module using npm install rollup-espruino it should work (that is, use v0.2.1).. The updated example configuration in the README should be correct though, while the one in v0.2.1 may no longer work.

    I'll be working on this plugin soon, and will iron out any issues then. Hopefully in a few days!

  • Thanks.

  • I've looked into loading Espruino modules straight from
    espruino.com/modules, and while that seems possible, I think It will
    have to wait untill I actually need is, I have enough on my plate as
    it is.

    If someone else wants to give it a shot, the relevant Rollup hooks
    appear to be resolveId and load.

    https://github.com/opichals/rollup-plugi­n-espruino-modules (not published to npm yet).

    It is generates a default config for Espruino which currently also does minification by default. It is just a first shot as it needs refinement with regards to configuration possibilities but it works quite nicely. It downloads the modules into the local ./modules folder

    I converted the webpack/babel example to use this and babel 7.x here https://github.com/opichals/espruino-rol­lup-babel-sample

    By using the import statements in the app code one can get use of tree shaking so that e.g. the SSD1306 driver's connectSPI is not bundled when not used.

    caveat: I haven't tried the resulting bundles on-board yet.

  • wow greate job, will give it a try in the next weeks

    I have decided to include some modules directly into the build like SSD1306 and tinyMqtt see ESP8266_4MB how to.

  • Using the external tools like rollup and the terser used to minify the bundle in the rollup plugin one can avoid hitting es6 (arrow functions) related minifcation issues for example. Also does the module contents merging mentioned in https://github.com/espruino/EspruinoTool­s/issues/27.

    Although it will require some work I think we should be able to get rollup (and perhaps even the babel functionality if deemed interesting) including the necessary plugins into the EspruinoTools and therefore into the EspruinoIDE. There is no reaction on the EspruinoTools google-closure-compiler-js PR . @Gordon what would be your position on that?

    @MaBe I'd rather not make the default firmware images bigger by including stuff people won't use. I'd instead aim for the 'Storage' module assigned flash space expansion for the boards where possible (like the ESP8266_4MB) and use tools like rollup to put stuff in.

  • Yeah, I know - sorry about that PR. I've been on the fence about adding the big chunk of extra code for far too long, but I'll try and get it in this/next week. It's just a shame we can't do some kind of 'load on demand' thing.

    I'm unsure about how rollup would work in the Web IDE though - doesn't it expect to be able to pull in NPM modules? I don't see how that'd work at all when running in the browser.

    Presumably we could add command-line options for EspruinoTools such that you could use it with rollup/babel/etc if they had been installed by NPM (without requiring that the espruino NPM package actually depended on all of them).

    @MaBe yeah, I definitely wouldn't want to start building modules in 'just because' - personally I'd have thought that having the ability to grab modules from Storage, and perhaps having a nice method of loading modules/files via the IDE would be far more useful for people?

  • I'm unsure about how rollup would work in the Web IDE though - doesn't it expect to be able to pull in NPM modules? I don't see how that'd work at all when running in the browser.

    It should be possible to make it work. Rollup doesn't pull anything on its own. Only the https://github.com/opichals/rollup-plugi­n-espruino-modules/blob/master/espruino-­tools.js is actually fetching stuff from espruino.com and we could update this for sure (using CORS it would work even from the IDE hosted anywhere). You can replace the default loader to do anything - it doesn't have to touch any filesystem IMO.

    With babel and the rest it might be a different story, but there is again the babel-standalone which is easily runnable in the browser if required.

    @Gordon Storage module resolution would be a nice thing to have if the IDE would allow synchronizing them would sound as an interesting feature.

  • Yeah, I know - sorry about that PR. I've been on the fence about adding the big chunk of extra code for far too long, but I'll try and get it in this/next week. It's just a shame we can't do some kind of 'load on demand' thing.

    It should be possible to lazy-load the closure compiler. My main concern was the time it takes to execute and the lack of any visual hint on that the browser is working on something.. Perhaps wasm would help eventually. Also this could be done in a webworker so that it would not block the js event queue.

  • @MaBe I'd rather not make the default firmware images bigger by including stuff people won't use. I'd instead aim for the 'Storage' module assigned flash space expansion for the boards where possible (like the ESP8266_4MB) and use tools like rollup to put stuff in.

    @MaBe yeah, I definitely wouldn't want to start building modules in 'just because'

    Ouch - never thought about including min.js modules into Espruino releases, this it's just for private builds.

    personally I'd have thought that having the ability to grab modules from Storage, and perhaps having a nice method of loading modules/files via the IDE would be far more useful for people?

    That would be a very nice feature!

  • My main concern was the time it takes to execute and the lack of any visual hint on that the browser is working on something

    We could at least add that as a status indicator... Does it take a very long time then? Surely it's got to be better than the current web request to the closure compiler (which is also without any status indication)

  • Surely it's got to be better than the current web request to the closure compiler (which is also without any status indication)

    IIRC the JavaScript transpilation result is significantly slower compared to the native Java service provided (GWT is used to transpile from Java).

    From this perspective I'd rather try babel-minify or the terser for minification. I used the terser in the rollup plugin.

    Rollup (or webpack) give you IMHO quite important tree-shaking. Especially important for the Espruino's microcontroller world.

    I think there is no point in trying to reinvent the wheel with regards to the bundling pipeline. What is the point in keeping this Espruino specific?

  • What is the point in keeping this Espruino specific?

    There isn't - it's more that the existing solution is there to work with require - if we only uploaded tree-shaken code then you'd never be able to do require(x) on the left-hand side of the IDE. Also I bet it doesn't preserve line-numbers for debug/etc, so could only ever be an option (not the default).

    Trying to pull in NPM modules could be quite painful - again, we wouldn't want to reimplement NPM, but trying to include NPM, babel, etc in an online IDE would probably make it huge and slow.

    Having a bundling solution does seem great though. Part of me wonders whether we wouldn't be better off if I just made it a service on the Espruino website? I could then use NPM, babel, closure, etc as-is. Only problem is that'd screw over ESP8266/ESP32 users because I'm not going to want to spend half my server time supporting them for free. I could happily open it to Patreon/etc users though, but I can imagine it'll still annoy a bunch of people.

  • work with require - if we only uploaded tree-shaken code then you'd never be able to do require(x) on the left-hand side of the IDE.

    Can even be made to output the Modules.addCached calls just as the current solution does. Can be added as an option if required.

    Also I bet it doesn't preserve line-numbers for debug/etc, so could only ever be an option (not the default).

    The toolchain should be configurable to the point it would allow this. Or source map support can be added to the IDE and elsewhere. So this could work even for minified code.

    trying to include NPM, babel, etc in an online IDE would probably make it huge and slow.

    Yes, this would need to be carefully selected for the IDE. But IMO doable.

    just made it a service on the Espruino website?

    I still think it is to be done so that you should not need any server-side. Just the IDE should be enough. E.g. rollup is not slow on its own AFACT. It just babel and/or the minification parts are taking more time (which I can imagine people can wait for) and can be big to download for the first time.

    I think I'd try to create some rollup POC PR for the IDE just to see how that would fare. What do you think?

  • I think I'd try to create some rollup POC PR for the IDE just to see how that would fare. What do you think?

    Yes, that'd be great! If we can fit it in in such a way that the IDE is still responsive I'm definitely up for including it - I just don't want to be in the position where the IDE is 5x the size, slower, and doesn't do all the stuff it did before :)

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Espruino plugin for Rollup module bundler

Posted by Avatar for Joakim @Joakim

Actions