Web IDE vs Espruino Tools minification

Posted on
  • Hi,
    I'm building a 'command-line-script' to download an application to multiple Bangle.js 2 devices using Espruino Tools on Linux. I ran into a strange problem.

    Consider this (isolated) piece of code :

    var dummy = 123;
    
    console.log(`dummy = ${dummy}`);
    
    

    When minifying this is the Web IDE, no problems (of course...).

    When running minification with the Espruino Tools, a strange error is reported (line 62 below):

    root@ubuntu:~# espruino --verbose --board BANGLEJS2 --config MINIFICATION_LEVEL=ESPRIMA --config MODULE_MINIFICATION_LEVEL=ESPRIMA --config PRETOKENISE=true --config SAVE_ON_SEND=1 --minify SmartBracelet/software/projects/esprima.­js -o SmartBracelet/software/es.js
    0.1.38
    Espruino Command-line Tool 0.1.38
    -----------------------------------
    
    Acorn library not found - you''ll need it for compiled code
    Found /usr/local/lib/node_modules/espruino/lib­s/targz.js
    ...
    Found /usr/local/lib/node_modules/espruino/cor­e/serial_winnus.js
    Not on Windows, Winnus not needed
    Found /usr/local/lib/node_modules/espruino/cor­e/terminal.js
    Found /usr/local/lib/node_modules/espruino/cor­e/utils.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/assembler.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/boardJSON.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/compiler.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/getGitHub.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/localModules.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/minify.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/pretokenise.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/saveOnSend.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/setTime.js
    Found /usr/local/lib/node_modules/espruino/plu­gins/unicode.js
    UTF8 Library loaded successfully
    Found /usr/local/lib/node_modules/espruino/plu­gins/versionChecker.js
    Initialising CodeWriter
    Initialising Config
    Initialising Env
    Initialising Flasher
    Initialising FlasherESP8266
    Initialising Modules
    Initialising Notifications
    Initialising Serial
      - Initialising Serial Noble Bluetooth LE
      - Initialising Serial Node Socket
      - Initialising Serial Web Audio
      - Initialising Serial Web Bluetooth
      - Initialising Serial Web Serial
    Initialising Utils
    Initialising Status
    Initialising Assembler
    Initialising BoardJSON
    Initialising Compiler
    Initialising GetGitHub
    Initialising LocalModules
    Initialising Minify
    Initialising Pretokenise
    Initialising SaveOnSend
    Initialising SetTime
    Initialising Unicode
    Initialising VersionChecker
    Initialising CoreModules
    Command-line option set Espruino.Config.MINIFICATION_LEVEL to "ESPRIMA"
    Command-line option set Espruino.Config.MODULE_MINIFICATION_LEVE­L to "ESPRIMA"
    Command-line option set Espruino.Config.PRETOKENISE to true
    Command-line option set Espruino.Config.SAVE_ON_SEND to 1
    Explicit board JSON supplied: "BANGLEJS2"
    Loading http://www.espruino.com/json/BANGLEJS2.j­son
    Board JSON loaded
    Manual board JSON load complete
    No port supplied, but output file listed - not connecting
    Minifying
    TypeError: Cannot read property 'length' of undefined
    TypeError: Cannot read property 'length' of undefined
        at Controller.traverse (/usr/local/lib/node_modules/espruino/no­de_modules/esmangle/node_modules/estrave­rse/estraverse.js:411:38)
        at Object.traverse (/usr/local/lib/node_modules/espruino/no­de_modules/esmangle/node_modules/estrave­rse/estraverse.js:556:27)
        at annotateDirective (/usr/local/lib/node_modules/espruino/no­de_modules/esmangle/lib/annotate-directi­ve.js:131:16)
        at Object.optimize (/usr/local/lib/node_modules/espruino/no­de_modules/esmangle/lib/esmangle.js:159:­16)
        at obfuscate (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:117:23)
        at minifyCodeEsprima (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:98:18)
        at minify (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:212:23)
        at Array.eval (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:60:7)
        at cb (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:98:15)
        at compileCode (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:154:7)
    Minification complete
    Since Acorn 8.0.0, options.ecmaVersion is required.
    Defaulting to 2020, but this will stop working in the future.
    Uploading 45 bytes to flash
    Writing output to SmartBracelet/software/es.js
    Done
    

    Platform is Ubuntu 18.04, Node version is V8.10.0, espruino (tools) is V0.1.38.

  • Wow,that is a strange one.

    Try re-installing now. It looks like the npm version of esmangle is broken (uses an old estraverse). Since EspruinoTools includes esmangle anyway, we now just use that version instead and it seems to work ok.

  • That already is better, but...
    Another error is presented (which I do not have in Web IDE). I can reproduce it with the following :

    var dummy = 123;
    console.log(`dummy = ${dummy}`);
    var mod = require("esprimaMod");
    

    with 'esprimaMod.js' as :

    function something()
    {
    }
    
    exports.connect = function ()
    {
    	return something;
    };
    

    and the resulting error at line 29 and 42:

    root@ubuntu:~# espruino --verbose --board BANGLEJS2 --config MINIFICATION_LEVEL=ESPRIMA --config MODULE_MINIFICATION_LEVEL=ESPRIMA --config PRETOKENISE=true --config SAVE_ON_SEND=1 --config MODULE_EXTENSIONS=.js --config  MODULE_URL=SmartBracelet/software/module­s --minify SmartBracelet/software/projects/esprima.­js -o SmartBracelet/software/es.js
    0.1.39
    Espruino Command-line Tool 0.1.39
    -----------------------------------
    
    Acorn library not found - you''ll need it for compiled code
    Found /usr/local/lib/node_modules/espruino/lib­s/targz.js
    ...
    Initialising CoreModules
    Command-line option set Espruino.Config.MINIFICATION_LEVEL to "ESPRIMA"
    Command-line option set Espruino.Config.MODULE_MINIFICATION_LEVE­L to "ESPRIMA"
    Command-line option set Espruino.Config.PRETOKENISE to true
    Command-line option set Espruino.Config.SAVE_ON_SEND to 1
    Command-line option set Espruino.Config.MODULE_EXTENSIONS to ".js"
    Command-line option set Espruino.Config.MODULE_URL to "SmartBracelet/software/modules"
    Explicit board JSON supplied: "BANGLEJS2"
    Loading http://www.espruino.com/json/BANGLEJS2.j­son
    Board JSON loaded
    Manual board JSON load complete
    No port supplied, but output file listed - not connecting
    Loading modules
    loadModule(esprimaMod)
     - esprimaMod requires []
    Minifyingn esprimaMod
    No errors in esprimaMod. Minified 81 bytes to 66 bytes.
    Minification complete
    Since Acorn 8.0.0, options.ecmaVersion is required.
    Defaulting to 2020, but this will stop working in the future.
    { SyntaxError: Unexpected token (1:54)
        at Parser.pp$5.raise (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:3416:15)
        at Parser.pp.unexpected (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:741:10)
        at Parser.pp.semicolon (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:718:66)
        at Parser.pp$1.parseExpressionStatement (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:1201:10)
        at Parser.pp$1.parseStatement (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:934:26)
        at Parser.pp$1.parseBlock (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:1217:23)
        at Parser.pp$4.parseFunctionBody (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:3242:24)
        at Parser.pp$1.parseFunction (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:1339:10)
        at Parser.pp$4.parseExprAtom (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:2772:19)
        at Parser.pp$4.parseExprSubscripts (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:2583:21) pos: 54, loc: Position { line: 1, column: 54 }, raisedAt: 55 }
    Error parsing JavaScript, but uploading anyway.<br/>SyntaxError: Unexpected token (1:54)
    Minifying
    Error: Line 1: Unexpected token {
    Error: Line 1: Unexpected token {
        at ErrorHandler.constructError (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:5012:22)
        at ErrorHandler.createError (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:5028:27)
        at Parser.unexpectedTokenError (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:1985:39)
        at Parser.throwUnexpectedToken (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:1995:21)
        at Parser.consumeSemicolon (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:2297:23)
        at Parser.parseLabelledStatement (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:4029:19)
        at Parser.parseStatement (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:4122:97)
        at Parser.parseStatementListItem (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:3393:31)
        at Parser.parseFunctionSourceElements (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:4195:29)
        at Parser.parseFunctionExpression (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:4408:26)
    Minification complete
    (node:69905) UnhandledPromiseRejectionWarning: SyntaxError: Unexpected character '«' (1:76)
        at Parser.pp$5.raise (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:3416:15)
        at Parser.pp$9.getTokenFromCode (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:5081:10)
        at Parser.pp$9.readToken (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:4765:17)
        at Parser.pp$9.nextToken (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:4756:17)
        at Parser.pp$9.next (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:4717:10)
        at Parser.pp$9.getToken (/usr/local/lib/node_modules/espruino/no­de_modules/acorn/dist/acorn.js:4721:10)
        at Object.next (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:115:20)
        at pretokenise (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:164:17)
        at Array.eval (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:31:7)
        at cb (eval at loadJS (/usr/local/lib/node_modules/espruino/in­dex.js:12:11), <anonymous>:98:15)
    (node:69905) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
    (node:69905) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
    

    No idea where that 'unexpected token {' would be at in above simple code...

    I honestly thought Espruino Tools used the very same chain of tools as Web IDE does, but apparently this is not the case ? I'm trying to use the tools for distributing an app to multiple Bangle.js 2's, but if Espruino Tools builds 'something different' as compared to Web IDE (where the app was fully debugged), I do not feel particularly comfortable...

  • Ok, that one is interesting. You may be able to get around it just by uninstalling the acorn module? It's not like you need the compiled code anyway.

    I honestly thought Espruino Tools used the very same chain of tools as Web IDE does

    Well, it uses the majority of the same stuff. However, as I remember (I could be wrong here) some developers came along and were saying "Why are you using built-in modules when there are ones on NPM that'll be better maintained?" so I ended up having this special case for those modules when running the CLI tool.

    As it happens, in this case the esmangle development has been really flaky and they haven't published to NPM properly, which is how we got in this mess. I went to significant lengths to get a properly working version for EspruinoTools which could be included, so it makes sense to stick with that I guess.

  • Just to say, I looked into your code and it does say Error parsing JavaScript, but uploading anyway. - I believe the issue may be that you are pretokenising, and that the compiled code step is run after the pretokenisation step.

    Try updating again now. This should be miles better - more info at https://github.com/espruino/EspruinoTool­s/issues/138

  • Yes, now it works ! The resulting number of bytes from the message 'No errors. Minified xxx bytes to yyy bytes.' is identical in both cases, so chances are both builds come up with the exact same result.

    As it happens, in this case the esmangle development has been really flaky

    I also struggle with the 'wonderful world' of open source modules. One can usually only guess at things like quality and longevity, but hey, apart from time spent, it's free....

    In the open source world, the Espruino project is a notable exception in terms of quality and support.

    Thank you again for the quick response.

  • Hello again,
    I ran into another difference while using Web IDE vs Espruino Tools. My programming flow for Bangle.js 2 bracelets involves building a single 'output file' for later download to multiple bracelets. It is mentioned on https://github.com/espruino/EspruinoTool­s that programming is as simple as :

    This file can then be sent directly to Espruino at some later time - sometimes just cat file.js > /dev/ttyACM0 is enough.

    Unfortunately I ran into an issue with 'special character strings'. Consider the following code (this the double arrow image used in Bangle's menu) :

    g.clear().drawImage("\x0c\x05\x81\x00 \x07\x00\xF9\xF0\x0E\x00@", 10, 10, {scale:4});
    

    This is prepared for download with :

    espruino --board BANGLEJS2 --config MINIFICATION_LEVEL=ESPRIMA --config SAVE_ON_SEND=1 /tmp/test.js -o /tmp/testOut.js
    

    'testOut.js' looks like :

    setTime(1634026260.035);E.setTimeZone(-7­)
    require("Storage").write(".bootcde","g.c­lear().drawImage('\\f\\x05\\x81\\0 \\x07\\0ùð\\x0E\\0@',10,10,{scale:4})",0­,67);
    load()
    

    When downloading this to the bracelet, the error 'Too much data for file size' is returned. After some tracing it turns out the download is 69 byte, not 67 (attached image). The image on the bracelet is - obviously - 'mangled'. A dump() looks messed up :

    |  __|___ ___ ___ _ _|_|___ ___
    |  __|_ -| . |  _| | | |   | . |
    |____|___|  _|_| |___|_|_|_|___|
             |_| espruino.com
     2v10.89 (c) 2021 G.Williams
    >
    >
    >dump()
    // Code saved with E.setBootCode
    g.clear().drawImage('\f\x05\x81\0 \x07\0 extends in extends throw\x0E\0@',10,10,{scale:4})=undefined­
    > 
    

    Using the Web IDE and 'Send to Espruino' Flash, the file downloads OK and is 67 byte in size (attached). The dump () :

    >
    >dump()
    // Code saved with E.setBootCode
    g.clear().drawImage('\f\x05\x81\0 \x07\0 ?[249] ?[240]\x0E\0@',10,10,{scale:4})=undefine­d
    > 
    

    The difference are the codes for the 'ùð' characters. The strange thing is that the console dumps in Web IDE (development tools) show the same characters in the 'Sending' prompt, but result in a correct file being stored.

    >>> Sending...
    settingsConsole.js:30 ---> "\u0010reset();\n\u0010print()\n\u0010re­quire(\"Storage\").write(\".bootcde\",\"­g.clear().drawImage('\\\\f\\\\x05\\\\x81­\\\\0 \\\\x07\\\\0ùð\\\\x0E\\\\0@',10,10,{scal­e:4})\",0,67);\n\u0010load()\n\n"
    

    So obviously 'someone' is altering the output of the minified source before sending it down BLE. How then is it possible to just 'cat' an output file to a bracelet ? What are the additional modifications a minified source needs before 'just copying' to a Bangle.js 2 ?


    2 Attachments

    • webide.png
    • file.png
  • How do you download that prepared file to Bangle.js? Using the command-line tools again? What's the command?

    I believe the issue is with unicode.js. This has been a continual source of frustration for me - it was added by Amperka for their Russian Espruino clone because (I think) they wanted print statements with Russian characters to work, but it messes with any strings you send over and it just needs to be removed IMO as I don't believe anyone actually uses it.

    So... You have a few options I think:

  • The 'testOut.js' file is sent down using c-code in our devices. These are industrial terminals using an ARM CPU. This download works as it should : connecting, waiting for the prompt then copying 'testOut.js' byte by byte down to Bangle.js 2. Bangle.js 2 responds as it should, so nothing wrong there.

    It is the 'testOut.js' file that is already 'wrong' : the erroneous bytes C3 B9 C3 B0 are the UTF8 representation for 'ùð' : ( https://onlineunicodetools.com/convert-u­nicode-to-utf8 ), and not the original F9 Fo. So using another way for download will not change anything I'm afraid.

    For now, I've changed my code to avoid strings with binary 'characters', using a btoa-version of the binary values. For the example above, this comes down to :

    g.clear().drawImage(atob("DAWBACAHAPnwDg­BA"), 10, 10, {scale:4});
    

    So I'm in the clear for now, but this is a bit of a clumsy workaround of course.

  • Ahh, I see. This is a tricky one actually.

    In a way, nothing is actually wrong. What's happened is the command-line tools are writing the file to disk in UTF8 format. If you were also to load the files in UTF8 format and then send character by character everything would be fine.

    I believe you could use the iconv command-line tool on Linux to convert the formats for you before your C code reads it.

    Maybe the tools should write the file in the raw ISO8859-1 format, however they are expected to read files in UTF8 so the current behaviour of writing as UTF8 seems kind of reasonable.

  • Yes I'm familiar with iconv (and libiconv), but to be able to convert UTF8 to what the (binary !) characters originally were, one needs to know the character encoding of the platform the UTF8 was generated on. That encoding was used to transform 0xF9 and oxF0 characters in the corresponding UTF8 characters and is needed to properly specify the 'decoding' libiconv must perform.

    It remains unexplainable that a string declared as

    x="\x0c\x05\x81\x00 \x07\x00\xF9\xF0\x0E\x00@"
    

    gets transformed into something UTF8 :

    '\f\x05\x81\0 \x07\0ùð\x0E\0@'
    

    Why not just leave binary valued characters in strings as they are ? After all, the string for a drawImage call is not really a string anyhow ...

    (Just an opinion of someone totally unfamiliar with the inner workings of Espruino and the Web IDE.)

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Web IDE vs Espruino Tools minification

Posted by Avatar for jgw @jgw

Actions