Better Web IDE

I finally got fed up with the hideous mess that the Web IDE now is.

I've just added a new branch to the IDE's repository: https://github.com/espruino/EspruinoWebIDE/tree/major_refactor

I'm basically starting from a blank canvas. I've:

• Moved everything into folders called unused
  
• Started off with what I hope is a simple base (https://github.com/espruino/EspruinoWebIDE/blob/major_refactor/js/espruino.js and https://github.com/espruino/EspruinoWebIDE/blob/major_refactor/js/plugins/_examplePlugin.js with https://github.com/espruino/EspruinoWebIDE/blob/major_refactor/js/core/config.js for storing configuration settings)
  
• Been re-adding functionality on top of the base in such a way that every feature (including terminal, editor, etc) is quite self-contained and in its own file. I'm aiming to get a bare minimum Web IDE working first.
  
• Tried at least partially to move everything back to a K&R coding style.
  
  

It's still not working properly, but I'd really appreciate it if people could take a look at the general layout and let me know what they think. This is the point at which stuff is being put back in, so now would be the time to make sure the base is as nice/simple/maintainable as possible.

Excellent. I find the webIDE painful too.
I may instead start working on a espuino-mode for emacs.. never done anything like that, but it seems the bits are already there (there's already serial terminal, and countless javascript and lisp modes that talk to an inferior lisp/other process)

Hmm, it's an idea. The only potentially difficult bit is trying to load modules off the net when they're required. Ideally there would be one piece of code that we could share between everything (I know somebody was already working on a node.js-enhanced command-line for it).

To be honest though I'd really like more people to contribute to the Web IDE. It's basically just Juergen at the moment - which is pretty sad given the amount of JS devs that are using it :)

Okay, cool. Yeah, I was also playing around with the straight terminal on my Mac over the weekend, trying to see if I could figure out how to read / write from the SD card.

I'm mainly looking to poke around in the WebIDE to use the blackly components, so it would be interesting to hear why people are using the WebIDE.

Per my comment on the forum, I should have more time to dedicate to this soon. Apologies if I set expectations high that I could get into it this week (as I had personally expected).

I tried something similar, but the terminal implementation on the espruino itself is the painful part. I was surprised to find out that the board is the one implementing the CLI and the serial connection simply sends/receives text characters.

Is it possible to use a communication protocol and bypass espruino's CLI? Or maybe configure espruino to not send the cli prefix >, nor the response prefix =. You could also view it as a consistency problem, espruino is sending \r\n at the end of intermediate lines, and \r> at the end of output, but interpreting the text is wrong and can lead to a lot of problems.

Overall, is there a way to connect to the board without using its CLI, in order to implement a (proper) cli?

Hey Gordon,

I wasn't sure whether to share this or not just yet as it's just a personal experiment at the moment to learn some new tech (node-webkit + angularjs + node-serialport) so I don't know where I intend to go with it, but as you started this thread, I thought what the hell.

Basically, I took the current IDE and redesigned it to be a bit more modern, and hopefully a bit friendlier (See screenshot).

So far I've got it connecting, and have the terminal sending / responding to commands, but it's by no means "usable".

I don't know how you feel about using node-webkit? It basically makes the app standalone, rather than being a chrome app (I find the chrome app a small barrier, much preferring to just download an app and go). In theory (but I've not done this yet) you should still be able to compile it for both PC and Mac / Linux, so should be cross platform.

Anyways, like I say, it's just a concept at the moment, but thought I'd share it.

Many thanks

Matt

1 Attachment

@louisrawlins you'd probably use the Web IDE because it can automatically load modules off the net, even on boards that don't have their own net connection. Also it's easy to get a syntax highlighted editor which will highlight syntax errors. You're right that if you are just playing around in the console window there's not really any benefit over using Screen (except on Windows where a terminal app doesn't come pre-installed).

@randunel you can just send echo(0) in which case you can just send commands. However I feel that it defies the whole point of Espruino - which is that you don't need a complex terminal app to be able to use it.

@mattbrailsford thanks for sharing! It looks great! Do you have something I could try out? Personally I'd like to keep the terminal on the left though so I don't have to change basically every tutorial ;)

node-webkit seems like a neat idea... I'd still like to keep the Chrome App as well though - while some people like installing software, many more don't - and a few have Chromebooks too! The Chrome store also provides me with a single place where I can distribute the software. If I've got to publish a new version to the Windows App Store, Mac App store and Ubuntu package repositories for each version it's going to be particularly painful.

Presumably there's no real reason why we can't make the Web IDE work on both platforms though? Both are using Webkit and JS, and the serial library is already abstracted out...

Please could you take a look at the refactoring branch? At the moment it should be quite simple to change it such that it can be used on both platforms?

It'd also be really really nice if I could use that IDE design :) How does the options menu work? That's the thing that's really bugging me at the moment...

Hey @Gordon,

I've just done a quick screenr to show the functionality I have so far which you can watch here.

Some interesting points. The port picker popup checks for connected ports every 3 seconds, so as you can see, it detects when I connect my Espruino to it. The Terminal has "tab completion" of known commands (you can see that in the video, i type dig > TAB > W > TAB). The connect button changes colour to show connection state. Click the button when connected, disconnects. I put the terminal on the right / bottom, as this is how most IDE's work with the "console" window being secondary (think chrome dev tools). I guess this could be an option though :) (I do want to develop modes where you can turn one off to give focus to the task at hand)

I haven't got round to designing the options window just yet, but I see it opening in a similar popup to the port picker (but bigger).

I guess there is no reason why this couldn't be used for both. And I'd be more than happy for the design to be used for the web IDE aswell.

I've put everything I've done in a github repo here https://github.com/mattbrailsford/EspruinoIDE if you want to give it a try. Just download and launch the run.bat file (this will only work on windows currently).

The concept for the design kinda came from Google Coder. I just wanted to make it clean, simple and fresh for people starting out, whilst still being functional for more advanced users (but lets face it, uber advanced devs are probably going to use their own tools).

Cheers

Matt

Oh, and by the way, issuing the digitalWrite commands, did indeed turn my LED1 on and off :)

It looks really nice! Thanks for putting the source online...

just wanted to make it clean, simple and fresh for people starting out, whilst still being functional for more advanced users (but lets face it, uber advanced devs are probably going to use their own tools).

That's exactly my feeling... It's really about being easy to get started with, but not getting in the way if you're more advanced. As you say, past a point people are just going to use something like emacs :)

I'm not quite sure how to merge your code in though... It looks like a total re-write, and I'm not sure it's really that compatible with being a Chrome Web App?

I'd need to take a look at how the Chrome Web Apps work (is there a guide to getting setup / running with the current code?). Ultimately, while there are a few bits specific to node-webkit, it's mostly angularjs which should work for both, however yea, this isn't how things are currently setup.

If I can get to grips with how Chrome Web Apps work, I can see how easy / hard it would be to port over and we can make a decision then on the way to go.

Matt

Chrome 'first app' covers complete basics of how the apps work. It's dead simple: http://developer.chrome.com/apps/first_app

For the current code:

• follow the instructions in the readme
  
• When running, right-click and 'inspect element', and you can get to the console
  
• You can change code on the disk, right-click and 'reload app', or I believe you can actually develop right inside Chrome if you need to.
  
  

But there's the refactor branch that will hopefully be a hell of a lot easier to get to grips with and modify, so I'd work from that.

The one annoyance right now is the way you don't seem to be able to load JavaScript files from within JS code. It might be I'm missing something, but right now it seems like you have to include scripts from main.html :(

(I think we should create separate threads of discussion if these go any further)

@Gordon I am aware of echo(0), but that does not help. I want to know when a command finishes, in a consistent manner. At the moment, I can only read the output, which is inconsistent when single commands return single or multiple lines, and inconsistent when a command spans over multiple lines. I saw that you got over this problem by waiting for 100ms in the Web IDE, but that's a slow hack, not a proper solution solution.

You said you wanted something easy to get started with, but not getting in the way if you want something more advanced. At the moment, the protocol only allows simple things and is getting in the way of something more advanced. I wanted to implement a more advanced cli, that's why I asked if there is a protocol other than espruino's cli.

My purpose was to have a cli which allowed command history, which sent minified commands, which only sent the commands once the code block was closed, and it should also implement internal commands such as .sendFile('path/to/file.js'). A GUI could be built on top of it, even integrated with chrome.

If that is not possible, then I guess we'll have to deal with the current behavior, and simply forward everything to espruino and wait for 100-1000 miliseconds after every command.

You can force a newline that doesn't execute with Alt+Enter (27 + 10). If you obey really simple rules (run it through a lexer and then bracket count) you'll be able to ensure that every command you send to Espruino does finish.

I don't think it does stop you implementing a nice CLI, as someone has already done with espruino-cli for node.

Or your other option is to just do Serial1.setConsole() to get the JS console out of the way, and then you can implement whatever you like on USB by creating a handler with USB.onData(...).

But I really don't think just implementing yet another CLI will help 99% of Espruino's users. It seems like several of the features you want could be implemented in Espruino itself, and then they'd be available to everyone regardless of their OS or what terminal they decided to use. There's already a command history by the way - or did you have something else in mind?

Building a lexer that handles serial output is painful, because you never know when the output finishes. Imagine sending multiple commands in quick succession which return one or multiple lines, building something to properly handle that is really tricky, especially if the first commands take longer to return.
I didn't know about the Serial1.setConsole, I will take a look and see what options are available.

I don't view implementing additional options on the espruino itself as a good idea, because these just take up memory and force other users to use the same options. For example, I would be interested in a "search-through-history" feature where you press ctrl+r like on linux, start typing some text and the matching commands are shown, and I already implemented something similar in here, but most users wouldn't even be aware of that.

Take a look at how other projects do it, the most popular servers are distributed with a separate client (mysql, redis, mongodb). They implement a protocol, and you can build alternate clients for them if you need to.

Anyway, I'm totally fine with the status quo, these are just my 2c :P if the "cli-on-espruino" will not change and cannot be avoided/disabled, we'll just have to adapt to it.

Well, I've added the Alt-Enter key code. If there's something like that that can be added - for instance some character combination at the start of a line => silently accept command and prefix output with a special character then both types of command-line can coexist.

Thing is, your example of the command history search could be implemented relatively easily on the Web IDE (by grepping for lines starting with > and :) and it would work without problems. Maybe it's not 'pretty', but to me it's prettier than having some app-specific protocol and then completely reimplementing the IDE :)

I'd like to "vote" for going without Chrome (you even have to register with Google for running the IDE) and maybe go native apps instead (that webkit approach with a lightweight native wrapper might be a good alternative).

Chrome seems a high price for having serial line access.

I am happy with the CLI though, so, just my 2ct, not an urgent matter.

-Stev

Hopefully we can have both (without massive amounts of code duplication!) - there are a huge amount of people for whom having the Chrome web app is a massive bonus though - particularly in education. I honestly think that given the choice of an app that's pre-installed (like screen on mac) or an app that needs installing from an unknown source, most people will just stick with screen - especially the more advanced users who are the ones that are campaigning for a non-chrome app :)

Just a quick note - the refactored Web IDE is almost usable now... No options pane or flasher yet, but what seems like a pretty clean separation of code.

Another update: The Web IDE refactor branch now looks like the attached.

I wanted to keep stuff really simple (still with the plugin interface) so it's got Matt's CSS, but without AngularJS I'm afraid.

Each module (like load/save) is self-contained (including any HTML it needs) so it should be pretty straightforward for people to add/change things I hope.

Overall it's looking good - I still have to implement the nice port selector and Settings menu, but if I don't get distracted then that could happen later today.

1 Attachment

Now we have a settings page + port selector... I've still got to implement some of the backend stuff for settings - but it's looking a mile better than it was before.

@mattbrailsford I have bastardised your design a bit I'm afraid... Please - if you have a second, feel free to make it look pretty again :)

It'd be awesome if this could be made to run on node-webkit though. I don't think it should be that difficult - apart from storing settings (which aren't implemented yet) I guess all you need is a wrapper for node-serialport.

2 Attachments

@Gordon OMG WHAT HAVE YOU DONE! hehe, only joking :)

I'll pull the source down and try and take a look at it over the weekend to neaten things up. Once that's done, I can then see what can be done with regards to a node-webkit version.

Matt

Hi Gordon,

I've pulled it down now to just get it setup and working which I now have. Before I go ahead and do anything though, I guess it would be a good idea to know what you would actually be happy for me to do? I mean, how much are you prepared to change? Not that I'm a master dev, but there are a few things going on that I would consider bad practice / don't make it easy for others to get involved (such as JS generating the majority of the HTML, vague folder structure, etc).

If you just want me to make it look pretty, I can do that, but I think it could do with a bit of tidy up / refactoring in other places too.

Also, do you have any objection to using other existing modules / plugins? (ie, in my demo, I was making more use of 3rd party plugins for common things rather than re-inventing the wheel such as a jQuery Terminal plugin, and a better code editor plugin).

As I say, it would just be good to get an idea of your expectations.

Cheers

Matt

Hi Matt,

I'm very happy for some tidying up (I want this to be as maintainable and easy for people to tweak as possible) - but please try to change it such that it really is easier for people to get into, and not just easier for you because you're used to working on websites with Angular :)

For example I actually found it extremely difficult to see what was going on in your project. This isn't meant as criticism, it's just that I'm not used to AngularJS:

The folder names are not obvious to me (I've had to look inside them to figure out what they're meant to be). I found the port picker view relatively easily, but the actual JavaScript that drives it is actually spread between the template and the JS file which is in a different folder with a different name and capitalisation. The port picker is also in AngularJS's special template HTML so I've got to delve around in AngularJS docs to see what's meant to happen...

I can't think I'm the only person that doesn't know AngularJS... I want to make the IDE really easy to understand and modify, even if you have a relatively simple grasp of JS, HTML and CSS (and probably jQuery). I think it's quite a hard balance (which I'm fully aware I haven't got right).

The movement of HTML creation into the JS was actually something I'd started doing in the refactor, to avoid stuff like this where there are a bunch of things defined and it's not clear which file is responsible for their implementation (especially if buttons can be turned on and off from the Settings menu). There's so much code in the IDE that you can't have a single JS file per view, and if you can't do that, things get difficult very quickly if you don't somehow define the HTML along with the code that expects to use it.

So what's the alternative to JS generating HTML? IMO, my usage of CSS/HTML is quite poor - but it feels like when used properly the actual HTML that's generated ends up being about as simple as you can get (and any abstraction might actually make it harder to understand what was going on).

As far as libraries - I'm happy to use other libraries, but again - personal taste. The only library in your vendor folder I know of is jQuery, so to me, every time one of them is used in code it's something I have to Google, try and find documentation for (which is hard in some cases) and understand before I can work with it.

• What does Ace actually do better than CodeMirror? I've already tweaked jshint to get rid of warnings for Espruino-specific things (eg. ES6's binary numbers), and the autocomplete suggestions are at least vaguely Espruino-related, so just changing for the sake of it would be a bit of a step backwards.
  
• The Terminal was written over a year ago when I couldn't find a terminal plugin that I could get working nicely. I guess things have moved on now - however it still has changes that make it well suited to Espruino (like the ability to insert HTML between terminal lines, which helps with the tutorial). If the terminal really adds something we could change it over - but I don't want to be in a position where any changes are actually removing functionality from the Web IDE.
  
  

Anyway, that's all a bit long - but I hope it explains where I'm coming from...

As an aside, it strikes me that we might be able to forward serial communications over HTTP (or WebSockets when Espruino implements them)... If we could do that, it might actually be possible to serve up the Web IDE from the Espruino itself - which would be pretty awesome :)

Hi Gordon,

That makes sense.

I guess the thing I like about angular is that it is built for web applications and fixes a lot of concerns (it brings things like two way binding and dependency injection). I've do a lot of web dev, and just find a lot of time is spent hooking things up, when binding can handle a lot of that for you. Equally, whilst the folder structure might be initially confusing, it follows some strict conventions, so as you use it, it's clear where everything should go.

This said, I'm no angular evangelist, I just know it makes my life easier for this type of application, but if it's not a route you want to go down, it's not a problem.

Regarding the HTML, I was having a thing about this earlier, and was wondering whether a technique similar to what you have already done with "data" tabs in the options, where you load it off disk? My reason for wanting this is because HTML in strings is just awful to work with (escaping quotes etc) and you loose all formatting (unless you format your string over multiple lines, which is still not very nice). I'm not sure what this would bring from a performance perspective though. If you did go this route though, maybe just having a "templates" or "partials" folder to place them in would work well.

Regarding the plugins, again, I'm not precious about them, I just think they look / work nicer, and those have better support behind them, so are less likely to become redundant. But I didn't realize you had to make changes to them for Espruino. It should be possible to make these existing plugins look a little nicer anyway.

I'll have a bit of a tinker over the weekend anyway with how it is now, and just see how I get on.

RE Having the ide running from the board, this would be totally awesome. I think that would make for a great way to get up and running with less friction.

Matt