Architecture: WiFi support for WiFi enabled boards ...

Posted on
Page
of 3
/ 3
Last Next
  • This post is primarily in relation to the work on the port to the ESP8266 but I believe it has general relevance.

    As boards start to have native capabilities beyond just GPIO and UART (and others ...) we start to find richer functions such as WiFi and TCP/IP stacks. How should support for these capabilities be architected into the project?

    For example, the ESP8266 has native WiFi capabilities. What should the API exposed to the JavaScript programmer look like? Do we have an "interface" pre-defined that is the "function set" for WiFi that support for a board containing WiFi should provide? Should this be a "module" that the programmer has to import or should it simply be "just there" as a core set of functions for JS programmers on that board?

    From a practical standpoint, I have the knowledge and skills to write the C level APIs to drive an ESP8266 to perform WiFi access point attachments and other WiFi related functions but don't know where to start to think about exposing those to the JavaScript programmer that will actually use them on an Espruino environment running on an ESP8266.

    Neil

  • My first thoughts would be to look at the apis for the wiznet and cc3k

    But there should probably be some thought given to whether that's what we want long term.

    I mean, it certainly needs to support the existing http system...

    But beyond that... What are the sorts of functionality to be exposed? Beyond simply making http requests, and acting as an http server, I mean

  • @DrAzzy,
    This is good ... so let's follow this through ... and thanks for the names of wiznet and cc3k, I didn't know where to start. If we look here: http://www.espruino.com/WIZnet

    We find a bootstrapping of:

    var eth = require("WIZnet").connect();
    

    Now ... for an ESP8266 (or a board with native WiFi), I would not imagine we need to require a module ... I would imagine it would "just be present". So I'm imagining an object called "WIFI" being already present in the JS context.

    So where in the example for WIZnet we next do things like "eth.getIP()" we might do things like "WIFI.getIP()".

    Looking at the CC3000 this looks close to a WiFi story. See:
    http://www.espruino.com/CC3000

    there we have

    var wlan = require("CC3000").connect();
    

    and I imagine that the ESP8266 WIFI object would be a predefined instance of something like that.

    But then we get into interesting notions such as the ESP8266 not only being able to be a station ... but also be able to be an access point that other WiFi stations can connect to.

    ... and all of a sudden ... we start entering the world of "architecture" and "specifications". What I would like to avoid is the notion that ESP8266 does it one way, CC3000 does it another way and un-named devices yet to come do it other ways. Ideally we define the capabilities of an abstract WiFi device and then define the functions on a representative class such that a concrete implementation can "just be" dropped in... where boards such as the ESP8266 already have concrete implementations already present for use without any require() statements.

    And this is where we as a community and project owners need to come together and put our heads together. I think progress on ESP8266 networking will have to be suspended until we can figure this one out as the decisions made here will reverberate down the line. Ive internally tested WiFi and TCP/IP working concurrently on the ESP8266 with Espruino and no issues ... so we are literally at the stage now of deciding how to externalize for consumption.

    Neil

  • Neil, I'd figure there'd be a driver, at least architecturally. It sort of sounds like you might be wishing for wifiDriverX, Y and Z to each provide a standard wifi interface, so that you don't have to care what driver is in play - just call wifi.connectAP() or some such. You could write such a driver layer, with each function implemented in terms of the respective wifi libs, but more layers don't come for free.

    Speaking of native ESP8266 coding, I've been playing with Lua on the ESP8266, which is interesting, but it's not Javascript. The especially interesting part for me is how the ESP8266 SDK (which enables application code to run on the same hardware being used for networking) is mapped onto the Lua runtime: C functions, registered with the SDK for specific events, calling into the Lua code through the standard Lua C API.

    Within this SDK framework, the SDK (not an OS) takes a turn processing network traffic, then calls into your application, where you take a turn doing stuff with pins and lights, say. 6 alarms are built in at some level, though I'd expect you only need two and a setTimeout data structure (that's the purpose of the alarms). Pin-triggered interrupt(s), etc... All analogous to how we arrange event-oriented programs on Espruino.

    All very encouraging in terms of making Espruino run there. Yielding to the SDK in the JS event loop is key.

    One downside of the CPU-sharing for the developer is that if your event-handler does very much work, you run the risk of interfering with network operation. That's where I'm actually quite keen on Espruino, in that you can easily optimize those particular time-sensitive functions with pre-compilation. Maybe nodemcu has an option for that too, but I haven't found such a thing.

  • I'd look at the existing ESP8266 module. Down the bottom there are a list of functions... createAP, getAPs, connect, etc. It'd be nice to make those consistent - whether we're running on the ESP8266 or just using it as a separate module.

    I've tried to arrange that the calls are at least named the same in ESP8266 and CC3000 (and WIZnet where functions are shared), although in the ESP8266 module everything uses callbacks, which is something I think we should stick with.

    As far as a built-in variable for ESP8266, I'd say don't do this. I tried it for the LCD on the 'HY' boards that have built in LCDs, and it was actually really ugly. Using require would definitely be a way to make the ESP8266 support fit in with everything else, and still having the connect() function could be useful - for instance it could actually initialise the WiFi adaptor (which you might not want to initialise by default). In the future it could take an object where you could specify things such as transmit power - so if you were doing a project where you knew you couldn't supply the full 400mA, you could specify that even before WiFi booted up.

    Implementing the actual API shouldn't be too hard. I've already split everything out in this directory so it'll just be a matter of making your own ESP8266 directory and filling in the relevant bits.

  • The wiznet implementation is probably the cleanest one to look at. Basically just 4 files:

    • jswrap_X.c/h the implementation of connect/getIP/etc
    • network_X.c/h the implementation of the socket API that's used

    If yours can fit in the same kind of style it opens a lot of possibilities - for instance it would be relatively easy to modify the existing code such that multiple network adaptors could coexist. You could have GSM and ESP8266, and could be using both at the same time.

    Just to add, we also have GSM, so:

    And they all use the same form of instantiation and have the API functions named the same where possible. It would be a big help to everyone if this one worked the same way as well.

  • If I understand the contents of discussion I would recommend to look at nodeMCU

    https://github.com/nodemcu/nodemcu-firmw­are/wiki/nodemcu_api_en

    There is a complete set of network functions defined that could be an inspiration for you.
    This should have some advantages:
    a/ Gordon could probably implement all by using "require" if ESP modul with AT commnads is used with Espruino,
    b/ Neil could implement these functions direct in JS for ESP (the functions are already written by creators of nodeMCU),
    c/ the users of both HW solutions will use the same language and interface.

    P.S. Sorry, English is not my mother language.


    1 Attachment

    • nodeMCU_INDEX.png
  • @profra
    Good post. I am not familiar with Lua which I think is the language that I think is used in this binding. NodeMCU has two stories ... one is an ESP8266 board (which should be able to run the Espruino port) and the other is their IDE/Lua language. The APIs that you listed match almost exactly the "C" language bindings supplied by the ESP8266 SDK. (The ESP8266 is made by a company called Espressif ... this is the raw IC that IS the ESP8266 which board makers then include in their boards. Espressif provides a C language set of libraries that can be linked with a C application to achieve WiFi, network, GPIO, timers etc etc ... it is this SDK API that is being leveraged by the Espruino support for the board specific capabilities).

    I had assumed that we had walked this space before and there would have been a prescribed strategy that the ESP8266 support would simply "follow" ... however, I'm starting to get the impression that this may be "relatively" new ground. What I think I'll do is do the deep research and make up a "report" on the different APIs that are being used now for networking in Espruino today as well as the APIs being used by ESP8266. However, I believe I want to stay away from anything that would smell of ESP8266 specific ... the goal should be that the ESP8266 should be "just another board" and even if I arrive at Espruino as a skilled ESP8266 kind of guy ... the story should be that this is an Espruino story. To make progress and allow testing, I'm thinking of continuing design and implementation with a class name that may be "EXPRIMENTAL_ESP8266.". Obviously that will not be the final method name or structure but changing names and parameters will be simple once we figure out the API structure we want while at the same time not blocking anyone who wants to continue with the board support. I don't want to stall through analysis paralysis and as long as the method names have no ambiguity that they have not been finalized, we should be ok and not accidentally make something that would lock us in ...

  • Yeah - we already do require() for built-in modules, for WizNET/CC3k, http...

    For the ESP8266, though, there are some extra functionality - you mention getting a console through telnet. That's a really cool functionality, and it'd be great if that was exposed - but I'm not quite sure what the right API for that is. We'd probably want it to act like a serial port, but we'd have to deal with the possibility of there not being anything connected to telnet.

  • I'm starting to get the impression that this may be "relatively" new ground

    There are 4 different ways to connect Espruino to the internet right now, and they all use the same basic API. I don't see there's any reason why the 5th way of connecting Espruino should magically have a different one.

    Sure there are some extra functions you'll want to expose, but it should be possible to do those in a similar way to the existing ones - and maybe we could tweak some of the existing ones such that they match up better (for instance making the functions appear async on CC3000/WIZnet, and making getIP on the ESP8266 return an object).

    It's useful to try and make each single function as extensible as possible. For instance on WIZnet/CC3000, the setIP function lets you set the IP, but also the Mac address (on WIZnet where it's possible), dns, subnet and gateway.

    you mention getting a console through telnet. That's a really cool functionality, and it'd be great if that was exposed

    ... but that should be something that was available for all network adaptor types. As there is already an abstraction layer for the socket API it seems completely nuts to implement the telnet functionality at a lower level so it only works on one device.

  • Howdy @Gordon,
    Completely with you. What I am going to do is study all the APIs ... and see if I can't get close to what the community wants. I'm a huge fan of written stuff and plan to document the hell out of the work I do ... Open Source if fantastic ... but no-one wants to write stuff up. This post was to see if anyone wanted to "help" do some of the research and say "this is the API you should use" ...

    As for the Telnet, I agree 100%. When I started the project, I couldn't get the serial stuff working but because the ESP8266 is a WiFi device, I could telnet into it and drive the JavaScript exactly that way. Now that I am starting to get something working, I am thinking that I can see that ESP8266 support can supply a Telnet story via JS level ... and hence NOTHING is likely to be needed at the board/code level.

    Neil

  • Great!

    I guess at some point we could write a Telnet module in JS that handled telnet - it might be it needs some firmware tweaks to work properly, but it would be a good start. Something very useful would be at minimum a password on the telnet connection ;)

  • A Telnet module that would give you something that looked like a serial port so you could put the console on it when someone connected would be very useful. I'm not sure where my reply here went, I remember posting here about how awesome Telnet console would be, but the post seems to have vanished. That would mean you wouldn't have to provide for easy physical access in order to make code changes....

  • Actually, the good news is that I had that working before I had serial working. The ESP8266 is great at being a network device and telnet client support was plain easy. What I'm hoping is that as part of the telnet support we can look into adding that as a supported mechanism for the IDE so Telnet would become just another "connection option".

  • we can look into adding that as a supported mechanism for the IDE

    Yes, there's actually something built in for that already. However because it's a web app there could maybe be some complications - I think the existing code could be node.js only...

  • Howdy Gordon, is there any basic write-up or other docs for network access as opposed to serial access for the IDE?

  • Nope - however the implementation of it is here - looks like it's actually designed to work with Chrome.

    To get it working, create your own local copy of the Web IDE, then change serial_chrome in main.html to serial_socket: https://github.com/espruino/EspruinoWebI­DE/blob/master/main.html#L75

    I have a branch where both can coexist, but I haven't merged it yet.

  • I took a pass at the esp8266 wifi library and reorganized Neil's code (the core of the code is largely unchanged). I'd like to discuss the result in the context of the generic wifi library. I tried to document each function as well as I can, but I'm sure there's room for improvement. The result can be seen at https://s3.amazonaws.com/s3.voneicken.co­m/espruino/functions.html#wifi
    For reference, the wiki discussion about all this was at https://github.com/espruino/Espruino/wik­i/Generic-WiFi-Design
    The reorg follows much of the wiki, but isn't the same by a long shot.

    One item I have not resolved is asynchronicity. It seems that all functions have to be async in order to work for wifi modules connected via serial. Is that true or does it not really matter when a high enough baud rate is used? I have to admit that I don't like the async getters because it turns simple things like finding out what the IP address is into callback hell. The best I can come up with is to add an optional callback parameter and to document that using the direct return value may not be as portable. This way the user can decide whether to use the more portable callback stuff or not. :-)

    Note that what I have isn't yet working. I worked on the docs first in order to get this discussion going so I'm not changing the whole thing twice..

    NB: I also tweaked the ESP8266 library to remove some overlap and there are remains of the ESP8266WiFi library in the doc file that will go away.

    NNB: I find the formatting of the documentation to be really painful. Specifically, I can't figure out how to reliably get a bullet list into the description section and I find the fact that each newline starts a new paragraph (as opposed to needing an empty line) frustrating too.

  • Great!

    For the async stuff - yes, you definitely need it async for devices connected on serial - as the processing of serial data only happens in the idle loop. Espruino (and also the ESP8266+GSM modules) would need some pretty major mods to fix that, but your optional callback sounds like a good idea... 'Use this to be portable, or if you don't care then just use the return value'.

    With the doc formatting, it's because I have a really hacky markdown parser in build_docs.py. It'd be great if there was something simple that worked properly and could be pulled in (I'm tempted to use use node.js for building the docs, and marked works really well).

  • Just some thoughts here:

    • Can getRSSI just be a field in getStatus? Seems odd to have a specific function for that one thing.
    • Maybe onChange should be removed and wifi should just have events emitted on it, so you could do wifi.on('connected', ...)? You can document events via JSON - have a look at jswrap_http (I think).
    • I know this wasn't in the Wiki version, but someone mentioned about positional parameters to connect and startAP - maybe the password/key should be in options instead, as it's not required if you connect to an insecure network?
  • Can getRSSI just be a field in getStatus? Seems odd to have a specific function for that one thing.

    Sounds plausible. getStatus is becoming quite the monster, I hope we don't run into issues with that...

    Maybe onChange should be removed and wifi should just have events emitted on it.

    Makes sense.

    someone mentioned about positional parameters to connect and startAP - maybe the password/key should be in options instead

    Is that one comment/change or two?

  • Quick questions:

    • should it be the "wifi" library or the "Wifi" library?
    • should IP addresses be represented as integers (which endian-ness) or strings or arrays?
  • I made the changes suggested by Gordon above. Same file (https://s3.amazonaws.com/s3.voneicken.co­m/espruino/functions.html)

    The two short questions are still open and the answer will cause further small changes.

    Gordon, would you mind making some small fixes to the pseudo-markdown so startAP and stopAP have decent bullet lists, or just tell me how to change the formatting so it works? (https://github.com/tve/Espruino/blob/mas­ter/libs/network/esp8266/jswrap_esp8266_­network.c#L710-L733 and https://github.com/tve/Espruino/blob/mas­ter/libs/network/esp8266/jswrap_esp8266_­network.c#L349-L365)

    Update: ooops, I forgot about the async optional parameter...
    Update2: now added

  • Honestly, up to you. Or WiFi. I was wondering about that but I have no real preference, just not wIfI or wIFI :)

    Does ESP8266 support IPv6 at all? Currently internally Espruino uses uint32_t for IP addresses (so that's what the utility functions that do conversions expect), but if there's any chance of IPv6 that needs to change.

    In terms of JS-land, strings all the way I think.

  • For the bullet points, I'm a bit lost. Actual code is here:

    https://github.com/espruino/Espruino/blo­b/master/scripts/build_docs.py#L75-L87

    So from what I can see if what you're doing, it should work if (like you seem to be doing) the line starts with *. Do you want to have a poke around? I can definitely stick it on my todo list, but I won't manage it tonight I'm afraid.

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Architecture: WiFi support for WiFi enabled boards ...

Posted by Avatar for Kolban @Kolban

Actions