Recording and analyzing sound

Hi Guys,

I have a sound which is between 150Hz-250Hz. I need two things:

1) The average frequency of the sound for the last 5 seconds. After a discussion with @Gordon it appeared that 1Khz sampling rate will be enough and this could be done on the Puck with FTT.
2) Get the digitalized sound from the microphone for the last 5 seconds.

The microphone is connected on D2(for analog), D1 (for power) and GND.

Thank you.

Something like this seems to work:

var w = new Waveform(512,{doubleBuffer:true,bits:16});
var a2 = new Uint16Array(512);
var a = new Int16Array(a2.buffer);
w.on("buffer", function(buf) {
  a2.set(buf);
  E.FFT(a2);
  var m=0,n=-1,b;
  for (var i=150;i<250;i++){b=Math.abs(a[i]);if(b>n){n=b;m=i;}}
  console.log(m.toFixed(0)+"Hz @ "+n);
});
w.startInput(D2,512,{repeat:true});

It's sampling at 512Hz, which seems to be good enough for what you want.

Ideally the input voltage range would be higher and you could go back to 8 bits though - I had to use 16 because my microphone isn't very good.

Also, there appears to be some bug in the FFT where it's returning signed values when it shouldn't be - I'll look into that, but for now the attached code seems to work ok.

Ok, FFT is now fixed, and will be in the 1v92 release.

When 1v92 is released you'll be able to use this (slightly cleaner & faster) code:

var w = new Waveform(512,{doubleBuffer:true,bits:16});
var a = new Uint16Array(512);
w.on("buffer", function(buf) {
  a.set(buf);
  E.FFT(a);
  var m=0,n=-1;
  for (var i=150;i<250;i++)if(a[i]>n)n=a[m=i];
  console.log(m.toFixed(0)+"Hz @ "+n);
});
w.startInput(D2,512,{repeat:true});

The 'for' loop is intentionally quite tight - Espruino's execution speed isn't that fast so I'm trying to make it as quick as possible.

The @ xxx part that's printed is the volume of the most prominent frequency. If no frequency is playing then you'll get a random frequency shown, but with a lower volume.

Also, just a note: don't expect great battery life when using this. Realistically you'll need to use a bigger power source if you want to get more than a day's life out of it.

Hi Gordon,

Really appreciate your feedback and quick reaction! I will require an average frequency for 5 seconds every hour, as we discussed it could double decrease the battery life, but the Puck open architecture give possibility for additional power supply, everything is a matter of evaluation.

Thank you again.

Ahh, forgot about that - yes, 5 seconds an hour will provide much better battery life.

It should be easy enough for you to tweak the code above to do a 5 second average (probably just keep a sum of the FFTs, and do the loop over them at the end).

Hi @Gordon,

Based on the code below:

var clipSeconds = 3, currentSecond = 0;
var w = new Waveform(512,{doubleBuffer:true,bits:16});
var a = new Uint16Array(512 * clipSeconds);
w.on("buffer", function(buf) {

  a.set(buf, currentSecond * 512);
  currentSecond++;

  if (currentSecond >= clipSeconds) {
    w.stop();

    E.FFT(a);
    var m=0,n=-1;
    for (var i=150;i<250;i++)if(a[i]>n)n=a[m=i];
    console.log(m.toFixed(0)+"Hz @ "+n);
  }
});
w.startInput(D2,512,{repeat:true});

My understanding is that all the sound as buffer will be stored in "a" from where once all the data is aggregated we could define the frequency.

When this code is deployed the flow hangs on "E.FFT(a);" and I cannot connect to the Puck anymore. Please advice.

Thank you.

Are you using the up to date (1v92) firmware?

If it still fails I'd try and simplify things a bit... if you have a massive buffer already, why not just write into it directly - for example:

var clipSeconds = 3;
var w = new Waveform(512*clipSeconds,{bits:16}­);
w.on("finish", function(buf) {
  var a = buf;
  E.FFT(a);
  var m=0,n=-1;
  for (var i=150;i<250;i++)if(a[i]>n)n=a[m=i];
  console.log(m.toFixed(0)+"Hz @ "+n);
});
w.startInput(D2,512);

It is possible that the buffer size is too large for E.FFT, but there are checks in place to avoid that kind of problem and to throw an error if there isn't enough memory available.

Failing that, is there a reason you don't want to perform the FFT in smaller chunks and average the result?

Hi @Gordon,

I have following use-cases.

• Client requests an average frequency for the last N seconds.

  var clipSeconds=5, currentSecond=0, sBuff=new Uint8Array(clipSeconds);
  var w = new Waveform(512,{doubleBuffer:true,bits:16});
  var a = new Uint16Array(512);
  w.on("buffer", function(buf) {
  a.set(buf);
  E.FFT(a);
  var m=0,n=-1;
  for (var i=150;i<250;i++)if(a[i]>n)n=a[m=i];
  sBuff[currentSecond]=m.toFixed(0);
  console.log(sBuff[currentSecond]+"Hz @ "+n);
  currentSecond++;
  if (currentSecond >= clipSeconds) {
  w.stop();
  var avg = E.sum(sBuff)/clipSeconds;
  console.log("Average " + avg.toFixed(0) + " Hz");
  }
  });
  w.startInput(D2,512,{repeat:true});
  

This code is tested and working on firmware (1v92)

• Client requests an audio for the last N seconds.

  var clipSeconds = 5;
  var w = new Waveform(512*clipSeconds,{bits:16});
  w.on("finish", function(buf) {
  console.log(buf.length);
  });
  w.startInput(D2,512);
  

For ought I see the "buf" contains the audio which should be stored and send to the client ?

I really appreciate your feedback!

Thank you.

Yes, that should be correct - you'll just have to be careful with how you send it to the client, since over Bluetooth LE it could take a while!

You could also consider just recording it in 8 bits?

Hi @Gordon,

Yes recording in 8 bits is reasonable.

I would like to send the clip to the client. I have following questions based on the code:

var soundBuff;
NRF.setServices({
  0xBCDE : {
    0xABCD : {
      value : "N/A",
      readable : true,   // optional, default is false
      writable : true,   // optional, default is false
      notify : true,   // optional, default is false
      onWrite : function(evt) { // optional
        var clipSeconds = evt.data[0];
        var w = new Waveform(512*clipSeconds,{bits:8});
        w.on("finish", function(buf) {
          soundBuff=buf;
          NRF.updateServices({
            0xBCDE : {
              0xABCD : {
                value : buf,
                notify: true
              }
            }
          });
        });
        w.startInput(D2,512);
      }
    }
  }
});

1. Is this the right way of doing this ?
   
2. How can I free memory once the data is sent to the client ?
   
3. How can I view the result with "nRF Connect"
   
   

Thank you.

Hi,

Yes, that looks fine - however there are limits on how much you can send by Bluetooth LE in one go - I believe 20 bytes at the moment - so you may need to send the data in several chunks, by just calling NRF.updateService from a setInterval.

The only thing causing your data not to be freed is soundBuff=buf; - if you didn't have that I believe everything would be freed automatically.

Actually seeing your data in nRF Connect is unlikely to be possible - while you can read the value of a characteristic, it's not set up to read repeated notifications - which is what you'd have to do to send the amount of data you want to send. You'd need your own app.

The other option is actually to use the Bluetooth UART that's on by default. For instance you could write:

function getData(clipSeconds) {
        var w = new Waveform(512*clipSeconds,{bits:8});
        w.on("finish", function(buf) {
          Bluetooth.write(buf);
        });
        w.startInput(D2,512);
      }

And then writing "getData(2)\n" to the UART's RX characteristic would cause the TX characteristic to output all the data. You could view that on the Nordic UART app.

Hi @Gordon,

Thank you for the advice! I installed "nRF UART 2.0" and tried

var v;
function g(s) {
  v=s;
  Bluetooth.write(s);
}

If you write without a semicolumn once you connected to the Puck it says "unexpected EOF". With a semicolumn I succeeded to light the LED on from here https://www.espruino.com/Puck.js+BLE+UART

If you write the function(with or without) a semicolumn in "nRF UART 2.0" or "nRF Connect" with notifications enabled I receive what I wrote, for example in "nRF UART 2.0" I have:

TX:g(5);
RX:g(5);

TX:g(9)\n
RX:g(9)\n

Do you think this could be Android specific issue as appeared in the Security thread ?

Thank you.

No, that's expected - it's because you're interacting with the JS interpreter, which echoes back what it receives. To disable that you can also send echo(0)\n.

The easiest way is to send "\x10g(5)\n" - sending character code 16 on a blank line will stop Espruino from echoing just for that line. Obviously it's harder to do that (if not impossible) via nRF UART though!

If you don't send a newline, the command won't be executed. Then when you do send a newline you'll get g(5);g(9) executed.

Hi @Gordon,

You are right. The main issue was how to enter new line character, but that can be done typing "g(5)" followed by "shift + enter" for new line character and then press "send" button and the characteristics return the expected result.

Thank you.

Hi @Gordon,

For some reason

E.FFT(a);

from above is not working on 1v95.20

Thank you.

How is it not working? Errors? Wrong values?

Hi @Gordon,
I missed that, the device is reset.
Thank you.

I'm not sure I understand... Is it that you updated the firmware but your old software was left in flash memory, so didn't work?

Hi @Gordon,

I updated my puck to 1v95.153 and uploaded this code:

function fq(sec){
  var cSec=0, sBuff=new Uint8Array(sec);
  var w=new Waveform(512,{doubleBuffer:true,bits:8});
  var a=new Uint16Array(512);
  w.on("buffer", function(buf) {
    a.set(buf);
    E.FFT(a);
    var m=0,n=-1;
    for (var i=150;i<250;i++)if(a[i]>n)n=a[m=i];
    sBuff[cSec]=m.toFixed(0);
    console.log(sBuff[cSec]+"Hz @ "+n);
    cSec++;
    if (cSec >= sec) {
      w.stop();
      var v=E.sum(sBuff)/sec;
      var rKey=new Date(Date.now()).toString();
      console.log(rKey,v);
    }
  });
  w.startInput(D2,512,{repeat:true});
}

which worked before.
When you call fq(3) for example, the puck got reset. The problem comes from E.FFT(a);

Thank you.

Could you try rebooting Puck.js with the button held down for ~10 seconds (until the red LED has stopped blinking). That should clear out any saved code.

I'm pretty sure the issue is that you had saved software from an older firmware version still in flash memory - and perhaps you weren't uploading from the right-hand side of the IDE (or reset on send was turned off in options).

I've just tried a latest build of Espruino and your code seemed to work fine.

Hi @Gordon,
I took another puck and put a completely new battery. Before uploading any code I follow your advice to clear completely the flash area holding down for ~10 seconds the button.

1) It is working on 1v194xxx(the version with which the puck was bought)
2) It is NOT working on 1v95.153(after dfu upgrade with nrf connect), after function execution it disconnects.
3)It is working on 1v92.41(after dfu downgrade with nrf connect).
4) It is NOT working on 1v95.153 (after dfu upgrade with nrf connect) I added LED2.write(1); after E.FFT(a); It again disconnects and the led is not on.

Please advice how to proceed ?

Thank you.

Sorry - I just checked this out on 1v95.153 and you're right. It seems to be related to the amount of memory needed to do an FFT on that many samples. In 1v95 I increased the amount of JS variables, which decreased the available stack.

It seems we got hit with two issues - there's a check to ensure that the stack never overflows, but it seems it was broken in this case so the error was never reported.

I've now fixed that and shifted some memory back towards the stack, and if you try one of the latest builds again it should work.

Hi @Gordon,

I checked out and build from source 1v95.71 and can confirm that this is working. Thank you!
My expectation was to build a version > 153 but found 71 ?

Thank you.

My expectation was to build a version > 153 but found 71 ?

I know! Honestly I'm not sure why that is. The number should be based on the number of GitHub commits since the last RELEASE tag. All I can think is for some reason the RELEASE_1V95 tag didn't get pushed to the Espruino repo until just now, so it had been counting the commits since 1V94 instead.

Hi @Gordon,

I tried to build from source 1v95.92 and have:

~/source/repos/github/espruino/Espruino/targetlibs/nrf5x_12/components/libraries/bootloader/ble_dfu/nrf_ble_dfu.c:50:20: fatal error: boards.h: No such file or directory
#include "boards.h"

compilation terminated.

Thank you.