ESP8266 upgrade woes

While hacking strings and character arrays around is pretty normal in the microcontroller world, when you're sending serial data as text, it's also a laborious pain to code up. Which is probably why most Arduino examples using the ESP8266 wifi module use the esp8266.h library.

Which is great.
Except it doesn't work with our firmware.

Once again, we're up against the idea of library abstraction.
While hiding everything away and making it accessible through a few functions makes your main code look neat, that only holds true for as long as the library works! And when you've built a project based on a library that doesn't work, it's often more hassle trying to untangle someone else's code than it would have been to build your own from the ground up.

Anyway, we struggled with the esp8266.h library for a couple of reasons: First up, our wifi SSID is 18 characters long. The esp8266.h library has buffers for 16 character strings. It's a quick fix, once you know where to look - but infuriating for a long while, when the command

AT+CWJAP = "SSIDquitealongname","password"

was being sent to the wifi module as

AT+CWJAP = "SSIDquitealongnamepassword","password".

No wonder we couldn't connect to the router! (the reason, presumably, being that our SSID character array did not contain a \0 trailing character to indicate the end of a string, so the code/library/mcu ran on into the next memory space, until it found the \0 byte at the end of our password. However, there's no guarantee that the SSID and password strings would be in continuous sections of memory space - it could just have easily been a load of random garbage!

To fix this, we had to change the esp8266.h file and increase the buffer size

And then in the esp8266.cpp file, because we didn't bother working out which functions were used to handle the SSID/password combination, wherever there was a character array of 16 characters, we increased the buffer size to 20.

This at least allowed us to connect to the router successfully and to obtain an IP address.
But even then, we had to do a bit more hackery. See, the ESP8266 library expects certain responses from the wifi module. One of these responses is that when the AT+CIFSR instruction is sent, the response includes the character pattern STAIP followed by the IP address

Our (updated) wifi modules don't do this. They simply split out the IP address followed by OK.
So instead of looking for a response of STAIP, we had to know the beginning range of our IP address, and look for that instead.

In fact, we found the esp8266 library full of little things like this - expected responses and strings either chopped up or entirely discarded, depending on what the firmware expected to receive (rather than what the actual response was).

In short, it looks like we're going to have to abandon the esp8266 library and write something ourselves. And if you've updated your firmware using the same files as we did a little while back (so your firmware boots up looking like the screenshot below) you might have to do the same!

Measuring rotation with Arduino and GY-521 (MPU-6050 Accelerometer + Gyro)

Blog posts have been a bit thin on the ground this month. Mostly that's due to a house move (Nerd Towers has once again relocated) as well as constructing a new workshop to house all our electronics goodies as well as recovering the laser cutter and CNC and other "cool tools" to get them all back into one place.

But we've also been pretty busy with real-life work.
Not just hacking code together, but actually putting some of our (not inconsiderable) experience to use, making cool stuff. Sadly, a lot of it is commercially sensitive. Hence the lack of posts, even when a few of us have been working on some really nerdy projects!

Anyway, here's a quick post about a generic idea we've been playing about with. It's basically a way of allowing us to use real-world objects as controllers. It was mentioned a little while back and we were using an accelerometer to measure the rotation of an object around the horizontal axis (holding an object horizontally and rotating it around the y-axis)

This time, we wanted to measure rotation around the vertical (or Z) axis.
This proved a little more difficult. Try as we might, no amount of messing about with accelerometers was going to get us a workable result. Maybe it's because we'd be moving the object at exactly 90 degrees to the direction in which the acclerometer worked (i.e. it can measure any amount of rotation in relation to gravity - the vertical plane - but can't work out rotation in relation to a plane at 90 degrees - i.e. the horizontal plane). In fact, we're pretty convinced that this is exactly why we couldn't get a workable solution.

So then we took to Inkscape and drew a PCB which allowed us to place 12 hall sensors in a ring. By placing a small magnet inside our object, we could place it inside the ring of hall sensors and rotate it. It didn't matter which way up the object/PCB was - as long as the object was inside the hall sensors, we could consistently detect it's rotation. Bingo! A working solution.

Except now we're looking to remove the need for the object to sit inside a ring of sensors, and for it to work on pretty much any surface. We tried a few different ideas but the one that worked consistently well was the GY-521 module.

Sure, it's an MPU-6050 Accelerometer - and we know accelerometers are no good for detecting rotation in the horizontal plane. But it also has an onboard gyro. Which can.

Now unlike accelerometers, which give you a position in space, based on the effect of gravity on each of three different axes, a gyro only reports angular velocity - i.e. when the direction of movement is changing in any axis. Which means it only gives meaningful values while its rotating and doesn't report it's position in space, when it's standing still.

To demonstrate this, we used the example code from the Arduino Playground page and mapped the output readings to a graph in Google Spreadsheets. We ran the code, rotated the sensor around the z-axis and drew a graph of results.

When we did this in the past - with the accelerometer for example - we at least got some recognisable kind of patterns (sine waves) with some extraneous noise. This time our values bear no relation to each other - it's just a jumble of noise! The spikes and troughs in the middle reading relate not to the position of the sensor in the vertical plane, but the velocity of the sensor as it rotated around the vertical axis.

With all this in mind, we put together some simple firmware for registering rotation "events":

#include<Wire.h>
const int MPU_addr=0x68;nbsp;  // I2C address of the MPU-6050
int16_t AcX,AcY,AcZ,Tmp,GyX,GyY,GyZ;

int curr_Y = 0;
int last_Y = 0;
int curr_val = 0;
int max_val = 12;
int min_speed = 50;


void sendMsg(int d){
   // this just simply reports whether we've started moving in a clockwise
   // or anti-clockwise direction and updates a counter/pointer accordingly
   
   if(d < 0){
       Serial.println("anti-clockwise");
       curr_val--;
       if(curr_val < 0){ curr_val = max_val;}  
   }else{
       Serial.println("clockwise");
       curr_val++;
       if(curr_val > max_val){ curr_val = 0;}  
   }
   Serial.print("value: ");
   Serial.println(curr_val);
}

void setup(){
   Wire.begin();
   Wire.beginTransmission(MPU_addr);
   Wire.write(0x6B);   // PWR_MGMT_1 register
   Wire.write(0);        // set to zero (wakes up the MPU-6050)
   Wire.endTransmission(true);
   Serial.begin(9600);
}

void loop(){
   Wire.beginTransmission(MPU_addr);
   Wire.write(0x3B);   // starting with register 0x3B (ACCEL_XOUT_H)
   Wire.endTransmission(false);
   Wire.requestFrom(MPU_addr,14,true);    // request a total of 14 registers
   AcX=Wire.read()<<8|Wire.read();    // 0x3B (ACCEL_XOUT_H) & 0x3C (ACCEL_XOUT_L)
   AcY=Wire.read()<<8|Wire.read();    // 0x3D (ACCEL_YOUT_H) & 0x3E (ACCEL_YOUT_L)
   AcZ=Wire.read()<<8|Wire.read();    // 0x3F (ACCEL_ZOUT_H) & 0x40 (ACCEL_ZOUT_L)
   Tmp=Wire.read()<<8|Wire.read();    // 0x41 (TEMP_OUT_H) & 0x42 (TEMP_OUT_L)
   GyX=Wire.read()<<8|Wire.read();    // 0x43 (GYRO_XOUT_H) & 0x44 (GYRO_XOUT_L)
   GyY=Wire.read()<<8|Wire.read();    // 0x45 (GYRO_YOUT_H) & 0x46 (GYRO_YOUT_L)
   GyZ=Wire.read()<<8|Wire.read();    // 0x47 (GYRO_ZOUT_H) & 0x48 (GYRO_ZOUT_L)
   
   // measure rotation around the Y-axis to work out
   // active rotation (the giro only measures AS THE
   // thing rotates, it doesn't report absolute position)
   
   // reduce the sensitivity
   GyX = (GyX >> 8) & 0xFF;
   GyY = (GyY >> 8);
   GyZ = (GyZ >> 8) & 0xFF;
   
   //Serial.print(" | GyX = "); Serial.print(GyX);
   
   curr_Y = 0;
   if(GyY < (0-min_speed) || GyY > min_speed ){
       curr_Y = GyY;

       if(curr_Y < 0 && last_Y < 0){
           // this is a continuation of a rotation movement
       }else if(curr_Y > 0 && last_Y > 0){
           // this is a continuation of a rotation movement
       }else if((curr_Y >= 0 && last_Y < 0) || (curr_Y <= 0 && last_Y > 0) || last_Y==0 ){
           // this is a change in rotation, so trigger "rotated"
           sendMsg(curr_Y);
           // now we don't want this to repeat to quickly so
           // let's just shove a (blocking) delay in here
           delay(250);nbsp;
       }
   }

   last_Y = curr_Y;
}

And here's a video showing it in action....

ESP8266 re-programmer PCB

A little while back we updated some firmware on one of our ESP8266 modules. Well now we've only gone and bought about twenty of them off ebay - and they all need updating!

While it's possible to do this with a usb-to-serial device and pull wires out and plug them in again in a little breadboard, it's also pretty time-consuming. And a bit fiddly.

And when real-life work gets in the way of nerding about, time-consuming and fiddly means more time working and less time nerding. Which means we needed something to make re-programming all these ESP8266 modules that little bit quicker. So we came up with this:

It's a simple re-programming board for the ESP8266 module. Just plug the module into the twin DIP socket, hold down the "program" button and hit reset. The wifi module boots up into "ready-to-program" mode and you can just dump your new firmware to it at the push of a button. No more messing about swapping wires and trying to time everything just right!

Until we got this little thing made, things were getting a little bit messy and out of control at work!

But with this programmer, we managed to re-flash about 20 wifi modules in less than 40 minutes (it takes 90 seconds or more to download the firmware onto the device, so this was pretty good going!) and get a couple of panels of PCBs ready to stick them onto...

Here's the PCB layout should you want to make one yourself.

Wifi Reprogrammer for ESP8266 module

Just don't forget to put a 1K pull-up resistor on the RESET and GPIO1 lines.
We've drawn the board with both 1206-sized pads (for all you surface mount junkies out there) as well as holes for through-hole resistors too.

The pin layout on the right matches the pin layout for an Arduino Pro mini, which in turn is pin-to-pin for most USB-to-serial modules commonly available on eBay and various other online electronics outlets.

We've found that for programming the wifi module, you don't even need a dedicated power supply - the 100mA from the USB port should be enough . If you do want to power the wifi modules externally (perhaps for testing after flashing the new firmware) simply remove the Vcc wire from your usb-to-serial connector and connect a battery (or other power supply) to the Vcc/Gnd pins.

OSC UDP TCP/IP and choosing the right technology

Back in the real world, we occasionally have to venture out from behind the nerd bench and making stuff for fun and enter into the commercial arena - it's what some of us have to do in order to pay the rent. It's also what some of us do for the sake of "staying relevant", keeping up with different technologies, and applying it in a real-world application.

The wealth of different technologies for controlling hardware has exploded in recent years. Maybe it's the IoT (internet of things) "branding" or maybe it's platforms like Arduino and Raspberry Pi that have led to a renewed interest in making things talk to each other - whatever it is, there's never been a better time for making hardware and integrating it with cool technology like smartphones, tablets, handsets and PC computers.

But with all this choice comes responsibility.
A responsibility to choose the most appropriate technology for the job in hand. Sometimes "most appropriate" simply means "one I know how to use". And for many things, that's fine. But something that has become apparent, as the IoT revolution has exploded, is that more and more people are selling their services as industrial/commercial "specialists" - and many haven't the first idea what they are doing!

Over the years we've built layers of abstraction into technology. A bit like cars. They're simpler to use and easier to work on (since everything comes in a single, encapsulated module that can just be swapped out). But with this ease of use comes a cost - when things go wrong, it's not so easy to fix! With cars you take it to the garage; a part may no longer be "fixable" as it might have been 30 years or more ago, so it's just thrown out and a replacement part fitted. That is the cost of simplification.
With technology the same thing seems to be happening. More and more, projects are built from encapsulated modules of code (or libraries if you like) which make everything nice and simple. Until it goes wrong. But, unlike a poorly performing car, it's not always so easy to throw out the "broken" module and replace it with a new one - sometimes it's not immediately obvious which module is broken. Sometimes there may not be a replacement library/module available. What then?

But worse than this, there seems to be a fundamental mis-understanding of which technology to use and when. At best it's a lack of awareness. A gap in knowledge or understanding of how things work "under the hood" meaning that the best-fitting solution is either not-known or overlooked. At worst, it's a lack of what we might call "giving a shit".

An example springs to mind - it's difficult to be too specific, since this is a real-life, commercial ongoing project. But the outline is something like this:

We've been asked to build some hardware that a user interacts with. Our hardware connects to a central controller which, when certain combinations of events occur, sends a message to a video player to play a specific sequence. During this time our hardware should remain silent. When the video has finished playing, our controller receives a message from the video player to say it's ready to accept incoming messages again.

That's the jist of things. There's a bit more to it, but those are the basics. We've been asked to send our messages to the video player using the OSC protocol. On the face of things, it seems ok. But when we look a little further into things, questions start to raise about whether or not this would be the most appropriate technology for the job....

Now first up, let's just say that TouchOSC is a great product.

It works across multiple platforms, iOS/Android etc. The interfaces look great and it has an inbuilt editor that lets you put sliders, dials, buttons and the like together really easily. It's dead easy to use and the end result looks pretty good (a bit same-y; once you've seen the TouchOSC app all interfaces tend to look the same, but it's very good for what it does).

But it our case, is it really the most suitable solution?
Firstly - and this is our biggest bugbear - TouchOSC uses UDP to send messages. It's a fire-and-forget protocol. You press a button, a message gets fired, and you just hope that it gets there.

UDP has it's uses. It's relatively fast (compared with TCP/IP). It's great for realtime gaming. And OSC has proven very popular amongst musicians and even lighting engineers. There's an old joke that nerds often tell to explain UDP:

"I'd tell you a joke about UDP. but I'm not sure if you'd get it".

Understand the joke and you pretty much understand UDP.
For some applications, UDP is a great fit. For games, for example. If you're continually updating a number of other players across a network of your character's location, UDP is perfect. It's fast, lightweight and does the job. It doesn't just address one target - it can be broadcast across an entire network easily. TCP/IP would be a poor substitute, with it's latency and error-checking and resend-on-fail, only being delivered to one recipient at a time, all adding to the time it takes to update some game-world co-ordinates. Repeat that for multiple players sharing a game, and you've got a pretty slow, unresponsive game. Compared to TCP/IP then UDP is fast.

For realtime audio, UDP is also pretty good - for the same reasons.
When you're whirling an onscreen rotary encoder, and the app is blarting out a stream of values as the encoder position changes, you want to be able to send a high volume of messages quickly. You want the realtime feedback (the change in volume) to be almost instantaneous, not laggy.

For UDP's strengths as a high-volume, high-speed transport layer, it also has one major weakness: you never know if the message was received. It's a bit like shouting into the darkness. There's no specific end-point, you just put a message onto a port number and anyone who is listening gets the message.

In contrast, sending data via TCP/IP is a bit more akin to using Royal Mail's parcel tracking service. It's sent to a specific address. It's slower. When the parcel arrives, confirmation is requested and sent back to the sender to acknowledge everything arrived as it should. Sending data via TCP/IP has an "overhead" but at least you know your data has reached its destination.

So why is using OSC and UDP such a pain for our particular application?

It's important to understand that we're not saying UDP sucks and TCP/IP is great. Just that there are better reasons for choosing one over another to match the requirements of the project.

For realtime gaming, UDP is ideal. If a packet of data doesn't arrive at the destination (if you should into the darkness but your voice is drowned out by a passing drunk singing "Danny Boy" walking past your door at 2am in the morning) it doesn't really matter. Because any second now, there'll be another packet of data coming along. And another. And at the receiving end, the missed data doesn't really cause a problem. If, for example, you're updating an on-screen avatar, whose location is being updated many many times a second, the odd dropped set of co-ordinates is hardly noticeable. The on-screen character might jump four pixels instead of two, or it might be so far in the distance as the jump in game-world position is unnoticeable.

Similarly if you're using an OSC controller over UDP to, say, make some lights go up and down, the odd missed packet of data doesn't really matter. If you turn a virtual rotary encoder and the lights don't immediately respond, because you're looking at them for feedback, you know to turn the wheel a little bit more - a whole heap of data gets blasted towards the lighting controller and it takes just one packet of data to update the light.

For high volume, high speed data, UDP is very useful.

However, for two-way communication, with long delays between messages, it's not quite so robust. In our situation, we're having to use a fire-and-forget "shouting into the darkness" communication method when what we really need to know is that our messages have been received (and, similarly, it's really important that we don't miss any messages coming back).

For this particular project, TCP/IP would be a much better communications protocol. We're sending low volume messages. Latency isn't a problem - if the response time between a user interacting with our hardware and the video starting to play was as much as a few hundred milliseconds, the end result would be no different!

But not being able to guarantee that messages are received could cause all kinds of headaches. Here's how the mode of operation should go.

• User interacts with hardware
• Message sent to video player
• Hardware stops responding to user while video plays
• Video player sends message after video ends
• Hardware responds to user input again
• Repeat ad infinitum

 It only takes one dropped message to cause the whole system to appear to have stopped working. If we reach a trigger point and our device firmware says "tell the video to play and stop responding to user input until we get a message back" what happens if our message to the video player fails to arrive?

Our hardware is now in a "suspended state" and no return message is ever going to be received to turn it back on (since the video player hasn't been told to play a video and thus send an "end of video" message back.)

Or maybe the video player does get the "play video" message. Everything is fine. We stop responding to user input, while the video is playing, as required. But what if the "end of video" message coming back is lost? Our hardware never wakes up again!

In either scenario, we eventually end up with hardware that appears to have stopped working. And all for the sake of choosing one communications protocol over another. When queried, we were told that "both systems have worked well for us and our OSC libraries use UDP so that's what we're using".

Of course, it would be possible to implement a feedback loop, from devices to video player and back again - send a message and if no response is received within a certain timeframe, resend it. But then acknowledgements coming back from the video player are broadcast across the entire network, to every connected device. So how do we know which acknowledgement is for which device? By implementing some kind of address system? So... pretty much recreating the TCP/IP protocol. But by shouting. And resending a lot of data. Suddenly our fast, zippy UDP transport layer is bogged down with noise and multiple packets of data......

This isn't an anti UDP rant or an anti-OCS moan, far from it.
But it's just to highlight - because there seem to be an awful lot of "computer people" out there unaware of what's going on under the hood - that sometimes pre-built libraries and handy, encapsulated modules of code, are not always the "best" fit.

Sometimes you actually need to understand what is going on. Especially if you're working in a commercial environment and billing a client for your time. Simply falling back on someone else's code and assuming everything is going to be alright because it worked for someone else in the past isn't good enough. Because they may have used it under a completely different set of circumstances, to achieve an entirely different result.

Please people. from one bunch of nerds to another, be mindful of what you're doing, why you're doing it, and choose the most appropriate technology - not just the quickest/cheapest/easiest to prototype with. That way we can all build an internet-of-things to be proud of, not just a buggers-muddle of poorly-designed devices all fighting for our bandwidth!

Upgrading firmware in ESP8266 wifi modules

It's been a while since we played around with our wifi modules but in recent months we've been inundated with Arduino projects in the real world; the latest involves running an AVR off a lipo battery (actually, it's a li-ion but you get the idea) to interface with an ESP8266 wifi module.

So far so good.
Except our Arduino Pro Mini modules are set up to run at 8Mhz from a 3.3V supply (an atmega328 is not guaranteed to be stable at above somewhere around 12Mhz on a 3.3V supply). And with the internal oscillator at 8Mhz on an AVR, the error rate is far too high for a 115,200bps baud rate.

Which  means either
a) change the baud rate on the ESP8266 or
b) change the controller to one that can run faster at lower voltages

Obviously a faster mcu running off a battery will drain it more quickly, so we're looking to reduce the baud rate if possible. Except our ESP8266 modules are running some pretty old firmware - and they don't support changing the baud rate.

ESP8266 AT Command Set by chris_holden2495 on Scribd

The datasheet says we should be able to issue the AT+CIOBAUD command to query/set the baud rate. Each time we tried this (and even tried AT+CIPBAUD in case it was a typo!) the result was ERROR.

It looked like we were stuck with 115200bps.
But after a bit of digging around on the intertubes, we found this website: http://www.xess.com/blog/esp8266-reflash/

And it talks through re-flashing the firmware on the wifi modules using a nice easy-to-use Windows interface (rather than the nasty run-it-and-hope Python scripts a lot of other sites recommend). It basically involves running the ESP8266Flasher.exe and downloading the v0.9.2.2.ATFirmware.bin - both of which we've bundled up and put here.

Firstly, wire up the module with the GPIO1 line pulled to ground (normally we leave this line floating). Then start the executable

Select the .bin file (we're using version 0.9.2.2 but other versions may be available on the interweb)

Hit the  "download" button. It takes about 2-3 minutes (for ever in computer time) to re-flash the wifi module. Booting up, however, returned nothing but garbage

Normally the ESP8266 spits out some gobbledegook when it boots up. But that's because the boot-up log is always spit out at 74880bps before the module returns to the baud rate set by the AT+CIOBAUD command.

But we expected at least some kind of "OK" or "ready" message. Instead we got nothing. Trying to enter AT commands resulted in more garbage. So out of curiosity, we set out serial monitor baud rate to 9600 and tried again. This time, the response was much more meaningful. Not only could we see that we were running newer firmware, but we also received valid responses from the AT+CIOBAUD? command

As it turns out, this new firmware defaults to 9600bps, which is what we were looking for. So there's no need for us to update the baud rate, once the firmware has updated. And - more importantly - we're able now to communicate with the wifi modules using an Arduino running at 8Mhz of it's own internal oscillator, powered by a lipo battery.

It's only a small victory. But a success all the same!