Sunday 16 October 2016

More matched betting

It turns out it was about a year ago (almost to the day) that a few of us had a go at Matched Betting. In fact, Matt had been doing it for a while before then, and it was only when he was showing off his £2k new PC with uber-graphics cards that we were convinced it wasn't all going to go horribly wrong and end up with him living in a cardboard box under a railway bridge.

But, sure enough, he knew what he was doing. And seemed to be making money quite easily. I had a go and, following his instructions, I made about £700 over two months. I stopped after doing all the "easy" sign-up offers (rather than continue with the reload offers) but this left me with £700 tied up in a couple of betting exchange accounts.

Rather than withdraw this cash, I did the same offers again, on behalf of friends and family. This time, having a large(ish) amount in the exchanges allowed me to do multiple offers at once, getting through the offers more quickly and turning free bets into cash. I turned over about £450 within ten days for my mother, £400 within about the same timeframe for one of my friends in Brighton, and about £300 for another friend, just in time for Xmas.

It's worth noting that as Matched Betting became "a thing" late last year, and as more and more of us were doing it, the rewards for the time invested went down and down - the bookies were onto it, pretty quickly!

However, having Christmas dinner for the entire family paid for by Mssrs Ladbrokes, Paddy Power and William Hill was quite a nice touch. So, naturally, we're going to give it another try this year. Now, of course, we can't go doing the same sign-up offers again, but - in preparation for running a few matched bets through the exchanges again this year- I was clearing up some old accounts, and in doing so was offered a couple of free bets to get an otherwise dormant account active again. It's thanks to this that I found a bit of an anomaly:

Betfair (the exchange we use to "lay" the bets off) was offering a £25 free sportsbook bet. Simply put, if I wagered £25 on a an event (I choose Liverpool to beat Man U on Monday) we were immediately given a £25 free bet. I did exactly this, then went to the Smarkets website and laid off Liverpool (i.e bet on United to win or draw) for the same value.

Because each exchange charges a percentage commission, this "matched bet" was always going to lose a little bit of money. It turned out that the combination of prices chosenm, I stood to lose £1.69 irrespective of the result of the Liverpool v ManU game. A kick in the pants. But worth it, to earn a free £25 bet.

So with the "qualifying bet" placed and matched on the exchange, I was pretty pleased to see the free bet immediately appear in the Betfair account. Now I'm already having to wait over 24 hours for the qualifying bet to resolve, and didn't want to wait longer than this again, so chose to place the free bet on a much higher odds result on a football game happening in just a couple of hours time.

I chose Villareal and bet that their opponents (at 5/1) would win. The game was to kick off just two hours after the bet was placed, so it would all by over in a little under four hours. Of course, my bet selection didn't win (Villareal won 4-nil in the end) so our free Betfair sportsbook bet lost, and the amount in our Smarkets account went up.

So I just had to wait for the Liverpool/Man U match to resolve, and I'd have turned the free bet into cash. Except I didn't actually have to wait. Thanks to the "cash in" feature on Betfair, I was able to "cancel" the qualifying bet:


And by betting on the reverse result on the Smarket exchange, effectively "cancel" that bet too.


And if we look at the costs involved:
To cash out the Betfair bet, they took £24.31 from our initial £25 bet - so it cost us £0.69. Irrespective of the result on Smarkets (whether Liverpool win, Man U win or it's a draw) we lose £0.47 at most (only £0.42 if it's anything other than a Liverpool win). Which means that, irrespective of the match result, we've lost a total of £1.16 in "cancelling" our qualifying bets.

But this is less than the £1.69 it would have cost us, if we'd let the bets stand, to resolve fully 24 hours later. In short, it worked out better for placing a qualifying bet on an event that occurred after the event I had used the free bet on.

Once the free bet was "spent" (i.e. it lost and the money appeared in the Smarkets betting exchange - after all, the bet is only really insurance in case the lay bet fails) I was able to "cancel" the qualifying bet at Betfair, and the lay bet on Smarkets. Not only to generate lower loss, but also to resolve the entire position 24 hours earlier than had I allowed the bets to run to their full conclusion.

It's a tactic I might look at again in the future.
And one that everyone else doing Matched Betting might want to consider too. After all, if you're trying to turn over a number of high-value bets quickly, getting the cash out of the bookies and back into the exchange balance, rather than have it riding on sports events is critical. Being able to cancel qualifying bets and get the money back 24 hours earlier means another day of opportunities tomorrow - rather than having to pass over them because of a lack of cashflow!

There are a few of use having a go at matched betting over the coming weeks. We might just post some results here, to log our progress and - hopefully - explain how it all works a little bit better (complete with cock-ups, mis-calculations and lucky breaks).

The aim is a rather modest £300 for Christmas. At least it means no-one has to worry about paying for a turkey big enough to feed 14 of us on the big day!

Wednesday 12 October 2016

Messing about with MAX7219 common anode 8x8 LED matrix and Arduino

Many years ago, we built a couple of word clocks. They were pretty good fun and we used both 74HC595 shift registers and the MAX7219 LED driver chip. As we've been working with Arduino so much in recent months, we figured it'd be interesting to port some earlier code to create a working prototype.

So here it is:

Firstly, we had no information on our LED matrix. It was in a drawer somewhere so we had to work out the connections. This involved nothing more than taking a 3v lead and a ground lead and touching them to each of the pins on the matrix, to see which LEDs lit up.
As each LED  lit up, we noted which pin was grounded and which had power, and from this we were able to calculate which pin was for which row and which pin was for which column.


Once we had worked out the rows and columns on the 8x8 matrix, it was a case of hooking up a few wires and giving it a try!


According to the MAX7219 datasheet, turning LEDs on and off involves writing data to memory addresses using a two-byte 16-bit value. Luckily for us, it's a relatively simple protocol.

The first byte is the memory address - or row - to write the "image data" to. Memory address 0x01 corresponds to row 1, 0x02 to row 2 etc.

The second byte contains the LED on/off data in binary format, where 1 = LED on and 0 = LED off. So if we wanted to switch on just the first two LEDs in row 3, we would send the two byte message

0x03, 0xC0 (where 0xC0 is binary 11000000)

So we built some simple bit-banging routines (no nasty libraries here!) to pump data into the MAX7219 chip.





#define data_pin    7
#define clock_pin 8
#define load_pin    9
#define led_pin    13
bool led = false;

void sendData(int k){
    int j;

    Serial.print(F("sending: "));
    digitalWrite(load_pin,LOW);
   
    for(int i=0; i<=15; i++){
        j = k & 0x8000;       
       
        if(j==0){
            digitalWrite(data_pin,LOW);
            Serial.print("0");
        }else{
            digitalWrite(data_pin,HIGH);
            Serial.print("1");
        }

        digitalWrite(clock_pin,HIGH);       
        delayMicroseconds(20);
        digitalWrite(clock_pin,LOW);       
        k = k << 1;               
    }
    Serial.println("");   
    toggleLoadPin();       
}

void toggleLoadPin(){
    digitalWrite(load_pin,HIGH);
    delay(1);
    digitalWrite(load_pin,LOW);   
}

void sendValue(int ad, int v){
    int k=0;   
    k=ad;
    k=k<<8;
    k=k|v;   
    sendData(k);   
}

void setup() {
    // put your setup code here, to run once:

    pinMode(data_pin,OUTPUT);
    pinMode(clock_pin,OUTPUT);
    pinMode(load_pin,OUTPUT);   

    Serial.begin(19200);
   
    // this turns off the LEDs on the matrix
    // (though we can still send data to the max7219)   
    // lc.shutdown(0,false);// turn off power saving, enables display
    // lc.setIntensity(0,15);// sets brightness (0~15 possible values)
    // lc.clearDisplay(0);// clear screen
   
    int k=0;
    int addr;
    int val;

   
    // no decode mode
    sendValue(0x09, 0x00);    // do not use BCD

    // full brightness
    sendValue(0x0A, 0x0F);

    // scan limit = 8 (use 8 rows)
    sendValue(0x0B, 0x07);

    // normal operation
    sendValue(0x0C, 0x01);    // display on
    sendValue(0x0F, 0x00);    // no test mode

    // display test
    sendValue(0x0F, 0x01);
    delay(1500);
    sendValue(0x0F, 0x00);
   
    // now light up all rows   
    sendValue(0x01,0x55);
    sendValue(0x02,0xAA);
    sendValue(0x03,0x0F);
    sendValue(0x04,0xF0);
    sendValue(0x05,0x00);
    sendValue(0x06,0xFF);
    sendValue(0x07,0x3C);
    sendValue(0x08,0xC3);
   
}

void loop(){   
    // put your main code here, to run repeatedly:

}

To pump data onto the LED display, simply call the function SENDVALUE (row, led_data)

Sunday 9 October 2016

Reading data from a web page using Arduino and an ESP8266 module (no esp8266.h library)

A little while back, we were playing about with our newly-updated ESP8266 modules, with version 0.9.4 firmware. They're great little modules, and make excellent socket-based clients. But our new firmware broke the esp8266 Arduino library. Which meant we had to roll our own code for sending, receiving and parsing AT commands.

The basics are pretty simple - chuck a few strings into a few functions, send AT commands, catch the responses as strings and use the usual built-in parsing functions.

But when you start downloading entire web page contents (or, tbh, anything more than a couple of hundred bytes at a time) the responses get a little... well, confused it putting it mildly.

Now no-end of people will say "avoid Strings on an embedded controller". I'm one of them. But this was as much an exercise in making shareable Arduino code as much as it was about making a working solution. So while the very first impulse, when things started to go wrong, was "sod this, let's shove it on a PIC and use byte arrays", we had to resist - and shove it on an Arduino and use byte arrays....

There's a bit of confusion between Strings and byte/char arrays in Arduino-land. We haven't the scope to cover C++ programming concepts, pointers and memory locations here. But there's probably just about enough space to throw down some (working) code from which you can build your own embedded, web-based projects.

So let's get down to it....

The thing to remember is that we're working on an embedded system. An mcu with a massive 2k of memory. That's not a lot. So even though we're sending data to and from a web site, we need to write our web pages so as to keep the total character count down to a minimum.

We've built our input buffer to be about 600 characters. We might get away with a few hundred more. 1024 characters at push - the ESP8266 "chunks" large responses into 1024 bytes anyway. But that doesn't leave a massive amount of RAM left for all the other stuff we need to do. And it only takes one string of data to exceed the expected length, and we get all kinds of weird bugs and faults and corrupted data.

So although this code can retrieve data from a web page, it does so on the understanding that the web page itself is very very lightweight.


#include <LiquidCrystal.h>
#include <EEPROM.h>

#define wifi_reset    10
#define led_pin        13
#define button1_pin 12

#include <SoftwareSerial.h>
SoftwareSerial mySerial(8, 9); // RX, TX

// ------------ change this stuff --------------
String ssid = "YOUR_SSID";
String pwd    = "YOUR_PWD";
String user = "Chris";
#define DEST_IP "188.226.228.99"
#define HTTP_REQ " HTTP/1.1\r\nHost: www.yourdomain.co.uk:80\r\n\r\n"
#define HTML_TIMEOUT    10
#define HTML_OVERHEAD 2
bool in_debug = true;
bool show_wifi_output = false;
// ---------------------------------------------

int state=0;
bool has_ip = false;
char b;
String cmd;
bool wifi_matched = false;
char wifi_response[500];
char *ptr;
int ptr_index;
String this_ip;

void emptyWifiResponse(){
    memset(wifi_response, 0, sizeof(wifi_response));
    ptr_index=0;
}

void writeAT(String at){
    emptyWifiResponse();
    if(in_debug==true){
        Serial.print(F("Sending: "));
        Serial.println(at);
    }   

    // empty the incoming serial buffer
    while(mySerial.available()>0){ mySerial.read(); }       

    // send the AT command to the wifi module
    mySerial.println(at);   
}

bool wifiResponseContainsHeader(){
    bool b=false;
    int k=0;
    if(wifi_response[ptr_index-14]==':'){ k++;}
    if(wifi_response[ptr_index-13]==' '){ k++;}
    if(wifi_response[ptr_index-12]=='t'){ k++;}
    if(wifi_response[ptr_index-11]=='e'){ k++;}
    if(wifi_response[ptr_index-10]=='x'){ k++;}
    if(wifi_response[ptr_index- 9]=='t'){ k++;}
    if(wifi_response[ptr_index- 8]=='/'){ k++;}
    if(wifi_response[ptr_index- 7]=='h'){ k++;}
    if(wifi_response[ptr_index- 6]=='t'){ k++;}
    if(wifi_response[ptr_index- 5]=='m'){ k++;}
    if(wifi_response[ptr_index- 4]=='l'){ k++;}

    // characters -3, -2, -1, 0 are the
    // combination of CrLf x 2
   
    if(k>=9){ b=true; }
    return(b);
}

bool wifiResponseContains(String findThis){
   
    // search the entire character array for the string
    char __findThis[sizeof(findThis)];
    findThis.toCharArray(__findThis, sizeof(__findThis));

    ptr=strstr(wifi_response, __findThis);
    if(ptr!=NULL){               
        return(true);
    }else{       
        return(false);
    }       
}

void wifiResponse_Remove(String findThis){
    // find a particular string in the char array and remove it
    // (reducing the size of the array as necessary)

    // we could use clever C functions like memmove etc. but for some
    // functions behaviour is undefined if strings overlap, and for some
    // they create a shadow copy before moving; we might not have the
    // memory available to do this, so lets use the array itself

    // get the index of the first character in the string
    // we're trying to remove
    char c;
    int c_index=0;
    int r_index=-1;
    int match_count = 0;
    int find_count = findThis.length();   
    int buff_len = 500-find_count;
    char __findThis[find_count+1];
   
    findThis.toCharArray(__findThis, find_count+1);   
   
    // walk through the response array, looking for the search string
    for(int i=0; i<500; i++){
        if(wifi_response[i]==__findThis[c_index]){           
            match_count++;
            c_index++;

            if(match_count==find_count){
                // we've just found our string
                r_index=(i-find_count)+1;
                // quit the for-next loop early
                i=500;               
            }
        }else{
            match_count=0;
            c_index=0;
        }       
    }

    if(r_index > -1){
        // remove x characters from the array by simply budging everything up
        for(int i=r_index; i<buff_len; i++){
            wifi_response[i] = wifi_response[i+find_count];
        }
        for(int i=buff_len+1; i<500; i++){ wifi_response[i]=0; }
    }
   
}

bool checkIP(){
    // first, send a CIFSR instruction then wait for a response
    this_ip="0.0.0.0";
    writeAT(F("AT+CIFSR"));
    readInput(4000, "192.168.", "0.0.0.0", true);
    if(wifi_matched==true){               
        if(wifiResponseContains("OK")==true){               
            wifiResponse_Remove("OK");
            wifiResponse_Remove("AT+CIFSR");

            // convert the char array into a string
            // this_ip=wifi_response;                               
            this_ip=wifi_response;
            return(true);       
           
        }else{           
            return(false);
        }
    }else{               
        return(false);
    }
}

void readInput(int timeout, String match, String err, bool exitOnFind) {       
    unsigned long stop = millis()+timeout;
   
    emptyWifiResponse();
    wifi_matched=false;

    if(in_debug==true){
        Serial.print(F("Looking for: "));
        Serial.println(match);
    }
   
    if(show_wifi_output==true){
        Serial.print(F("raw wifi >"));
    }else{
        Serial.print(F("wifi response: "));
    }

    // point to the start of the character array
    ptr_index=0;
   
    do {
        while(mySerial.available()>0){
           
            char b=mySerial.read();           
            if(show_wifi_output==true){ Serial.print(b);}
           
            if(b==0x0a || b==0x0d || b < 0x20 || b > 0x80){
                // ignore these characters               
            }else{
                wifi_response[ptr_index] = b;
                ptr_index++;
           
            }
            if(wifiResponseContains(err)==true){
                // error condition has been matched   
                if(in_debug==true){ Serial.println(F("Error condition matched")); }
                stop=millis()-1;

                // read the rest of the buffer just to empty it
                emptyWifiResponse();               
               
                while(mySerial.available()>0){
                    b = mySerial.read();
                    if(b==0x0a || b==0x0d || b < 0x20 || b > 0x80){
                        // ignore these characters               
                    }else{
                        wifi_response[ptr_index] = b;
                        ptr_index++;           
                    }
                }
                break;
               
            }           
           
            if(wifiResponseContains(match) == true ){
                // requested string has been matched
                // so now return everything after the matched word   
                if(in_debug==true){
                    Serial.println(F("Match condition found"));
                    Serial.println();
                }
                wifi_matched = true;
                if(exitOnFind==true){
                    stop=millis()-1;
                    // read the rest of the buffer just to empty it               
                    while(mySerial.available()>0){
                        b = mySerial.read();
                        if(b==0x0a || b==0x0d || b < 0x20 || b > 0x80){
                            // ignore these characters               
                        }else{
                            wifi_response[ptr_index] = b;
                            ptr_index++;
                        }
                    }
                    break;
                }
               
            }
        }
    } while (millis() < stop);   

    // this is a bit hacky, but it'll have to do.
    // sometimes, when we're expecting OK we get "no change"
    // but if there's been no change, that's the same as
    // OK, so let's deal with that

    if(show_wifi_output==true){
        Serial.println(F("<"));
        Serial.print(F("Buffered response: "));
    }

    if(in_debug==true){ Serial.println(wifi_response); }
   
    if(match=="OK" && wifiResponseContains("no change")==true){
        if(in_debug==true){ Serial.print(F(" OK/no change (close enough) ")); }
        wifi_matched=true;
    }

    if(in_debug==true){
        Serial.print(F("match found: "));
        Serial.println(wifi_matched);   
    }
   
}


void readCloseInput(int timeout, String match, String err, bool exitOnFind) {       
    unsigned long stop = millis()+timeout;   
    cmd="";
    bool closeOK = false;

    // because the response from an AT command is small, we're using this
    // simplified function (complete with Strings!) to check the response
    // to the CIPCLOSE command, without wiping out the character array
    // buffer (which at this point contains the HTML we want to parse)
   
    do {
        while(mySerial.available()>0){
           
            char b=mySerial.read();           
            if(show_wifi_output==true){ Serial.print(b);}
           
            if(b==0x0a || b==0x0d || b < 0x20 || b > 0x80){
                // ignore these characters               
            }else{
                cmd.concat(b);           
            }
            if(cmd.indexOf(err)>=0){
                // error condition has been matched   
                if(in_debug==true){ Serial.println(F("Error condition matched")); }
                stop=millis()-1;

                // read the rest of the buffer just to empty it               
                while(mySerial.available()>0){ b = mySerial.read(); }
                break;
               
            }           
           
            if(cmd.indexOf(match)>=0 ){
                // requested string has been matched
                // so now return everything after the matched word   
                if(in_debug==true){ Serial.println(F("Match condition found")); }
                closeOK = true;
                if(exitOnFind==true){
                    stop=millis()-1;
                    // read the rest of the buffer just to empty it               
                    while(mySerial.available()>0){ b = mySerial.read(); }
                    break;
                }               
            }
        }
    } while (millis() < stop);   

    if(in_debug==true){
        Serial.print(F("Closed ok: "));
        Serial.println(closeOK);   
    }
}


void flashLED(int k){
    for(int i=0; i<k; i++){
            digitalWrite(led_pin,HIGH);
            delay(250);
            digitalWrite(led_pin,LOW);
            delay(500);
    }   
}


void connectToRouter(){   
        String p;
       
        // hardware reset the module
        if(in_debug==true){ Serial.println(F("Resetting wifi"));}
        digitalWrite(wifi_reset, LOW);
        delay(500);
        digitalWrite(wifi_reset, HIGH);
       
        // Look for ready string from wifi module
        readInput(4000, "Ready", "Error", true);
        if(in_debug==true){
            Serial.print(F("Wifi read: "));
            Serial.println(wifi_response);
        }   

        if(wifi_matched==true){           
            writeAT(F("AT+CWMODE=1"));
            readInput(2500, "OK", "Error", true);
            if(in_debug==true){
                Serial.print(F("Wifi: "));
                Serial.println(wifi_response);
            }   

            if(wifi_matched==true || wifi_matched!=true){               

                delay(1000);
                has_ip=checkIP();                   
               
                if(has_ip==false){                                                   
                    p = "AT+CWJAP=\"";
                    p.concat(ssid);
                    p.concat("\",\"");
                    p.concat(pwd);
                    p.concat("\"");                   
                    writeAT(p);   

                    // there's not really an OK message with CWJAP
                    // just returns a string when completed
                    readInput(16000, "OK", "FAIL", true);
               
                    if(wifiResponseContains("FAIL")==true){
                        // can't connect to the access point
                        Serial.println(F("Can't get onto the access point"));   
                   
                    }else if(wifi_matched==true){               
                        // so now query the system for an IP address again
                        has_ip=checkIP();                   
                        if(has_ip==true){                       
                            Serial.print(F("This ip address is: "));
                            Serial.println(this_ip);                                                   
                        }else{
                            // access point refused
                            Serial.println(F("Refused access to AP"));
                        }
                    }else{
                        Serial.println(F("No response from AP"));
                    }
                }               
            }
           
        }else{       
            Serial.println(F("Unable to reboot wifi module"));
        }
}

void getWebPageContents(String url){

            cmd="AT+CIPSTART=\"TCP\",\"";
            cmd.concat(DEST_IP);
            cmd.concat("\",80");           
            writeAT(cmd);           
           
            readInput(8000, "Linked", "ERROR", true);
            if(wifi_matched==true){

                // tell the server how many bytes to expect
                // by first building the request then getting
                // a character count (including trailing \r\n)               
                delay(100);
               
                cmd = "GET ";
                cmd += url;
                cmd += "?user=";
                cmd += user;
                cmd += HTTP_REQ;               
                   
                String c = "AT+CIPSEND=";
                c.concat(String(cmd.length()));               
                writeAT(c);                                                               
                readInput(4000, ">", "Error", true);
               
                if(wifi_matched==true){
                                                   
                    // now send the request to actually get some data
                    delay(500);                   
               
                    emptyWifiResponse();               
                    while(mySerial.available()>0){ mySerial.read(); }       

                    // send the AT command to the wifi module
                    if(in_debug==true){
                        Serial.print(F("Sending: "));
                        Serial.println(cmd);
                    }
                    mySerial.print(cmd);

                    // now we can't just use our readInput as the response could be in multipart chunks
                    // so we need to parse this separately
                    getHTMLResponse();
                   
                }else{
                    Serial.println(F("Can't enter send data mode"));
                    emptyWifiResponse();   
                }

                // now close the connection
                delay(300);               
               
                // empty the incoming serial buffer
                while(mySerial.available()>0){ mySerial.read(); }       
                // send the AT command to the wifi module
                mySerial.println(F("AT+CIPCLOSE"));

                // while it's tempting here to use the readInput function
                // to capture the response from the CIPCLOSE instruction
                // to do so would mean we lose the entire HTML page we've
                // just downloaded, so we use a different function here
                readCloseInput(4000, "Unlink", "ERROR", true);                               
               
            }else{
                Serial.print(F("Can't contact "));
                Serial.println(DEST_IP);
                emptyWifiResponse();   
            }
           
}

void getHTMLResponse(){
   
    unsigned long stop = millis()+(HTML_TIMEOUT*1000);
    emptyWifiResponse();
   
    int rec_state = 0;
    int resp_len = 0;
    bool quit=false;
    int char_received;
    bool html_header_found;

    emptyWifiResponse();
    ptr_index=0;
   
                                                   
    do {
        while(mySerial.available()>0){
            char b=mySerial.read();

            // try just concatenating the character b to a string in this
            // function and you'll quickly see why we're using a fixed
            // length character array to receive the HTML from the server!
           
            switch(rec_state){

                case 0:
                // get the first part of the response up to +IPD           
                wifi_response[ptr_index] = b;
                ptr_index++;               
               
                if(wifiResponseContains("+IPD,")==true){
                    // this is the start of the server HTML response
                    if(in_debug==true){ Serial.println(F("+IPD, string found")); }
                    emptyWifiResponse();
                    rec_state=1;
                }
                break;

                case 1:
                // get the number of characters included in the reponse
                // (so we know when to stop reading)
                wifi_response[ptr_index] = b;
                ptr_index++;
                if(wifiResponseContains(":")==true){

                    // remove the colon from the end of the string
                    // and parse it into a value
                    ptr_index--;
                    wifi_response[ptr_index]=0;
                   
                    // turn the character array into a string
                    // then use toInt to get the value
                    String t = "";
                    for(int it=0; it<ptr_index; it++){
                        t.concat(wifi_response[it]);
                    }

                    if(in_debug==true){
                        Serial.print(F("HTML character count: "));
                        Serial.println(t);
                    }
                    resp_len=t.toInt();   
                   
                    if(in_debug==true){
                        Serial.print(F("HTML response length: "));           
                        Serial.println(resp_len,DEC);                       
                    }               
                       
                    emptyWifiResponse();
                    ptr_index=0;
                   
                    char_received=0;
                    html_header_found=false;
                    rec_state=2;
                }
                break;

                case 2:
                // just keep receiving characters, adding them to the receive
                // buffer, up to the end of the response
                if(quit==false){               
                    wifi_response[ptr_index] = b;
                    ptr_index++;
                    char_received++;
                                       
                    // if the received string is too long, the memory overrun can cause
                    // all kinds of weird stuff to happen, so keep it nice and short               
               
                    // an HTML1.1 reponse separates the header of the message from the body
                    // using a double-CRLF entry

                   
                    if(html_header_found==false){
                        if(wifiResponseContainsHeader() == true){
                            if(in_debug==true){ Serial.println(F("End of HTML header found")); }
                            html_header_found=true;
                            emptyWifiResponse();
                        }
                    }
                       
               
                    // once you've received X characters, jump out of the
                    // timeout delay loop (ignore the last two characters
                    // as they're a CrLf combination)
                    if(char_received>=(resp_len-HTML_OVERHEAD) || b=='|'){
                        if(in_debug==true){   
                            Serial.println(F("Full message received. Quitting read routine"));
                        }
                        quit=true;
                    }
                }                               
                break;
               
            }
           
        }
    } while (millis() < stop && quit==false );

    // null terminate the char array so we can Serial.print it
    wifi_response[ptr_index]=0;

    if(in_debug==true){
        Serial.print(F("characters received: "));
        Serial.println(char_received);
        Serial.print(F("HTML response: "));
        Serial.println(wifi_response);           
    }
   
}

void setup() {

    Serial.begin(19200);
    mySerial.begin(9600);
   
    // set up the pins
    pinMode(wifi_reset, OUTPUT);
    digitalWrite(wifi_reset, LOW);

    pinMode(button1_pin, INPUT_PULLUP);

    Serial.println(F("Let's go"));
    state=0;   

}

void loop() {

    // if we're not connected to a router, re-try periodically
    if(has_ip==false){       
        flashLED(1);       
        connectToRouter();           
        delay(3000);
       
    }else{
       
        switch(state){

            case 0:
            // wait for the user to press a button to call the web page
            b = digitalRead(button1_pin);
            if(b==LOW){ state = 1; }
            break;

           
            case 1:
            // after a successful call to this function, our character
            // array buffer contains the contents of the HTML page so
            // we can do whatever we like with it
            getWebPageContents(F("/timekeeper/test.php"));

            // do stuff with the wifi_response[i] buffer here to parse
            // the response(s) from the web server
            Serial.println(F("-----------------------"));
            Serial.print(F("wifi_buffer contents: "));
            Serial.println(wifi_response);
            Serial.println("");           
            state=0;
           
        }       
    }   
}


Now we're fully aware that this code still has plenty of String uses throughout it. And Strings are bad. But our first attempt used nothing but Strings and - up to the point of retrieving an actual HTML page - worked really well. So we're in the process of updating the code to turn ALL instances of Strings into fixed length character arrays. But since the AT commands and responses are relatively small, and our incoming HTML buffer is a fixed 600 characters or so, there's plenty of available RAM to handle all the dynamic memory swapping that Arduino does to handle strings. So this is where we're up to at the minute.

Enjoy.