April 2011


This has been quite a short week due to the Easter bank holiday on Monday and the public holiday for the royal wedding today. I decided to spend the three days working on the Bubblino & Friends iPhone app that I built back in January.

Soon after January I talked to local artist and illustrator Sophie Green about her providing new artwork for the app and within a week or so she delivered some really nice pics for Bubblino, Pirate Parrot and another character that’ll be new with this version. I also chatted to a voiceover guy who came up with some great quality audio for Pirate Parrot.

Unfortunately I then got busy and so nearly three months passed without progress. This week I’ve finally got going on it again and have made loads of progress. I’ve added the new character and new art throughout. There’s a new tasteful tweet view UI and a nice simple form for building up advanced search queries that even searches using your phone’s location.

The update is essentially done now so I can’t wait to get it on the store, there’s just one more feature I keep forgetting about and some testing to be done but hopefully in just over a week there’ll be a new Bubblino & Friends available to buy on the app store!

Oh, and introducing my new robot friend:

He’s currently going by the name “Mr Roboto” but I’m open to suggestions for something better!

(The text for these weeknotes was all entered through a Kindle device. The screen is so great in the sunlight that I’m desperate to find ways to work on it. The keyboard is a little difficult so I may have to see if it’s possible to use an external one.)

This should be a fairly short post as the relays are quite simple. I decided to intercept the power to the heating by taking apart an old mechanical timer like these, stripping out all the mechanics and making use of the exposed connections which simply needed shorting to turn the socket on. With the mechanics removed there was plenty of space to fit the relay and the DC connector. I decided, for no particular reason, to use some 3.5mm headphone type jacks to handle turning the relays on and off. wiring it all up was fairly simple, the relay has 5 connections two for the DC connection and 3 for the AC connections, the AC ones are wired up differently depending on whether you want the AC circuit to be on or off when the DC signal is present. I also wired a diode across the DC connections on the relay as protection against any reverse current generated by the coil when the DC power is turned off. After reading the rating on the central heating system’s plug I went for (what I thought was) a 13A rated relay, when I opened the plug later on to rewire it though I found it only had a 3A fuse inside! At least my relay unit will be more capable if I want to use it elsewhere.

The place where my central heating plugs in actually has just two sockets. With this relay and the heating plugging into one that meant I only had one spare. I had already decided to use Powerline Ethernet to get network access to my boiler and so I didn’t actually have anywhere left to plug the Arduino in. I thought it would be a shame to have to load in an extension just for the sake of the Arduino so I began exploring other options. The first thing I looked at was tapping 5v from somewhere in the Powerline Ethernet device I was using, having taken it apart and looked up various of the ICs on the internet though I decided that probably wasn’t a good idea. I’m sure there’s somewhere I could’ve taken it from but considering the device was mostly about varying voltage and playing with everything from mains voltage to DC on the IC and a different level of DC over ethernet I decided to leave this one alone. I ended up looking on Farnell to see if they had any self contained switching PCB power supplies and with some searching help from Adrian McEwen again we found this 5VDC 2.75W switching PSU. I managed to fit this and an old USB socket into my old mechanical timer so that it was not only relay controlled but also provided 5V DC of power via USB, just the thing for an Arduino! The result is quite a nicely put together unit, the only problem being the black insulation tape I’m using to cover up the big circular hole where the mechanics were, I really ought to sort that out sometime. I quite liked the size and capability of the PSU though and may even get some more to power the XBee modules if the batteries don’t power them for long enough.

So, bill of materials for this unit are as follows, I’m not going to bother putting a circuit diagram in as it’s so simple but post comments if you have any questions.

Part Quantity
Mechanical timer 1 These are not the ones I used, just an example, so I can’t guarantee they’ll have the same amount of space, in fact they do look a bit smaller than mine.
5VDC/250VAC PCB Relay 1 Data-sheet – I had thought this was 15A rated but looking again seems it might be only 6A
Diode 1 Not entirely sure what type of diode this was, just what Adrian had lying around
3.5mm Jack 1 Data-sheet
3.5mm Plug 2 needed for this project (one either end of the connecting cable), this is a pack of ten though Data-sheet
5VDC 2.75W switching regulated PSU 1 Data-sheet
Female USB Type A socket 1 Just one I had lying around

For the light switch I went with this 5A solid state relay which I rejected for the central heating control when I thought I needed a 13A relay. Again it just required a simple circuit with a 3.5mm jack and a protection diode across. I’m housing all of this inside a thermoplastic junction box. it does the trick but probably isn’t ideal and has unsightly rubbery cones coming off each side which you can cut down to fit cables. Bill of parts for this one would be:

Part Quantity
Thermoplastic junction box 1 Not ideal but does the job
Solid state 3-15VDC/240VAC Relay 1 Data-sheet
Diode 1 Not entirely sure what type of diode this was, just what Adrian had lying around
3.5mm Jack 1 Data-sheet
3.5mm Plug 2 needed for this project (one either end of the connecting cable), this is a pack of ten though Data-sheet

So next blog post should cover the base station, and that’s when thing start to get a bit more interesting.

Spent much of this week finishing off the next stage of development for the Chess Viewer. One of the key things we wanted to get into this version was the ability to buy books from Everyman Chess’s website. If you look at their iPhone app page you’ll see that they have a selection of books available in the right format for the iOS app. Each of these is sold for $19.99. At the moment it would be quite a faff to buy the book in Safari on your phone and then download it and get it into the app, the new functionality aims to make this much easier by letting you browse the books on the phone and purchase them with your iTunes details. I was a little nervous about how difficult this could be having read some horror stories, but it did all seem relatively straight-forward so I had only allowed 2-3 days for it. In the end it did take just about 3 days and wasn’t so bad, I think I was well prepared having read some great web pages about it so that when I did have problems I knew how to handle them. This page had some great information and a class to use to work with the app store. This in-app purchases walkthrough was also invaluable. The first of those links seemed to be aiming for a different sort of thing than I was so I actually used the second more but reading both was good to know what to expect. The second one was good for taking you through the steps you’d need to do, including setting up the items to be purchased in iTunes Connect, and also telling you what you’d need to do before testing. A second page by the same guy gives a good breakdown for how to handle “Invalid Product ID” messages from the app store. I did get to a point where I was repeatedly getting no valid product IDs from Apple’s servers and so was seeing no products in my store, in this case (following the instructions on that page) I needed to delete the app and reinstall it to get things going again.

I got all of this working and managed to finish the other features and fix a few remaining bugs. I’m really happy with how things are looking, the app has gone from being a fairly basic looking iPhone app (with lots going on under the hood) to a much better presented app (with even more going on under the hood!) Hopefully we won’t find too many bugs while testing over the next few weeks and can get the new version on the store soon.

A few more libraries that really helped me were ASIHTTPRequest – a great iOS class for handling web requests which made it really easy for me to submit files back to the server if people have problems with them, I’m sure I’ll be using this one again. Also ZIStoreButton – a class that mimics the buy button on the app store, it shows up as a blue button with a price in and when tapped changes to green and says “Buy Now”. I’ve linked to my own github fork there as I did make a few changes to make it more compatible with pre-iOS4 devices and to (IMHO) better match the style of the app store button.

Apart from this I attended the 4th Maker Night of the year. That was good fun, we had a great turn-out and had people working on all sorts of different things. I found I wandered between projects catching up with people and helping out but did get time to help complete a few pieces for the Cupcake 3D printer we’ve been building. Also this week, on a similar vein, I’ve been writing up my experiences with my new “home management” system – YAHMS. There’ll be more to come from that when I get time.

My wireless temperature probes work by using an Xbee module to transmit readings from a TMP36 down to the Arduino base station. The XBees aren’t too cheap, coming in around £19 or $23 so I tried to be cheap and ordered mine from Sparkfun, I bought this XBee (series 1) with Chip Antenna and these breakout boards. I was intending to solder the module directly onto the board and didn’t take notice of what Sparkfun says about “please order the accompanying 2mm sockets (you’ll need 2!) and 0.1″ headers below.” (these ones). Once I saw what a stupid idea it was to solder the module (means you can’t switch them around between boards, handy for testing and reprogramming) I ended up buying these headers from Farnell.

As mentioned, to take the temperature reading I’m using a TMP36, these take a voltage of around 3-5.5V as input and will output a voltage between 0-2.7V to indicate the temperature, which can be from -40 – 125°C. In the end I needed three of these for this project, two for another project and managed to kill a friends’ (or it was already dead) so I just bought 10 to make sure I had enough.

XBee modules require regulated 3.3V input, as I’m using the very basic breakout board that just breaks the pins out without adding any functionality I used a MCP1700(-3302E/TO) which accepts up to 6V. In my first order I just bought two, one for each temperature probe. The first time I tried one I managed to connect it to the batteries the wrong way round (in fact the wires on the battery pack were colour coded wrong!) which unfortunately killed the regulator. With this experience and a few others I’ve realised that for something that only costs £0.36 it’s never worth buying “just enough”, always get plenty. The data sheet for that part suggests putting 1 µF capacitors on the input and output, I had issues finding any with that rating that were cheap enough so ended up going with 3.3 µF capacitors. I eventually ordered 10 regulators so ordered 20 capacitors, might as well make sure I had enough to match!

In my initial experiments, when I was still using solar cells, I realised that although the regulator would regulate a large voltage down to 3.3V, if the input voltage was too low the output voltage would actually come out less than 3.3V. This gave me some interesting values when the XBee would just about continue to work, but the reference voltage against which the TMP36’s output was being measured would also drop resulting in temperatures shooting up as the batteries ran out!

I found the solar cells weren’t charging enough to power the XBee throughout the day, let alone through the night, so I decided to switch to batteries. I bought some of these 6xAA battery packs, I later realised I only need 4 batteries to power my regulator (and in fact 6 would be too many, I’d been thinking of Arduinos which would need more like 9V input) but it’s simple enough to short a few connections so that the packs work with 4 batteries instead. One problem I did have with these was that I didn’t get any connectors so I was trying to solder the wires directly onto the pack, this really didn’t work well because the plastic started melting before the solder did, so next time I’ll be buying some PP3 battery connectors like these.

To get around the issue of the voltage dropping below 3.3V I decided to use a voltage detector to detect when the input voltage was getting low and turn the whole circuit off, I used these TC54VN detectors to check for an input of at least 4.2V, meaning that my circuit should now provide between 4.2V and ~6V (4×1.5V AA batteries) to the regulator, or nothing. The voltage detectors I got are “Open Drain” which essentially means that when the voltage is above 4.2V, “VOUT” is floating, i.e. does nothing. When the voltage goes below 4.2V VOUT is pulled to ground. I actually expected VOUT to output the input voltage when the input voltage was high enough but it turns out I would need a TC54VC if I wanted that. The Open Drain version worked well enough once we figured it out (I had a lot of help from Adrian McEwen on this one). I’m using a transistor to turn on the voltage regulator. I pass the battery input to VOUT via a resistor and then onto base on the transistor via another resistor, this way the transistor is turned on, until the voltage drops and VOUT is grounded, short-circuiting the voltage meaning it doesn’t get to the transistor.

The picture above shows the circuit diagram as I was working on it, ignore the Arduino-style pins at the top and bottom. The two circuit diagrams should be just about the same, the higher one was supposed to be clearer. Here’s a fritzing diagram which should hopefully be more useful:

You’ll notice that I’m also running a wire to pin 20 on the XBee, this is A0 and will take an analog read from between the two resistors, they’re acting as a voltage divider which should mean I’ll be able to monitor the battery level and get an idea when I’ll need to change the batteries in advance. It should be possible to calculate, using V=IR, what reading I’ll get as the batteries approach 4.2V.

This seems like a good time for a bill of parts, so here goes:

Part Quantity (for a single probe)
XBee (series 1) with Chip Antenna 1 Data-sheet
XBee breakout board 1
2mm 10 way headers 2 Sparkfun link
TMP36 (temperature sensor) 1 Data sheet
MCP1700-3302E/TO (voltage regulator) 1 Data sheet
3.3 µF capacitors 2 Data sheet
TC54VN voltage detector 1 Data Sheet
P2N2222AG – NPN Transistor 1 Data Sheet
6xAA battery pack 1
PP3 battery connector 1

I’ll also mention now that the Lady Ada article about TMP36 sensors was also really useful for teaching me how these work.

So that’s the power circuit and the temperature sensor all connected up to the XBee but you still need to program the XBee so that it’ll do something useful with that data. Again Lady Ada was really useful here, I followed the instructions on the Tweet-a-Watt which outline how to send current data via XBee. I used essentially the same programming to get my XBee to report the temperature and battery usage data from AD0 and AD4. I also bought the Rough Cut of Building Wireless Sensor Networks from O’Reilly. Unfortunately that only covers Series 2 XBees whereas I’m using Series 1, but the sections about how to wire an XBee up to USB for programming and what apps to use on the computer to speak serial to the XBee were really useful.

To connect the XBee to the computer I actually used an Arduino programmed with an empty sketch (basically void setup() {} void loop() {}). Pin 2 (DOUT) on the XBee was plugged into Pin 0 (RX) on the Arduino and pin 3 (DIN) on the XBee was plugged into Pin 1 (TX) on the Arduino. In this way I’m basically just piggybacking on the Arduino so that it provides a USB -> Serial interface. I then used the Serial Monitor in the Arduino IDE to send commands to the XBee and monitor the response. There’s more about this in the Tweet-A-Watt article. The program I sent to my probes was the following:


ATMY=1,SM=4,ST=3,SP=FF,D4=2,D0=2,IT=13,IR=1,ID=1234

Which broken down means:

  • AT – Attention!
  • MY=1, – the ID of this unit
  • SM=4, – Sleep Mode SM (4 = Cyclic sleep)
  • ST=3, – Sleep Time (3 milliseconds after wakeup to go back to sleep)
  • SP=FF, – Sleep Period (0xFF – 255 x 10 ms = ~2.5 seconds)
  • D4=2, – enabling pin AD4
  • D0=2, – enabling pin AD0
  • IT=13, – number of samples (0x13 – 19 samples per packet)
  • IR=1, – sample rate (1ms between samples)
  • ID=1234, – the PAN ID for the network of XBees

Once that has been sent to the XBee it will go into sleep mode. This will mean that the XBee cannot be re-programmed unless you reset it. I actually haven’t figured out how to do that yet and so can’t modify the program on my XBees! I advise that you make sure you’re ready before sending it. It will sleep for 2550ms and then wake up, perform 19 samples with 1ms delay between each, transmit them and then go back to sleep again.

Once you’ve sent that to one of your XBees you can then place the module into the temperature sensing circuit and it should start transmitting temperatures. On the base station you would connect an unprogrammed XBee to an Arduino as above and send the following to it, this just sets the PAN ID and the modules own ID (to zero):


ATMY=0,ID=1234

Now if you monitor the output on the serial connection you should see data being sent about every 2.5s. You can then use the XBee library for Arduino to parse this, or use the code for YAHMS as will be explained later in this series. If you are going to use the XBee library, be aware that the version available for download (0.2.1) actually contains a bug meaning it will give the same output for all pins, use the version in SVN instead which has been fixed.

Still to come, the relays and the base station, watch this space!

I’ve mentioned a few times on recent blog posts about the Arduino project I’ve been working on to provide an internet controlled system for my central heating. I’m glad to be able to say I’ve finally got this working, but not only that, I’ve also opted to release the source code so that others can make use of it. The source code comes in two parts, the code to run on your Arduino and the code to run on the server side. I’m releasing both parts as Open Source Software as I realise that I’m not necessarily going to get time to add features that others need and by opening up the source code I can allow more people to get involved. I’ll be running the server side on yahms.net and will be letting anyone sign up to use the site. Right now that’s not hugely useful as new programs can only be added by inserting directly in the database but hopefully I or someone else will get around to knocking up some more UI and an API. I haven’t really mentioned what the site or the system actually does so I’ll give some brief summaries about the system and the hardware and then will be posting full build logs for all pieces of the hardware shortly.

YAHMS

A little more about the system in general then. I decided before Christmas that I wanted to be able to control my central heating from the Internet. What made me finally decide to do it was when I realised that my existing controller is simply plugged into a 3-pin socket meaning that all I needed to do was set the existing controller to be on constantly and then find a hardware solution that would allow me to get in between the 3-pin plug and socket to control the system. One of the things that made me decide I should build the system was the spate of cold weather we had. It seemed to come on all of a sudden so there was a number of mornings we’d wake up at 7am and find that because the weather was so cold the heating hadn’t been on for long enough. To solve this I decided I needed some intelligent temperature control that would turn the heating on earlier when we’re having a cold morning. A similar but different problem is when we go on holiday for a few days and come back to a cold house, with the heating controlled from the internet we’ll be able to have the heating fully off while we’re away but make it turn on a few hours before we get home.

So, the smallest part of the system is the temperature “probes”. Partly just to experiment with Xbee but also to get an idea for how the house retained heat I decided I wanted to put a few wireless temperature probes around the house. In theory they should be very simple just using an Xbee with a TMP36 temperature sensor feeding into them, in actual fact I’ve ended up with about 10 separate parts, most of which are there to make sure I get a steady regulated power source to the Xbee. Originally I had been hoping to power them from a battery-backed solar panel but in the end I’ve had to go with batteries.

To control my hall light I’m using a solid-state relay wired into the power line for the light. I had hoped to be able to make this non-invasive by wiring into the light switch but had difficulty getting the wire through the wall to the switch so ended up having to cut the line and re-connect it in a junction box which I could then wire up to my relay. The light is turned on by short-circuiting the two wires on this line and can be done either by turning my relay on or by using the existing switch. This means you don’t have to use the internet to turn the light on, but once the internet has turned it on you can’t turn it off locally, but that’s not a big deal.

To intercept the power to the heating I used an old mechanical timer with all the mechanics removed. This left me with a nice simple device which I only needed to short circuit two connections to turn on the circuit. A simple relay did the job here too and I had enough space left to add in a power module so that I can provide 5v USB power from it too, but more about that in the dedicated post. I can leave the existing heating controls set to “Constant” so that as soon as I turn the socket on, the heating comes on too.

Finally, to control the two relays and receive signals from the Xbee modules I put together a simple Arduino shield. It has an Xbee module connected to receive the signals, some wired jacks to send signals to the relays and also a temperature sensor, just “because”. The Arduino also has an Ethernet shield so the software is able to download configuration from yahms.net and turn the relays on and off in response to that config, it can also submit sensor readings back over the connection too.

So that’s the general overview, I’ll go into more detail in the next few posts and talk more about the software as well as the hardware and what they’re both capable of.

Another week spent on the Chess Viewer app. One of the features that I’m adding is support for searching within the chess games, specifically of the filenames and the headers that are generally attached to each game. In the original app I was reading all of this information into memory and storing it in a big in-memory data structure of arrays and dictionaries. This made the searching fast but also made it complicated and could result in having very big data structures in memory. To fix this I decided to use Core Data.

Core Data is the “Model” part of the Model View Controller paradigm in iOS and is the recommended way to store your data. It’s generally backed by an sqlite database so can be quite quick but as it’s an Object Model can be simple to work with from code without having to use SQL. I ported the code over to use it pretty quickly and then sent a build over to the client to try out. Unfortunately the app crashed for them soon after launching, in fact they couldn’t get it to launch at all. It turned out that as the code was parsing the chess files and building up the Core Data data model it was using excessive amounts of memory. It could also be really slow, some files (specifically The Week in Chess) can hold around a thousand games, each of those games will then have an average of ten headers each meaning you end up with around ten thousand headers to store. This alone causes the parsing to be slow and use memory, but if you have multiple of these files (and TWIC is updated weekly) it could cause real problems. This was mostly a problem on the iPhone 3G but it would be silly to refuse to support it.

In the end I decided to use sqlite without going via Apple’s Core Data model. Perhaps there was a way to optimise the existing code but I couldn’t find it in the days I spent on it. Considering the search function was supposed to be a half day task I really needed to move on! Previously I’ve tried using the sqlite functions directly, these are a real pain to work with so it’s fortunate that I came across FMDB quite recently. This is a library that wraps the C functions of sqlite with a nice Objective-C interface making things much easier to work with. Again I managed to port the code over to use this without too much difficulty… I still had to make various optimisations to make sure that the big imports don’t take too long and that the searching would be as fast as possible. It’s “funny” when spending so much time on something like this, it’s only really going to affect people upgrading to the latest version that have big files, as this is when the database will be built initially. When downloading files in the future it shouldn’t be so bad considering there’ll only generally be one file being processed at a time.

In case anyone else has issues I thought I’d paste in some snippets (uninteresting bits of code replaced with comments) to show how I ended up doing things:


    // OPEN THE DATABASE

    // All the insert statements are the same so this makes sure
    // sqlite doesn't have to parse them repeatedly
    [db setShouldCacheStatements:YES];

    // This stops sqlite from writing to disk so much, increases the
    // chance of db corruption but if we die half-way through an
    // update we'll need to start from scratch anyway
    sqlite3_exec([db sqliteHandle],
        "PRAGMA synchronous=OFF", NULL, NULL, NULL);
    sqlite3_exec([db sqliteHandle],
        "PRAGMA journal_mode=MEMORY", NULL, NULL, NULL);
    sqlite3_exec([db sqliteHandle],
        "PRAGMA temp_store=MEMORY", NULL, NULL, NULL);
    // Use memory as much as possible, more on sqlite pragmas here

    // I actually tried to make sure I didn't use any autoreleased
    // objects but there was one I couldn't avoid and could be
    // some in code I didn't control anyway.
    // Having my own pool means I can release these periodically
    NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];

    // This monitors how many inserts we've done
    NSUInteger numUpdates = 0;

    // Tell the database not to commit the inserts straight away
    [db beginTransaction];

    // FOR EACH FILE

        // FOR EACH GAME IN THE FILE

            // INSERT ROW IN DATABASE FOR FILE

            // FOR EACH HEADER IN THE GAME

                // INSERT ROW IN DATABASE FOR HEADER

                ++numUpdates;
                if (memoryWarning || numUpdates > 5000) {
                    [db commit];
                    [db beginTransaction];
                    memoryWarning = NO;
                    numUpdates = 0;
                    [pool drain];
                    pool = [[NSAutoreleasePool alloc] init];
                }

So what’s happening there is every 5000 inserts for a file we’re committing the transaction to the database and clearing the auto release pool. This way we should hopefully not run out of memory but will run as fast as possible. You’ll notice there’s a memoryWarning flag too, this gets set separately by a notification, so if we get a memory warning before we’ve done 5000 transactions we’ll still commit the transaction and clear the pool to clear out as much as possible.

And, finally, more progress on my Arduino control system, I’ve got all the bits working now so that I can control my central heating and hall light from a website. The UI still needs a lot of work but it essentially matches the abilities of the original central heating controller and that’s the minimum I need in the short term. I’ll be putting some blog posts up with build logs soon.

Spent this week making improvements to the Chess Viewer that I released for Everyman Chess a few weeks ago. It was good to spend some time re-factoring an app that I actually started building about a year ago.

Monday was largely spent redoing the code for the “ChessBoardView” to use Core Animation Layers. I had to make some quite fundamental changes so that I could track the movement of the pieces but at the end of it I had a board that shows nice animations as pieces move around and are taken, and also added the ability to flip the board around to see if it from the point of view of the black player.

On Tuesday I added a feature to prompt the user of the app to review the app on the app store. Lots of apps have started doing this but I couldn’t find a simple small library to implement this, so I did it myself. The code for MKEReviewRequester is now available on Github, there’s some instructions in the README file but it’s fairly simple. Set some variables when the app launches, call a method every time a significant app action occurs (an ebook is read or a game is played) and then call another method at the point you want to display the alert to users. The code was all based around some code I found on this blog post, I just wrapped it together into a nice simple class.

Got lots more done on the app too but have run into a problem with preparing Ad-Hoc builds on my Xcode4 install so annoyingly I can’t show the updates to the client. Need to check whether I can distribute a different app and also try my old laptop, hopefully one of those will work.

Today (Friday) I’ve finally managed to get the memory usage on one of my Arduino projects down so that it fits within the 2K of RAM provided. I should hopefully be writing up some build notes on that with more details shortly.