Web Stuff


I recently added a wiki on my MapMe.At site and found it quite tricky to get working and difficult to find just the right information I needed so I thought I’d write it up.

MapMe.At is still on Rails 2 which seemed to mean I couldn’t install Gollum as part of the site.

I created a separate Rails 3 project that runs alongside MapMe.At and simply hosts Gollum, using instructions from here: http://stackoverflow.com/questions/13053704/how-to-properly-mount-githubs-gollum-wiki-inside-a-rails-app

I wanted it to use the user information in MapMe.At’s session hash so switched MapMe.At to use activerecord based sessions and used information on here to make the rails 2 session load in rails 3: http://www.kadrmasconcepts.com/blog/2012/07/19/sharing-rails-sessions-with-php-coldfusion-and-more/

I’m not actually using the rails2 session as the main session, I just load the information in. I have the following in config/initializers/session_store.rb

module ActionController
  module Flash
    class FlashHash < Hash
      def method_missing(m, *a, &b)
      end
    end
  end
end

MapmeAtWiki::Application.config.session_store :cookie_store, key: '_mapme_at_wiki_session'

Then in routes.rb:

class App < Precious::App

  before { authenticate }

  helpers do
    def authenticate
      oldsess = ActiveRecord::SessionStore::Session.find_by_session_id(request.cookies["_myapp_session"])
      if oldsess and oldsess.data and oldsess.data[:user_id]
        u = User.find(oldsess.data[:user_id])
        email = "#{u.username}@gitusers.mckerrell.net"
        session["gollum.author"] = { :name => u.full_name, :email => email }
      else
        response["Location"] = "http://mapme.at/me/login?postlogin=/wiki/"
        throw(:halt, [302, "Found"])
      end
    end
  end

end

MapmeAtWiki::Application.routes.draw do
  # The priority is based upon order of creation:
  # first created -> highest priority.

  App.set(:gollum_path, Rails.root.join('db/wiki.git').to_s)
  App.set(:default_markup, :markdown) # set your favorite markup language
  App.set(:wiki_options, {:universal_toc => false})
  mount App, at: 'wiki'

end

I wanted a wiki on the site to allow my users to help out with documenting the site. Adding their own thoughts and experiences and perhaps fixing typos I might make. I’m not sure that’s really started happening yet but at least I have a nice interface for writing the documentation myself!

So, what have I been doing since November 27th?

1. Secret Location Startup

2. New version of Basic Sat Nav/Basic GPS launched 19th December with audio support and Spanish translations!

3. “HTML5″ and Android version of Examstutor.com apps almost finished!

4. Bubblino & Friends iPhone app went from idea to submission in less than 24 hours, and is live on the App Store!

5. CamViewer for Foscam Webcams goes live after a short dev time and gets hundreds of downloads and rave reviews on the App Store!

6. After over a year “in development” Chess Viewer has gone live offering an advanced viewer for chess e-books popular amongst chess fanatics. Further development is planned including in-app purchase of e-books from Everyman Chess.

And all of this has taken me to the point where I’ve had 50 apps live on the app store! Not all of these are live right now as some of the Revise apps were replaced by Examstutor.com apps but I currently have 18 Examstutor.com, 15 Reviseapps.com, Basic Sat Nav, Bubblino & Friends, CamViewer, Chess Viewer, Credit Cards, iFreeThePostcode = 39 apps live right now and another one should be coming this week. Even discounting the carbon copies that’s still 9 separate code-bases, and again doesn’t include the app I completed for 7digital which Apple are unfortunately barring from the store. I’ve been busy and hopefully will continue to be with both client work and my own projects as this year progresses.

If you want me to help with a mobile web, or native mobile app, then get in touch by emailing me at john “At” mckerrell “dot” net

I’ve been spending a few days this week trying to get my Examstutor.com apps working on Android. I had already decided to do this using HTML and spent the past two weeks creating a “HTML5″ version of the app. I was surprised but pleased about how quickly I managed to complete this, partly due to it always being easier doing something the second time. This HTML5 version uses localStorage a lot to store the test modules and various other bits of data. It’s probably not something that I would actually release as currently it stores too much data in localStorage, but I built it in such a way that I can build plug-ins that work on different devices meaning that I only need to recreate the device specific code for storing and retrieving data and the rest of the code that handles the interface should be the same.

Once I got the HTML5 version done I had to begin on the Android app. The app is very basic with a WebView (a WebKit control) taking up the whole screen. I then load the HTML5 app inside there and leave it do the rest. To provide the device specific code I’ve created a Java class with methods that match the plugin interface. I then add an instance of that class to the WebView using addJavascriptInterface. In theory I thought that would be all I’d need to do to get it working, due to a few idiosyncrasies of addJavascriptInterface that wasn’t the case, as I’ll explain in the following few paragraphs.

The first issue I found was that the object that is exposed in the JavaScript appears not to act like a normal JavaScript object. The way I had my code arranged there was a global singleton ExamsTutor object and then the plugin would be another singleton called ExamsTutorDevicePlugin. To save having to decide which object’s methods to call, ExamsTutor would copy all the functions from the plugin into itself, as follows:

    for( var key in device ) {
        obj[key] = device[key];
    }

In that example I’ve already assigned ExamsTutorDevicePlugin to device and obj is going to be the ExamsTutor singleton.

After that code I would expect a call like ExamsTutor.someDeviceSpecificMethod to work. Unfortunately I found it wasn’t and when I added some logging statements it turned out that the code was never going into the loop. In the end I decided to add a JavaScript singleton that would wrap the Android object, a little annoying that I need to do this but as you’ll see it ended up being useful later, here’s a snippet from that class:

var ExamsTutorDevicePlugin = (function() {
    var a = ExamsTutorAndroidPlugin;
    return {
        log: function(message) { a.log(message); },
        requiresInitialPathway: function() { return a.requiresInitialPathway(); },
        showPathwayDialog: function() { a.showPathwayDialog() },
        currentPage: function(page) { return a.currentPage(page) },
        ...

The next issue was the types that could be sent to and returned from the android interface. In my plugin interface I had already arranged to send various JavaScript Objects and Arrays in and out of the functions. After some testing I found that addJavascriptInterface only allows basic data types such as boolean, int and String. Fortunately this is simple enough to fix. I’m already using jQuery inside my web app and have the JSON plugin so I can use $.parseJSON and $.toJSON to make sure that I only pass strings to and from the Java. I was worried that this would result in me having to do lots of packing and unpacking on both sides of the interface but actually on the Java side I will generally just be storing the JSON to files so it shouldn’t be much of an issue. Another snippet from my JavaScript singleton with this in place would be:

        defaultPathway: function() { return parseJSON(String(a.defaultPathway())) },
        setDefaultPathway: function(pathway) { a.setDefaultPathway($.toJSON(pathway)) },

You might notice in that snippet that I’ve used parseJSON rather than $.parseJSON. This was due to the final issue I’m going to describe here. For some reason the string objects returned by the Android interface don’t seem to react to the typeof operator in the way I might expect, and in the way jQuery’s parseJSON method was expecting. The first thing that $.parseJSON does before parsing is some sanity checking to make sure it has a string:

		if ( typeof data !== "string" || !data ) {
			return null;
		}

For some reason, calling typeof on the strings returned by Android was giving "object" and so this check was failing and jQuery was giving up on the parse. Fortunately this was simple enough to handle. I added a local parseJSON method to coerce the json to a string and also to handle the exception that might be fired by jQuery:

    function parseJSON(json) {
        var data = null;
        try {
            data = $.parseJSON(json);
        } catch(e) {
            data = null;
        }
        return data;
    }

With that I managed to get the app to the point where I can select a Revision Pathway and have it download the information for that pathway and download the test modules from the Internet. There’s still plenty more to be done but as you’ll see from the screenshots it’s not looking bad already.

Last week I mentioned that we’d got the examstutor.com apps finished and submitted. This week they went live. Apple approved them all on Tuesday evening at which point they moved into the “Processing for App Store” state. Unfortunately it then took about 24 hours for all the apps to get through this stage as 6 of them ended up sitting thereĀ for ages. We finally got everything sorted out and put the apps live on Wednesday night. The apps have started selling straight away which is a great relief though not a huge surprise as the old apps were still selling ok. This is the first time in a while that I’ve had a free app on the store that I really care about the sales of so it’s interesting to see how the “Login” versions are doing. Currently they’re “outselling” the paid-for apps by way over 10 times but this isn’t a huge surprise. It’s good to see that people are interested though and hopefully some of those will convert to sales. Examstutor.com is now featuring the apps on the homepage and has a good set of pages describing the apps so it’s no surprise to learn that existing subscribers are already downloading the free apps and logging into them. If you want to see more information then take a look at the iPhone apps section on Examstutor.com

Getting these 18 apps live actually took me to 47 separate apps live on the app store, which I like to think – even if some of the apps are just different content – is a pretty good number. Because the Examstutor.com apps are intended to replace the old “Revise” apps I have now taken the A-level versions of those down but I’m still at 36 live apps. 17 of my own (3 not available in the UK), 18 examstutor.com apps and one Credit Cards app I did for Moneyextra.com. Many of these may be repeated but if we ignore the repetitions that’s 5 different apps I’ve got live on the store now and 3 others that are completed but waiting to go live for various reasons. Quite happy with my portfolio and hope I get to continue building interesting apps in the new year.

A quick update on my clock. I ended up deciding to bolt the motor onto the mechanism rather than using glue again. After various problems including broken drill bits I managed to get it secured and turning nicely. Unfortunately when I tried to get it working with the software it stopped working again. I’m now not sure if there’s a problem with my circuit board or somehow there’s a new problem in the software. I’ve had to leave it alone for a while though as I have other things to work on but hopefully I’ll get it going eventually.

I’ve also been working on a brand new start-up idea that someone came to me with a few months ago. I’ll be doing the bulk of the technical work including a Ruby on Rails back-end and supporting mobile apps. I can’t really go into too much detail about it right now but it’s hopefully going to be an interesting return to the location space for me. Watch this space for more info as things progress on that.

So these weeknotes are rapidly, or rather slowly, becoming “monthnotes”. The past few weeks have seen a few interesting things happen though so I thought I’d get another post up. Last week saw the completion of version 1.0 of the Examstutor iPhone apps. We managed to get the content for all of the 9 subjects we were hoping to support finished and so submitted 18 apps! We’re now just waiting to see what Apple will say, hopefully they’ll go straight through without any rejections but you can never be too sure. We’re submitting two apps per subject. One is quite simply a paid-for app, you pay your money and you get access to hundreds of questions on your subject. The other will be a free app in the store but is intended to be used by schools and colleges (and, in fact, individuals) who have signed up with examstutor.com for use of their online and mobile service. On downloading the free app they will be prompted for a username and password, when they enter this they will get full access to all the content “free”. We’ve included a few tests (generally around 20 questions) for people to use on the free app though so that if they download it by accident they still get to try it out and can then decide to upgrade if they want.

Last week I also managed to get the final few issues sorted out with my chess app, it’s looking better and hopefully I’ll be able to submit to the store pretty soon.

I also got most of the work done adding features to the Ruby on Rails site I mentioned in my last blog post. I needed to generate PDF invoices for a selection of addresses and services in the database and managed to get the work done pretty quickly. This was the whole point of converting from the awful PHP to a new codebase but I was still surprised by how quickly I got it done. I mainly need to make some changes to the PDF document to better match what the client wants and then let them try the system out and see what they think. I’ve used Prawn for the PDF generation and have been quite happy with it finding it pretty easy to use.

Final thing to mention is my location clock that has been quite popular on this blog previously. I originally built it at Howduino Liverpool in May 2009 – around 18 months ago. It has sat around with barely any updates since then and so was still made up of a big messy bundle of wires.

With the second Liverpool Howduino event coming up I decided I really had to get around to soldering this up, and I wanted to get it done before the event so that I could play with something else on the day (Xbee modules, lasers and a second four handed clock powered by servos were all on my list). Leaving it to the last minute I finally got started on building a circuit board on Thursday and did manage to get it finished off on Friday. Unfortunately I had a few issues with it, various connections connected that shouldn’t have been, and others that should have been were not. I then found that for some reason the software that had been working fine before had stopped working too. Fortunately I finally managed to get it all working on Saturday so now have a fully working circuit board and software.

So why am I not happy now? When I came into the office this morning I picked up the clock and started looking to see if I could get a chime to fit in the casing and where would be best to fit the arduino and cut holes for the ethernet, USB and power ports. I brushed past the motor and it wobbled, I thought “hmm, that seemed loose…. OMG, THAT’S LOOSE!!” After something like a year of being firmly stuck on by Araldite the stepper motor has now detached itself from the mechanism so my clock is still not working. I’m going to go out at lunchtime and see what glue I can find that might be able to stick a metal motor onto a brass mechanism, or possibly try to think of an alternative!

Quite a varied week this week, though as the title suggests, it did involve some nightmares, and as the map should suggest it involved a bit of travel.

Supposedly my main focus was to get the web based admin interfaces for the revision apps sorted out. As I mentioned last week I found I was using the wrong version of a plugin so I tried using the right one and it worked better. As it happened I still wasn’t entirely happy with the interface offered so I’ve kept using my own form fields for some parts and used the defaults elsewhere. By the end of Monday I’d got enough of the admin interfaces done to declare them “finished” and then moved onto the next task. That task is to allow uploading of three large blocks of text and have the code parse the text to work out what the questions should be. This is to allow the teacher I’m working with to use his existing word documents to upload questions without having to make any changes. I got that nearly finished this week but got a bit sidetracked on a few other issues. I’m hoping to have that finished today and get a few other bits done so that I can have the whole first milestone finished this weekend.

On Tuesday I took a trip down to London. The main reason I was going was to meet the people at 7digital. The iPhone app I was working on earlier this year was for them so we decided that I should go to visit them and finally meet them in person. The meeting was fairly short but went well and we discussed a few of the features they’d like to get into the next version. The first version of the app is actually still in the review process for the app store (as far as I know) but hopefully it’ll get through soon.

While in London I also met up with the guy that I’ve been working on a chess app with. We discussed some of the features in the current version of the app and highlighted a few hopefully fairly simple steps that we can take to get the app to version 1.0 and ideally get me paid!

While on the trip to London I also had a play with Twitter’s OAuth support. Twitter have decided that Basic Auth is too insecure and leads to leakage of password and so everyone needs to use OAuth in the future. This is definitely a good idea but meant that I needed to change the way I was accessing Twitter with mapme.at. I found a fork of the main twitter4r library with OAuth support and managed to get it working quite easily in a small standalone script. When it came to incorporating this into mapme.at though I came across some issues. The first, very basic, script I modified worked fine but the next one gave me errors. After trying to find the problem over the course of a few days (while trying to do other work) I finally found the issue. The fireeagle library that I was using was “monkey patching” the OAuth library. Basically all it was doing was wrapping one method call so that it could change a few properties on the object that was returned. Unfortunately the fireeagle changes were based on an old version of the OAuth library and so ended up breaking the main OAuth library. Once I’d fixed that I had a few more issues as I remembered I had in the past made a few changes of my own to the twitter library, both by modifying the library in place and by monkey patching it in my own code. I ended up creating my own branch on github that has fixes in for broken friendship methods, an extra friendship method and a few other small fixes.

Room 29

So I managed to miss last week’s week notes out. I’m not really sure why that was, I think because over the weekend I sometimes feel I don’t want to do “work”, i.e. the weeknotes, but then during weekdays I want to do proper work, not weeknotes. Oh well, here I am now so I might as well get on with it.

Week 107 was actually pretty good, fairly relaxed but quite productive too. I finally started work on the new version of my GCSE & A-Level revision iPhone apps. I spent the whole of the Monday brainstorming the feature-set that I’m aiming for and then building this into a list of Milestones and breaking down the first few milestones into more detail. Once I’d done that I entered all the information into the Trac setup that I’ve previously mentioned. I quite like how Trac is all set up for handling milestones and versions of apps as it has worked out really well for managing this project.

Although these are iPhone apps the first thing I needed to do was to create a web app for managing the questions. So far I’ve just been taking the questions from the teacher as a collection of word documents and then in a part-manual, part-automated process turned these into XML files for the iPhone app to read. I wanted to make a web-based system that the teacher could use himself to upload questions so that he can sort everything to do with the questions out, taking up less of my time. Along with properly planning this project out and setting milestones I also decided I really needed to try building this project using “Test Driven Development”, i.e. writing a set of automated tests for functionality and then building out the functionality until the tests pass. This is actually the first time I’ve tried doing this for a whole project and it’s been an interesting ride so far. I’ve found that it’s really slowed me down, partly just from having to learn about the various Ruby modules and Rails plugins and getting myself up to speed with them, but I do think I’m getting better quality code and structure as a result so I’m going to continue doing it.

The first features I completed were the ability to import the existing XML files into the database and then export “test modules” from the database into the same XML format. This is about making sure we can manage questions from the database in the future and that, if necessary, I can continue to use the existing iPhone code to launch apps. I then started work on the admin interfaces. I’ve done so many admin interfaces over the years, especially back when I was doing PHP. I was hoping the state of things had changed so that it would be nice and easy to set something up to do all the work for me, especially considering rails has scaffolding which I haven’t used before but have read about and is supposed to do this sort of thing. I happened to find out about the Ruby Toolbox that week and that had an Admin Interfaces section which suggested active_scaffold was the most popular plugin to use.

I started playing with active_scaffold this week with mixed results. It was really simple to get set up but then some of the features didn’t seem to work. I managed to work out eventually that I needed another plugin to support it – recordselect. I got that installed but then it didn’t seem to work right, the ajaxy bits just didn’t do what they were supposed to and I was getting nowhere. I ended up dropping the ajaxy bits but did post a message to the active_scaffold mailing list. I’ve now had a response telling me that I need to use a different fork of the recordselect plugin, the main version only supports Rails 3 whereas I’m still using Rails 2. Hopefully once I do that things will go better.

I got sidetracked on Thursday this week by having a look at a few bugs in the chess app I’ve been working on. It seems I’d missed out some functionality of the PGN format and it was tripping up when trying to read some more complicated files. I spent all day Thursday and some of Friday on this but finally got it sorted on Saturday morning. I’ve now sent a new version over to the client and I’ll find out what they think when I meet them on Tuesday.

Other interesting things that happened in the past two weeks… in week 107 I had feedback for mapme.at requesting support for xAuth in the API and XML output. On Saturday morning I had some time so I had a look into it. The XML output turned out to be trivial. The JSON output is quite simple, I construct native datastructures like arrays and hashes and then call .to_json on the resulting object. Turns out if I call .to_xml on the same object I get perfectly reasonable XML out. The xAuth support was a little more tricky. xAuth is a mechanism whereby mobile and desktop apps can take a username and password from a user and then make a call to a server to exchange this for a token that can be used to make requests. This is better because the app then doesn’t need to store the username and password and they’re also not being passed over the network repeatedly. It also means that you don’t have to go through the convoluted OAuth process of app -> web browser -> app. I managed to get both these features out on Saturday morning and the person who had requested them was quite surprised and happy. Hopefully he’ll be using them in a Windows Mobile 6.5 twitter app soon. One other thing I tried to do was to convert mapme.at from using Basic Auth with Twitter to using OAuth instead, Twitter are turning off Basic Auth support at the end of the month and so I really need to switch soon. Unfortunately it looked like the twitter4r library that I use doesn’t yet support OAuth so I decided to postpone the change, I’m really going to have to do something this week, either find an OAuth compatible version of twitter4r or use an alternative library.

When looking back at my previous post I noticed that a good chunk of it was about how the mapme.at server went down and took ages to come back up. Yesterday I upgraded some packages on the server again and rebooted it, happy in the knowledge that the disk check completed last time and so that should’ve found whatever the problems were. Unfortunately at 12:30 this morning (just before going to bed) I got a tweet, an SMS and an email from pingdom telling me that mapme.at was down. I looked into it and saw that yet again it had mounted the disk as read only. I began a disk check and went back to bed. It’s still running now but hopefully will be done in the next two hours. I’m going to have to get in touch with Hetzner and demand that they give me new drives I think as I can’t have this going down for so long so often.

Again to finish on some good news, I have an office! You can see it in the photo above, and Adrian McEwen practicing his DJing (or perhaps pretending to sit at a desk). A bunch of us freelancers in Liverpool have been umming and ahhing over getting one for months now. Hakim happened to be walking past an office building on Duke Street with a sign mentioning office space to let. We had a look at a selection of the offices they had available and settled on one with enough room for four of us with plenty of natural light (though hopefully not too much so we can still see our screens). We received the keys on Thursday and most of us moved in properly on Friday. It’ll be interesting to see how this goes for me, I’ve predominantly worked from home for the past 5 years so I’m not sure how I’ll find working in an office again. Right now it’s looking like being a lot of fun and good to see people every day but considering none of us really got much work done on Friday we’re going to have to be more careful in the future. I’m sure as things calm down and we get used to it we’ll be really productive, and hopefully able to make use of each other’s slightly differing talents.

I’ll be dropping some stuff off in the office today and then heading to How? Why? DIY! to see what it’s all about. If you’re in Liverpool you should come along!

Well, as you may have noticed I missed last week’s notes. I’ve had a pretty harried few weeks trying to close out the two iPhone projects I’ve been working on. Mainly this was simply because I’d finished the time that I’d assigned to the projects and so any further work was not making me any money, but also July is the end of my financial year and I really wanted to get this work invoiced for so that I could include it in this year’s earnings (the second year for my company). Not only was I trying to get these two client projects finished but I also wanted to make some progress on the chess app that has remained untouched for a few months now.

It’s difficult to say for sure but it looks like the two client projects are just about finished. One has zero live bugs and the other just has a few with questions attached which I’m hoping should result in no more work. One thing I’m finding though is that I’m definitely getting my time estimates for projects wrong. These two client projects I originally estimated as 5 and 10 days respectively. The 10 day project was eventually increased to 13 days (I submitted a “complete” app only to be sent wireframes for how the app should work, not sure if there was a communication issue here) but all in all I’ve probably spent at least 50% more time again on each one. I suppose what I’ve been doing is thinking of the time it’ll take for me to get an app together that fits the spec without considering time for the client testing and for the client to make the various modifications they’re likely to request. I’m definitely going to be more careful about any estimates I submit in the future.

I did spend some time getting a “full game view” working on the chess app. This view is intended to show the complete description of a chess game including all commentary and variations. As the game is progressed using the controls the current move should be highlighted and if any move is clicked then the visible game should be altered to show that move. I actually managed to get this pretty much done and working in half a day last week with another half day spent on some other tweaks. I sent it off to the guy I’m working with on this and he tried it out. There did seem to be a couple of issues with handling of incompatible file types (it was crashing rather than giving an error message) and a few issues with handling very large files but in general it seems to be working. I now need to experiment a bit with the placement of the full game view and then I think we’ll just be working out what other tweaks we need to do to it before we can release it.

Last week I also spent an interesting two days with Dave Verwer of Shiny Development. He’s asked me to help him out with some iPhone training so we spent the two days going through his materials and making sure I understood everything and was clear on how he presented it. Dave has been developing in Objective C for many years, initially for the Mac but with the iPhone SDK since it was released, so it was really good just spending the time with him and learning even more about the iPhone SDK from him. I’m basically going to act as the overflow taking on any courses when Dave is already booked up, the first course is booked in for October so it’s going to be really interesting to see how that goes. If you’re interested in getting trained up in Objective C and the iPhone SDK then you should definitely give Dave a call. Whether you want in-house training for a company or you’re an individual who wants to join one of the public training sessions you’ll have a great experience and be building iPhone apps in no time.

Final thing to mention is something I finally got set up today. I’ve been using SVN for a long time for various personal projects including mapme.at but I’ve also more recently been using Git. Git is great as a distributed VCS giving me the ability to check-in code on the train and branch projects with ease, but the lack of an enforced server-side component has led to me being a bit sloppy about how often my code gets backed up to a server. Today I finally worked out how to properly host git modules on a server. I’m actually doing it over SSH so it’s fairly simple, I just need to create a “bare” repository on the server:

mkdir foo.git
cd foo.git
git --bare init

Then the following commands link my local repository to the remote one and pushes all my changes up:

git remote add origin ssh://git.example.com/path_to_repository/foo.git
git push origin master

I think when I’d tried this previously I was using an old version of Git that didn’t properly support --bare so I’m glad that everything worked easily this time. I then took this one step further by setting up trac as a bug tracking system. It wasn’t too hard to set up on ubuntu using these instructions they’re a bit out of date though so once I’d setup mod_python I then found I needed to do that bit again and use WSGI instead using these instructions. I’ve also installed the Git plugin so that I can browse my Git repository from within trac and also most importantly I can resolve and update bugs from my commit messages using the post-receive script on that page (which must be named post-receive and put in the repository.git/hooks/ folder).

So a busy few weeks. As I’m coming to the end of projects I’m getting to the post-project comedown that I tend to have after being so busy and then becoming quiet again. I’ll still need to finish off any final bugs on these two client projects and get this chess app more complete. I also have emails from clients with various bits and pieces that need doing but I should finally be able to get started on some new projects, I just have to work out which ones!

Quite often when people ask me about new features on mapme.at they get what is becoming a stock answer of “it’s on the list”. In most cases this really is true, the problem is that the list is very very long and I’m getting very little time to work on it. I’ve decided that I need to sort this out and the way I’m going to do this is by recruiting some help. In a previous job in Manchester we worked with the University of Manchester to recruit their second year students on summer internships, and in later years even full sandwich year placements. This was a great experience for both the students and for us who had to train them and work with them. I’m looking to this as a possible solution to my problem on mapme.at.

Firstly – the unpaid bit – at the moment mapme.at is not generating any revenue. I’m fine with that and am fully intending to take my own sweet time to push it in that direction. That does mean that any money spent on it comes out of my own pocket. I think that working on mapme.at could be a great opportunity for someone beginning a career in software development. They would be working on a popular site getting plenty of traffic and API usage. They would also get the opportunity to work with me, a software developer and entrepreneur who has worked on leading websites in a career spanning over ten years.

What would you be working on? Well mapme.at is a popular and successful location based social networking site. The site started in June 2007 and since then has added many users who make use of the various functionality offered. The site interacts with many other services including Twitter, Foursquare and Google Latitude to give a single central store for users’ location history. It offers users the ability to restrict what level of detail they show to different contacts and also gives interesting and useful visualisations of their data. The API allows anyone to build an app that interacts with the data, querying or even updating many aspects of a user’s data, with their permission of course.

So what am I looking for? When I’ve done this before I’ve worked with 2nd year Computer Science students giving them an opportunity to work during the summer holiday. This is the type of person that I’m expecting to hear from. It’s quite possible that people in different stages of their career will be interested, perhaps a school leaver that’s not looking to go to University or someone who’s already been working in the field and is looking for a change. Recent graduates would also be likely candidates. I really need someone who has experience working with software development and can demonstrate an aptitude for it. They don’t necessarily have to have used Ruby on Rails before but some experience of web based development and use of relational databases would be expected. I will expect to provide some basic training and support, though I’d also hope that they will be the type of person to pick up new technologies quickly.

As I’ve mentioned the ideal would be for someone to come in and work for me unpaid over the summer for a few months. They will gain some great experience in working on a web startup and I’ll get some development done on mapme.at. I definitely wouldn’t want to leave anyone out of pocket though. I’d certainly be willing to pay someone’s expenses if they’re going to have to travel to get to Liverpool each day, though I wouldn’t go so far as to pay for someone’s accommodation so they’re definitely going to need to live somewhere nearby. I should also mention now that I will be expecting this person to work with me on-site in an office in central Liverpool so I’m not looking to work with anyone remotely.

So, are you interested or do you know anyone who might be? If so get in touch by emailing me at my first name “at” my second name dot net, or just by leaving a comment on this posting.

So, a much quicker update after last week’s delayed entry. This past week I’ve been working on a project for Clear Digital. A relatively simple project that required setting up a WordPress blog and re-skinning it to match the client’s requirements. I hadn’t played with WordPress so much in quite a while so it was an interesting experience. Turned out not to be too difficult, making use of plenty of existing plugins to extend functionality. On a recommendation from Dave Coveney I used the Thematic theme. This is more of a tool than an actual theme itself. The theme you get is very simple but it allows lots of hooks to extend the theme and customise it how you like. I think there’s lots of themes that are based off this but I chose to create a new theme building on top of the very basics that Thematic provides, the better to match the client’s requirements. WordPress provides “widgets” which are small UI elements that you can drop onto the page in various places. Things like tag clouds, a calendar of your blog posts, a list of Categories, and lots of others. Thematic provides quite a few different places that you can drop Widgets making it even easier to customise your blog.

In case it helps someone else, here’s a complete list of the plugins I used:

As well as this project, I also started out on a new personal project last Sunday. I intended it to be just a quick thing to try something out but it’s started taking a lot more time and resources than I expected. As you may know I have quite a few iPhone apps in the app store. Right now I’ve got 22 live on my own account and another that I did for a client under their account. Though Apple provide perfectly good sales statistics they don’t give any indication of how well you’re doing in their “Top 100″ charts. Though much of the desire to know your position is due to vanity there are some uses to knowing, you can use it for marketing and if you reach the top 20 it’s a good reassurance that you’re going to make a reasonable sum of money from sales.

Apple don’t provide this information but a number of other people do. APPlyzer offer access to some of the data for free and require you to pay for more. An iPhone app called “PositionApp” also gives you some information and allows you to select favourite apps but still didn’t give me the information in a way that I liked, so I decided to write my own.

I had already found a perl script that would download the information for the Top 100s and would give me information for a specific app, category and country if I wanted. I was originally running this twice a day but unfortunately I hadn’t updated it to list some of my latest apps so when I found that two of my apps were in the Top 100 in the UK Education category I decided I needed a better option. If I was going to download the Top 100s I really ought to be putting them in a database them so that I could do more with them in the future.

I started by writing a script that would do the basic download of the XML and for some reason decided to throw the XML in the database for later parsing. Actually a large part of my reasoning behind this was having minimal time but wanting to leave something downloading data as I went off to OGGCamp. As it turned out storing the XML in the database was an incredibly bad idea, after a short while I had thousands of entries with 600KB of data in each meaning that an SQL query to request the latest download to check if it had changed took 15 seconds!

So, version two, parse the data straight away. The data was in XML so obviously the safest way to parse it was to use a proper XML parser. Because the file was pretty big I decided to go with a SAX style parser. After spending a while doing this and getting a completed parser going, I found that my XML parsing was taking over a minute! I’d already noticed that sometimes the HTTP request from Apple could take up 15 seconds and doing that 5000 times (for all the categories and countries) was going to take a long time, so an additional minute was terrible news!

Next day I decided to skip the “proper XML parsing” and go with a regex. After half an hour of coding I had something that would parse the entire 600KB file in less than a second, much better.

I’ve now been running this script four times a day for nearly a week. I’ve downloaded approximately 20 batches of data in that time. Each batch is pretty big as I’m querying 40 categories in 62 countries for two types of app (free and paid), which comes to 4960 requests four times a day! Each of those requests then generates 100 positions entries meaning I now have over 10 million position entries in my database. This quantity of data has been causing its own problems but so far I’m keeping on top of them. Yesterday I added a few more indexes to the tables and converted the tables from InnoDB to MyISAM. This gave much better results. The 6pm batch yesterday took 5 hours to run whereas the midnight batch took 1 hour 45 minutes and the 6am batch took just an hour to run. I’m also coincidentally hoping to move to a more powerful server this weekend so that should help too.

So, future plans for this data? Well basically I’m not sure how much effort I’m willing to put into it. The main thing that I want to get out of it is positions for all of my apps on some sort of regular basis, and the ability to query history for apps even if I haven’t specifically remembered to add them to my list. Other people might have other ideas of things to do though so I’m intending to dump the data out into some basic form, CSV most likely, and make it available to download. Hopefully I’ll get around to putting a web interface on this to allow people to look for information on their own apps or even register to get emailed position updates but any of that will be time permitting, and I’ve got lots of work to keep me busy!

If you’re interested to know though, Basic Sat Nav is continuing to do well in the UK Navigation category, hovering around the number 10 mark and hovering around the 60 mark in Ireland. My GCSE and A-Level revision apps are doing nicely in the run up to the exams, none of them getting particularly high in the Top 100 but most of them making appearances in various positions. Even iFreeThePostcode is sitting at number 60 in the UK Free Navigation category.

I’ll be talking about this project a little at NSManchester on Monday night so go along to that if you’re interested to know more.

Next Page »