• Week Notes Week 100(!)

    Jun 28, 2010

    Ah, week 100, I did notice this milestone coming up last week and thought that I should do something special for it. Unfortunately I then forgot. I’ve actually had a fairly quiet week of getting my head down and working so I don’t really have much to report. Maybe I’ll make 128 the milestone instead or 104 as that would be my 2 year anniversary. Or probably I just won’t worry about such things.

    So, this week I spent all 5 days working on the new client iPhone app. After my good progress last week I managed to get the essential functionality completed by Tuesday and so I compiled a list of things I’d missed on the first pass or bugs I’d found, and started working through it. I made good progress on some bits but had a few issues working with the client’s API for one call so ended up spending more time than I was expecting on that. It all boiled down to accessing an API call on a different server which had a different OAuth implementation. It looks like they have a slight bug in their implementation that meant I had to be very careful about the order of my URL parameters. I managed to get that sorted though and sent a version of the app over for them to try out on Friday night.

    I’ve got plenty of things to work on today but hopefully they won’t be as tricky as that last task. Today is in theory the last scheduled day I’ve got on this app but it looks like I’ll definitely be spending a little more time tidying up the loose ends. I finally got sign off for the other client app I was working on weeks ago so I’ll be going straight onto that one as soon as I can.

  • Unpaid Ruby on Rails Internship in Liverpool

    Jun 25, 2010

    Quite often when people ask me about new features on mapme.at they get what is becoming a stock answer of “it’s on the list”. In most cases this really is true, the problem is that the list is very very long and I’m getting very little time to work on it. I’ve decided that I need to sort this out and the way I’m going to do this is by recruiting some help. In a previous job in Manchester we worked with the University of Manchester to recruit their second year students on summer internships, and in later years even full sandwich year placements. This was a great experience for both the students and for us who had to train them and work with them. I’m looking to this as a possible solution to my problem on mapme.at.

    Firstly - the unpaid bit - at the moment mapme.at is not generating any revenue. I’m fine with that and am fully intending to take my own sweet time to push it in that direction. That does mean that any money spent on it comes out of my own pocket. I think that working on mapme.at could be a great opportunity for someone beginning a career in software development. They would be working on a popular site getting plenty of traffic and API usage. They would also get the opportunity to work with me, a software developer and entrepreneur who has worked on leading websites in a career spanning over ten years.

    What would you be working on? Well mapme.at is a popular and successful location based social networking site. The site started in June 2007 and since then has added many users who make use of the various functionality offered. The site interacts with many other services including Twitter, Foursquare and Google Latitude to give a single central store for users’ location history. It offers users the ability to restrict what level of detail they show to different contacts and also gives interesting and useful visualisations of their data. The API allows anyone to build an app that interacts with the data, querying or even updating many aspects of a user’s data, with their permission of course.

    So what am I looking for? When I’ve done this before I’ve worked with 2nd year Computer Science students giving them an opportunity to work during the summer holiday. This is the type of person that I’m expecting to hear from. It’s quite possible that people in different stages of their career will be interested, perhaps a school leaver that’s not looking to go to University or someone who’s already been working in the field and is looking for a change. Recent graduates would also be likely candidates. I really need someone who has experience working with software development and can demonstrate an aptitude for it. They don’t necessarily have to have used Ruby on Rails before but some experience of web based development and use of relational databases would be expected. I will expect to provide some basic training and support, though I’d also hope that they will be the type of person to pick up new technologies quickly.

    As I’ve mentioned the ideal would be for someone to come in and work for me unpaid over the summer for a few months. They will gain some great experience in working on a web startup and I’ll get some development done on mapme.at. I definitely wouldn’t want to leave anyone out of pocket though. I’d certainly be willing to pay someone’s expenses if they’re going to have to travel to get to Liverpool each day, though I wouldn’t go so far as to pay for someone’s accommodation so they’re definitely going to need to live somewhere nearby. I should also mention now that I will be expecting this person to work with me on-site in an office in central Liverpool so I’m not looking to work with anyone remotely.

    So, are you interested or do you know anyone who might be? If so get in touch by emailing me at my first name “at” my second name dot net, or just by leaving a comment on this posting.

  • Week Notes Week 99

    Jun 21, 2010

    Spent most of the week on a new client iPhone app. Quite an interest project for a fairly big client, hopefully it should result in a large number of downloads when I get it finished. The first thing I needed to do was to interface with their OAuth API. This was as fiddly as ever, not helped by the fact that their API returned XML during the OAuth handshaking when every (well, both) OAuth API I’ve worked with in the past has returned URL encoded key/value pairs. Got past that though and managed to get the OAuth stuff working well. I then integrated the code that I worked on recently for showing data in a hierarchical format. That worked pretty well, I’m really stretching the limits of what that code can do now but considering that code was based on the iPhone’s iPod application and the app I’m building is related to music playing it should work out quite well.

    In essence the app needs to do the following:

    1. Authenticate with the client’s API

    2. Download user information

    3. Present that information to the user

    4. Allow the user to select items to download from the server

    5. Download the selected items

    6. Allow the user to interact with the downloaded items

    I’m now 4 days into the project (took Friday off) and I believe I’ve completed the first 4 of those tasks with just the last two to go. I’m hoping the downloading will be simple enough as I’ve covered something very similar in the chess app just a few weeks ago, and then hopefully the interacting won’t take too long either. I’m really trying to get the bulk of the work on this one finished early so there’s more time to spend testing and fixing bugs later. As you’ve probably noticed I’ve found recently that the iPhone apps have been taking longer than I’ve been predicting!

    Friday was spent working on my existing client’s slightly buggy iPhone app. It was being reported that the current version of the app with my changes was running slower than the last version before I got involved. Obviously this reflects badly on me so I was eager to find out what I was doing wrong. The problem basically came down to a point where the original developer had allowed an in-memory dataset to become invalid. It didn’t matter in that version of the app though because no-one would be able to see that it was invalid, whereas my version allowed some editing. This meant that when I added a new item to the dataset that overlapped on an existing one I was downloading all the data from the server again. This meant two network connections per-action and so was making the action doubly slow, especially noticeable over a 3G or GPRS connection. Fortunately I was able to handle the overlapping on the device so that only in very specific cases do I need to request all the data again (basically when one item gets broken into two items, they will both get new identifiers so I need to call back to the server to find out what the new identifiers are).

    This week I’m back on the new client app all week, really hoping to get it finished, possibly even a day or two early to allow maximum time for testing.

  • Week Notes Week 98

    Jun 14, 2010

    Quite a varied week this week, but also a short week due to attending a wedding on Thursday so having Thursday and Friday off. The project that was due by the 10th did get put back to “sometime in July” which is actually a good thing really as it clears space for the project I need to work on for the next few weeks. The other project that I was waiting on sign-off for still didn’t get signed off so I actually almost ended up with some free time.

    Monday I worked on my Chess app. I completed the “version 1” that I’d been working on which involves managing and viewing chess games and downloading from the Internet. As it happened my timing was quite good (if 1 month late) as I was able to get the app demoed to the clients on Wednesday. I’m still waiting to hear the feedback from that meeting but a quick summary was that “the main features [are] there, but that at the same time plenty of UI adjustments worth looking at”. Which is probably good news.

    Tuesday I caught up with some email sending in the morning but decided to take another look at my app store positions app I worked on a few weeks ago. I’ve now been downloading top 100s for the Apple app store for the past month which means I am storing a lot of data but wasn’t really doing anything with it. My friend Colm recently launched a World Cup iPhone app - Total Football 2010 - and it’s been interesting watching his success in the various Sport Top 100s around the world. I decided I really needed to do something with my data so knocked up a script to send daily email summaries of app positions.

    First changes were to add some database tables for storing user info. I went with a basic “users” table for storing an email address and various other information that might be useful if I make this a web-facing app. Also a “user_apps” table for linking Apple application IDs to my user IDs. Both tables have a boolean column so that I can turn off the emails per-app or per-user. I then put together my script. The main thing I wanted to do was to get the current position of the app in the top 100 and a graph showing the recent performance. I decided that a sparkline was the best way to do this. I also wanted to use google charts API to save me needing to install any software to generate the graphs. Fortunately I found this page on Google Charts Sparklines and was able to tweak the settings to get exactly what I wanted. After a few runs I decided to add a few more data-points showing max and min positions, and also showing the dates that the data was taken from. Each email is supposed to show the data from the last week but if a user has no data for a time then it might show less.

    I’m quite happy with the results, and you can see an example in the screenshot above (big dip caused by my scripts failing to run for a few days :-( ), but as was mentioned by Dave Verwer there’s really quite a lot of information. When I was working on my own data I wasn’t surprised to get lots of results considering I have 28 apps live, but even when running Colm’s data with his single app he still gets loads of results because he’s being successful in lots of countries. One feature suggested by Dave was adding a summary at the top giving stats such as “iPhoneApp1 has increased by 20 places. iPhoneApp2 has dropped by 80 places” which does seem like a good idea but will require some work. Next time I spend a random half day on this I may well add that!

    Finally, on Tuesday night I got some feedback about mapme.at which led to a long email conversation because it turns out the CBC (Canadian Broadcasting Corporation) wanted to use the site to track a road trip being taken by one of their presenters! They really like the fact that it can show you a whole journey and overlay the tweets and check-ins that you make along the way. Unfortunately because they wanted to show the social media on the map they couldn’t just use the existing export functionality because that will only show a trace. To help them out I made a special version of the existing “history” page which I termed the “headless” version. This version takes off the top and left bars of the page leaving only the map, the timeline and the lifestream of social media on the right. I also modified the page so that instead of just saying “I am here” it now shows your username and profile pic instead. They seemed really happy with the changes and I’m hoping this will lead to some increased traffic and more users for the site. If you’re interested you can take a look at the page here. They’re using Google Latitude to trace the journey and it looks like they didn’t turn it on for the first stage but I’m hoping that from here on out they’ll have it running continuously, still looks pretty good though.

    This week I’m beginning a new 2 week project working on a music app for a pretty big client. Not going to go into too much detail straight away but hopefully I’ll get to talk about it more when I get a little way through it.

  • Week Notes Week 97

    Jun 5, 2010

    A short week but by no means a quiet week. All iPhone again too.

    I started off by working just over a day on a new client app, supposed to be 5 days with nothing too tricky (now delayed waiting for my client’s client’s sign off). A few interesting situations came up with the novel UI they were looking for but nothing I couldn’t handle, including:

    • Mock-ups showed a screen into which you enter a ten digit number. Mock-up showed three boxes to split into three, three and four digits. I “interpreted” this to mean that so long as the digits were split up to make them easily readable that would be enough. I added a handler for the textField:shouldChangeCharactersInRange:replacementString: delegate method so that any time characters were added or removed I would allow the change but then reformat the text.

    • In the mock-ups it also showed a number pad on top of the toolbar. As far as I can tell it’s not possible to move the standard Apple number pad, not without doing hacky things like inspecting the hierarchy of views and pulling out the number pad and moving it to another place. I also wasn’t sure that Apple would be too happy about this change in their sacrosanct UI principles. I did consider allowing the number pad to just pop up and cover up the toolbar but on this view it’s actually important for the user to have access to the toolbar buttons to get help. As the whole point of this screen is to enter a number it didn’t really seem appropriate to have the user close the number pad to get the toolbar and then have to open the number pad again once they’d viewed the help. I ended up handling it by moving the toolbar up at the same time that the keyboard appears. This way I can use the standard keyboard and still have the toolbar visible. I’m not 100% sure it works but it makes sure the info button is available and still allows the standard number pad to be used.

    You should be able to get an idea for what I’ve done from the screenshot, I’ve blanked out some sections as I’m under NDA but the bits I’ve discussed are still clear.

    I had intended to spend the whole week on this app but the sign-off issue I mentioned above meant I needed to pause. It actually worked out quite well though as it gave me more time to spend on the hierarchy app I’ve been working on. I added some features that allow you to search for items and have different “categories”. The categories allow you to follow different paths for filtering information. It is quite similar to the iPod app on the iPhone which gives you, amongst others, the following paths:

    • Artists: Artists > Albums > Songs

    • Albums: Albums > Songs

    • Songs: Songs

    • Genres: Genres > Artists > Albums > Songs

    I’m quite happy with how I’ve built the app, it’s really configurable and allows new categories to be created with ease by just specifying a name, icon and path.

    An interesting issue I came across with this app was that I wanted to demo it on the iPad using the standard iPhone app scaling to make it bigger and easier to see from a distance. As it stood though the app was a “Universal” iPad & iPhone app, but because I hadn’t written any iPad code if I tried to run it I’d just get a big blank screen. I fiddled about though and found out how to downgrade it, having not found any information on the web about doing this I promptly blogged about it.

    Friday ended up as a bit of a manic day, I was travelling down to Oxford to have a meeting with a client on Saturday morning. I was also attending the Cathedral Valley lunch. Before I could do either though I ended up having to meet with client to try to fix their iPhone ad-hoc provisioning problems. Unfortunately as it turned out we ended up having just ten minutes before I needed to leave for my lunch and the most we established was that it wasn’t working!

    Heading back from Oxford now after a really successful client meeting. I met up with the teacher that I’ve been working with on the Revise GCSE & A-Level apps and also someone else that we’re looking to work with. We discussed the revision apps and came up with lots of new features that could go into them. As part of the full meeting we also ended up discussing not one but two potential new projects. Definitely a successful day and lining up lots of work to do for the summer.

    Next week? I don’t like to predict as it never works out. I have one client app that’s supposed to be live by the 10th but which I’m waiting for feedback on so not really expecting much to happen with that, hopefully the project from this week will get sign off and I’ll be able to finish that. Really need to get some things finished off as the following week I should be starting on a really interesting music app.

  • "Downgrading" a Universal iPad/iPhone app to iPhone only

    Jun 3, 2010

    For the past month or so I’ve been creating all my new iPhone projects using the “Universal” setting in Xcode. When doing this you are then given two essentially separate codebases to complete, one for the iPhone version of your app and the other for the iPad version. Obviously you can share code between the two but the root of each app (the “App Delegate”) is separate. I’ve been creating apps this way knowing that at some point in the future I’m going to want to upgrade them and thinking that I might as well start from the Universal app now so that everything’s ready.

    Unfortunately this does mean that if I try to run these apps on an iPad, to demo to people for instance, I end up with a big white screen as I haven’t actually implemented any of the iPad specific code. This would also be an issue if I wanted to release one of these apps to the app store without writing the iPad code.

    I had a quick Google around for a solution but didn’t come up with anything so I decided to have a poke about to see if I could figure it out myself. Turns out it’s quite easy.

    The first thing I looked at was the Build Settings for the app, looking through I found one labelled “Targeted Device Family” which was set to “iPhone/iPad”. That seemed an obvious candidate so I changed it to iPhone and tried the app on my iPad. This loaded the app in a half-size iPhone style window but when the app loaded up I was still left with a white window and no content. Seems it was running my (non existent) iPad code in an iPhone style window. I didn’t see anything else useful in the Build Settings so I had a look at my Info.plist file. There’s a number of settings in there that seem to reference the iPad so I edited it in a text editor and prefixed all those settings with “backup-“ so that they would no longer be noticed. Recompiled the app and loaded on the iPad and it worked fine.

    As a bonus hint, I made my icon 72x72px even though the iPhone icons are 57px. This then worked fine and looked good on both the iPad and the iPhone. Fairly obvious really but worth knowing.

    So, to recap, to convert a Universal app so that it appears like an iPhone app when run on an iPad:

    • In build settings modify the “Targeted Device Family” setting it to “iPhone”

    • Open your Info.plist file, ideally in a text editor, and rename the properties ending in ~ipad so that Xcode will ignore them. You could of course delete these but I wanted to keep them handy for when I do convert this to a proper iPad app in the future

    • Remember to make your icon 72x72px so that it looks good on the iPad too

    Disclaimer: I’m only doing this for demoing at the moment, and haven’t had any issues. I suggest you test it out for a while before submitting an app with these settings to the app store. I take no responsibility for anything that goes wrong if you try these instructions!

    UPDATE - I noticed that if I don’t change the Targeted Device Family and leave it at “iPhone/iPad” you actually get iPhone style but full size and full res. Generally this can look quite odd but it’s useful to know anyway.

  • Week Notes Week 96

    Jun 1, 2010

    So, quite a varied week with a surprising ending.

    On Monday I tried to get the version 1 of my chess app finished. I actually started a new project from scratch for this. Everything so far has been about building modules for parsing files or for visualising different aspects of the chess board and game. This new project was intended to cover other aspects of the game including downloading files from the internet and managing them on the device. I spent the day on it and got a good way through it, completing the download aspect and beginning the file management, but I didn’t manage to get close to a “finished” version so I’ll need to continue work on that when I get time.

    Most of the week I devoted to working on a new client app. They need the app to allow you to navigate through a hierarchy of information resulting in some detailed information and the ability to view web pages. I think this is actually quite a popular format of app and is probably the type of thing most people do as one of their first apps. For some reason it’s taken me 18 months to get around to it. It came along quite nicely and I’m quite happy with how I’ve done it, does seem the type of thing that might come in handy in the future too. Part of the app was an embedded web browser, loading standard web pages in a view with basic back/forward/stop/refresh controls. I had a look around the web and was surprised to find that no-one has open sourced such a thing. It didn’t really take me very long to do it myself but as it’s something in use in so many apps I would have expected someone to do it. I might have to do it myself, if I get around to it.

    Thursday morning I had an interesting time at an event organised by North West Vision+Media on Developments & Opportunities in the Education Market. With my ever expanding range of iPhone education apps (currently 26 apps live) I thought this would be an interesting event to go to and try to find some new opportunities in the market. The event was really interesting with different people having different perspectives and agendas. I actually found it quite inspiring and wrote lots of notes on ways to improve the experience of my iPhone apps and ways to expand the scope and sales potential. Much of that was not actually related to the speakers, just that the opportunity to sit and think about education helped to focus some thoughts I’ve been having recently. I was a little dismayed by a few choice quotes from Ray Barker of the British Educational Suppliers Association regarding how the future of education is walled gardens and that the culture of free is nearly over. Walled gardens have never really been a good thing, look at AOL and Compuserve whose walls fell when the World Wide Web came along and offered open access to everyone. Apart from this though I had a good experience at the event and even got the opportunity to speak in response to a question of how mobile phones are being used in education. Also after the event I chatted to someone who’s been working with a school that has given iPod Touches to every student and doesn’t know what to do with them, I’m hoping there’s some real opportunities to be had there.

    Finally Friday. I had fully intended to spend Friday at home working on my client app, that was until Dave Verwer of Shiny Development called me up at lunchtime on Thursday and asked if I wanted to be on BBC Radio Manchester the following day talking about the iPad. It seemed quite an unusual opportunity but I jumped at the chance to hawk my own apps and services over the airwaves. Unfortunately they wanted me to be available in person to chat on the breakfast show so it meant being up early enough for the 7am train. In the end I had quite an interesting chat with the host Allan Beswick. He was calling the people queueing up outside Apple stores (and by extension, me) barmy, but did seem a little excited about the device. I got a few comments in about the evils of anti-competitive practices and mentioned my own development services but didn’t get the chance to mention my own apps! Shortly after that I got another call from Dave Verwer asking if I would like to go on Radio 5Live, so from regional to national radio! That one didn’t go so well, unfortunately I think my phone had poor signal so they couldn’t get hold of me when they originally tried, so when I did get on the radio I managed a few sentences to answer a question before they carried on talking to Rory Cellan-Jones in the studio. A short while after that one I then got another call from someone at BBC West Midlands also looking for someone to come on and chat about the iPad. This one, mid-afternoon on Friday, went the best of the three I think. I answered a few of the presenter’s questions, managed to talk about my own apps quite a lot and also discussed the money-making opportunities with the app store.

    If you’re interested in listening to these follow the links below to the BBC iPlayer, the links will probably only work for the next few days:

  • Week Notes Week 95

    May 22, 2010

    Again this week I’d been intending to work on a new client iPhone project but unfortunately it hasn’t been signed off by my client’s client, but that did leave me free to put more time into the chess app, or that was the idea (again, see last week). This week I managed two pretty solid days on the chess app and made some real progress. I’ve now got an area at the bottom of the screen showing the moves for the game and a set of buttons on the side for moving forwards and backwards in the game and for taking any alternative branches. I’ve created an internal wrapper for the game which models the game and you can send messages to it telling it to go forwards or backwards in the game, when it does this it also alerts any other objects observing it that the state has changed meaning I can automatically update the moves view and chess board view when the game is progressed.

    Something I’ve spent a fair amount of time on is making sure that all the rotation code works. It can be quite fiddly making sure that when you rotate the phone all the UI components end up in the right place. The SDK does provide you with various hooks that are supposed to make things easier but in the end I’ve largely had to manually set positions of elements to make sure they go into the right places.

    There’s still plenty to do on the app. To complete the elements I’ve done so far I need to add the ability to read comments that might be interspersed in the game, and to indicate better when these comments or alternative branches are available. When that’s done I really need to start working on something for managing the games that you have stored on your phone and allow downloading more, neither particularly simple jobs. I then also need to offer more functionality for reading through the game so there’s still a good way to go.

    On Monday, after having my old server’s disk fill up for the umpteenth time, I decided I might as well get the server move done and dusted. I’d previously copied most of the files from my old server to my new one so all that was required was configure all the services - the web server, database server, mail server, etc. - perform another sync, copy the databases across and then switch the DNS. It actually went fairly smoothly in the end and I think I managed to keep the downtime to a minimum. The final rsync only took about 5 minutes but then the database dump and import took about 10-15 minutes each. Wasn’t helped that every time I imported the database my connection to the server seemed to hang so the import would finish (the magic of GNU Screen) but I wouldn’t actually see until I tried to do something in the window and SSH finally told me that the server had disconnected.

    I don’t think I had any other real problems though and I definitely managed to get everything transferred before my app store positions script started running, so definitely within about 4 hours everything was ready. I’ve actually switched to running directly on the bare metal so I’ve gone from a virtual machine on an older server to running directly on a quite modern server. I’m glad to report that the app store positions downloads are now taking 50 minutes so basically half the time they took on the old server. Queries of the database also seem to be much faster which is really good, maybe that’ll help me to get around to writing some externally accessible query mechanisms, like a website.

    The rest of the time was spent on my revision apps. On Tuesday I spent some time preparing a press release which you can find here. It’s the first press release I’ve put out about the apps and is largely intended to get a bit more attention to them just as the exam period is coming around. The apps are doing quite well and seeing increases as we get to the exam period but I’d still like to see some bigger increases to be completely happy. On Friday I prepared a few more apps for submission, it’s probably getting a bit late now but I had the material ready so seemed a shame not to use it. One was for “Human Biology” for AS-level students - another Biology app but this time focused on the human side of things. The others were for the American market. I’m actually quite excited about this, it’s a huge market and has taken relatively little effort to prepare new apps tailored to US students. The questions aren’t too different so it just required some tweaking of the grading algorithms and then repackaging. Whenever Apple get around to reviewing them we should see Grade 10 and Grade 12 Biology, Chemistry and Physics appearing in the store. Again it’s probably a bit late in the school year but it will be good to see some healthy sales in another market if this works out.

    So that’s about it for the past week. Not sure what I’ll spend most time on in the next week, I do need to get version 1 of the chess app “finished” but I now have 3 other iPhone apps basically confirmed and needing completing by the end of June. Think I’ll be keeping busy for a good while yet!

  • Weeknotes Week 94

    May 17, 2010

    Well, this past week was supposed to be devoted to a new client iPhone app. “Unfortunately” I didn’t get sign-off for it but that did mean I could keep going with my chess app. With various other bits and pieces to cover I’ve managed to spend about a day, maybe two on that in the last week. Last Monday I also ended up finishing off the wordpress blog project for Clear Digital and then taking a trip over to Manchester. I had an interesting meeting with a potential client who want to do a really big iPad project. They initially wanted me for 12 weeks full time which actually spooked me a bit as I don’t usually do full time. I’ve yet to hear what’s happening with that but could be an interesting project to work on if they do want me to go ahead with it. After that meeting I met up with Dave Verwer and went to NSManchester in the evening. While at NSManchester I gave a very hastily put together presentation on the iPhone app store positions app that I worked on the previous week. Talk seemed to go well and I had time to chat with some interesting people in the pub afterwards too.

    I got home from Manchester at about 11:30 and began my planned server migration. My current server is hosted by Hetzner and is a “DS3000”, AMD Athlon 64 3700+ with 2GB memory and 2x160GB drive (probably, I bought mine over a year ago so specs may have changed). They now have an “EQ4” which offers Intel i7-920 quadcore with 8GB memory and 2x750GB drive for exactly the same price, though with a setup fee. I’m currently hosting mapme.at on one of these and it’s running really well so I decided to upgrade my other server too. On the older server I’ve been using VMware to host most of my stuff in a virtual machine. The idea for this was that when it did come to moving servers in the future (i.e. now) I could do it by simply copying the VM across and starting it up. By about 3:30am on Tuesday morning after wrestling with VMware and networking for many hours I was getting pretty tired. I got a few hours sleep (as much as my cat would let me before she decided she needed feeding) and then tried again in the morning. After another few hours I decided that VMs were not the way to go :-/ Considering I host everything on ubuntu and that’s super easy to set up anyway the ease of setup isn’t really that big a deal, also having to copy massive virtual disk images wasn’t proving to be fun anyway. I’m going to host my services on the bare metal which means I can switch from one server to the other by doing a simple rsync to get up to date, re-syncing databases and then switching DNS. Unfortunately I haven’t yet had time to do this, ideally I’d do it overnight like I attempted last week but considering the app store positions app keeps filling up the disk and knocking out services I’m worrying less about the downtime, it should be minimal anyway now it’ll just take some resyncing.

    I ended the week by attending the Liver & Mash event organised by Mandy Phillips. The event was in the spirit of previous “mashed libraries” events which have tried to introduce librarians to the concept of mashups and the many ways in which they can be useful. I wasn’t really sure what to expect from the event but had agreed to talk on “Maps” so prepared some slides and went along. In the end the event was really useful, it wasn’t really too dissimilar from other web/technology events I’ve been to. Everybody was really interested in mashups using various web APIs, hardware hacking with Arduino and other techologies, and pretty much anything that interests geeks. The libraries side of it gave it some focus but was easy to get to grips with for someone like me who has had minimal experience of libraries recently.

    Most of the talks were given in three tracks and the rooms were assigned by order of popularity. As it happened my talk on maps was voted most popular and I was asked to give it in the main room to everybody! The talk seemed to be well received, looking at the twitter back channel, with most people finding the various examples I gave interesting. I only had 15 minutes so gave some basic background on maps in general and where my experience of online mapping started. I gave some examples of using mapping APIs and OpenStreetMap then finished off with a quick overview of mapme.at and my experiences of tracking my location. As usual slides can be found on slideshare (usual problems with videos, though these can be found on my user page on vimeo).

    In the afternoon I also ran an hour and a half workshop on mapping. I’m not sure how well this went as we were in a fairly small room and I hadn’t particularly prepared any tasks for attendees to try out. I tried to go through some of the best ways to use mapping APIs (use mapstraction!) How to get involved with OpenStreetMap, how to edit the map using GPS, Aerial imagery or even the new Ordnance Survey data. I also covered the various ways to use OSM data including loading it into mapstraction or using the Cloudmade APIs to generate custom map styles and retrieve data through the web services. I got plenty of questions from my audience though and gave answers for all of them so hopefully they enjoyed it.

    All-in-all I think the day went really well. Unfortunately when my talk finished at 3:30 I had to rush out straight away and didn’t get to enjoy the evening revelry. Instead I hopped into the car to rush to London for a leaving party!

  • Week Notes Week 93

    May 8, 2010

    So, a much quicker update after last week’s delayed entry. This past week I’ve been working on a project for Clear Digital. A relatively simple project that required setting up a Wordpress blog and re-skinning it to match the client’s requirements. I hadn’t played with Wordpress so much in quite a while so it was an interesting experience. Turned out not to be too difficult, making use of plenty of existing plugins to extend functionality. On a recommendation from Dave Coveney I used the Thematic theme. This is more of a tool than an actual theme itself. The theme you get is very simple but it allows lots of hooks to extend the theme and customise it how you like. I think there’s lots of themes that are based off this but I chose to create a new theme building on top of the very basics that Thematic provides, the better to match the client’s requirements. Wordpress provides “widgets” which are small UI elements that you can drop onto the page in various places. Things like tag clouds, a calendar of your blog posts, a list of Categories, and lots of others. Thematic provides quite a few different places that you can drop Widgets making it even easier to customise your blog.

    In case it helps someone else, here’s a complete list of the plugins I used:

    As well as this project, I also started out on a new personal project last Sunday. I intended it to be just a quick thing to try something out but it’s started taking a lot more time and resources than I expected. As you may know I have quite a few iPhone apps in the app store. Right now I’ve got 22 live on my own account and another that I did for a client under their account. Though Apple provide perfectly good sales statistics they don’t give any indication of how well you’re doing in their “Top 100” charts. Though much of the desire to know your position is due to vanity there are some uses to knowing, you can use it for marketing and if you reach the top 20 it’s a good reassurance that you’re going to make a reasonable sum of money from sales.

    Apple don’t provide this information but a number of other people do. APPlyzer offer access to some of the data for free and require you to pay for more. An iPhone app called “PositionApp” also gives you some information and allows you to select favourite apps but still didn’t give me the information in a way that I liked, so I decided to write my own.

    I had already found a perl script that would download the information for the Top 100s and would give me information for a specific app, category and country if I wanted. I was originally running this twice a day but unfortunately I hadn’t updated it to list some of my latest apps so when I found that two of my apps were in the Top 100 in the UK Education category I decided I needed a better option. If I was going to download the Top 100s I really ought to be putting them in a database them so that I could do more with them in the future.

    I started by writing a script that would do the basic download of the XML and for some reason decided to throw the XML in the database for later parsing. Actually a large part of my reasoning behind this was having minimal time but wanting to leave something downloading data as I went off to OGGCamp. As it turned out storing the XML in the database was an incredibly bad idea, after a short while I had thousands of entries with 600KB of data in each meaning that an SQL query to request the latest download to check if it had changed took 15 seconds!

    So, version two, parse the data straight away. The data was in XML so obviously the safest way to parse it was to use a proper XML parser. Because the file was pretty big I decided to go with a SAX style parser. After spending a while doing this and getting a completed parser going, I found that my XML parsing was taking over a minute! I’d already noticed that sometimes the HTTP request from Apple could take up 15 seconds and doing that 5000 times (for all the categories and countries) was going to take a long time, so an additional minute was terrible news!

    Next day I decided to skip the “proper XML parsing” and go with a regex. After half an hour of coding I had something that would parse the entire 600KB file in less than a second, much better.

    I’ve now been running this script four times a day for nearly a week. I’ve downloaded approximately 20 batches of data in that time. Each batch is pretty big as I’m querying 40 categories in 62 countries for two types of app (free and paid), which comes to 4960 requests four times a day! Each of those requests then generates 100 positions entries meaning I now have over 10 million position entries in my database. This quantity of data has been causing its own problems but so far I’m keeping on top of them. Yesterday I added a few more indexes to the tables and converted the tables from InnoDB to MyISAM. This gave much better results. The 6pm batch yesterday took 5 hours to run whereas the midnight batch took 1 hour 45 minutes and the 6am batch took just an hour to run. I’m also coincidentally hoping to move to a more powerful server this weekend so that should help too.

    So, future plans for this data? Well basically I’m not sure how much effort I’m willing to put into it. The main thing that I want to get out of it is positions for all of my apps on some sort of regular basis, and the ability to query history for apps even if I haven’t specifically remembered to add them to my list. Other people might have other ideas of things to do though so I’m intending to dump the data out into some basic form, CSV most likely, and make it available to download. Hopefully I’ll get around to putting a web interface on this to allow people to look for information on their own apps or even register to get emailed position updates but any of that will be time permitting, and I’ve got lots of work to keep me busy!

    If you’re interested to know though, Basic Sat Nav is continuing to do well in the UK Navigation category, hovering around the number 10 mark and hovering around the 60 mark in Ireland. My GCSE and A-Level revision apps are doing nicely in the run up to the exams, none of them getting particularly high in the Top 100 but most of them making appearances in various positions. Even iFreeThePostcode is sitting at number 60 in the UK Free Navigation category.

    I’ll be talking about this project a little at NSManchester on Monday night so go along to that if you’re interested to know more.

subscribe via RSS or via JSON Feed