Category Archives: Development

Plotting Weekly Mobile Retention from the Localytics API using R and ggplot2

Part of building mobile web apps is understanding the myriad of mobile analytics and in part visualizing the data to shed light on trends that my otherwise be difficult to see in tabular data or even a colorful cohort table. I’ve been building a dashboard using R, RStudio, Shiny, and Shiny Dashboards aggregating data from MSSQL, Postgres, Google Analytics, and Localytics.
Continue reading

Could Facebook have scaled using MSSQL instead of MySQL?

The issue of scaling has been discussed recently at work and particularly in light of some of the recent details that have come out from Facebook on their data centers.

The question I want to know is do you think Facebook have scaled having been built on MSSQL as compared to MySQL? Did access to the source play an important role? Should it still be called MySQL or has FB changed it sufficiently that it’s really no longer MySQL?


Leveraging the Facebook API on Google AppEngine with jQuery Mobile

In the past I’ve built starter kits for Facebook development in ASP.NET and building CruiseControl plugins. My latest interest has been experimenting with Google AppEngine as my day job is all python appserver stuff so it’s a pretty logical fit. I’m slowly putting together all of the pieces I’d like to have in website starter kit including support for jQuery Mobile and Facebook Graph API.

I have a simple proof-of-concept app working here. Btw, as this is a work-in-progress it YMMV and the app may or may not be in a working state so apologies in advance.

Technologies used:

Google AppEngine

Cheetah Templates

Facebook Graph API

jQuery Mobile

Pretty printing a Python dictionary to HTML

Here’s a routine I wrote to pretty print a Python dict into an HTML table and though I’d share.

    def prettyTable(dictionary, cssClass=''):
        ''' pretty prints a dictionary into an HTML table(s) '''
        if isinstance(dictionary, str):
            return '<td>' + dictionary + '</td>'
        s = ['<table ']
        if cssClass != '':
            s.append('class="%s"' % (cssClass))
        for key, value in dictionary.iteritems():
            s.append('<tr>\n  <td valign="top"><strong>%s</strong></td>\n' % str(key))
            if isinstance(value, dict):
                if key == 'picture' or key == 'icon':
                    s.append('  <td valign="top"><img src="%s"></td>\n' % Page.prettyTable(value, cssClass))
                    s.append('  <td valign="top">%s</td>\n' % Page.prettyTable(value, cssClass))
            elif isinstance(value, list):
                for i in value:
                    s.append('<tr><td valign="top">%s</td></tr>\n' % Page.prettyTable(i, cssClass))
                if key == 'picture' or key == 'icon':
                    s.append('  <td valign="top"><img src="%s"></td>\n' % value)
                    s.append('  <td valign="top">%s</td>\n' % value)
s.append('</table>') return '\n'.join(s)

Setting up debugging for Google AppEngine projects in Eclipse

When I made the move from Windows to OSX and Python development one of the things I wanted to experiment with has been Google’s AppEngine. I installed the SDK and setup the plugin for Eclipse but ran into a few issues I wanted to make note of since I think other could probably benefit from it as well. I’ll mention I’m on OSX 10.6.8 using Eclipse for Java Version Helios SR 2.

Creating a new AppEngine project in Eclipse

With the plugin installed you get an AppEngine project template listed under PyDev.

Screen shot 2011-08-30 at 8.48.55 PM

Clicking Next displays the standard Eclipse dialog for a new project where you enter the name and optionally a few other project settings. However, it’s the dialog after that where I had my first question where you’re prompted for the Google App Engine Directory which in my case corresponds to:


Google instructions for development in Eclipse didn’t answer this question and it wasn’t immediately obvious to me. At any rate, once this path is set correctly you’ll immediately see a list of AppEngine modules to be included on the PYTHONPATH:

Screen shot 2011-08-30 at 9.14.38 PM

Configuring the Project’s Python Interpreter

After I successfully created the AppEngine project I wanted to look at the appengine-boilerplate for HTML5 which is based on HTML5 boilerplate which is well worth checking out. Next, I used the Google AppEngine Console to create a new application called “helloworld954” then copied the boilerplate files into my project and edited the app.yaml file to set the application name:

application: helloworld954

I then ran the project and everything looked ok but when I hit the page from the browser (http://localhost:8080) I got:

ImportError: No module named cgi

Which I subsequently discovered more information about here.

Now, IIRC my MacBook Pro came with Python 2.6 as the default version though v2.5 was installed as well and given AppEngine is, as of this writing, based on v2.5 I figured rather than tweaking as mentioned on that prior link I figured I’d first try the proper Python version. I right clicked my project from the package explorer and selected Properties then clicked the PyDev – Interpreter/Grammar option then clicked the “Click here to configure an interpreter not listed” link (blue link below).

Screen shot 2011-08-30 at 9.36.44 PM

Next, on the preferences dialog (which I won’t screenshot here as it’s too big) I clicked New… and added a new interpreter option for “Python v2.5” with a path of:


Btw, I think I tried the symlink but that wasn’t allowed (again, I think).

At any rate, once I saved those changes voila the boilerplate project started working. Ah, one step I overlooked was setting up a debug configuration for this project:

Screen shot 2011-08-30 at 9.49.13 PM

The main module for the application has to be Google’s file which is located here on my machine:


Also, under the (x)= Agruments tab I set the Program Arguments to “.” for the current working directory and for “Working Directory” I selected “Other” with a value of “$(workspace_loc:starterkit}” Note, starterkit is the name of my project.

The final result:

Screen shot 2011-08-30 at 9.53.42 PM

Hopefully, this helps someone else, it will certainly help me months from now when I trying to do the same thing again so I won’t have try and recall all these steps.

ASP.NET Facebook Starter Kit updated to VS.NET 2010

Recently, I received an email via my blog asking for a VS.NET 2010 version of my Facebook Starter Kit. Since I no longer work on Windows it hasn’t been a high priority so I offered instructions on how to extract the project from the .vsi file and load the project manually. Much to my (pleasant) surprise I got a follow-up email with an updated starter kit for VS.NET 2010 updated to .NET v4.

If anyone decides they want to upgrade the FDT to the latest version I’d be happy to post an update for that as well. The starter kit is available on Facebook is nearing 5600 users! An interesting side note, I’m no longer able to edit the properties of this application on Facebook because it contains the word “facebook” in the URL which is no longer allowed.


Setting up a new SVN repository

Since I rarely tend to setup new $g(SVN source repositories) I find myself googling for the syntax of the various commands so I’m writing this post to myself for future reference. I’m starting a new website project at work and needed a new repository created with a few external folder references to another project. After creating a bootstrap set of directories and files from pieces of another project then needed to create the new repository. My folder structure looks like this:

~/work/publishing <- location of new project files/folders

cd ~/work
svn import -m "Initial revision" publishing https://svn/repos/publishing/trunk

The project has two folders which will come from an existing project that won’t be modified in this new project so I’m using svn:externals to reference those folders. I’m not sure this was the best way to do it but since I have several other people who will be using the repository I decided to check it out locally:

cd ~/work
svn co https://svn/repos/publishing pub

Next, I changed to the subfolder where the two external folders are located:

cd ~/work/pub/trunk/application

Then using svn propedit I created two external links though the first time through I got it wrong. Note, the quotes are important here:

SteveT:application strefethen$ svn propset svn:externals 'model https://svn/repost/proj/trunk/application/lib' .
property 'svn:externals' set on '.'
SteveT:application strefethen$ svn up
svn: warning: Error handling externals definition for 'model':
svn: warning: OPTIONS of 'https://svn/repost/proj/trunk/application/model': 200 OK (https://svn)
At revision 24431.

The problem in this case was my mistyping “repos” adding a “t” (bolded above). Next, I made the mistake of using propset twice for two different externals (model and lib) which yielded the following when I tried to update:

SteveT:application strefethen$ svn up
svn: warning: Error handling externals definition for 'model':
svn: warning: URL '' at revision 24431 doesn't exist
At revision 24431.

At this point, I needed to use svn propedit to create two external folder references but I didn’t have any editor set to make these changes which yielded:

SteveT:application strefethen$ svn propedit svn:externals .svn: None of the environment variables SVN_EDITOR, VISUAL or EDITOR are set, and no 'editor-cmd' run-time configuration option was found. I edited my .bash_profile adding an SVN_EDITOR environment variable
mate ~/.bash_profile


export SVN_EDITOR="mate -w"

Save an close TextMate. Then source my .bash_profile to get the changes:

source ~/.bash_profile

Now I could finish editing the svn:externals as follows:

svn propedit svn:externals .

The lib external was already defined from my previous propset command so I just needed to the model external making my svn:externals look like this:

lib https://svn/repos/proj/trunk/application/lib
model https://svn/repos/proj/trunk/application/model

Now when I update both external folders are updated:

SteveT:application strefethen$ svn up
Fetching external item into 'model'

External at revision 24431.

Fetching external item into 'lib'
External at revision 24431.

At revision 24431.

Finally, I committed my changes to the application folder for these externals. 

SteveT:application strefethen$ svn commit
Log message unchanged or not specified
(a)bort, (c)ontinue, (e)dit:
Sending        application
Committed revision 24432.

Not sure this is going to be helpful for anyone else but I can certainly make use of it in the future.

Useful Facebook FQL queries

This is a post for me to aggregate various FQL queries.

Fetch the Facebook like count for a given URL

SELECT like_count FROM link_stat 
WHERE url=""

Facebook user’s connections

SELECT target_id,target_type,is_following 
FROM connection WHERE source_id = facebook_userid

Facebook Share count on a page

SELECT share_count, like_count, comment_count, total_count 
FROM link_stat 
WHERE url=""

Fetch open graph id for a url:

SELECT id,type,site FROM object_url 
WHERE url=""

View which groups a user belongs to

SELECT name,gid FROM group 
WHERE gid IN (SELECT gid FROM group_member 
WHERE uid = facebook_userid)

Fetch a “friend map”

Replace UID with logged in user id. Via this page.

SELECT uid1, uid2
FROM friend
WHERE uid1 IN (SELECT uid1 FROM friend WHERE uid2=UID)
AND uid2 IN (SELECT uid1 FROM friend WHERE uid2=UID)
AND uid1 < uid2

Names of the places you’ve checked in

SELECT name FROM page 
WHERE page_id IN 
(SELECT page_id FROM checkin WHERE author_uid = me())

Names of the Places your Friends have checked in

SELECT name FROM page 
WHERE page_id IN (SELECT page_id FROM checkin WHERE author_uid IN 
(SELECT uid2 FROM friend WHERE uid1 = me()))

You can play around with executing these here including your Facebook user id if you’re logged in.

Updating my Starter Kit to Facebook Developer ToolKit v3.0 Take 1

Over the break I took a look at updating my Facebook Developer Toolkit (FDT) Starter kit to the latest v3.0 release that was announced at PDC. My first thought was to download the sources via SVN but unfortunately, that failed so I resorted to using the SDK Source download which I’d rather not do as it leaves no easy way to keep up-to-date with recent checkins.

Anyway, I built the sources which “just worked”, surely a good sign.

Next, I worked to compile and run my Starter Kit application and here is what I found (at least so far):

  • Delete old referenced assemblies
  • Change using statements:
    • using facebook; –> using Facebook
    • using facebook.web; –> using Facebook.Web;
    • using facebook.Schema; –> using Facebook.Schema;
  • Master.API.uid; –> Master.Api.Users.GetLoggedInUser();
  • Master.API.friends.getUserObjects(); –> Master.Api.Friends.GetUserObjects();
  •; -> Master.Api.Photos.GetAlbums();
  • facebook.Schema.user –> Facebook.Schema.user
  • Master.API.users.getInfo(Master.API.uid); –> Master.Api.Users.GetInfo(Master.Api.Users.GetLoggedInUser());
  • facebook.fql q = new facebook.fql(Master.API);
    • Add using Facebook.Rest
    • facebook.fql q = new facebook.fql(Master.API); –> Fql q = new Fql(Facebook.Session.FacebookSession);
  • Convert Site.Master page Page_Load method to a constructor setting RequireLogin = true;
  • Regular Expression bug

Until I get some time to resolve the RegEx issue the rest will be on hold for a bit though things are looking pretty good and the above didn’t take long at all. I have noticed a few worrisome comments on the discussion forum.