Working on building mobile apps for the last several years I thought I would publish a list of some of the things I’ve learned here in the mobile trenches. Without further adieu and in no particular order… Btw, welcome your feedback/additions.
Part of building mobile web apps is understanding the myriad of mobile analytics and in part visualizing the data to shed light on trends that my otherwise be difficult to see in tabular data or even a colorful cohort table. I’ve been building a dashboard using R, RStudio, Shiny, and Shiny Dashboards aggregating data from MSSQL, Postgres, Google Analytics, and Localytics.
The issue of scaling has been discussed recently at work and particularly in light of some of the recent details that have come out from Facebook on their data centers.
The question I want to know is do you think Facebook have scaled having been built on MSSQL as compared to MySQL? Did access to the source play an important role? Should it still be called MySQL or has FB changed it sufficiently that it’s really no longer MySQL?
In the past I’ve built starter kits for Facebook development in ASP.NET and building CruiseControl plugins. My latest interest has been experimenting with Google AppEngine as my day job is all python appserver stuff so it’s a pretty logical fit. I’m slowly putting together all of the pieces I’d like to have in website starter kit including support for jQuery Mobile and Facebook Graph API.
I have a simple proof-of-concept app working here. Btw, as this is a work-in-progress it YMMV and the app may or may not be in a working state so apologies in advance.
Here’s a routine I wrote to pretty print a Python dict into an HTML table and though I’d share.
def prettyTable(dictionary, cssClass=''): ''' pretty prints a dictionary into an HTML table(s) ''' if isinstance(dictionary, str): return '<td>' + dictionary + '</td>' s = ['<table '] if cssClass != '': s.append('class="%s"' % (cssClass)) s.append('>\n') for key, value in dictionary.iteritems(): s.append('<tr>\n <td valign="top"><strong>%s</strong></td>\n' % str(key)) if isinstance(value, dict): if key == 'picture' or key == 'icon': s.append(' <td valign="top"><img src="%s"></td>\n' % Page.prettyTable(value, cssClass)) else: s.append(' <td valign="top">%s</td>\n' % Page.prettyTable(value, cssClass)) elif isinstance(value, list): s.append("<td><table>") for i in value: s.append('<tr><td valign="top">%s</td></tr>\n' % Page.prettyTable(i, cssClass)) s.append('</table>') else: if key == 'picture' or key == 'icon': s.append(' <td valign="top"><img src="%s"></td>\n' % value) else: s.append(' <td valign="top">%s</td>\n' % value) s.append('</tr>\n')
s.append('</table>') return '\n'.join(s)
When I made the move from Windows to OSX and Python development one of the things I wanted to experiment with has been Google’s AppEngine. I installed the SDK and setup the plugin for Eclipse but ran into a few issues I wanted to make note of since I think other could probably benefit from it as well. I’ll mention I’m on OSX 10.6.8 using Eclipse for Java Version Helios SR 2.
Creating a new AppEngine project in Eclipse
With the plugin installed you get an AppEngine project template listed under PyDev.
Clicking Next displays the standard Eclipse dialog for a new project where you enter the name and optionally a few other project settings. However, it’s the dialog after that where I had my first question where you’re prompted for the Google App Engine Directory which in my case corresponds to:
Google instructions for development in Eclipse didn’t answer this question and it wasn’t immediately obvious to me. At any rate, once this path is set correctly you’ll immediately see a list of AppEngine modules to be included on the PYTHONPATH:
Configuring the Project’s Python Interpreter
After I successfully created the AppEngine project I wanted to look at the appengine-boilerplate for HTML5 which is based on HTML5 boilerplate which is well worth checking out. Next, I used the Google AppEngine Console to create a new application called “helloworld954” then copied the boilerplate files into my project and edited the app.yaml file to set the application name:
I then ran the project and everything looked ok but when I hit the page from the browser (http://localhost:8080) I got:
ImportError: No module named cgi
Which I subsequently discovered more information about here.
Now, IIRC my MacBook Pro came with Python 2.6 as the default version though v2.5 was installed as well and given AppEngine is, as of this writing, based on v2.5 I figured rather than tweaking dev_appserver.py as mentioned on that prior link I figured I’d first try the proper Python version. I right clicked my project from the package explorer and selected Properties then clicked the PyDev – Interpreter/Grammar option then clicked the “Click here to configure an interpreter not listed” link (blue link below).
Next, on the preferences dialog (which I won’t screenshot here as it’s too big) I clicked New… and added a new interpreter option for “Python v2.5” with a path of:
Btw, I think I tried the symlink but that wasn’t allowed (again, I think).
At any rate, once I saved those changes voila the boilerplate project started working. Ah, one step I overlooked was setting up a debug configuration for this project:
The main module for the application has to be Google’s dev_apperser.py file which is located here on my machine:
Also, under the (x)= Agruments tab I set the Program Arguments to “.” for the current working directory and for “Working Directory” I selected “Other” with a value of “$(workspace_loc:starterkit}” Note, starterkit is the name of my project.
The final result:
Hopefully, this helps someone else, it will certainly help me months from now when I trying to do the same thing again so I won’t have try and recall all these steps.
Recently, I received an email via my blog asking for a VS.NET 2010 version of my Facebook Starter Kit. Since I no longer work on Windows it hasn’t been a high priority so I offered instructions on how to extract the project from the .vsi file and load the project manually. Much to my (pleasant) surprise I got a follow-up email with an updated starter kit for VS.NET 2010 updated to .NET v4.
If anyone decides they want to upgrade the FDT to the latest version I’d be happy to post an update for that as well. The starter kit is available on Facebook is nearing 5600 users! An interesting side note, I’m no longer able to edit the properties of this application on Facebook because it contains the word “facebook” in the URL which is no longer allowed.
Since I rarely tend to setup new $g(SVN source repositories) I find myself googling for the syntax of the various commands so I’m writing this post to myself for future reference. I’m starting a new website project at work and needed a new repository created with a few external folder references to another project. After creating a bootstrap set of directories and files from pieces of another project then needed to create the new repository. My folder structure looks like this:
~/work/publishing <- location of new project files/folders
cd ~/work svn import -m "Initial revision" publishing https://svn/repos/publishing/trunk
The project has two folders which will come from an existing project that won’t be modified in this new project so I’m using svn:externals to reference those folders. I’m not sure this was the best way to do it but since I have several other people who will be using the repository I decided to check it out locally:
cd ~/work svn co https://svn/repos/publishing pub
Next, I changed to the subfolder where the two external folders are located:
Then using svn propedit I created two external links though the first time through I got it wrong. Note, the quotes are important here:
SteveT:application strefethen$ svn propset svn:externals 'model https://svn/repost/proj/trunk/application/lib' . property 'svn:externals' set on '.' SteveT:application strefethen$ svn up svn: warning: Error handling externals definition for 'model': svn: warning: OPTIONS of 'https://svn/repost/proj/trunk/application/model': 200 OK (https://svn) At revision 24431.
The problem in this case was my mistyping “repos” adding a “t” (bolded above). Next, I made the mistake of using propset twice for two different externals (model and lib) which yielded the following when I tried to update:
SteveT:application strefethen$ svn up svn: warning: Error handling externals definition for 'model': svn: warning: URL 'https://web.merchantcircle.com/repos/proj/trunk/application/model' at revision 24431 doesn't exist At revision 24431.
At this point, I needed to use svn propedit to create two external folder references but I didn’t have any editor set to make these changes which yielded:
SteveT:application strefethen$ svn propedit svn:externals .svn: None of the environment variables SVN_EDITOR, VISUAL or EDITOR are set, and no 'editor-cmd' run-time configuration option was found. I edited my .bash_profile adding an SVN_EDITOR environment variable mate ~/.bash_profile
export SVN_EDITOR="mate -w"
Save an close TextMate. Then source my .bash_profile to get the changes:
Now I could finish editing the svn:externals as follows:
svn propedit svn:externals .
The lib external was already defined from my previous propset command so I just needed to the model external making my svn:externals look like this:
lib https://svn/repos/proj/trunk/application/lib model https://svn/repos/proj/trunk/application/model
Now when I update both external folders are updated:
SteveT:application strefethen$ svn up Fetching external item into 'model' External at revision 24431. Fetching external item into 'lib' External at revision 24431. At revision 24431.
Finally, I committed my changes to the application folder for these externals.
SteveT:application strefethen$ svn commit Log message unchanged or not specified (a)bort, (c)ontinue, (e)dit: c Sending application Committed revision 24432.
Not sure this is going to be helpful for anyone else but I can certainly make use of it in the future.
This is a post for me to aggregate various FQL queries.
Fetch the Facebook like count for a given URL
SELECT like_count FROM link_stat WHERE url="http://www.stevetrefethen.com/blog/"
Facebook user’s connections
SELECT target_id,target_type,is_following FROM connection WHERE source_id = facebook_userid
Facebook Share count on a page
SELECT share_count, like_count, comment_count, total_count FROM link_stat WHERE url="http://www.stevetrefethen.com/school/"
Fetch open graph id for a url:
SELECT id,type,site FROM object_url WHERE url="http://www.stevetrefethen.com/school/"
View which groups a user belongs to
SELECT name,gid FROM group WHERE gid IN (SELECT gid FROM group_member WHERE uid = facebook_userid)
Fetch a “friend map”
Replace UID with logged in user id. Via this page.
SELECT uid1, uid2 FROM friend WHERE uid1 IN (SELECT uid1 FROM friend WHERE uid2=UID) AND uid2 IN (SELECT uid1 FROM friend WHERE uid2=UID) AND uid1 < uid2
Names of the places you’ve checked in
SELECT name FROM page WHERE page_id IN (SELECT page_id FROM checkin WHERE author_uid = me())
Names of the Places your Friends have checked in
SELECT name FROM page WHERE page_id IN (SELECT page_id FROM checkin WHERE author_uid IN (SELECT uid2 FROM friend WHERE uid1 = me()))
You can play around with executing these here including your Facebook user id if you’re logged in.
Over the break I took a look at updating my Facebook Developer Toolkit (FDT) Starter kit to the latest v3.0 release that was announced at PDC. My first thought was to download the sources via SVN but unfortunately, that failed so I resorted to using the SDK Source download which I’d rather not do as it leaves no easy way to keep up-to-date with recent checkins.
Anyway, I built the sources which “just worked”, surely a good sign.
Next, I worked to compile and run my Starter Kit application and here is what I found (at least so far):
- Delete old referenced assemblies
- Change using statements:
- using facebook; –> using Facebook
- using facebook.web; –> using Facebook.Web;
- using facebook.Schema; –> using Facebook.Schema;
- Master.API.uid; –> Master.Api.Users.GetLoggedInUser();
- Master.API.friends.getUserObjects(); –> Master.Api.Friends.GetUserObjects();
- Master.API.photos.getAlbums(); -> Master.Api.Photos.GetAlbums();
- facebook.Schema.user –> Facebook.Schema.user
- Master.API.users.getInfo(Master.API.uid); –> Master.Api.Users.GetInfo(Master.Api.Users.GetLoggedInUser());
- facebook.fql q = new facebook.fql(Master.API);
- Add using Facebook.Rest
- facebook.fql q = new facebook.fql(Master.API); –> Fql q = new Fql(Facebook.Session.FacebookSession);
- Convert Site.Master page Page_Load method to a constructor setting RequireLogin = true;
- Regular Expression bug
Until I get some time to resolve the RegEx issue the rest will be on hold for a bit though things are looking pretty good and the above didn’t take long at all. I have noticed a few worrisome comments on the discussion forum.