I’ve been working on a small web scraping project using Scrapy in Visual Studio Code using the Python Extension. I’m using a virtualenv with Scrapy installed and my launch.json file looks like this:
Working on building mobile apps for the last several years I thought I would publish a list of some of the things I’ve learned here in the mobile trenches. Without further adieu and in no particular order… Btw, welcome your feedback/additions.
Part of building mobile web apps is understanding the myriad of mobile analytics and in part visualizing the data to shed light on trends that my otherwise be difficult to see in tabular data or even a colorful cohort table. I’ve been building a dashboard using R, RStudio, Shiny, and Shiny Dashboards aggregating data from MSSQL, Postgres, Google Analytics, and Localytics.
The issue of scaling has been discussed recently at work and particularly in light of some of the recent details that have come out from Facebook on their data centers.
The question I want to know is do you think Facebook have scaled having been built on MSSQL as compared to MySQL? Did access to the source play an important role? Should it still be called MySQL or has FB changed it sufficiently that it’s really no longer MySQL?
In the past I’ve built starter kits for Facebook development in ASP.NET and building CruiseControl plugins. My latest interest has been experimenting with Google AppEngine as my day job is all python appserver stuff so it’s a pretty logical fit. I’m slowly putting together all of the pieces I’d like to have in website starter kit including support for jQuery Mobile and Facebook Graph API.
I have a simple proof-of-concept app working here. Btw, as this is a work-in-progress it YMMV and the app may or may not be in a working state so apologies in advance.
Here’s a routine I wrote to pretty print a Python dict into an HTML table and though I’d share.
def prettyTable(dictionary, cssClass=''): ''' pretty prints a dictionary into an HTML table(s) ''' if isinstance(dictionary, str): return '<td>' + dictionary + '</td>' s = ['<table '] if cssClass != '': s.append('class="%s"' % (cssClass)) s.append('>\n') for key, value in dictionary.iteritems(): s.append('<tr>\n <td valign="top"><strong>%s</strong></td>\n' % str(key)) if isinstance(value, dict): if key == 'picture' or key == 'icon': s.append(' <td valign="top"><img src="%s"></td>\n' % Page.prettyTable(value, cssClass)) else: s.append(' <td valign="top">%s</td>\n' % Page.prettyTable(value, cssClass)) elif isinstance(value, list): s.append("<td><table>") for i in value: s.append('<tr><td valign="top">%s</td></tr>\n' % Page.prettyTable(i, cssClass)) s.append('</table>') else: if key == 'picture' or key == 'icon': s.append(' <td valign="top"><img src="%s"></td>\n' % value) else: s.append(' <td valign="top">%s</td>\n' % value) s.append('</tr>\n')
s.append('</table>') return '\n'.join(s)
When I made the move from Windows to OSX and Python development one of the things I wanted to experiment with has been Google’s AppEngine. I installed the SDK and setup the plugin for Eclipse but ran into a few issues I wanted to make note of since I think other could probably benefit from it as well. I’ll mention I’m on OSX 10.6.8 using Eclipse for Java Version Helios SR 2.
Recently, I received an email via my blog asking for a VS.NET 2010 version of my Facebook Starter Kit. Since I no longer work on Windows it hasn’t been a high priority so I offered instructions on how to extract the project from the .vsi file and load the project manually. Much to my (pleasant) surprise I got a follow-up email with an updated starter kit for VS.NET 2010 updated to .NET v4.
If anyone decides they want to upgrade the FDT to the latest version I’d be happy to post an update for that as well. The starter kit is available on Facebook is nearing 5600 users! An interesting side note, I’m no longer able to edit the properties of this application on Facebook because it contains the word “facebook” in the URL which is no longer allowed.
Since I rarely tend to setup new $g(SVN source repositories) I find myself googling for the syntax of the various commands so I’m writing this post to myself for future reference. I’m starting a new website project at work and needed a new repository created with a few external folder references to another project. After creating a bootstrap set of directories and files from pieces of another project then needed to create the new repository. My folder structure looks like this:
~/work/publishing <- location of new project files/folders
cd ~/work svn import -m "Initial revision" publishing https://svn/repos/publishing/trunk
The project has two folders which will come from an existing project that won’t be modified in this new project so I’m using svn:externals to reference those folders. I’m not sure this was the best way to do it but since I have several other people who will be using the repository I decided to check it out locally:
cd ~/work svn co https://svn/repos/publishing pub
Next, I changed to the subfolder where the two external folders are located:
Then using svn propedit I created two external links though the first time through I got it wrong. Note, the quotes are important here:
SteveT:application strefethen$ svn propset svn:externals 'model https://svn/repost/proj/trunk/application/lib' . property 'svn:externals' set on '.' SteveT:application strefethen$ svn up svn: warning: Error handling externals definition for 'model': svn: warning: OPTIONS of 'https://svn/repost/proj/trunk/application/model': 200 OK (https://svn) At revision 24431.
The problem in this case was my mistyping “repos” adding a “t” (bolded above). Next, I made the mistake of using propset twice for two different externals (model and lib) which yielded the following when I tried to update:
SteveT:application strefethen$ svn up svn: warning: Error handling externals definition for 'model': svn: warning: URL 'https://web.merchantcircle.com/repos/proj/trunk/application/model' at revision 24431 doesn't exist At revision 24431.
At this point, I needed to use svn propedit to create two external folder references but I didn’t have any editor set to make these changes which yielded:
SteveT:application strefethen$ svn propedit svn:externals .svn: None of the environment variables SVN_EDITOR, VISUAL or EDITOR are set, and no 'editor-cmd' run-time configuration option was found. I edited my .bash_profile adding an SVN_EDITOR environment variable mate ~/.bash_profile
export SVN_EDITOR="mate -w"
Save an close TextMate. Then source my .bash_profile to get the changes:
Now I could finish editing the svn:externals as follows:
svn propedit svn:externals .
The lib external was already defined from my previous propset command so I just needed to the model external making my svn:externals look like this:
lib https://svn/repos/proj/trunk/application/lib model https://svn/repos/proj/trunk/application/model
Now when I update both external folders are updated:
SteveT:application strefethen$ svn up Fetching external item into 'model' External at revision 24431. Fetching external item into 'lib' External at revision 24431. At revision 24431.
Finally, I committed my changes to the application folder for these externals.
SteveT:application strefethen$ svn commit Log message unchanged or not specified (a)bort, (c)ontinue, (e)dit: c Sending application Committed revision 24432.
Not sure this is going to be helpful for anyone else but I can certainly make use of it in the future.
This is a post for me to aggregate various FQL queries.
Fetch the Facebook like count for a given URL
SELECT like_count FROM link_stat WHERE url="http://www.stevetrefethen.com/blog/"
Facebook user’s connections
SELECT target_id,target_type,is_following FROM connection WHERE source_id = facebook_userid
Facebook Share count on a page
SELECT share_count, like_count, comment_count, total_count FROM link_stat WHERE url="http://www.stevetrefethen.com/school/"
Fetch open graph id for a url:
SELECT id,type,site FROM object_url WHERE url="http://www.stevetrefethen.com/school/"
View which groups a user belongs to
SELECT name,gid FROM group WHERE gid IN (SELECT gid FROM group_member WHERE uid = facebook_userid)
Fetch a “friend map”
Replace UID with logged in user id. Via this page.
SELECT uid1, uid2 FROM friend WHERE uid1 IN (SELECT uid1 FROM friend WHERE uid2=UID) AND uid2 IN (SELECT uid1 FROM friend WHERE uid2=UID) AND uid1 < uid2
Names of the places you’ve checked in
SELECT name FROM page WHERE page_id IN (SELECT page_id FROM checkin WHERE author_uid = me())
Names of the Places your Friends have checked in
SELECT name FROM page WHERE page_id IN (SELECT page_id FROM checkin WHERE author_uid IN (SELECT uid2 FROM friend WHERE uid1 = me()))
You can play around with executing these here including your Facebook user id if you’re logged in.