About/Contact

Steve Trefethen

Steve Trefethen is CTO at Wanderful Media.
Contact me

View my LinkedIn profile



Calendar

<<  May 2013  >>
MoTuWeThFrSaSu
293012345
6789101112
13141516171819
20212223242526
272829303112
3456789

View posts in large calendar

Disclaimer

The posts on this weblog are provided AS IS with no warranties, and confer no rights. The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.



Using flot to chart data using a date range

June 28 2011 11:04AM

jquery flot chart I’m working on developing Bloglines these days and one of the features I wanted as an admin was the ability to see various pieces of data related to the site over time. Ideally what I wanted were some charts like those on $g(Google Analytics) for things like:

  • users joining/day
  • votes cast/day
  • blogs submitted per day

I've previously experimented with the $l(Google Chart Tools) and that’s where my search started but that lead to a bit of a dead end. I found a related post on Tom Fotherby’s blog but as Greg Fitzgerald pointed out there are still a few more issues to be worked out. Since we use jQuery on the site I started searching along that vein which led me to flot and more specifically this example (caution the examples site seems really slow) which fit perfectly.

Looking at the code and markup I quickly found this was something I could have working right away. The data in JavaScript looks like this:

var d = [[1196463600000, 0], [1196550000000, 0], [1196636400000, 0], ...];
 

Where the first value is a $g(JavaScript timestamp) and the second is the actual data to be plotted. There are a couple of notes regarding the timestamp mentioned here:

The timestamps must be specified as Javascript timestamps, as milliseconds since January 1, 1970 00:00. This is like Unix timestamps, but in milliseconds instead of seconds (remember to multiply with 1000!).

As an extra caveat, the timestamps are interpreted according to UTC to avoid having the graph shift with each visitor's local time zone. So you might have to add your local time zone offset to the timestamps or simply pretend that the data was produced in UTC instead of your local time zone.

We’re using $g(Postgres) and the tables I need to query all have a date_created field of timestamp without timezone. Here’s the SQL to fetch the data:

SELECT extract(epoch from date_trunc('day', date_created)) * 1000, count(*) from blog group by extract(epoch from date_trunc('day', date_created)) * 1000 order by extract(epoch from date_trunc('day', date_created)) * 1000 DESC

From the results I use the following python method to create the above data structure which is ready to feed into the flot chart:

    @staticmethod
    def statsByDay(query):
        data = PubBase.sqlQuery(query)
        dl = []
        for d in data:
            if d[0] != None and d[1] != None:
                dl.append([int(d[0]), int(d[1])])
        return dl         

I’ve been working with a lot of JavaScript frameworks/libraries lately and this is on that integrated into the project pretty flawlessly so kudos to the flot folks!

FacebookDel.icio.usDigg It!

Using the Dealmap API from Python

June 04 2011 8:32PM

Update Aug 1, 2011:With Google's acquisition of Dealmap one would have to assume this will become a Google API. The deal raises a number of interesting questions related to how deals would be served to specific clients. Should be interesting to watch.

Update Jun 11, 2011: I made one more tweak I neglected to mention to the Dealmap python API which I highly recommend if you intend to use it in a production environment which is to modify the call to urllib2.urlopen and add the timeout parameter. In fact, in my revision I've modified all of the calls into the API to accept a timeout parameter which is passed along to the urllib2.urlopen call.

I’ve been looking into the $l(DealMap API) and found what appears to be a semi/partially/maybe/sorta-official python implementation though there appears to be a missing module reference called “Util” containing an $g(ordered dictionary) and some XML serialization bits. I posted a message to the Google group for the project and even followed up with Dealmap directly but unfortunately haven’t gotten any response, not exactly a great sign though seeing as this is a pretty straightforward $g(REST API) let’s move on shall we...

There’s a fairly simple workaround for this unknown (at least to me) module using Beautiful Soup (gotta love that domain name) and a few simple wrapper classes making it easier to work with the API. Beautiful Soup is defined thusly:

Beautiful Soup is an HTML/XML parser for Python that can turn even invalid markup into a parse tree. It provides simple, idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.

In dealmap.py I removed all of the Util module references and changed the deserialize calls to return an instance of a BeautifulStoneSoup wrapper class called Deals:

class Deals(BeautifulStoneSoup):
    def __init__(self, dealmarkup=""):
        BeautifulStoneSoup.__init__(self, dealmarkup, convertEntities=BeautifulSoup.XML_ENTITIES)
        self._deals = None

    def getDeal(self, index):
        deals = self.getDeals()
        return Deal(deals[index])
        
    def getDeals(self):
        if self._deals == None:
            self._deals = self.findAll('deal')
            l = []
            for d in self._deals:
                l.append(Deal(d))
            self._deals = l
        return self._deals

Here’s a method from the Service class where I replaced the call to deserialize(…) with my new Deals class:

def search_deals(self, activities, capabilities, expirationDate, location, query="*", distance=5.0, startIndex=0, pageSize=20):
    searchDeals = self.__build_get_url(self.__dealmapUrls["search_deals"],
                                     l=location,
                                     q=query,
                                     d=distance,
                                     si=startIndex,
                                     ps=pageSize,
                                     a=activities,
                                     c=capabilities,
                                     ed=expirationDate,
                                     key=self.__apikey
                                     )
      
    result = self.__dealmap_get_request(searchDeals)
    obj = Deals(result)
    return obj

In the Deals class above you might have noticed I reference a Deal (singular) class which is a thin wrapper for accessing the properties of the Deal Tag object returned via BeautifulSoup using a __getattr__ override:

class Deal(object):
    def __init__(self, dealtag):
        self._dealtag = dealtag
        
    def __getattr__(self, name):
     

Accessing information from a deal now looks like this:

dealmap = Service("dealmap_api_key")
deals = dealmap.search_deals(None, None, None, location="+37.0491490732-122.025146484")
d = deals.getDeal(0)
print d.moreinfolink

You can specify any of XML child property names for a Deal and they’ll be returned (moreinfolink is such an example).

Fair, warning what’s posted here has essentially no error checking.

Btw, having been looking at the API I’ve found the performance to be much better if you provide a lat/lng location rather than city/state or zip where the latter seems to be particularly slow.

If you’ve worked with this API I’d be interested to get your impressions.

FacebookDel.icio.usDigg It!

How many browser tabs to you typically have open?

May 27 2011 10:57AM

I work standing up in an office of half-height cubes so everyone pretty much sees what I’m working on as they pass by and I consistently hear “wow, that’s a lot of tabs!”. $g(Chrome) is my browser of choice where I have 6 tabs I’ve pinned (to the left) a set of tabs I’m generally saving because they are topics of near term interest (middle) and working tabs I’m refreshing frequently as I work on Bloglines.

chrometabs

 

How do you use tabs? What strategies/tools do you use to manage this sort of flow?

FacebookDel.icio.usDigg It!

Using the Python Cheetah Template compiler

May 25 2011 12:35AM

The web application I’m working on uses Cheetah - “The Python Powered Template Engine” for web page generation and while the updated documentation is a great improvement over the old docs there remain some gaps that weren't very clear to me when I first got started…

image

Of course, Cheetah’s got a lot of features so I’m certain there are things I’m missing which is part of the point of this post to start a conversation about how I’m using Cheetah and elicit feedback and exchange ideas/best practices.

The first piece I’ll toss out is simply a routine to call the Cheetah compiler on a list of templates. Running this routine alone on the CI server has caught a bunch of syntax errors that wouldn’t have been caught using runtime compiled templates.

def compileTemplates(templateFiles, outpath="ui/compiled/"):
    ''' Compiles templateFiles if they're out of date '''
    if templateFiles == []:
        raise Exception("No template files specified")

    for t in templateFiles:
        outfile = os.path.split(t)[1]
        outfile = os.path.splitext(outfile)[0] + ".py"
        if not os.path.exists(outpath + outfile) or os.path.exists(outpath + outfile) and os.path.getmtime(outpath + outfile) < os.path.getmtime(config.templatePath(t)):
            t = config.templatePath(t)
            print "Template: " + t
            source = Cheetah.Compiler.Compiler(file=t, moduleName=os.path.splitext(outfile)[0]).moduleDef()
            f = open(outpath + outfile, "w+t")
            f.write(source)
            f.close()

The routine checks to see if the template file (.tmpl) has been modified since the last compilation and optionally recompiles. 

FacebookDel.icio.usDigg It!

ASP.NET Facebook Starter Kit updated to VS.NET 2010

May 24 2011 11:40PM

Recently, I received an email via my blog asking for a VS.NET 2010 version of my Facebook Starter Kit. Since I no longer work on Windows it hasn’t been a high priority so I offered instructions on how to extract the project from the .vsi file and load the project manually. Much to my (pleasant) surprise I got a follow-up email with an updated starter kit for VS.NET 2010 updated to .NET v4.

If anyone decides they want to upgrade the FDT to the latest version I’d be happy to post an update for that as well. The starter kit is available on Facebook is nearing 5600 users! An interesting side note, I’m no longer able to edit the properties of this application on Facebook because it contains the word “facebook” in the URL which is no longer allowed.

Enjoy!

FacebookDel.icio.usDigg It!

Upgrading to BlogEngine.NET v2 on discountasp.net

April 12 2011 11:14PM

I’ve been testing a v2.0 install of BlogEngine.NET for quite awhile but first had to get this problem fixed and fortunately, a contributor to the community blogged a response (thank you). While the solution was pretty straightforward having moved completely to OSX back in October getting setup to simply recompile the core BE assembly was a little more involved requiring a VM etc.

For me the single largest benefit is the addition of recaptcha for reducing comment spam which in recent months had grown completely out of control though there’s plenty more to like about the new version.

To make the upgrade I installed BE v2 in a new directory on my discountasp.net account and copied my post data and images over to the new install. I also setup my extensions and theme and got all the various bits in place before pulling the trigger. Once everything was ready I tried to rename my /blog folder using $g(FireFTP) which resulted in an “Access denied” error. It turns out that I hadn’t removed the IIS application from the /blog folder via the control panel. Once I did that I was able to rename /blog to /B515 and replace it with my new v2 install folder.

FacebookDel.icio.usDigg It!

Need for help with IIS URL rewrite rule for BlogEngine.NET v2.0

April 09 2011 11:41PM

Executive Summary

Essentially, I’m looking for help with creating an IIS URL rewrite rule to support the following:

Existing modified BlogEngine.NET v1.5 style post URL:

http://www.stevetrefethen.com/blog/ASQLObjectcaseinsensitiveLIKEquery.aspx

BlogEngine v2.0 style post URL:

http://www.stevetrefethen.com/blog/post/ASQLObjectcaseinsensitiveLIKEquery.aspx

I need a rule that will accept the prior and redirect/rewrite to the latter. I’d prefer whatever option has a minimal SEO affect.

Any ideas?

Details

My blog software is BlogEngine.NET v1.5 and I want to upgrade to 2.0 to take advantage of a number of new features. When I moved to BlogEngine.NET several years ago one of the things I wanted to maintain was support for my dasblog style URL’s which didn’t include “/post” in the path. Back then I tweaked BlogEngine to remove the “/post” path from post URLs but when looking to upgrade to v2.0 I see there were significant changes so my code wouldn’t port easily especially considering I’m no longer running Windows except in a VM on OSX and only have access to the VS.NET Express which doesn’t support multiple projects in a solution making debugging this issue pretty difficult since the code is in a satellite assemble and not the core website.

As an alternative I’m considering simply rewriting/redirecting my old URL’s to include “/post” in the path though I’ve done some playing around in the IIS Manager but have yet to achieve the desired result.

That’s where you (hopefully) come in. I’m pretty sure this is possible but need help creating the necessary rule(s). I’ve got BE v2.0 setup on my domain so I can conduct testing using this alternative path quickly.

Thanks in advance…

Just FYI, I’m getting so much comment spam on BE v1.5 I’ve closed comments after two weeks so if they’re closed please use my contact link instead.  Thanks.

FacebookDel.icio.usDigg It!