Category Archives: Appengine

Cost of WordPress blog running on Google AppEngine

Having recently migrated all of my blog/site content to Google AppEngine away from my prior $10/month hosting I was curious about what the monthly cost of running the site would be. My blog went live on AppEngine on Nov 13th and with the default instance settings was running ~$2.00/day at roughly 50-55 instance hours the vast majority of which is servicing requests from crawlers.

Tuning Google AppEngine, For Cost

Idle Instances

As mentioned above I started with the default settings and the first significant change was to set Idle Instance Max to 1 as running multiple instances chews through the free 28 hours at an xN multiplier rate. This change alone had a significant impact on reducing overall instance hours though clearly impacts the performance of my blog. I’ve tested the performance and for a blog I feel it’s suitable so I’m comfortable leaving things at this level for awhile.

robots.txt and dos.yaml

I’ve been running another site on AppEngine which has both significantly more pages and traffic than my blog though I’ve been able to optimize it to stay under 28 instance hours and one key factor has monitoring bot traffic and blocking via either robots.txt or dos.yaml.

Google Cloud SQL Costs

So far for the month of November which is missing roughly 12.5 days worth of actual traffic to my blog I’ll be around $15. In addition to normal site traffic (crawlers & users) I’m currently running JetPack which appears to ping back to my domain for tracking stats etc. I’ll need to evaluate the possible benefits of using dedicated memcache over the Cloud SQL expense as most of the site’s content is static. I’ve configured WordPress as recommended by Google using Memcached plugin as well as Batcache.

SSL Administration

I’ll admit I mistakenly hadn’t thought too much about SSL prior to finally switching my domain over to Google’s servers. At that point, I faced the issue of having to buy SSL support for AppEngine which would certainly add to the monthly cost of running my blog. Seeing as how I’m the only administrator this wasn’t exactly an attractive option given the traffic challenges my blog has faced coupled with the desire to keep costs down. I’ve instead opted to use the free SSL on the appspot.com version of my site to manage WordPress administration which works *fairly* well though there are places (like the Themes Editor) which expect to load all content from the hosting domain which in my case doesn’t work. I suspect there are other issues here as well that I simply haven’t run into yet but time will certainly reveal those issues too.

At this point, I’m fairly happy with the switch as WordPress is far more advanced that BlogEngine.NET 2.X not to mention it has clients for iPad and iPhone which is likely where I’ll be posted from the vast majority of the time. That said, I’ll continue watching the hosting costs and learning more about the SSL issues.

Updated Results (Dec 6, 2013)

I’ve been running with the above modifications since Nov. 21, 2013 and my blog is running about 24-26 instance hours per day, within Google’s free 28 hours. It looks like my associated Cloud SQL instance is running about $14-16/month which is roughly comparable to where I was hosting before. Prior to these changes my instance cost was nearing $2/day and typically exceeding 50 instance hours.

WordPress on Google AppEngine and Migrating from BlogEngine.NET

In 2004 I started blogging on my own domain using discountasp.net as they offered a competitive hosting package based on Microsoft technology. Fast forward nearly a decade and I haven’t used Microsoft technology for nearly 5 years and maintaining my Blogengine.NET blog had become an increasing challenge. Upgrades were very difficult and I found myself stuck on a tweaked version that couldn’t have easily be upgraded. Recently, Google added support for PHP on Google Appengine and posted steps to get WordPress running so I applied for and got early access and I’ve been slowing migrating all of my content.

I’m writing this post prior to completing the migration as the number of steps involved has been growing and I figured it might be helpful to track what was required so I don’t forget and to share what I found. The list here is not exhaustive by any stretch and it’s likely written in such a way for me to remember than perhaps general utility though nonetheless here it is.

Once I got a basic install working which was rather straight forward I spent several days getting reacquainted with WordPress which I’d used many years go though only locally. Needless, to say it’s come a long way and with an iOS client on both the iPhone and iPad it looks really promising.

WordPress Plugins

  • Memcached plugin (and no this doesn’t activate!)
  • BlogML pluggin
  • Redirection plugin
  • Creative Commons
  • Google Analytics for WordPress
  • Adsense Explosion
  • Pushover Notifications

Importing My Blog Content

First was the task of researching tools to import BlogEngine.NET’s BlogML format for which I found a plugin. The plugin wasn’t just drop in and go and I ran into issues with it processing my BlogML.xml file which required manually editing the XML as well as finding and patching XPath.class.php to v1.148 as it was outdated in the sources I’d found.

I also had to patch function parse_blogml in blogml-importer.php to read my BlogML.xml file from my WordPress dedicated Google Cloud Storage bucket like this (and yes, figuring out this URL took a fair bit of Googling where wpsdt-blog-files refers to my Google Cloud Storage bucket):

$sXml = file_get_contents('gs://wpsdt-blog-files/BlogML.xml');

Next I found my BlogML.xml file was too large so I had to chunk it down into several different smaller files in order for the import to proceed a process which took a long time as AppEngine repeatedly timed out importing the .xml file though it eventually worked. Oh yes, it took quite a bit of trial and error to figure out the various file permissions that had to be granted to get access from a WordPress plugin to Google Cloud Storage including the fact that I found myself cursing Google’s new Cloud Console with buttons to change permissions that only appear once you check the bucket you want to change. Perhaps needless to say it would have been awful nice if they were just disabled so you at least you knew they existed.

Setting up Redirects

I exported BlogEngine.NET to BlogML.xml and extracted all post URL’s to build a CSV mapping to import into the Redirect plugin as WordPress and BlogEngine.NET don’t have compatible slugs. (I’m still manually correcting the dash “-” placement in over 600 slugs as no simple pattern would have worked).

Setup redirect for BlogEngine.NET dynamic image serving /blog/image.asx?picture=(.*) -> /AppData/files/\1

Setup redirect for blog archives /blog/Archives.aspx -> /blog/2013 (close enough)

Redirect BlogEngine.NET syndication link /blog/SyndicationService.asmx/GetRss -> /blog/feed

Fix up FeedBurner RSS source link /blog/feed

Set Site Address (URL) to /blog so I could continue to use appspot.com’s support for SSL for site administration without paying for an SSL cert on www.stevetrefethen.com and saving myself the cost of the cert and what looks like $9/month for SSL support on AppEngine.

Of course, I learned a hard lesson along the way by changing both the WordPress Address and Site Address URL’s in such a way that created an infinite redirect loop. To fix this I had to install Google’s CLI tools for accessing MySQL on Google Cloud SQL and manually update the wp_options table to correct my mistake.

School Blog Content

While I was working on the the Scotts Valley Parcel Tax effort I setup a secondary BlogEngine.NET install (before it supported multiple blogs) and all of that content had to be exported, imported, and redirected as well so repeat many of the above steps.

Wiki Content

I was a fan of ScrewTurn wiki and had been running a few installs of that as well so I moved a number of those pages here manually largely but cut/paste as there is/was not export/import available.

Handling Static Image Files

Over the years I’ve had various image files scattered around my site thus I had to copy these paths from my old site into the new WordPress install which included:

  • /files
  • /blog/AppData/files
  • /blog/images

Upload various media files to my WordPress Google Cloud Storage bucket.

Domain Name Transfer

I’d not done a domain name transfer before so this was my first such experience and while it all went smoothly the only problem was a very poorly written email from OpenSRS (whose website looks like it’s from the early 90′s) that mislead me thinking the transfer would occur after 7 days. The email read (emphasis mine):

If you wish to cancel the transfer, please contact us before Tue Nov 12 04:13:22 2013 by going to our website, <link removed>

Needless, to say I didn’t click the link until after the deadline passed only then to realize the link had a checkbox I needed to click and submit.

Years ago I’d gotten a Google Apps for Domains account for my Gmail hosted personal email address so once the domain was transferred I just needed to edit the zone file and get it pointed to my WordPress AppEngine project.

Missing Handler

After running into a few problems I found his issue with app.yaml

# 09/29/2013
# https://code.google.com/p/googleappengine/issues/detail?id=9972
# missing handler
- url: /wp-(.+).php
  script: wordpress/wp-\1.php
- url: /(.+)?/?
  script: wordpress/index.php

New Contact Form

After a bit of digging I found a plugin called Contact Form 7 which I’ve used to replace my old BlogEngine.NET contact form.

Conclusion

All in all I’d say this was a pretty damn painful process with a lot of trial & error that took me a few months of late evenings to get going. I’ve been blogging since June 2004 and accumulated an awful lot of cruft over the years that I simply didn’t remember which didn’t help.

The site has now been running for a few days and I’m in the process of trying to reduce the cost of running it on AppEngine as it’s currently over $2/day whereas discountASP.NET I was running about $20/month with SQL support. Just applied a few changes based on this post.

Found this post which I totally agree with.

[Updated: 12/15/2013] Added Contact form info
[Updated: 01/07/14] Updated Google AppEngine plugin to 1.2 running on 3.6.1

WordPress running on Google AppEngine

After several months of slowly migrating all of my blog content and testing Google’s WordPress support on AppEngine I’ve finally pulled the trigger on moving my blog from BlogEngine.NET hosted on discountASP.NET to AppEngine.

w00t!

As time permits I’ll post details here regarding caveats I ran into (some of which I’m still wrangling). At any rate, I’m glad to be off of .NET and have learned a lot about some new parts of Google’s Cloud Infrastructure.

Configuring virtualenv to run Google appengine samples

In my experiments with Google AppEngine I wanted to configure a virtualenv using python 2.5 for running Google’s samples. Using the existing python install for OSX I was running into a few errors such as:

ImportError: No module named django

and

ImportError: No module named cgi

A bit of Googling turned up this post which details all of the necessary steps. In step 2 the path I used for google_appengine was:

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine

Leveraging the Facebook API on Google AppEngine with jQuery Mobile

In the past I’ve built starter kits for Facebook development in ASP.NET and building CruiseControl plugins. My latest interest has been experimenting with Google AppEngine as my day job is all python appserver stuff so it’s a pretty logical fit. I’m slowly putting together all of the pieces I’d like to have in website starter kit including support for jQuery Mobile and Facebook Graph API.

I have a simple proof-of-concept app working here. Btw, as this is a work-in-progress it YMMV and the app may or may not be in a working state so apologies in advance.

Technologies used:

Google AppEngine

Cheetah Templates

Facebook Graph API

jQuery Mobile

Setting up debugging for Google AppEngine projects in Eclipse

When I made the move from Windows to OSX and Python development one of the things I wanted to experiment with has been Google’s AppEngine. I installed the SDK and setup the plugin for Eclipse but ran into a few issues I wanted to make note of since I think other could probably benefit from it as well. I’ll mention I’m on OSX 10.6.8 using Eclipse for Java Version Helios SR 2.

Creating a new AppEngine project in Eclipse

With the plugin installed you get an AppEngine project template listed under PyDev.

Eclipse New Project dialog

Clicking Next displays the standard Eclipse dialog for a new project where you enter the name and optionally a few other project settings. However, it’s the dialog after that where I had my first question where you’re prompted for the Google App Engine Directory which in my case corresponds to:

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/

Google instructions for development in Eclipse didn’t answer this question and it wasn’t immediately obvious to me. At any rate, once this path is set correctly you’ll immediately see a list of AppEngine modules to be included on the PYTHONPATH:

AppEngine Module selection dialog

Configuring the Project’s Python Interpreter

After I successfully created the AppEngine project I wanted to look at the appengine-boilerplate for HTML5 which is based on HTML5 boilerplate which is well worth checking out. Next, I used the Google AppEngine Console to create a new application called “helloworld954” then copied the boilerplate files into my project and edited the app.yaml file to set the application name:

application: helloworld954

I then ran the project and everything looked ok but when I hit the page from the browser (http://localhost:8080) I got:

ImportError: No module named cgi

Which I subsequently discovered more information about here.

Now, IIRC my MacBook Pro came with Python 2.6 as the default version though v2.5 was installed as well and given AppEngine is, as of this writing, based on v2.5 I figured rather than tweaking dev_appserver.py as mentioned on that prior link I figured I’d first try the proper Python version. I right clicked my project from the package explorer and selected Properties then clicked the PyDev – Interpreter/Grammar option then clicked the “Click here to configure an interpreter not listed” link (blue link below).

PyDev Package properties dialog

Next, on the preferences dialog (which I won’t screenshot here as it’s too big) I clicked New… and added a new interpreter option for “Python v2.5” with a path of:

/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python

Btw, I think I tried the symlink but that wasn’t allowed (again, I think).

At any rate, once I saved those changes voila the boilerplate project started working. Ah, one step I overlooked was setting up a debug configuration for this project:

Eclipse Debug Configuration

The main module for the application has to be Google’s dev_apperser.py file which is located here on my machine:

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py

Also, under the (x)= Agruments tab I set the Program Arguments to “.” for the current working directory and for “Working Directory” I selected “Other” with a value of “$(workspace_loc:starterkit}” Note, starterkit is the name of my project.

The final result:

Google AppEngine HTML5 boilerplate

Hopefully, this helps someone else, it will certainly help me months from now when I trying to do the same thing again so I won’t have try and recall all these steps.