About/Contact

Steve Trefethen

Steve Trefethen is CTO at Wanderful Media.
Contact me

View my LinkedIn profile



Calendar

<<  May 2013  >>
MoTuWeThFrSaSu
293012345
6789101112
13141516171819
20212223242526
272829303112
3456789

View posts in large calendar

Disclaimer

The posts on this weblog are provided AS IS with no warranties, and confer no rights. The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.



Using Google Playground Examples in VS.NET

October 06 2009 5:43AM

image I’ve been working on ways to leverage Google’s Calendar API to publish Falafel’s training calendar entries to the web. One of the things I stumbled into was the Google Code Playground which is a cool tool for testing out various Google API’s. The Playground includes Firebug lite which is also something I hadn’t run into, I love finding new cool things!

At any rate, this post is about how to get from the Playground to something live. In the screenshot to the right you can see the JavaScript source code in a code editor at the top right hand side of the screen. The missing piece here is that to get the code from the Playground to work on your own site you need to do a few things. First, you have to get an API Key. Once you’ve done that you can then add a <script> tag to include the Google GData API in your page.

    <script src="http://www.google.com/jsapi?key=<your_key>" type="text/javascript"></script>

After, that you need to initialize the GData API and get the code from the Playground to execute. To do that I did two things.

  1. Wrapped the JavaScript code from the Playground page in a function.
  2. Added the following code to my page with a reference to call the JavaScript function
    <script type="text/javascript">
        //<![CDATA[

        google.load("gdata", "1");
        google.setOnLoadCallback(loadItems);
        //]]>
    </script>

”loadItems” above is the name of the function I created in Step 1. Next, I had to add a DIV tag to the page as a target for the new content:

<div id="content">
</div>

At this point, the page (plain HTML) can be debugged from VS.NET and/or uploaded to your domain and function correctly. Anyway, I learned a few things along the way and thought this might be useful.

FacebookDel.icio.usDigg It!

Mail Server Log MIME Header Parsing

September 06 2009 5:12AM

imageThe other day a friend, whose not a developer, approached me looking for help with a problem they were having dealing a few large server log files. The file contained 100’s of $g(MIME headers) listed one after the other each with a starting comment and separated by a blank line. I was given a sample containing 1400 MIME headers which at first I opened in Windows Notepad and looked like what you see to the right. They had been trying to work with this file using Excel but not having much luck because the headers were inconsistent sizes and I imagine extracting the right fields was clearly a problem. The desired result was a format from which they could perform some analysis of the data and particularly of the X-Originating-IP field.

After about 10 seconds of staring at the data in Notepad I opened it in Notepad++ and things looked a bit more sane and it sort of dawned on me what my friend’s first thoughts probably were when they first glanced at this file. Excel looked better but didn’t make the process any easier.

image

A few choices entered my mind:

  • Write a simple parser, regex etc.
  • Look for an existing MIME parser
  • Use VS.NET editor Macros to extract the content
  • grep/findstr

My focus desire was to:

  • not spend much time
  • produce a CSV file
  • do something my friend could duplicate (lessen “support”)

I opted to search for a MIME parser largely because I figured one written in C# had to exist then write a tool to spit out a CVS file. My first Google search was “parse email header C#” which gave me a few interesting links but nothing that really caught my eye. The next attempt was “parse MIME header C#”:

image

Bingo.

The CodeProject article is largely code snippets and the first one looked interesting.

Mime m = Mime.Parse("message.eml");
// Do your stuff with mime

I thought, if there is .eml message parsing then I’m good regardless of the fact that it was expecting a file. I downloaded the source and it compiled without error, always encouraging. Next, I looked into the parsing support:

// Summary:
//     Parses mime message from byte[] data.
//
// Parameters:
//   data:
//     Mime message data.
public static Mime Parse(byte[] data);
//
// Summary:
//     Parses mime message from stream.
//
// Parameters:
//   stream:
//     Mime message stream.
public static Mime Parse(Stream stream);
//
// Summary:
//     Parses mime message from file.
//
// Parameters:
//   fileName:
//     Mime message file.
public static Mime Parse(string fileName);

Sweet.

For testing I saved off a single MIME header and created a simple console application to try and parse a fake .eml file which worked like a charm. All that was left to do was write some code to read the log file one header at a time and spit out a .CSV file.

I made one minor change to the MIME parsing code which was to change it’s HeaderFieldCollection from an IEnumerable to IEnumerable<HeaderField> so as to leverage LINQ to search for the “X-Originating-IP”. Of course, I later found out that the code attached to the article is outdated.

At any rate, I quickly had the file parsed, output to .CVS using a simple console application with input and output filename params which I mailed off. So, if you’re looking for MIME header parsing this library worked well for the 1400 headers I tried and I’m glad I could offer this tiny bit of help in a situation that sounds very serious for the folks involved.

Btw, kudos to Ivar Lumi for making this available, heck I think writing this post took longer than developing the solution.

FacebookDel.icio.usDigg It!

Facebook Developer Toolkit now supports JSON results on FQL queries

December 03 2008 5:49PM
Facebook’s query language (FQL) allows fetching results as $g(JSON) by supplying an additional "format=json" parameter as part of the request. I submitted a patch a few days ago that adds a UseJson property to the FQL object which was recently added to the Toolkit.

So, to fetch JSON results for FQL queries you can use the following:

facebook.fql q = new facebook.fql(Master.API);
q.UseJson = true;
string result = q.query("SELECT ...");
Be sure to grab the latest sources of the toolkit. Btw, the main reason for even mentioning it is that unless you’re reviewing the toolkit source code it’s unlikely you’d even know it’s there so this post will hopefully help people searching Google for the answer.

Take a look at this post to see how the JSON format can be useful for using Facebook data using an HttpHandler.

FacebookDel.icio.usDigg It!

Using Google Maps for Geocoding in C# Part 2: Deserializing to a class

December 02 2008 3:32PM

In my previous post, I illustrated how to $g(geocode) a city state combination using Google’s Map API and a comment was asked why not deserialize to a class. Well, I could have done that but in the app I’m playing around with that wasn’t really necessary although there is more useful data than just the coordinates so here’s part two.

Once you have the XML data you can construct a C# class that would allow for deserializing to an object. Generating the C# classes is straight forward and requires you to run XSD.EXE (installed with VS.NET) on the XML file to generate an .xsd file. Next, run XSD.EXE again on the .xsd with /classes to generate C# classes. Below is the output from the command line of XSD.EXE and you’ll notice I got an error the first time around (for the work around keep reading):

[c:\temp]"\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin"\xsd geo.xml
Microsoft (R) Xml Schemas/DataTypes support utility
[Microsoft (R) .NET Framework, Version 2.0.50727.42]
Copyright (C) Microsoft Corporation. All rights reserved.
Writing file 'C:\temp\geo.xsd'.

[c:\temp]"\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin"\xsd geo.xsd /cl
asses
Microsoft (R) Xml Schemas/DataTypes support utility
[Microsoft (R) .NET Framework, Version 2.0.50727.42]
Copyright (C) Microsoft Corporation. All rights reserved.
Schema validation warning: The 'urn:oasis:names:tc:ciq:xsdschema:xAL:2.0:Address
Details' element is not declared. Line 23, position 22.

Warning: Schema could not be validated. Class generation may fail or may produce
 incorrect results.

Error: Error generating classes for schema 'geo'.
  - The element 'urn:oasis:names:tc:ciq:xsdschema:xAL:2.0:AddressDetails' is mis
sing.

If you would like more help, please type "xsd /?".

To work around the above error I deleted the value of the xmlns attribute on the <AddressDetails> node then reran the above commands resulting in this C# file.

Finally, to deserialize the XML I used the following code which is a modified version of the function from my previous post (with minimal error checking I might add). Btw, here is a page that discusses .NET XML serialization:

private string GeoCode(string city, string state, string country, string zip)
{
    string url = "http://maps.google.com/maps/geo?output=xml&key=<your_API_key>&q=" + System.Web.HttpUtility.UrlEncode(city + " " + state);
    WebRequest req = HttpWebRequest.Create(url);
    WebResponse res = req.GetResponse();
    StreamReader sr = new StreamReader(res.GetResponseStream());
    try
    {
        System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(typeof(kml));
        using (System.IO.StringReader reader = new System.IO.StringReader(sr.ReadToEnd()))
        {
            kml item = (kml)serializer.Deserialize(reader);
            if(item.Items.Count() > 0 && item.Items[0].Placemark.Count() > 0 && item.Items[0].Placemark[0].Point.Count() > 0)
                return item.Items[0].Placemark[0].Point[0].coordinates;
            else 
              return "";
        }
    }
    finally
    {
        sr.Close();
    }
}
FacebookDel.icio.usDigg It!

Using Google Maps for Geocoding in C#

December 01 2008 9:05AM

Recently, I wanted to find out how to programmatically find the longitude and latitude given a city and state. From my previous work using Google Maps I knew this was possible. I did some digging and found Google Maps API supports a particular URL for geocoding a specific location. The URL looks like this:

http://maps.google.com/maps/geo?output=xml&key=<your_google_api_key>&q=<city_state>

Simply specify your $g(Google API Key) and your URL encoded city and state and you will get a result similar to the following (for "scotts valley ca"):

<?xml version="1.0" encoding="UTF-8" ?>
<kml xmlns="http://earth.google.com/kml/2.0"><Response>
  <name>scotts valley ca</name>
  <Status>
    <code>200</code>
    <request>geocode</request>
  </Status>
  <Placemark id="p1">
    <address>Scotts Valley, CA, USA</address>
    <AddressDetails Accuracy="4" xmlns="">
      <Country>
        <CountryNameCode>US</CountryNameCode>
        <CountryName>USA</CountryName>
        <AdministrativeArea>
          <AdministrativeAreaName>CA</AdministrativeAreaName>
          <Locality>
            <LocalityName>Scotts Valley</LocalityName>
          </Locality>
        </AdministrativeArea>
      </Country>
    </AddressDetails>
    <Point>
      <coordinates>-122.0120480,37.0555750,0</coordinates>
    </Point>
  </Placemark>
</Response>
</kml>

Hint: For illustration purposes this works without using an API Key although you should review Google’s Maps TOS.

Getting Geocode Information Programmatically

I wrote the following C# function (which lacks error checking) to fetch the above XML response from the Google Maps service. As you can see I’m using a simple regular expression to locate and return the actual coordinates.

private string GeoCode(string city, string state)
{
    string url = "http://maps.google.com/maps/geo?output=xml&key=<your_API_key>&q=" + System.Web.HttpUtility.UrlEncode(city + " " + state);
    WebRequest req = HttpWebRequest.Create(url);
    WebResponse res = req.GetResponse();
    StreamReader sr = new StreamReader(res.GetResponseStream());
    try
    {
        Match coord = Regex.Match(sr.ReadToEnd(), "<coordinates>.*</coordinates>");
        if(!coord.Success) return "";
        return coord.Value.Substring(13, coord.Length - 27);
    }
    finally
    {
        sr.Close();
    }
}

This turned out to be a lot less painful than I had originally thought it might be all thanks to Google’s Maps service. Have you done anything similar/easier/better?
[Update: Dec, 2 2008] Using Google Maps for Geocoding in C# Part 2: Deserializing to a class
[Update: Dec, 16 2008] Status codes returned by this service.

FacebookDel.icio.usDigg It!

Using LINQ on results from SubSonic's generic ExecuteTypedList

October 27 2008 5:43AM

Lately, I’ve been working on the Purchase Order portion of the EDI system I’ve blogged about. The system makes use of SubSonic, a .NET DAL framework, which comes in handy when dealing with the DB of an ERP system with 429 tables and approaching 2000 SPROCs.

Recently, I worked on the $g(EDI 856) which is an "Advanced Ship Notice" (ASN) document that includes shipping details for the PO. Included in the line item details, as stored in the DB, are "special services" which have to be separated out from normal line items within the document. The are reason they’re stored together is historic though tangential to my point. Fetching the 856 line item data is done via a stored proc and the result set is an aggregate of fields from a number of tables as well as calculated values. Fortunately, Subsonic can generate wrappers for calling SPROCs though doesn’t provide a wrapper for the result set in cases such as this.

Working with SPROC results using SubSonic

Subsonic generates a generic method called ExecuteTypedList<> that takes a class that will be the type of an IList returned from the call like this:

SPs.SaRptShipConfirmDetails(OrderHeaderID, true).ExecuteTypedList<ShipmentDetail>(); 

SaRptShipConfirmDetails is a wrapper for the like named SPROC and this call returns an IList<ShipmentDetail> where ShipmentDetail declared as follows:

public class ShipmentDetail
{
    public int LineNumber { get; set; }
    public string SKU { get; set; }
    public string ProductDescription { get; set; }
    public int Ordered { get; set; }
    public int Committed { get; set; }
    public int Shipped { get; set; }
    public decimal Price { get; set; }
    public decimal Amount { get; set; }
    public string CountryOfOrigin { get; set; }
    public decimal UnitWeight { get; set; }
    public string SizeName { get; set; }
    public int OrderDetailID { get; set; }
    public int ProductID { get; set; }
    public string BuyerProductCode { get; set; }
    public string UOMCode { get; set; }
    public string EDIUOMCode { get; set; }
    public int ShipmentDetailId { get; set; }
}

The get/set declarations above are required allowing ExecuteTypedList<> to determine where to stuff the returned SPROC column values into the ShipmentDetail structure. In fact, the SubSonic sources for the underlying SqlQuery.BuildTypedResult method read "//coerce the values, using some light reflection". From there you can easily use LINQ to query the IList results as desired. Below is a very simple example of filtering out these "special services" by ProductID:

var lineitems = from p in Details
                    where !p.ProductID.Equals(52276) &&
                    !p.ProductID.Equals(53257)
                    select p;

Clearly this is a simple example and it would be easy enough to write logic to skip the two ProductIDs indicated though simply playing around with LINQ in this manner has been interesting. For example, it gave me an opportunity to experiment with code completion in such a case which was interesting. I definitely recommend setting up a similar example where it’s easy to test the results and give yourself time to get used to "thinking" LINQ.

Using LINQ to query for specific instances of a class within an array

Another case where LINQ came in handy was in finding data read into an in memory structure of an EDI document. I’ve created a class hierarchy for working with $g(EDI segments) which make up an EDI document and with LINQ can now easily query a loaded document looking for specific segment instances like this:

public EdiRecord[] FindSegments<T>()
{
    var query = from EdiRecord s in this
                where s.GetType() == typeof(T)
                select s;
    return query.ToArray<EdiRecord>();
}

FindSegments is a generic method that takes the class that I’m looking for an returns an array of all the instances of that class type found. I’ll admit that starting to use LINQ has been an interesting if not counter intuitive experience for me as it’s hard not to think of doing this sort of thing the "old fashioned" way and immediate start writing that code.

Lastly, a minor blog note, I’m testing a new syntax highlighter via Windows Live Writer so it will be interesting to see how well it works (or not).

Anyway, what’s been your experience using LINQ?

FacebookDel.icio.usDigg It!

Tags: ,

Retrieving XML from Microsoft SQL Server

July 28 2008 9:43AM

A

fter completing work on the EDI Invoicing system I’ve recently started work on the other half of the equation, Purchase Order processing. There’s an existing system in place but it’s old, brittle and doesn’t support everything the system requires these days. The new CCNET based Invoice processing is humming along nicely and has handled over $3 million in invoices.

EDI CCNET Web Dashboard

Getting XML from a SqlDataReader

Like the invoicing system the purchase order side needs to query the database fetching PO data for conversion to EDI and transmission to the trading partner. I’ve been looking for clean ways to produce XML from a SqlDataReader and discovered this post by Roy Osherove. IMO, the interesting piece of that post isn’t necessarily Roy’s VB code but what’s found in the comments. Specifically, the comment I’m referring to is related to using a DbDataAdapter descendent like this:

public class DataReaderAdapter : DbDataAdapter { public int FillFromReader(DataTable dataTable, IDataReader dataReader) { return this.Fill(dataTable, dataReader); } protected override RowUpdatedEventArgs CreateRowUpdatedEvent(
DataRow dataRow,
IDbCommand command, StatementType statementType,
DataTableMapping tableMapping)
{ return null; } protected override RowUpdatingEventArgs CreateRowUpdatingEvent(
DataRow dataRow,
IDbCommand command, StatementType statementType,
DataTableMapping tableMapping)
{ return null; } protected override void OnRowUpdated(RowUpdatedEventArgs value) { } protected override void OnRowUpdating(RowUpdatingEventArgs value) { } }

A descendent is necessary here to help expose the inherited, protected Fill(...) method. Once you have the data in a DataTable you can simply call WriteXml() and viola! But there is an easier way (at least for MSSQL)!

Fetching XML from SQL Server using "FOR XML"

After mentioning the above to John Waters, Falafel’s CTO, he proposed querying directly FOR XML (literally), an MSSQL feature I wasn’t unaware of (no surprise there having been a non-database guy for so long). Using SqlCommand’s ExecuteXmlReader makes creating an XmlDocument very easy as you can see below. Simply setup the SqlCommand normally and use its ExecuteXmlReader() method:

using (XmlReader reader= command.ExecuteXmlReader())
{
   try
    {
       reader.Read();
       return new XmlDocument().Load(reader);
    }
   finally
    {
        reader.Close();
    }
}

The SQL statement looks like this:

1 SELECT * FROM SA_OrderHeader WHERE OrderHeaderID = @OrderHeaderID FOR XML, AUTO ROOT('details')

Notice "AUTO ROOT" which allows you to control the name of the root node of the returned XML document, very handy.

Having said all this I’m sure there are other ways to accomplish the same thing so please let me know what you’ve come up with.

FacebookDel.icio.usDigg It!