VMware vSphere SDKs explained

I currently manage the vSphere/VMware Cloud on AWS SDK team and I find that looking at the SDK’s page on VMware Code it can be a bit daunting to figure out exactly which SDK you might need. So, to clarify the landscape for vSphere specific SDKs a bit I thought I’d flesh out some of the unwritten details.

SDKs & Tools for Calling VMware SOAP API’s

As of this writing the bulk of VMware’s vSphere API’s are SOAP API’s which can be used from a variety of languages built using VMware’s WSDL including the Management SDK (for Java and .NET), the family of  “vmomi” tools: Pyvmomi, Govmomi, rbvmomi and last but not least the Perl SDK.

Ok, on to REST APIs…

SDKs for VMware REST APIs

Prior to the release of vSphere 6.5 in 2016, VMware released a set of “vCloud Suite *” SDK’s for use with the Tagging and Content Library REST API’s. With the release of 6.5 VMware created a new set of SDKs named “vSphere Automation SDK for *” where “*” is a language like Java, Python or Ruby. These new SDK’s were released on Github and are available here.

VMware Cloud on AWS APIs

At VMworld in 2017 VMware announced the release of VMware Cloud on AWS (VMC) and with it a new set of API’s for managing this new IaaS environment. As part of this expansion we’ve since added support for these API’s to the vSphere Automation SDK’s to include language bindings for the VMC Console API’s as well as NSX-T Policy API’s.

So there you have it. Hopefully, this helps explain some of the links on the SDK page the VMware {code} website.

Please post a comment with any questions.

VMware Automation SDK for Python API Reference Documentation

Accessing VMware vcenter REST API Authentication from curl

Here’s a simple example of calling the vSphere REST API using curl. These commands first authenticate to the API which creates a vmware-api-session-id cookie which is stored to cookie-jar.txt then makes a request to get a list of VMs:

curl -k -i -u administrator@vsphere.local:password -X POST -c cookie-jar.txt https://vcenter/rest/com/vmware/cis/session
curl -k -i -b cookie-jar.txt https://vcenter/rest/vcenter/vm

NOTE: Use with caution as your credentials will likely be caught in your command line history!

Here’s a related post on vSphere SDKs.

Highway 17 Santa Cruz Mountains Traffic Commuter Resources

Resources for information on Highway 17 Traffic over the Santa Cruz mountains to/from Los Gatos.

Installing R and Shiny on Cloud 9

I wanted to see if I could run an instance of RStudio’s Shiny server on Cloud 9 and after a bit of finagling with the right set of steps to get it set up I have an instance running. Here’s what I did starting from a workspace using the stock HTML template.

strefethen:~/workspace/ $ sudo sh -c 'echo "deb http://cran.rstudio.com/bin/linux/ubuntu trusty/" >> /etc/apt/sources.list
strefethen:~/workspace/ $ gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9
strefethen:~/workspace/ $ sudo apt-get update
strefethen:~/workspace/ $ sudo su - \
-c "R -e \"install.packages('shiny', repos='https://cran.rstudio.com/')\""
strefethen:~/workspace/ $ sudo su - -c "R -e \"install.packages('shiny', repos='https://cran.rstudio.com/')\""
strefethen:/etc/shiny-server $ sudo shiny-server
[2016-09-02 02:25:04.460] [INFO] shiny-server - Shiny Server v1.4.4.801 (Node.js v0.10.46)
[2016-09-02 02:25:04.463] [INFO] shiny-server - Using config file "/etc/shiny-server/shiny-server.conf"
[2016-09-02 02:25:04.535] [WARN] shiny-server - Running as root unnecessarily is a security risk! You could be running more securely as non-root.
[2016-09-02 02:25:04.539] [INFO] shiny-server - Starting listener on 0.0.0.0:8081

I need to look into the user “shiny” to see about fixing the above warning. Then edit /etc/shiny-server/shiny-server.conf and change the port from 3838 -> 8081 so Cloud 9 will server the content and start the server:

sudo shiny-server

Browse to http://<project-name>-<username>.c9users.io:8081/ and you should see:

Default Shiny app running on Cloud 9

React Native bundle loading failing on a physical device

I’m building a React Native application and recently updated to v0.31.0 and at first things were working well debugging on the device benefiting from a feature of the react-native-xcode.sh script. The script copies your Dev machine’s IP address to a text file called ip.txt which is used for establishing the connection to your machine from your device since localhost points to the wrong place. Here’s the line of code in ./node_modules/react-native/packager/react-native-xcode.sh:

echo "$IP.xip.io" &gt; "$DEST/ip.txt"

Continue reading React Native bundle loading failing on a physical device

Using Google Sheets, Pivot Tables and Charts as a Startup Dashboard – Part II

In my previous post I wrote about the process that lead me to build a dashboard but first I want to talk a bit about the structure of the data in the Google Sheet where the whole process started. I first started by looking to quickly create a few charts to visualize some of our KPI‘s. To source the data I created a text file containing the SQL statements and used psql to fetch Postgres data which I dumped it to data into .CSV files for import into separate “data” only tabs in Google Sheets.

psql -h pgserver -d mydb -U myuser -w -t -A -F $ '\t' -f ~/Campains.sql > campaign.csv

The first tab was the “primary” dataset which contained a wide (A to AX) set of columns with a blend of content from the various linked “data” tabs and is where I derived all of the pivot tables with a primary key in the first column and with this initial set of at I was able to start building charts to help visualize the data.

Of course, once you’ve answered one question it leads to follow-on questions which require more data leading to more questions. Before long I was querying a dozen tables from Postgres and MSSQL and importing the data into these “data” tabs. For data tabs with a 1-1 relationship based on primary key I would aggregate the data onto the main sheet with a formula like “Imported Data’!B4” or in cases where not all keys were present via a lookup like =IFERROR(VLOOKUP($A:$A,”Data Sheet”!$A:$E,3,FALSE),0) setting the result accordingly when the primary key wasn’t found.

Ultimately, flattening the data made it easy to construct pivot tables for aggregate totals, averages, counts, and median values etc. from which I could build a variety of charts a sampling of which I’ve included below.Here’s a small sample of the kinds of charts built from pivot tables. Yes, I’ve clipped/changed some of the legends knowingly obscuring the underlying meaning of the chart.

Here’s a small sample of the kinds of charts built from pivot tables. Yes, I’ve clipped/changed some of the legends knowingly obscuring the underlying meaning of the chart.

Monthly Totals

I built a variety of pivot tables for the Wanderful Marketing team (sans charts) for easy analysis of Cash Dash campaigns from a variety of angles such as by a given retailer by offer type, amount, reward, launch day of the week and a variety of campaign performance metrics that I’d calculated within the sheet. Ultimately, the usefulness of this data caught on and a number of teams were not only reviewing the data but asking for additional analysis and updates.

While I was able to automate some portions of updating this sheet, its associated tabs etc. Google Sheet’s charts and pivot tables don’t automatically expand as the size of your data grows which made it a laborious task to “re-scope” them as more data was added not to mention I knew the 2M cell limit was looming in the distance.

In a follow-on post I’ll talk about how I began the shift to automating this using R and a Shiny Dashboard running on an OSX Mac mini.

From Google Sheets to an Automated Shiny Dashboard (Part I)

Over the past several months I’ve been building what’s become a large Google Sheet (which is now at the 2M cell limit) tracking Find&Save Cash Dash offers. Like a lot of data projects this one started out small but over many months grew to a point where it no longer makes sense to cut/paste SQL data into a Google Sheet for pivot tables and charts. I wanted a solution that could pull data from REST APIs, MSSQL and Postgres and present data in an internal dashboard. Unfortunately, Google Sheet’s JDBC support doesn’t include Postgres otherwise Google Apps Script might have been a viable choice for automating data collection notwithstanding size limitations. Finally, another issue with Sheets was that I couldn’t seem to get pivot tables to resize automatically as more data was added. So, 3 strikes and Google Sheets was out.

Continue reading From Google Sheets to an Automated Shiny Dashboard (Part I)

Plotting Weekly Mobile Retention from the Localytics API using R and ggplot2

Part of building mobile web apps is understanding the myriad of mobile analytics and in part visualizing the data to shed light on trends that my otherwise be difficult to see in tabular data or even a colorful cohort table. I’ve been building a dashboard using R, RStudio, Shiny, and Shiny Dashboards aggregating data from MSSQL, Postgres, Google Analytics, and Localytics.

Within the Product section of the dashboard I’ve included a retention chart and found some great articles at R-Blogger like this one. The retention data comes from the Localytics API which I discussed previously though getting the data into the proper format took a few steps. Let’s start with the data, here’s an example of the REST response from Localytics looks like for a weekly retention cohort:

{
"results": [
{
"birth_week": "2014-09-08",
"users": 1,
"week": "2014-12-29"
},
{
"birth_week": "2014-09-29",
"users": 1640,
"week": "2014-12-29"
},
{
"birth_week": "2014-10-06",
"users": 2988,
"week": "2014-12-29"
},
{
"birth_week": "2014-10-13",
"users": 4747,
"week": "2014-12-29"
},
{
"birth_week": "2014-10-20",
"users": 2443,
"week": "2014-12-29"
},
…

Below is the main function to fetch the Localytics sample data and convert it into a data frame that’s suitable for plotting. Now, admittedly I’m not an R expert so there may well be better ways to slice this JSON response but this is a fairly straight forward approach. Essentially, this fetches the data, converts it from JSON to an R object, extracts the weeks, preallocates a matrix and then iterates over the data filling the matrix to build a data frame.

retentionDF <- function() {
  # Example data from: http://docs.localytics.com/dev/query-api.html#query-api-example-users-by-week-and-birth_week
  localyticsExampleJSON <- getURL('https://gist.githubusercontent.com/strefethen/180efcc1ecda6a02b1351418e95d0a29/raw/1ad93c22488e48b5e62b017dc5428765c5c3ba0f/localyticsexampledata.json')
  cohort <- fromJSON(localyticsExampleJSON)
  weeks <- unique(cohort$results$week)
  numweeks <- length(weeks)
  # Take the JSON response and convert it to a retention matrix (all numeric for easy conversion to a dataframe) like so:
  # Weekly.Cohort Users Week.1
  # 1    2014-12-29  7187   4558
  # 2    2015-01-05  5066     NA
  i <- 1
  # Create a matrix big enough to hold all of the data
  m <- matrix(nrow=numweeks, ncol=numweeks + 1)
  for (week in weeks) {
    # Get data for all weeks of this cohort
    d <- cohort$results[cohort$results$birth_week==week,][,2]
    lencohort <- length(d)
    for (n in 1:lencohort) {
      # Skip the first column using "+ 1" below which will be Weekly.Cohort (date)
      m[i,n + 1] <- d[n]
    }
    i <- i + 1
  }
  # Convert matrix to a dataframe
  df <- as.data.frame(m)
  # Set values of the first column to the cohort dates
  df$V1 <- weeks
  # Set the column names accordingly
  colnames(df) <- c("Weekly.Cohort", "Users", paste0("Week.", rep(1:(numweeks-1))))
  return(df)
}

To make things easy I put together a gist and if you’re using R you can runGist it yourself. It requires several other packages so be sure to check the sources in case you’re missing any.
Fair warning the Localytics API demo has very limited data so the chart, let’s just say simplistic however given many weeks worth of data it will fill out nicely (see example below).

> library(shiny)
> runGist("180efcc1ecda6a02b1351418e95d0a29")
Localytics Retention Plot Example
Retention Chart