I’ve been working on a small web scraping project using Scrapy in Visual Studio Code using the Python Extension. I’m using a virtualenv with Scrapy installed and my launch.json file looks like this:
Resources for information on Highway 17 Traffic over the Santa Cruz mountains to/from Los Gatos.
- On Twitter
- On the Web
- CalTrans Highway 17 Conditions
- Scotts Valley Nixle feed
- Los Gatos Nixle feed
- Caltrans Road Information
- Bay Area Traffic (Highway 17)
- CHP Traffic Incident Information Page
- District 5 Planned Lane Closures
- Santa Cruz County Road Closures
- cruz511 Travel Information
- Local Conditions
- CalTrans Closed Circuit TV (install VLC Media Player on mobile to view feeds)
- Google Maps
I wanted to see if I could run an instance of RStudio’s Shiny server on Cloud 9 and after a bit of finagling with the right set of steps to get it set up I have an instance running. Here’s what I did starting from a workspace using the stock HTML template.
strefethen:~/workspace/ $ sudo sh -c 'echo "deb http://cran.rstudio.com/bin/linux/ubuntu trusty/" >> /etc/apt/sources.list strefethen:~/workspace/ $ gpg --keyserver keyserver.ubuntu.com --recv-key E084DAB9 strefethen:~/workspace/ $ sudo apt-get update strefethen:~/workspace/ $ sudo su - \ -c "R -e \"install.packages('shiny', repos='https://cran.rstudio.com/')\"" strefethen:~/workspace/ $ sudo su - -c "R -e \"install.packages('shiny', repos='https://cran.rstudio.com/')\"" strefethen:/etc/shiny-server $ sudo shiny-server [2016-09-02 02:25:04.460] [INFO] shiny-server - Shiny Server v126.96.36.1991 (Node.js v0.10.46) [2016-09-02 02:25:04.463] [INFO] shiny-server - Using config file "/etc/shiny-server/shiny-server.conf" [2016-09-02 02:25:04.535] [WARN] shiny-server - Running as root unnecessarily is a security risk! You could be running more securely as non-root. [2016-09-02 02:25:04.539] [INFO] shiny-server - Starting listener on 0.0.0.0:8081
I need to look into the user “shiny” to see about fixing the above warning. Then edit /etc/shiny-server/shiny-server.conf and change the port from 3838 -> 8081 so Cloud 9 will server the content and start the server:
Browse to http://<project-name>-<username>.c9users.io:8081/ and you should see:
I’m building a React Native application and recently updated to v0.31.0 and at first things were working well debugging on the device benefiting from a feature of the react-native-xcode.sh script. The script copies your Dev machine’s IP address to a text file called ip.txt which is used for establishing the connection to your machine from your device since localhost points to the wrong place. Here’s the line of code in ./node_modules/react-native/packager/react-native-xcode.sh:
echo "$IP.xip.io" > "$DEST/ip.txt"
In my previous post I wrote about the process that lead me to build a dashboard but first I want to talk a bit about the structure of the data in the Google Sheet where the whole process started. I first started by looking to quickly create a few charts to visualize some of our KPI‘s. To source the data I created a text file containing the SQL statements and used psql to fetch Postgres data which I dumped it to data into .CSV files for import into separate “data” only tabs in Google Sheets.
psql -h pgserver -d mydb -U myuser -w -t -A -F $'\t' -f ~/Campaigns.sql > campaign.csv
Working on building mobile apps for the last several years I thought I would publish a list of some of the things I’ve learned here in the mobile trenches. Without further adieu and in no particular order… Btw, welcome your feedback/additions.
Over the past several months I’ve been building what’s become a large Google Sheet (which is now at the 2M cell limit) tracking Find&Save Cash Dash offers. Like a lot of data projects this one started out small but over many months grew to a point where it no longer makes sense to cut/paste SQL data into a Google Sheet for pivot tables and charts. I wanted a solution that could pull data from REST APIs, MSSQL and Postgres and present data in an internal dashboard. Unfortunately, Google Sheet’s JDBC support doesn’t include Postgres otherwise Google Apps Script might have been a viable choice for automating data collection notwithstanding size limitations. Finally, another issue with Sheets was that I couldn’t seem to get pivot tables to resize automatically as more data was added. So, 3 strikes and Google Sheets was out.
Part of building mobile web apps is understanding the myriad of mobile analytics and in part visualizing the data to shed light on trends that my otherwise be difficult to see in tabular data or even a colorful cohort table. I’ve been building a dashboard using R, RStudio, Shiny, and Shiny Dashboards aggregating data from MSSQL, Postgres, Google Analytics, and Localytics.
Recently, I’ve been building a dashboard for Find&Save using R, RStudio and Shiny Dashboards and one of our core mobile metrics is 30 day active users for both iOS and Android which I can get from Localytics (our analytics platform) via their API. Googling around I found I’m not reinventing the wheel (via SO) though I did run into a problem where the response failed to convert properly thus the post.
Here’s the original function from SO with some slight tweaks for my needs: