{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# 'plottoolbox ...' using the command line interface\n",
    "This notebook illustrates some of the plotting features of tstoolbox on the command line.\n",
    "\n",
    "The plot functionality in tstoolbox was moved to plottoolbox and this notebook was converted to using plottoolbox, but kept in the tstoolbox package for reference on how the *toolbox packages can work together.\n",
    "\n",
    "For detailed help type 'plottoolbox --help' at the command line prompt.\n",
    "\n",
    "The available plot types are: time, xy, double_mass, boxplot, scatter_matrix, lag_plot, autocorrelation, bootstrap, histogram, kde, kde_time, bar, barh, bar_stacked, barh_stacked, heatmap, norm_xaxis, norm_yaxis, lognorm_xaxis, lognorm_yaxis, weibull_xaxis, weibull_yaxis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's get some data using `tsgettoolbox`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/var/home/timcera/published/tsgettoolbox/src/tsgettoolbox/utils.py:30: UserWarning: netCDF4 module not installed. Will be unable to handle NetCDF returns from NCSS.\n",
      "  from siphon.ncss import NCSS\n",
      "/var/home/timcera/published/tsgettoolbox/src/tsgettoolbox/utils.py:30: UserWarning: netCDF4 module not installed. Will be unable to handle NetCDF returns from NCSS.\n",
      "  from siphon.ncss import NCSS\n"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tsgettoolbox wdfn_daily --monitoring_location_id=\"USGS-02233484\" --time=\"2000-01-01/..\" > 02233484.csv\n",
    "tsgettoolbox wdfn_daily --monitoring_location_id=\"USGS-02233500\" --time=\"2000-01-01/..\" > 02233500.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the top of the data files using the 'head' command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==> 02233484.csv <==xec!\u001b\\\n",
      "Datetime,USGS-02233484_00060_00003:ft^3/s,USGS-02233484_00065_00003:ft,USGS-02233484_63160_00003:ft\n",
      "2001-12-04,111,11.73,\n",
      "2001-12-05,109,11.69,\n",
      "2001-12-06,106,11.65,\n",
      "2001-12-07,105,11.64,\n",
      "2001-12-08,106,11.65,\n",
      "2001-12-09,104,11.63,\n",
      "2001-12-10,105,11.65,\n",
      "2001-12-11,108,11.69,\n",
      "2001-12-12,108,11.69,\n",
      "\n",
      "==> 02233500.csv <==\n",
      "Datetime,USGS-02233500_00060_00003:ft^3/s,USGS-02233500_00065_00003:ft,USGS-02233500_63160_00003:ft\n",
      "2000-01-01,179,3.54,\n",
      "2000-01-02,172,3.44,\n",
      "2000-01-03,164,3.35,\n",
      "2000-01-04,159,,\n",
      "2000-01-05,152,,\n",
      "2000-01-06,148,,\n",
      "2000-01-07,143,3.04,\n",
      "2000-01-08,140,2.98,\n",
      "2000-01-09,139,2.94,\n"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head 02233484.csv 02233500.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the \"--columns LIST\" option, you can use the column numbers (data columns start at 1) or the column names.  The following example also illustrates how 'tstoolbox read ...' can be used to combine data sets, with the data piped to 'tstoolbox plot ...'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default 'tstoolbox plot ...' is a time-series plot where the datetime column is the x-axis, and all data columns are plotted on the y-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b]666;vte.shell.preexec!\u001b\\"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tstoolbox read 02233484.csv,02233500.csv | plottoolbox time --ofilename econ.png --columns USGS-02233484_00060_00003:ft^3/s,USGS-02233500_00060_00003:ft^3/s --ytitle BFlow "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![econ.png](econ.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 'norm_xaxis', 'norm_yaxis', 'lognorm_xaxis', 'lognorm_yaxis', 'weibull_xaxis', and 'weibull_yaxis' plot each data column against a transformed, sorted, ranking of the data.\n",
    "\n",
    "For all of these plot types the datetime column is ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b]666;vte.shell.preexec!\u001b\\"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tstoolbox read 02233484.csv,02233500.csv | plottoolbox norm_xaxis --plotting_position weibull --ofilename weibull.png --columns USGS-02233484_00060_00003:ft^3/s,USGS-02233500_00060_00003:ft^3/s --ytitle Flow "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![weibull.png](weibull.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following illustrates how to use the '--start_date ISO8601' and '--end_date ISO8601' to limit the x-axis of the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b]666;vte.shell.preexec!\u001b\\"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tstoolbox read 02233484.csv,02233500.csv | plottoolbox time --start_date 2009-01-01 --end_date 2010-01-01 --ofilename econ_clip.png --columns USGS-02233484_00060_00003:ft^3/s,USGS-02233500_00060_00003:ft^3/s --ytitle Flow "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![econ_clip.png](econ_clip.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The '--type xy' plot requires the data columns to be arranged 'x1,y1,x2,y2,x3,y3,...'.  You can use the '--columnns LIST' option to rearrange or duplicate columns as needed.  For example, if you have your data setup as 'x,y1,y2,y3,...', you can rearrage to what the '--type xy' requires by '--columns 1,2,1,3,1,4,...'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b]666;vte.shell.preexec!\u001b\\"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tstoolbox read 02233484.csv,02233500.csv | plottoolbox xy --ofilename econ_xy.png --columns 2,1,5,4 --linestyle ' ' --markerstyle auto"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![econ_xy.png](econ_xy.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b]666;vte.shell.preexec!\u001b\\"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "tstoolbox read 02233484.csv,02233500.csv | plottoolbox boxplot --ofilename boxplot.png --columns 1,4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![boxplot.png](boxplot.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The '--type heatmap' plot only work with a single, daily time-series.  You can pick the series you want to plot using the '--columns INTEGER' option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TypeError: Image data of dtype object cannot be converted to float\n"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "plottoolbox heatmap --columns 1 --ofilename heatmap.png < 02233484.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![heatmap.png](heatmap.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The '--type kde_time' plot will make a time-series plot combined with a probability density plot.  The probablity density plot is estimated using Kernel Density Estimation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b]666;vte.shell.preexec!\u001b\\"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "plottoolbox kde_time --columns 1 --ofilename kde_time.png < 02233484.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![ked_time](kde_time.png)"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,sh:percent"
  },
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}