{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# tsgettoolbox and tstoolbox - Command Line Interface\n",
    "## 'tsgettoolbox nwis ...': Download data from the National Water Information System (NWIS)\n",
    "This notebook is to illustrate the command line usage for 'tsgettoolbox' and 'tstoolbox' to download and work with data from the National Water Information System (NWIS).  There is a different notebook to do the same thing from within a Python program called tsgettoolbox-nwis-api.\n",
    "\n",
    "First off, always nice to remind myself about the options.  Each sub-command has their own options kept consistent with the options available from the source service.  The way that NWIS works is you have one major filter and one or more minor filters to define what sites you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsgettoolbox nwis_dv --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say that I want flow (parameterCd=00060) for site '02325000'.  I first make sure that I am getting what I want by allowing the output to be printed to the screen.  Note the pipe ('|') that directs output to the 'head' command to display the top 10 lines of the time-series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datetime,USGS-02325000-00060\r\n",
      "2000-01-01,82\r\n",
      "2000-01-02,81\r\n",
      "2000-01-03,80\r\n",
      "2000-01-04,79\r\n",
      "2000-01-05,75\r\n",
      "2000-01-06,75\r\n",
      "2000-01-07,74\r\n",
      "2000-01-08,73\r\n",
      "2000-01-09,75\r\n"
     ]
    }
   ],
   "source": [
    "tsgettoolbox nwis_dv --sites 02325000 --startDT '2000-01-01' --parameterCd 00060 | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then I redirect to a file with \"> filename.csv\" so that I don't have to wait for the USGS NWIS services for the remaining work or analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "tsgettoolbox nwis_dv --sites 02325000 --startDT '2000-01-01' --parameterCd 00060 > 02325000_flow.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 'tstoolbox ...': Process data using 'tstoolbox'\n",
    "Now lets use \"tstoolbox\" to plot the time-series.  Note the redirection again, this time for input as \"< filename.csv\".  Default plot filename is \"plot.png\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "tstoolbox plot < 02325000_flow.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](plot.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'tstoolbox plot' has many options that can be used to modify the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: tstoolbox plot [-h] [--ofilename <str>] [--type <str>] [--xtitle <str>]\r\n",
      "  [--ytitle <str>] [--title <str>] [--figsize <str>] [--legend LEGEND]\r\n",
      "  [--legend_names <str>] [--subplots] [--sharex] [--sharey] [--style <str>]\r\n",
      "  [--logx] [--logy] [--xaxis <str>] [--yaxis <str>] [--xlim XLIM] [--ylim\r\n",
      "  YLIM] [--secondary_y] [--mark_right] [--scatter_matrix_diagonal <str>]\r\n",
      "  [--bootstrap_size BOOTSTRAP_SIZE] [--bootstrap_samples BOOTSTRAP_SAMPLES]\r\n",
      "  [--norm_xaxis] [--norm_yaxis] [--lognorm_xaxis] [--lognorm_yaxis]\r\n",
      "  [--xy_match_line <str>] [--grid GRID] [-i <str>] [-s <str>] [-e <str>]\r\n",
      "  [--label_rotation <int>] [--label_skip <int>] [--force_freq FORCE_FREQ]\r\n",
      "  [--drawstyle <str>] [--por] [--columns COLUMNS] [--invert_xaxis]\r\n",
      "  [--invert_yaxis] [--plotting_position <str>]\r\n",
      "\r\n",
      "Plot data.\r\n",
      "\r\n",
      "optional arguments:\r\n",
      "  -h | --help\r\n",
      "      show this help message and exit\r\n",
      "  --ofilename <str>\r\n",
      "      Output filename for the plot. Extension defines the type, ('.png').\r\n",
      "      Defaults to 'plot.png'.\r\n",
      "  --type <str>\r\n",
      "      The plot type. Defaults to 'time'.\r\n",
      "      Can be one of the following:\r\n",
      "      time\r\n",
      "        standard time series plot\r\n",
      "\r\n",
      "      xy\r\n",
      "        (x,y) plot, also know as a scatter plot\r\n",
      "\r\n",
      "      double_mass\r\n",
      "        (x,y) plot of the cumulative sum of x and y\r\n",
      "\r\n",
      "      boxplot\r\n",
      "        box extends from lower to upper quartile, with line at the median.\r\n",
      "        Depending on the statistics, the wiskers represent the range of\r\n",
      "        the data or 1.5 times the inter-quartile range (Q3 - Q1)\r\n",
      "\r\n",
      "      scatter_matrix\r\n",
      "        plots all columns against each other\r\n",
      "\r\n",
      "      lag_plot\r\n",
      "        indicates structure in the data\r\n",
      "\r\n",
      "      autocorrelation\r\n",
      "        plot autocorrelation\r\n",
      "\r\n",
      "      bootstrap\r\n",
      "        visually asses aspects of a data set by plotting random selections of\r\n",
      "        values\r\n",
      "\r\n",
      "      probability_density\r\n",
      "        sometime called kernel density estimation (KDE)\r\n",
      "\r\n",
      "      bar\r\n",
      "        sometimes called a column plot\r\n",
      "\r\n",
      "      barh\r\n",
      "        a horizontal bar plot\r\n",
      "\r\n",
      "      bar_stacked\r\n",
      "        sometimes called a stacked column\r\n",
      "\r\n",
      "      barh_stacked\r\n",
      "        a horizontal stacked bar plot\r\n",
      "\r\n",
      "      histogram\r\n",
      "        calculate and create a histogram plot\r\n",
      "\r\n",
      "      norm_xaxis\r\n",
      "        sort, calculate probabilities, and plot data against an x axis normal\r\n",
      "        distribution\r\n",
      "\r\n",
      "      norm_yaxis\r\n",
      "        sort, calculate probabilities, and plot data against an y axis normal\r\n",
      "        distribution\r\n",
      "\r\n",
      "      lognorm_xaxis\r\n",
      "        sort, calculate probabilities, and plot data against an x axis lognormal\r\n",
      "        distribution\r\n",
      "\r\n",
      "      lognorm_yaxis\r\n",
      "        sort, calculate probabilities, and plot data against an y axis lognormal\r\n",
      "        distribution\r\n",
      "\r\n",
      "      weibull_xaxis\r\n",
      "        sort, calculate and plot data against an x axis weibull distribution\r\n",
      "\r\n",
      "      weibull_yaxis\r\n",
      "        sort, calculate and plot data against an y axis weibull distribution\r\n",
      "\r\n",
      "  --xtitle <str>\r\n",
      "      Title of x-axis, default depend on type.\r\n",
      "  --ytitle <str>\r\n",
      "      Title of y-axis, default depend on type.\r\n",
      "  --title <str>\r\n",
      "      Title of chart, defaults to ''.\r\n",
      "  --figsize <str>\r\n",
      "      The 'width,height' of plot as inches. Defaults to '10,6.5'.\r\n",
      "  --legend LEGEND\r\n",
      "      Whether to display the legend. Defaults to True.\r\n",
      "  --legend_names <str>\r\n",
      "      Legend would normally use the time-series names associated with the input\r\n",
      "      data. The 'legend_names' option allows you to override the names in\r\n",
      "      the data set. You must supply a comma separated list of strings for\r\n",
      "      each time-series in the data set. Defaults to None.\r\n",
      "  --subplots\r\n",
      "      boolean, default False. Make separate subplots for each time series\r\n",
      "  --sharex\r\n",
      "      boolean, default True In case subplots=True, share x axis\r\n",
      "  --sharey\r\n",
      "      boolean, default False In case subplots=True, share y axis\r\n",
      "  --style <str>\r\n",
      "      Comma separated matplotlib style strings matplotlib line style per\r\n",
      "      time-series. Just combine codes in 'ColorLineMarker' order, for\r\n",
      "      example r--* is a red dashed line with star marker.\r\n",
      "      ┌──────┬─────────┐\r\n",
      "      │ Code │ Color   │\r\n",
      "      ├──────┼─────────┤\r\n",
      "      │ b    │ blue    │\r\n",
      "      ├──────┼─────────┤\r\n",
      "      │ g    │ green   │\r\n",
      "      ├──────┼─────────┤\r\n",
      "      │ r    │ red     │\r\n",
      "      ├──────┼─────────┤\r\n",
      "      │ c    │ cyan    │\r\n",
      "      ├──────┼─────────┤\r\n",
      "      │ m    │ magenta │\r\n",
      "      ├──────┼─────────┤\r\n",
      "      │ y    │ yellow  │\r\n",
      "      ├──────┼─────────┤\r\n",
      "      │ k    │ black   │\r\n",
      "      ├──────┼─────────┤\r\n",
      "      │ w    │ white   │\r\n",
      "      ╘══════╧═════════╛\r\n",
      "\r\n",
      "      ┌─────────┬───────────┐\r\n",
      "      │ Number  │ Color     │\r\n",
      "      ├─────────┼───────────┤\r\n",
      "      │ 0.75    │ 0.75 gray │\r\n",
      "      ├─────────┼───────────┤\r\n",
      "      │ ...etc. │           │\r\n",
      "      ╘═════════╧═══════════╛\r\n",
      "\r\n",
      "      ┌──────────────────┐\r\n",
      "      │ HTML Color Names │\r\n",
      "      ├──────────────────┤\r\n",
      "      │ red              │\r\n",
      "      ├──────────────────┤\r\n",
      "      │ burlywood        │\r\n",
      "      ├──────────────────┤\r\n",
      "      │ chartreuse       │\r\n",
      "      ├──────────────────┤\r\n",
      "      │ ...etc.          │\r\n",
      "      ╘══════════════════╛\r\n",
      "\r\n",
      "      Color reference: <\u001b[4;36mhttp://matplotlib.org/api/colors_api.html\u001b[0m>\r\n",
      "      ┌──────┬──────────────┐\r\n",
      "      │ Code │ Lines        │\r\n",
      "      ├──────┼──────────────┤\r\n",
      "      │ •    │ solid        │\r\n",
      "      ├──────┼──────────────┤\r\n",
      "      │ --   │ dashed       │\r\n",
      "      ├──────┼──────────────┤\r\n",
      "      │ -.   │ dash_dot     │\r\n",
      "      ├──────┼──────────────┤\r\n",
      "      │ :    │ dotted       │\r\n",
      "      ├──────┼──────────────┤\r\n",
      "      │ None │ draw nothing │\r\n",
      "      ├──────┼──────────────┤\r\n",
      "      │ ' '  │ draw nothing │\r\n",
      "      ├──────┼──────────────┤\r\n",
      "      │ ''   │ draw nothing │\r\n",
      "      ╘══════╧══════════════╛\r\n",
      "\r\n",
      "      Line reference: <\u001b[4;36mhttp://matplotlib.org/api/artist_api.html\u001b[0m>\r\n",
      "      ┌──────┬────────────────┐\r\n",
      "      │ Code │ Markers        │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ .    │ point          │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ o    │ circle         │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ v    │ triangle down  │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ ^    │ triangle up    │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ <    │ triangle left  │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ >    │ triangle right │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ 1    │ tri_down       │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ 2    │ tri_up         │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ 3    │ tri_left       │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ 4    │ tri_right      │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ 8    │ octagon        │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ s    │ square         │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ p    │ pentagon       │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ •    │ star           │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ h    │ hexagon1       │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ H    │ hexagon2       │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ •    │ plus           │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ x    │ x              │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ D    │ diamond        │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ d    │ thin diamond   │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ _    │ hline          │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ None │ nothing        │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ ' '  │ nothing        │\r\n",
      "      ├──────┼────────────────┤\r\n",
      "      │ ''   │ nothing        │\r\n",
      "      ╘══════╧════════════════╛\r\n",
      "\r\n",
      "      Marker reference: <\u001b[4;36mhttp://matplotlib.org/api/markers_api.html\u001b[0m>\r\n",
      "  --logx\r\n",
      "      DEPRECATED: use '--xaxis=\"log\"' instead.\r\n",
      "  --logy\r\n",
      "      DEPRECATED: use '--yaxis=\"log\"' instead.\r\n",
      "  --xaxis <str>\r\n",
      "      Defines the type of the xaxis. One of 'arithmetic', 'log'. Default is\r\n",
      "      'arithmetic'.\r\n",
      "  --yaxis <str>\r\n",
      "      Defines the type of the yaxis. One of 'arithmetic', 'log'. Default is\r\n",
      "      'arithmetic'.\r\n",
      "  --xlim XLIM\r\n",
      "      Comma separated lower and upper limits (--xlim 1,1000) Limits for the\r\n",
      "      x-axis. Default is based on range of x values.\r\n",
      "  --ylim YLIM\r\n",
      "      Comma separated lower and upper limits (--ylim 1,1000) Limits for the\r\n",
      "      y-axis. Default is based on range of y values.\r\n",
      "  --secondary_y\r\n",
      "      Boolean or sequence, default False Whether to plot on the secondary y-axis\r\n",
      "      If a list/tuple, which time-series to plot on secondary y-axis\r\n",
      "  --mark_right\r\n",
      "      Boolean, default True : When using a secondary_y axis, should the legend\r\n",
      "      label the axis of the various time-series automatically\r\n",
      "  --scatter_matrix_diagonal <str>\r\n",
      "      If plot type is 'scatter_matrix', this specifies the plot along the\r\n",
      "      diagonal. Defaults to 'probability_density'.\r\n",
      "  --bootstrap_size BOOTSTRAP_SIZE\r\n",
      "      The size of the random subset for 'bootstrap' plot. Defaults to 50.\r\n",
      "  --bootstrap_samples BOOTSTRAP_SAMPLES\r\n",
      "      The number of random subsets of 'bootstrap_size'. Defaults to 500.\r\n",
      "  --norm_xaxis\r\n",
      "      DEPRECATED: use '--type=\"norm_xaxis\"' instead.\r\n",
      "  --norm_yaxis\r\n",
      "      DEPRECATED: use '--type=\"norm_yaxis\"' instead.\r\n",
      "  --lognorm_xaxis\r\n",
      "      DEPRECATED: use '--type=\"lognorm_xaxis\"' instead.\r\n",
      "  --lognorm_yaxis\r\n",
      "      DEPRECATED: use '--type=\"lognorm_yaxis\"' instead.\r\n",
      "  --xy_match_line <str>\r\n",
      "      Will add a match line where x == y. Default is ''. Set to a line style\r\n",
      "      code.\r\n",
      "  --grid GRID\r\n",
      "      Boolean, default True Whether to plot grid lines on the major ticks.\r\n",
      "  -i <str> | --input_ts <str>\r\n",
      "      Filename with data in 'ISOdate,value' format or '-' for stdin.\r\n",
      "  -s <str> | --start_date <str>\r\n",
      "      The start_date of the series in ISOdatetime format, or 'None' for\r\n",
      "      beginning.\r\n",
      "  -e <str> | --end_date <str>\r\n",
      "      The end_date of the series in ISOdatetime format, or 'None' for end.\r\n",
      "  --label_rotation <int>\r\n",
      "      Rotation for major labels for bar plots.\r\n",
      "  --label_skip <int>\r\n",
      "      Skip for major labels for bar plots.\r\n",
      "  --force_freq FORCE_FREQ\r\n",
      "      Force this frequency for the plot. WARNING: you may lose data if not\r\n",
      "      careful with this option. In general, letting the algorithm\r\n",
      "      determine the frequency should always work, but this option will\r\n",
      "      override. Use PANDAS offset codes,\r\n",
      "  --drawstyle <str>\r\n",
      "      'default' connects the points with lines. The steps variants produce\r\n",
      "      step-plots. 'steps' is equivalent to 'steps-pre' and is maintained\r\n",
      "      for backward-compatibility. ACCEPTS:\r\n",
      "      ['default' | 'steps' | 'steps-pre' | 'steps-mid' | 'steps-post']\r\n",
      "\r\n",
      "  --por\r\n",
      "      Plot from first good value to last good value. Strip NANs from beginning\r\n",
      "      and end.\r\n",
      "  --columns COLUMNS\r\n",
      "      Columns to pick out of input. Can use column names or column numbers. If\r\n",
      "      using numbers, column number 1 is the first data column. To pick\r\n",
      "      multiple columns; separate by commas with no spaces. As used in\r\n",
      "      'pick' command.\r\n",
      "  --invert_xaxis\r\n",
      "      Invert the x-axis.\r\n",
      "  --invert_yaxis\r\n",
      "      Invert the y-axis.\r\n",
      "  --plotting_position <str>\r\n",
      "      'weibull', 'benard', 'tukey', 'gumbel', 'hazen', 'cunnane', or\r\n",
      "      'california'. The default is 'weibull'.\r\n",
      "      ┌────────────┬─────────────────┬───────────────────────┐\r\n",
      "      │ weibull    │ i/(n+1)         │ mean of sampling      │\r\n",
      "      │            │                 │ distribution          │\r\n",
      "      ├────────────┼─────────────────┼───────────────────────┤\r\n",
      "      │ benard     │ (i-0.3)/(n+0.4) │ approx. median of     │\r\n",
      "      │            │                 │ sampling distribution │\r\n",
      "      ├────────────┼─────────────────┼───────────────────────┤\r\n",
      "      │ tukey      │ (i-1/3)/(n+1/3) │ approx. median of     │\r\n",
      "      │            │                 │ sampling distribution │\r\n",
      "      ├────────────┼─────────────────┼───────────────────────┤\r\n",
      "      │ gumbel     │ (i-1)/(n-1)     │ mode of sampling      │\r\n",
      "      │            │                 │ distribution          │\r\n",
      "      ├────────────┼─────────────────┼───────────────────────┤\r\n",
      "      │ hazen      │ (i-1/2)/n       │ midpoints of n equal  │\r\n",
      "      │            │                 │ intervals             │\r\n",
      "      ├────────────┼─────────────────┼───────────────────────┤\r\n",
      "      │ cunnane    │ (i-2/5)/(n+1/5) │ subjective            │\r\n",
      "      ├────────────┼─────────────────┼───────────────────────┤\r\n",
      "      │ california │ i/n             │                       │\r\n",
      "      ╘════════════╧═════════════════╧═══════════════════════╛\r\n",
      "\r\n",
      "      Where 'i' is the sorted rank of the y value, and 'n' is the total number\r\n",
      "      of values to be plotted.\r\n",
      "      Only used for norm_xaxis, norm_yaxis, lognorm_xaxis, lognorm_yaxis,\r\n",
      "      weibull_xaxis, and weibull_yaxis.\r\n"
     ]
    }
   ],
   "source": [
    "tstoolbox plot --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "tstoolbox plot --ofilename flow.png --ytitle 'Flow (cfs)' --title '02325000: FENHOLLOWAY RIVER NEAR PERRY, FLA' --legend False < 02325000_flow.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](flow.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monthly Average Flow\n",
    "You can also use tstoolbox to make calculations on the time-series, for example to aggregate to monthly average flow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datetime,USGS-02325000-00060_mean\r\n",
      "2000-01-31,80\r\n",
      "2000-02-29,89.7931\r\n",
      "2000-03-31,80.0323\r\n",
      "2000-04-30,81.7667\r\n",
      "2000-05-31,90.8387\r\n",
      "2000-06-30,94.4\r\n",
      "2000-07-31,85.9032\r\n",
      "2000-08-31,83.0323\r\n",
      "2000-09-30,128.067\r\n"
     ]
    }
   ],
   "source": [
    "tstoolbox aggregate --groupby M --statistic mean < 02325000_flow.csv | head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "tstoolbox aggregate --groupby M --statistic mean < 02325000_flow.csv | tstoolbox plot --ofilename plot_monthly.png --drawstyle steps-pre"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](plot_monthly.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}