wdmtoolbox.wdmtoolbox.csvtowdm

wdmtoolbox.wdmtoolbox.csvtowdm(wdmpath, dsn, start_date=None, end_date=None, columns=None, force_freq=None, groupby=None, round_index=None, clean=False, target_units=None, source_units=None, input_ts='-')

Write data from a CSV file to a DSN.

File can have comma separated ‘year’, ‘month’, ‘day’, ‘hour’, ‘minute’, ‘second’, ‘value’ OR ‘date/time string’, ‘value’

Parameters:

• wdmpath (str) – Path and WDM filename.

• dsn (int) – The Data Set Number (DSN) for the time series in the WDM file. This number must be greater or equal to 1 and less than or equal to 32000. HSPF can only use for input or output DSNs of 1 to 9999, inclusive.

• input_ts (str) –

  [optional though required if using within Python, default is ‘-’ (stdin)]

  Whether from a file or standard input, data requires a single line header of column names. The default header is the first line of the input, but this can be changed for CSV files using the ‘skiprows’ option.

  Most common date formats can be used, but the closer to ISO 8601 date/time standard the better.

  Comma-separated values (CSV) files or tab-separated values (TSV):

  File separators will be automatically detected.
  
  Columns can be selected by name or index, where the index for
  data columns starts at 1.
  

  Command line examples:

  Keyword Example	Description
  –input_ts=fn.csv	read all columns from ‘fn.csv’
  –input_ts=fn.csv,2,1	read data columns 2 and 1 from ‘fn.csv’
  –input_ts=fn.csv,2,skiprows=2	read data column 2 from ‘fn.csv’, skipping first 2 rows so header is read from third row
  –input_ts=fn.xlsx,2,Sheet21	read all data from 2nd sheet all data from “Sheet21” of ‘fn.xlsx’
  –input_ts=fn.hdf5,Table12,T2	read all data from table “Table12” then all data from table “T2” of ‘fn.hdf5’
  –input_ts=fn.wdm,210,110	read DSNs 210, then 110 from ‘fn.wdm’
  –input_ts=’-’	read all columns from standard input (stdin)
  –input_ts=’-’ –columns=4,1	read column 4 and 1 from standard input (stdin)

  If working with CSV or TSV files you can use redirection rather than use –input_ts=fname.csv. The following are identical:

  From a file:

  command subcmd –input_ts=fname.csv

  From standard input (since ‘–input_ts=-’ is the default:

  command subcmd < fname.csv

  Can also combine commands by piping:

  command subcmd < filein.csv | command subcmd1 > fileout.csv

  Python library examples:

  You must use the `input_ts=...` option where `input_ts` can be
  one of a [pandas DataFrame, pandas Series, dict, tuple, list,
  StringIO, or file name].
  

• start_date (str) –

  [optional, defaults to first date in time-series, input filter]

  The start_date of the series in ISOdatetime format, or ‘None’ for beginning.

• end_date (str) –

  [optional, defaults to last date in time-series, input filter]

  The end_date of the series in ISOdatetime format, or ‘None’ for end.

• columns –

  [optional, defaults to all columns, input filter]

  Columns to select out of input. Can use column names from the first line header or column numbers. If using numbers, column number 1 is the first data column. To pick multiple columns; separate by commas with no spaces. As used in toolbox_utils pick command.

  This solves a big problem so that you don’t have to create a data set with a certain column order, you can rearrange columns when data is read in.

• force_freq (str) –

  [optional, output format]

  Force this frequency for the output. Typically you will only want to enforce a smaller interval where toolbox_utils will insert missing values as needed. WARNING: you may lose data if not careful with this option. In general, letting the algorithm determine the frequency should always work, but this option will override. Use PANDAS offset codes.

• groupby (str) –

  [optional, default is None, transformation]

  The pandas offset code to group the time-series data into. A special code is also available to group ‘months_across_years’ that will group into twelve monthly categories across the entire time-series.

• round_index –

  [optional, default is None which will do nothing to the index, output format]

  Round the index to the nearest time point. Can significantly improve the performance since can cut down on memory and processing requirements, however be cautious about rounding to a very course interval from a small one. This could lead to duplicate values in the index.

• clean –

  [optional, default is False, input filter]

  The ‘clean’ command will repair a input index, removing duplicate index values and sorting.

• target_units (str) –

  [optional, default is None, transformation]

  The purpose of this option is to specify target units for unit conversion. The source units are specified in the header line of the input or using the ‘source_units’ keyword.

  The units of the input time-series or values are specified as the second field of a ‘:’ delimited name in the header line of the input or in the ‘source_units’ keyword.

  Any unit string compatible with the ‘pint’ library can be used.

  This option will also add the ‘target_units’ string to the column names.

• source_units (str) –

  [optional, default is None, transformation]

  If unit is specified for the column as the second field of a ‘:’ delimited column name, then the specified units and the ‘source_units’ must match exactly.

  Any unit string compatible with the ‘pint’ library can be used.