# WLTP
[![JupyterLab for WLTP](https://mybinder.org/badge_logo.svg)][1]

Notebooks for processing UNECE's WLTP reference implementations in Python and MSAccess DB


[TOC]


## Contents

- Python code implementing the WLTP algorithm (CURRENTLY PARTLY IMPLEMENTED `master` branch).
- The original files of the algo (document & MSAccess db).
- Notebooks to process the vehicle data, build HDF5-dbm launch the algo, etc:

      RunVehicle.ipynb            # sample code to run sample vehicle data
      Cycler.ipynb                # (WIP) run a vehicle from the h5db, step-by-step.
      CarsDB-compare.ipynb        # compare results stored h5db with AccDb
      CarsDB-msaccess.ipynb       # populate h5db with AccDB test-car input & output
      CarsDB-pyalgo.ipynb         # populate h5db with results from the old python code
      HDF5-APIh-help.ipynb        # list useful HDF5 methods
      VMax.ipynb                  # VMax experiments
      Rounding.ipynb              # showcase ROUNDING across MSAccess, Python, Matlab & C#
      ../tests/vehdb.py           # support code for reading & validating vehicles from HD5 files
      *.py                        # "py:percent" git-tracked files paired with *.ipynb notebooks
      VehData/                    # folder with raw input & results (e.g AccDB)
          +--WltpGS-msaccess.h5   # the h5db storing *accdb* inputs & results
          +--WltpGS-pyalgo.h5     # the h5db storing *pyalgo* results
      AccDB_src/                  # AccDB code & queries extracted and stored as text
      binder/                     # configurations for launching demo in https://mybinder.org
      conda/
          +--environment.yaml     # reproduce this conda-env with: `conda create -f conda/environment.yaml`
      README.md                   # this file

> **Note:**
> It is the `*.py` files that are stored in the git-repo of this project;
> they are synced automatically from `.ipynb` notebooks by the *jupytext*
> jupyter-lab extension.


## HDF5

The 2019 reference implementation is written in `WLTP_GS_calculation_15032019_for_prog_code_subgroup.accdb`
*MSAccess* MSAccess database, which facilitates the development & execution of the algorithm
by storing any and all data needed during those phases.

To substitute those facilities i used the *pandas*+*HDF5* file-format to store data,
persistently, across code runs.

- Help on `pandas` HDF5 facilities: https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-hdf5
- Help on the underlying `tables` library: https://www.pytables.org/
- Use []`vitables` GUI application](http://vitables.org/Screenshots/)
  to inspect the file (install it with: `conda install vitables`).
- All major API methods of those libraries are listed  at the top
  of the `CarsDB.ipynb` notebook.

### HDF5 Trees

Every `VehData/WltpGS-*.h5` has this structure:

    vehicles/
        +--v001/            the case (not vehicle_num)
        |   +--prop         (INP/OUT, series) all kv-pairs from input-specs & scalar outputs from accdb
        |   +--wot          (INPUT, df) the wot for the vehicle of the case, index by n
        |   +--cycle        (OUTPUT, df) cycle-run generated by AccDB/Python, etc
        |   +--override     (INP, series) pyalgo must also override these kv-pairs
        +...

Only the `VehData/WltpGS-msaccess.h5` contains all `/vehicles/001/{prop,wot,override}`;
the rest contain just `/vehicles/001/{cycle,ospec}`, since they reuse `prop` & `wot`
from *msaccess to produce their outputs.


## Install and launch the *notebooks*
These notebooks run in a *Jupyter-lab server*.
There are many ways to install and launch such a server.

- Launch [a *live* demo of this repository in *mybinder.org*][1].

- Use *pip*, assuming working Python-3.6+ & NodeJS (easier under *Linux*):

      pip install -r ../binder/requirements.txt

  and then follow the sections below from *step-3*.

- Use a *conda environment*, as detailed in the next sections:

### 0. Install the *base* conda
Install [*miniconda3*](https://docs.conda.io/en/latest/miniconda.html)
(or the full *Anaconda* distribution), if you don't yet have it.
Instructions vary for your Operating System.

### 1. Create a new conda environment
Reproduce the *exact same* conda-env that will be called *jupyter*::

    conda env create -f conda/environment.yaml
    conda activate jupyter

> **Tip:**
> Use these commands to create an environment with the latest necessary packages
> (without pinned versions)::
>
>     conda create   -n jupyter
>     conda activate jupyter
>     conda install  --override-channels -c ankostis -c conda-forge -c defaults --file conda/conda-reqs.txt
>     pip install    blackcellmagic nb-black columnize
>
> **Note (to the maintainer):**
> Maintain the env-file by running this terminal command after any program (un)install::
>
>     conda env export -n jupyter | grep -vE 'nodejs|m2-' > conda/environment.yaml


On *Windows* you need additionally *Git* and *NodeJS*.
If you don't have these, you may install them also through *conda* system::

    conda install m2-git nodejs m2-base

(`m2-base` contains optional Unix command-line utilities)


#### 2. Clone the git repo with the Notebooks

If you don't yet have the sources of this repository on your machine,
change to a directory you want work, and run these commands::

    git clone http://github.com/JRCSTU/wltp wltp.git --branch=jupyter --depth=1
    cd wltp.git


### 3. Enable *Jupyterlab* extensions

> (working `nodejs` & `npm` commands are needed there)

Install widgets extension and rebuild the jupyterlab
(if you skip this, jupyter will ask you to *BUILD* on your first launch):

    ## Once for all extensions of a jupyter installation
    jupyter nbextension enable --py --sys-prefix widgetsnbextension    # for jupyter-notebook server
    jupyter labextension install @jupyter-widgets/jupyterlab-manager   # for jupyter-lab server

Alternatively, you may follow the explicit instructions of each extension,
linked site below:

- [`jupytext`](https://github.com/mwouts/jupytext): needed to convert git-cloned files into *notebooks*.
  As a quick reference, do these steps:

  - You need the jupyterlab config-file; run this to generate it, if it does not exist::

        jupyter lab --generate-config

  - Append this to the bottom of your jupyter-config file::


        ## Git only "py:percent" diffs
        # From https://towardsdatascience.com/version-control-with-jupyter-notebooks-f096f4d7035a
        c.NotebookApp.contents_manager_class="jupytext.TextFileContentsManager"
        c.ContentsManager.default_jupytext_formats = ".ipynb,.py"

- [`qgrid`](https://github.com/quantopian/qgrid#installation)::

      jupyter nbextension enable --py --sys-prefix qgrid                 # for jupyter-notebook server
      jupyter labextension install qgrid                                 # for jupyter-lab server

- [`jupyter-matplotlib`](https://github.com/matplotlib/jupyter-matplotlib)::

      jupyter labextension install jupyter-matplotlib

> **Tip:**
> Make sure you restart the server after installing lab extensions - restarting
> just the the kernel is not enough.


### 4. Convert git-cloned scripts into *notebooks* `jupytext`

Run this command in the directory of the gir-repo you cloned::

      jupytext --to ipynb *.py

> **Tip**:
> Alternatively, you may *right-click* a `.py` "py:percent" file and
> click **Open with | Notebook**, and **Save** to generate a freshly created notebook
> (and remember to close the `.py` file).


### 5. Launch *Jupyterlab* server
Change directory to the folder you wish to store the notebooks,
and launch jupyterlab with::

    jupyter lab

A new tab should open automatically in your browser.


## Code to extract data from MSAccess

In case it's needed in the future
...but cannot run in this server bc `pyodbc` lib cannot runs on *Linux* :-(

```python
## Can run only on WINDOWS only!!
#
import pyodbc

def mdb_connect(db_file, user='admin', password = '', old_driver=False):
    """
    :param db_file:
        must be an absolute path
    """
    driver_ver = '*.mdb'
    if not old_driver:
        driver_ver += ', *.accdb'

    odbc_conn_str = ('DRIVER={Microsoft Access Driver (%s)}'
                     ';DBQ=%s;UID=%s;PWD=%s' %
                     (driver_ver, db_file, user, password))

    return pyodbc.connect(odbc_conn_str)

fname='WLTP_GS_calculation_15032019_for_prog_code_subgroup.accdb'
conn = mdb_connect(fname)
crsr = conn.cursor()

print([i.table_name for i in crsr.tables(tableType='TABLE')]
print([i.column_name for i in crsr.columns('gearshift_table_all')]
```

[1]: https://mybinder.org/v2/gh/ankostis/wltp/master?urlpath=lab/tree/Notebooks/README.md