# WLTP [![JupyterLab for WLTP](https://mybinder.org/badge_logo.svg)][1] Notebooks for processing UNECE's WLTP reference implementations in Python and MSAccess DB [TOC] ## Contents - Python code implementing the WLTP algorithm (CURRENTLY PARTLY IMPLEMENTED `master` branch). - The original files of the algo (document & MSAccess db). - Notebooks to process the vehicle data, build HDF5-dbm launch the algo, etc: RunVehicle.ipynb # sample code to run sample vehicle data Cycler.ipynb # (WIP) run a vehicle from the h5db, step-by-step. CarsDB-compare.ipynb # compare results stored h5db with AccDb CarsDB-msaccess.ipynb # populate h5db with AccDB test-car input & output CarsDB-pyalgo.ipynb # populate h5db with results from the old python code HDF5-APIh-help.ipynb # list useful HDF5 methods VMax.ipynb # VMax experiments Rounding.ipynb # showcase ROUNDING across MSAccess, Python, Matlab & C# ../tests/vehdb.py # support code for reading & validating vehicles from HD5 files *.py # "py:percent" git-tracked files paired with *.ipynb notebooks VehData/ # folder with raw input & results (e.g AccDB) +--WltpGS-msaccess.h5 # the h5db storing *accdb* inputs & results +--WltpGS-pyalgo.h5 # the h5db storing *pyalgo* results AccDB_src/ # AccDB code & queries extracted and stored as text binder/ # configurations for launching demo in https://mybinder.org conda/ +--environment.yaml # reproduce this conda-env with: `conda create -f conda/environment.yaml` README.md # this file > **Note:** > It is the `*.py` files that are stored in the git-repo of this project; > they are synced automatically from `.ipynb` notebooks by the *jupytext* > jupyter-lab extension. ## HDF5 The 2019 reference implementation is written in `WLTP_GS_calculation_15032019_for_prog_code_subgroup.accdb` *MSAccess* MSAccess database, which facilitates the development & execution of the algorithm by storing any and all data needed during those phases. To substitute those facilities i used the *pandas*+*HDF5* file-format to store data, persistently, across code runs. - Help on `pandas` HDF5 facilities: https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-hdf5 - Help on the underlying `tables` library: https://www.pytables.org/ - Use []`vitables` GUI application](http://vitables.org/Screenshots/) to inspect the file (install it with: `conda install vitables`). - All major API methods of those libraries are listed at the top of the `CarsDB.ipynb` notebook. ### HDF5 Trees Every `VehData/WltpGS-*.h5` has this structure: vehicles/ +--v001/ the case (not vehicle_num) | +--prop (INP/OUT, series) all kv-pairs from input-specs & scalar outputs from accdb | +--wot (INPUT, df) the wot for the vehicle of the case, index by n | +--cycle (OUTPUT, df) cycle-run generated by AccDB/Python, etc | +--override (INP, series) pyalgo must also override these kv-pairs +... Only the `VehData/WltpGS-msaccess.h5` contains all `/vehicles/001/{prop,wot,override}`; the rest contain just `/vehicles/001/{cycle,ospec}`, since they reuse `prop` & `wot` from *msaccess to produce their outputs. ## Install and launch the *notebooks* These notebooks run in a *Jupyter-lab server*. There are many ways to install and launch such a server. - Launch [a *live* demo of this repository in *mybinder.org*][1]. - Use *pip*, assuming working Python-3.6+ & NodeJS (easier under *Linux*): pip install -r ../binder/requirements.txt and then follow the sections below from *step-3*. - Use a *conda environment*, as detailed in the next sections: ### 0. Install the *base* conda Install [*miniconda3*](https://docs.conda.io/en/latest/miniconda.html) (or the full *Anaconda* distribution), if you don't yet have it. Instructions vary for your Operating System. ### 1. Create a new conda environment Reproduce the *exact same* conda-env that will be called *jupyter*:: conda env create -f conda/environment.yaml conda activate jupyter > **Tip:** > Use these commands to create an environment with the latest necessary packages > (without pinned versions):: > > conda create -n jupyter > conda activate jupyter > conda install --override-channels -c ankostis -c conda-forge -c defaults --file conda/conda-reqs.txt > pip install blackcellmagic nb-black columnize > > **Note (to the maintainer):** > Maintain the env-file by running this terminal command after any program (un)install:: > > conda env export -n jupyter | grep -vE 'nodejs|m2-' > conda/environment.yaml On *Windows* you need additionally *Git* and *NodeJS*. If you don't have these, you may install them also through *conda* system:: conda install m2-git nodejs m2-base (`m2-base` contains optional Unix command-line utilities) #### 2. Clone the git repo with the Notebooks If you don't yet have the sources of this repository on your machine, change to a directory you want work, and run these commands:: git clone http://github.com/JRCSTU/wltp wltp.git --branch=jupyter --depth=1 cd wltp.git ### 3. Enable *Jupyterlab* extensions > (working `nodejs` & `npm` commands are needed there) Install widgets extension and rebuild the jupyterlab (if you skip this, jupyter will ask you to *BUILD* on your first launch): ## Once for all extensions of a jupyter installation jupyter nbextension enable --py --sys-prefix widgetsnbextension # for jupyter-notebook server jupyter labextension install @jupyter-widgets/jupyterlab-manager # for jupyter-lab server Alternatively, you may follow the explicit instructions of each extension, linked site below: - [`jupytext`](https://github.com/mwouts/jupytext): needed to convert git-cloned files into *notebooks*. As a quick reference, do these steps: - You need the jupyterlab config-file; run this to generate it, if it does not exist:: jupyter lab --generate-config - Append this to the bottom of your jupyter-config file:: ## Git only "py:percent" diffs # From https://towardsdatascience.com/version-control-with-jupyter-notebooks-f096f4d7035a c.NotebookApp.contents_manager_class="jupytext.TextFileContentsManager" c.ContentsManager.default_jupytext_formats = ".ipynb,.py" - [`qgrid`](https://github.com/quantopian/qgrid#installation):: jupyter nbextension enable --py --sys-prefix qgrid # for jupyter-notebook server jupyter labextension install qgrid # for jupyter-lab server - [`jupyter-matplotlib`](https://github.com/matplotlib/jupyter-matplotlib):: jupyter labextension install jupyter-matplotlib > **Tip:** > Make sure you restart the server after installing lab extensions - restarting > just the the kernel is not enough. ### 4. Convert git-cloned scripts into *notebooks* `jupytext` Run this command in the directory of the gir-repo you cloned:: jupytext --to ipynb *.py > **Tip**: > Alternatively, you may *right-click* a `.py` "py:percent" file and > click **Open with | Notebook**, and **Save** to generate a freshly created notebook > (and remember to close the `.py` file). ### 5. Launch *Jupyterlab* server Change directory to the folder you wish to store the notebooks, and launch jupyterlab with:: jupyter lab A new tab should open automatically in your browser. ## Code to extract data from MSAccess In case it's needed in the future ...but cannot run in this server bc `pyodbc` lib cannot runs on *Linux* :-( ```python ## Can run only on WINDOWS only!! # import pyodbc def mdb_connect(db_file, user='admin', password = '', old_driver=False): """ :param db_file: must be an absolute path """ driver_ver = '*.mdb' if not old_driver: driver_ver += ', *.accdb' odbc_conn_str = ('DRIVER={Microsoft Access Driver (%s)}' ';DBQ=%s;UID=%s;PWD=%s' % (driver_ver, db_file, user, password)) return pyodbc.connect(odbc_conn_str) fname='WLTP_GS_calculation_15032019_for_prog_code_subgroup.accdb' conn = mdb_connect(fname) crsr = conn.cursor() print([i.table_name for i in crsr.tables(tableType='TABLE')] print([i.column_name for i in crsr.columns('gearshift_table_all')] ``` [1]: https://mybinder.org/v2/gh/ankostis/wltp/master?urlpath=lab/tree/Notebooks/README.md