Installing the beta "demeshened" versions of yt (yt-4.0) and trident

Author: Cameron Hummels

Updated: 3/19/2019

Note: If you only wish to install the demeshened version of yt (and not trident), you can stop after step 3.

Traditionally, yt interacts with particle-based datasets by depositing all of the particle fields to an octree-based grid, and then it samples this grid like it would any other grid-based dataset. Unfortunately, the grid-deposition step is costly both in terms of time and memory, and there can be some smoothing effects introduced to particle-based datasets in this process.

Recently, efforts have been made to change how yt interacts with particle-based datasets by avoiding this grid-deposition step altogether and interacting only with the particles themselves. Projections and rays are calculated by determining the line integral through the smoothing kernel of each particle along a line of sight, obviating the need for a grid structure at all. These efforts, known as "demeshening" have been led by Nathan Goldbaum, Meagan Lang, and Matthew Turk. Work by Bili Dong and Cameron Hummels has ensured that these improvements can be harnessed by Trident too. For more details on how demeshening works, please refer to the relevant YTEP https://ytep.readthedocs.io/en/latest/YTEPs/YTEP-0032.html and SciPY presentation https://www.youtube.com/watch?v=pkZgQIGac6I .

Because the demeshened version of yt has some backwards-incompatible changes, it will not go into the mainline of yt until version yt 4.0. Fortunately, it is now available for use in the yt-4.0 branch of the main yt repository. See below for more information about installing and use of yt-4.0 and the corresponding branch in Trident (sph-viz).

As an example of the difference between traditional yt with demeshened yt, here are two projections generated with this script of a publicly available medium-resolution FIRE dataset. Notably, the demeshened version takes substantially less time and memory to generate and has no grid artifacts.

import yt
fn = 'FIRE_M12i_ref11/snapshot_600.hdf5'
ds = yt.load(fn)
_, c = ds.find_max(('gas', 'density'))
p = yt.ProjectionPlot(ds, 'x', ('gas', 'density'), center=c, width=(1, 'Mpc'))
p.set_zlim(('gas', 'density'), 2e-6, 2e-2)
p.show()

Standard yt:

title

Demesh yt:

title

Warning--demeshened yt is still beta code but progressing fast!

At this point, nearly all of yt's functionality is working in the demeshened version, but the code is still under development. Results should be cautiously interpreted. That said, it is being used for production level work by many users (including the author), and we encourage people to begin using it for its many benefits in working with particle-based data. If you encounter a bug or a problem with your use case or simulation frontend, it would help us immensely if you could report it. Please contact the yt-users and/or trident-users email lists, jump into slack, or create a github issue with demeshening tag with the relevant information regarding how to reproduce your bug:

[email protected] 
[email protected]

1) Install Mini-conda on your system if not installed

Conda is a package manager providing a clean, stand-alone installation of python that is self-contained in its installation directory. yt & trident require a modern installation of python to work. conda provides that installation. It is possible to install yt without conda with an existing python installation by jumping to the next step.

Check if you have conda already installed

$ conda -h

If conda is installed, move to the next step. Otherwise install Mini-conda.

Install Mini-conda

Use the appropriate conda install script for your architecture. We recommend getting the latest version of conda for Python3 for your architecture here: https://repo.continuum.io/miniconda/

For modern macs:

$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh

For modern linux machines:

$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh

At the end of the installation step, allow conda to add its installation directory to the $PATH.

2) Install all of yt's dependencies using pip

yt had a number of dependencies, so we need to isntall them prior to installing yt itself:

$ pip install numpy jupyter sphinx gitpython h5py matplotlib cython nose scipy astropy sympy mpi4py

3) Get and install the demeshened version of yt (yt-4.0)

Right now, the demeshened version of yt operates as a separate branch in the main yt repository. Here are some git commands for accessing and installing from this branch, entitled yt-4.0:

$ git clone https://github.com/yt-project/yt.git
$ cd yt
$ git checkout yt-4.0
$ git pull
$ pip install -e .
$ cd ..

Also note these useful commands for changing between active branches

To change to the main version (master branch) of yt:

$ git checkout master           # go back to using the main yt version as your active yt installation
$ git pull origin master        # pull main yt-dev and merge with local master
$ pip install -e .              # Re-install the master branch of yt (rebuilds cython)

To change to the demeshened version (yt-4.0) of yt:

$ git checkout yt-4.0           # go back to using the yt-4.0 yt version as your active yt installation
$ git pull origin yt-4.0        # pull yt-4.0 and merge with local master
$ pip install -e .              # Re-install the yt-4.0 branch of yt (rebuilds cython)

4) Get demeshened version of Trident

Currently, the demeshened version of Trident exists as a branch in the main Trident repository. Here are some git commands for accessing this demeshened branch, entitled sph-viz:

If you have stable version of trident, uninstall and continue

$ pip uninstall trident

Get development version of trident and switch to demeshened branch (sph-viz)

$ git clone https://github.com/trident-project/trident.git
$ cd trident
$ git checkout sph-viz
$ pip install -e .
$ cd ..

Run Trident for the first time

$ python
>>> import trident

Follow the instructions to download the ion_balance table and verify the code.

You should be good to go. Try generating a yt.ProjectionPlot, which works better and faster than the one using the octree grid. You should also be able to create Trident LightRays using trident.make_simple_ray() or any of the functionality of Trident should work in the demeshened context.

5) (optional) Test out Trident demeshened functionality on test FIRE dataset

Grab the test FIRE dataset FIRE_M12i_ref11 from the yt sample datasets, run our test script on it, and look at the results.

$ wget http://yt-project.org/data/FIRE_M12i_ref11.tar.gz
$ tar -zxvf FIRE_M12i_ref11.tar.gz
$ cp /path/to/trident/example/gizmo_script.py .
$ python gizmo_script.py
$ open projection.png
$ open spec_raw.png
$ open spec_final.png