If you’re missing a library or program, and that library or program happens to be written in C, you either need root to install it from your package manager, or you will descend into a lovecraftian nightmare of attempted local builds from which there is no escape. You say you need lxml on shared hosting and they don’t have libxml2 installed? Well, fuck you.
"It's easy to build a VM if you automate the install process, and providing that install script for even one OS can demystify the install process for others; conversely, just because you provide a VM doesn't mean that anyone other than you can install your software"
— C. Titus Brown, "Virtual machines considered harmful for reproducibility"
Let's install conda-build
!
!conda install -y conda-build -q -n root
Using Anaconda Cloud api site https://api.anaconda.org Fetching package metadata: ........ Solving package specifications: ......... # All requested packages already installed. # packages in environment at /home/juanlu/.miniconda3: # conda-build 1.20.0 py34_0
conda packages are created from conda recipes. We can create a bare recipe using conda skeleton
to build it from a PyPI package.
!conda skeleton pypi pytest-benchmark > /dev/null
Using Anaconda Cloud api site https://api.anaconda.org
!ls pytest-benchmark
bld.bat build.sh meta.yaml
These are the minimum files for the recipe:
meta.yaml
contains all the metadatabuild.sh
and bld.bat
are the build scripts for Linux/OS X and Windows respectivelymeta.yaml
file¶It contains the metadata in YAML format.
package
, source
and build
specify the name, version and source of the packagerequirements
specify the build (install time) and run (runtime) requirementstest
specify imports, commands and scripts to testabout
adds some additional data for the package!grep -v "#" pytest-benchmark/meta.yaml | head -n24
package: name: pytest-benchmark version: "3.0.0" source: fn: pytest-benchmark-3.0.0.zip url: https://pypi.python.org/packages/source/p/pytest-benchmark/pytest-benchmark-3.0.0.zip md5: f8ab8e438f039366e3765168ad831b4c build: preserve_egg_dir: True requirements: build: - python - setuptools - pytest >=2.6 run: - python - setuptools - pytest >=2.6
build.sh
and bld.bat
files¶They specify how to build the package.
!cat pytest-benchmark/build.sh
#!/bin/bash $PYTHON setup.py install # Add more build steps here, if they are necessary. # See # http://docs.continuum.io/conda/build.html # for a list of environment variables that are set during the build process.
!grep -v "::" pytest-benchmark/bld.bat
"%PYTHON%" setup.py install if errorlevel 1 exit 1
Adapted from http://conda.pydata.org/docs/building/recipe.html#conda-recipe-files-overview
Seems legit!
!conda build pytest-benchmark --python 3.5 > /dev/null # It works!
Using Anaconda Cloud api site https://api.anaconda.org + /home/juanlu/.miniconda3/envs/_build/bin/python setup.py install warning: no directories found matching 'examples' warning: no files found matching '.isort.cfg' warning: no files found matching '.pylintrc' warning: no previously-included files matching '*.py[cod]' found anywhere in distribution warning: no previously-included files matching '__pycache__' found anywhere in distribution warning: no previously-included files matching '*.so' found anywhere in distribution warning: no previously-included files matching '*.dylib' found anywhere in distribution
!ls ~/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2
/home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2
!conda install pytest-benchmark --use-local --yes
Using Anaconda Cloud api site https://api.anaconda.org Fetching package metadata: .......... Solving package specifications: ......... Package plan for installation in environment /home/juanlu/.miniconda3/envs/py35: The following NEW packages will be INSTALLED: pytest-benchmark: 3.0.0-py35_0 Linking packages ... [ COMPLETE ]|###################################################| 100%
Let's upload the package first using anaconda-client
:
!conda install anaconda-client --quiet --yes
Using Anaconda Cloud api site https://api.anaconda.org Fetching package metadata: ........ Solving package specifications: ......... # All requested packages already installed. # packages in environment at /home/juanlu/.miniconda3/envs/py35: # anaconda-client 1.4.0 py35_0
!anaconda upload ~/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2
Using Anaconda Cloud api site https://api.anaconda.org detecting package type ... conda extracting package attributes for upload ... done Uploading file Juanlu001/pytest-benchmark/3.0.0/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 ... uploaded 54 of 54Kb: 100.00% ETA: 0.0 minutes Upload(s) Complete Package located at: https://anaconda.org/juanlu001/pytest-benchmark
And now, let's install it!
!conda remove pytest-benchmark --yes > /dev/null
Using Anaconda Cloud api site https://api.anaconda.org
!conda install pytest-benchmark --channel juanlu001 --yes
Using Anaconda Cloud api site https://api.anaconda.org Fetching package metadata: .......... Solving package specifications: ......... Package plan for installation in environment /home/juanlu/.miniconda3/envs/py35: The following NEW packages will be INSTALLED: pytest-benchmark: 3.0.0-py35_0 Linking packages ... [ COMPLETE ]|###################################################| 100%
You can run your tests with Python, Perl or shell scripts (run_test.[py,pl,sh,bat]
)
Using conda convert
for pure Python packages, we can quickly provide packages for other platforms
!conda convert ~/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 --platform all | grep Converting
Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to osx-64 Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to linux-32 Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to linux-64 Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to win-32 Converting /home/juanlu/.miniconda3/conda-bld/linux-64/pytest-benchmark-3.0.0-py35_0.tar.bz2 from linux to win-64
meta.yaml
¶Metadata files support templating using Jinja2!
python
as a build or run dependency!If the recipe builds on a fresh, headless, old Linux it will work everywhere
conda-forge is a github organization containing repositories of conda recipes. Thanks to some awesome continuous integration providers (AppVeyor, CircleCI and TravisCI), each repository, also known as a feedstock, automatically builds its own recipe in a clean and repeatable way on Windows, Linux and OSX.
Features:
What I love:
conda (2012?) and conda-build (2013) are very young projects and still have some pain points that ought to be addressed
meta.yaml
, the only option is to keep a copy of all dependenciesThe state of Python packaging is improving upstream too!
Still, there are some remaining irks: