 |
Gateau User Manual
Atmospheric simulation of astronomical signals
|
|
|
Gateau User Manual
Atmospheric simulation of astronomical signals
|
|
The gateau development lifecycle is partially automated using Github Actions. It is possible to see all action runs on the actions page:
We use one docker image for both creating gateau sdists and compiling the doxygen documentation. This image is uploaded to GHCR (github container registry). This is done in the build-image.yml github workflow. The workflow runs on any push that updates Dockerfile or the workflow file itself.
Although creating the sdist tarball does not require compiling gateau, due to how meson-python works, all of the compile-time dependencies need to be present in the image.
We go a step further and include absolutely everything needed to build gateau in the container. It will succeed in building wheels (compiled distributions) even with no internet access. We do this for archival purposes.
It is possible to download the built image from the packages page:
Building the sdist is done in the build-sdist.yml github workflow. That workflow downloads the docker image from GHCR and uses it to build the sdist. A new release is then automatically created, and the sdist tarball are attached as release artifacts. This workflow runs on any git tag starting with v.
The sdist tarballs can be downloaded from the releases page:
Documentation is built automatically using doxygen and published to github pages. This is done using the build-doxy.yml github action, which runs on any git tag starting with v. The built documentation can be viewed on the pages site:
Old versions of the documentation can be downloaded as gzip archives from the "artifacts" section of a specific action's run. For example:
Testing the sdist and publishing to pypi is not automated and has to be done manually.
pip install gateau-x.x.x.tar.gzpython -m unittestIf all tests pass, the sdist is good.
Procedure for publishing to pypi is same as for any other python package.
pip install twinepypi-xxxxxx with your token. Do not change the TWINE_USERNAME=__token__ string.Most Python modules that have a backend written in a compiled language like C++ are Python Extension Modules, also known as Python Extensions. Python Extension Modules facilitate the interaction between the compiled code and Python by linking against Python's C ABI.
Gateau is not a Python Extension Module because it does not link against the Python ABI. Instead, the compiled part is bundled as a shared object (libgateau.so), which interacts with the pure Python part of Gateau using the ctypes module.
There is no conventional name for this type of Python module because almost nobody does it this way. In the absence of a standardized terminology, we have decided to call this type of python module a "ctypes wrapper module".
Since version 0.2.3, Gateau is distributed as an sdist. We do this to support a wide variety of target platforms and CUDA versions.
Prior to 0.2.3, Gateau was distributed as a compiled wheel, which links against hdf5 and libgsl statically, and libcufft dynamically. We decided to abandon this approach when we realised that there are multiple versions of libcufft that we have to support.
pip is not smart enough to probe versions of specific libraries (other than the standard c library) before installing a package. So even if we built multiple versions of Gateau for multiple versions of libcufft, the user would still need to manually check which versions of libcufft they have and choose the corresponding version of gateau. This is a pain in the neck for the user. Or, more precisely, it is a bigger pain in the neck than apt-getting a few build dependencies before pip installing gateau.
The git tag cuda-static-deadend contains a meson.build file which successfully produces gateau wheels with static libcufft that function correctly regardless of the system's libcufft version. However, these wheels are illegal to distribute due to license incompatibility.
In order to build Gateau, the following dependencies are required:
These can likely be installed from your package manager.
CUDA is required to build and run Gateau. This can be installed from NVIDIA's website. All minor versions of CUDA 11 and CUDA 12 should work, but if you don't know which to choose, go with 12.3 (because that's what we use for Gateau).
The entire CUDA toolchain is quite large (~7GB). If you only want to build gateau without running it, you do not need the entire toolchain. You only need nvcc, cufft, and curand. So, if you want to save space and time, you can install only those components instead of the full toolchain. To do this, follow the deb (network) (or equivalent for your distro) steps on the CUDA website, but, instead of running
run
IMPORTANT Once installed, make sure nvcc is in your $PATH.
Gateau uses meson as its build system, You can likely install meson from your package manager.
Once the dependencies, meson, and CUDA have been installed, you can install Gateau in development mode like this:
Meson integrates tightly with the python packaging system. You do not need to manually recompile Gateau every time you make changes; recompilation will be triggered automatically every time import gateau is invoked.
If you would still like to manually trigger recompilation, you can do so by running pip install -e . again. Build caches is stored in the build directory in the project root. It is safe to delete.
For documentation of our failure to build Gateau for Windows, see Gateau on Windows .
Below are some common issues and solutions.
When trying to compile gateau:
Either Cuda is not installed, or it is installed, but not in $PATH.
If cuda is not installed, then install it. If it is installed, but you cannot run nvcc from a shell, you need to update the $PATH variable to include cuda's bin directory. For example:
When trying to compile gateau:
or
1.8.17