 |
Gateau User Manual
Atmospheric simulation of astronomical signals
|
|
|
Gateau User Manual
Atmospheric simulation of astronomical signals
|
|
We have tried to build Gateau for Windows 10. Sometimes we managed to get it to compile, but we never managed to get it to run. All relevant work is on windows2 and windows2-vcpkg branches. windows branch is for gateau 0.1.6. Henceforth, all relative/paths/to/files refer to these branches, not main.
We use qemu emulator to run windows 10 virtual machine, and forward the repo directory into the vm using virtiofs, mounted at Z:\ in the vm. There is a script at win/scripts/run-interactive-vm.sh that checks host dependencies (qemu, virtiofs, etc.), and launches vm and virtiofs. vm runs on qcow2 overlay on top of base image so it is easy to rollback accidental changes by deleting overlay file. Read the run-interactive-vm.sh script for more info.
There are archives of almost all external resources that we have used (windows image, visual studio installer, virtio guest tools, C libraries, cuda installers) under various subdirectories of win and archives of built (dysfunctional) wheels under archive in the repo root. These are git-annex links. Some of these files we are not allowed to distribute because of licenses. If you wish to continue where we stopped in porting Gateau to windows and you would like to get these files (i.e. assuming they are no longer available on the internet) please let us know and we will share what we can.
We have tried following environments to compile Gateau:
| Visual Studio | CUDA | Python | HDF5 | libgsl | libaec | zlib | compiles? | runs? |
|---|---|---|---|---|---|---|---|---|
| 2019 | 11.1 | 3.12 | 2.0.0 [1] | 2.3.0 [3] | 1.0.4 [5] | x.x.x [7] | YES | NO [10] |
| 2019 | 11.1 | 3.12 | 1.14.6 [1] | 2.3.0 [3] | 1.0.4 [5] | x.x.x [7] | YES | NO [10] |
| 2019 | 11.1 | 3.12 | 2.0.0 [2] | 2.8 [4] | 1.1.4 [6] | 1.3.1 [8] | YES | NO [10] |
| 2022 | 12.3 | 3.12 | 2.0.0 [1] | 2.3.0 [3] | 1.0.4 [5] | x.x.x [7] | NO [9] | NO [10] |
NOTES:
[9]: We could not get it to compile with cuda12.3. We have following error:
``‘ nvcc error : 'cudafe++’ died with status 0xC0000005 (ACCESS_VIOLATION) ninja: build stopped: subcommand failed. ```
full log: win/log/cuda-123-cudafe-access-violation.txt
[10]: libgateau.dll compiles but cannot be loaded by python with generic "cannot load dll" error. First we assumed it is issue with dependency dll's not being found; we have tried both static linking of dependencies (hdf5, gsl, aec, zlib), and bundling dynamic libraries into the wheel. But the error persists, to the cause must be something else. We have also ruled out missing DLLs using Dependencies tool.
The generic error message does not gives us any indication of what to try next. Someone with more windows experience who knows how to debug these types of errors might be able to get it working eventually.
libaec and zlib are external dependencies of hdf5 on windows. These need to be downloaded separately (we have tried various sources, including vcpkg package manager; see table above). Failing to have these available will result in built-time errors like this:
full log: win/logs/cuda-111-hdf5-1-14-6-linker-errors.txt
When troubleshooting dll loading, we have tried using (delvewheel)[https://pypi.org/project/delvewheel/] to bundle DLLs. It does not work; delvewheel show simply does not detect any dependencies in the wheel, even though delvewheel needed does show dependencies in the libgateau.dll file when built with dynamic linking. Maybe this is because delvewheel expects the dll to link against python abi whic we are not doing.
Instead of using delvewheel, we have tried static linking and manual dependency bundling, as mentioned in norun.
There are two differences between gcc and visual studio 2019 compiler when it comes to Gateau:
__impl_StrStrIA symbol is not available in vs compilerWe work around issue no.1 by downgrading to cpp14 and using experimental filesystem header instead of std::filesystem (at the time of writing, filesystem is the only cpp17 feature Gateau uses).
We work around issue no.2 using some arcane pragma magic.
In addition to changes in meson.build, the following snippet is added to fio.h (GATEAU_WIN_BUILD is a macro set it meson.build if windows is detected):
Failing to do the pragma incantation results in linker error like this:
The hdf5 releases page has different versions of hdf5 for windows: one built using cl compiler, and one built using intel compiler. Trying to use the intel compiler version with visual studion will result in error like this:
So do not use intel compiler version, use cl version. See win/logs/cuda-111-hdf5-2-libircmt.txt for full log.
1.8.17