343 lines
11 KiB
Markdown
343 lines
11 KiB
Markdown
|
# Precompilation guide
|
||
|
|
|
||
|
|
This guide has two sections, the first one is intended for precompiler module developers. It covers a minimal example of creating a precompiler module. The second section is intended for library developers who want their library to be able to use precompiled artefacts in a simple way.
|
||
|
|
|
||
|
|
## Library Developer
|
||
|
|
|
||
|
|
This guide assumes you have already added `elixir_make` to your library and you have written a `Makefile` that compiles the native code in your project. Once your native code compile and works as expected, you are now ready to precompile it.
|
||
|
|
|
||
|
|
A full demo project is available on [cocoa-xu/cc_precompiler_example](https://github.com/cocoa-xu/cc_precompiler_example).
|
||
|
|
|
||
|
|
### Setup mix.exs
|
||
|
|
|
||
|
|
To use a precompiler module such as the `CCPrecompiler` example above, we first add the precompiler (`:cc_precompiler` here) and `:elixir_make` to `deps`.
|
||
|
|
|
||
|
|
```elixir
|
||
|
|
def deps do
|
||
|
|
[
|
||
|
|
# ...
|
||
|
|
{:elixir_make, "~> 0.6", runtime: false},
|
||
|
|
{:cc_precompiler, "~> 0.1", runtime: false, github: "cocoa-xu/cc_precompiler"}
|
||
|
|
# ...
|
||
|
|
]
|
||
|
|
end
|
||
|
|
```
|
||
|
|
|
||
|
|
Then add `:elixir_make` to the `compilers` list, and set `CCPrecompile` as the value for `make_precompiler`.
|
||
|
|
|
||
|
|
```elixir
|
||
|
|
@version "0.1.0"
|
||
|
|
def project do
|
||
|
|
[
|
||
|
|
# ...
|
||
|
|
compilers: [:elixir_make] ++ Mix.compilers(),
|
||
|
|
# elixir_make specific config
|
||
|
|
make_precompiler: {:nif, CCPrecompiler},
|
||
|
|
make_precompiler_url: "https://github.com/cocoa-xu/cc_precompiler_example/releases/download/v#{@version}/@{artefact_filename}",
|
||
|
|
make_precompiler_filename: "nif",
|
||
|
|
make_precompiler_priv_paths: ["nif.*"],
|
||
|
|
make_precompiler_unavailable_target: :compile,
|
||
|
|
# ...
|
||
|
|
]
|
||
|
|
end
|
||
|
|
```
|
||
|
|
|
||
|
|
Another required field is `make_precompiled_url`. It is a URL template to the artefact file.
|
||
|
|
|
||
|
|
`@{artefact_filename}` in the URL template string will be replaced by corresponding artefact filenames when fetching them. For example, `cc_precompiler_example-nif-2.16-x86_64-linux-gnu-0.1.0.tar.gz`.
|
||
|
|
|
||
|
|
Note that there is an optional config key for elixir_make, `make_precompiler_filename`. If the name (file extension does not count) of the shared library is different from your app's name, then `make_precompiler_filename` should be set. For example, if the app name is `"cc_precompiler_example"` while the name shared library is `"nif.so"` (or `"nif.dll"` on windows), then `make_precompiler_filename` should be set as `"nif"`.
|
||
|
|
|
||
|
|
Another optional config key is `make_precompiler_priv_paths`. For example, say the `priv` directory is organised as follows in Linux, macOS and Windows respectively,
|
||
|
|
|
||
|
|
Also, you can specify how to recover from unavailable targets using the `make_precompiler_unavailable_target` config key. Allowed values are `:compile` and `:ignore`. Defaults to `:compile`.
|
||
|
|
|
||
|
|
It is also possible to pass in a 2-arity function to `make_precompiler_unavailable_target`: the first argument is the triplet of the unavailable target, and the second argument is a list that contains all available targets given by the precompiler.
|
||
|
|
|
||
|
|
```
|
||
|
|
# Linux
|
||
|
|
.
|
||
|
|
├── assets
|
||
|
|
│ ├── model.onnx
|
||
|
|
│ └── data.json
|
||
|
|
├── lib
|
||
|
|
│ ├── libpriv1.so
|
||
|
|
│ ├── libpriv2.so
|
||
|
|
│ └── libpriv3.so
|
||
|
|
└── nif.so
|
||
|
|
|
||
|
|
# macOS
|
||
|
|
.
|
||
|
|
├── assets
|
||
|
|
│ ├── model.onnx
|
||
|
|
│ └── data.json
|
||
|
|
├── lib
|
||
|
|
│ ├── libpriv1.dylib
|
||
|
|
│ ├── libpriv2.dylib
|
||
|
|
│ └── libpriv3.dylib
|
||
|
|
└── nif.so
|
||
|
|
|
||
|
|
# Windows
|
||
|
|
.
|
||
|
|
├── assets
|
||
|
|
│ ├── model.onnx
|
||
|
|
│ └── data.json
|
||
|
|
├── lib
|
||
|
|
│ ├── libpriv1.dll
|
||
|
|
│ ├── libpriv2.dll
|
||
|
|
│ └── libpriv3.dll
|
||
|
|
└── nif.dll
|
||
|
|
```
|
||
|
|
|
||
|
|
By default, everything in `priv` will be included in the precompiled tar file. However, files in `assets` can be very large or platform-independent, therefore, we would like to only include the `nif.so` (`nif.dll`) file and everything in the `lib` directory in the precompiled tar file to reduce the footprint. In this case, we can set `make_precompiler_priv_paths` to `["nif.so", "nif.dll", "lib"]`.
|
||
|
|
|
||
|
|
Of course, wildcards (`?`, `**`, `*`) are supported when specifying files. For example, `["nif.*", "lib/*.so", "lib/*.dll", "lib/*.dylib"]` will include `nif.so` (Linux/macOS) or `nif.dll` (Windows), and `.so` or `.dll` files in the `lib` directory.
|
||
|
|
|
||
|
|
Directory structures and symbolic links are preserved.
|
||
|
|
|
||
|
|
### (Optional) Test the NIF code locally
|
||
|
|
|
||
|
|
To test the NIF code locally, you can either set `force_build` to `true` or append `"-dev"` to your NIF library's version string.
|
||
|
|
|
||
|
|
```elixir
|
||
|
|
@version "0.1.0-dev"
|
||
|
|
|
||
|
|
def project do
|
||
|
|
[
|
||
|
|
# either append `"-dev"` to your NIF library's version string
|
||
|
|
version: @version,
|
||
|
|
# or set force_build to true
|
||
|
|
force_build: true,
|
||
|
|
# ...
|
||
|
|
]
|
||
|
|
end
|
||
|
|
```
|
||
|
|
|
||
|
|
Doing so will ask `elixir_make` to only compile for the current host instead of building for all available targets.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
$ mix compile
|
||
|
|
cc -shared -std=c11 -O3 -fPIC -I"/usr/local/lib/erlang/erts-13.0.3/include" -undefined dynamic_lookup -flat_namespace -undefined suppress "/Users/cocoa/git/cc_precompiler_example/c_src/cc_precompiler_example.c" -o "/Users/cocoa/Git/cc_precompiler_example/_build/dev/lib/cc_precompiler_example/priv/nif.so"
|
||
|
|
$ mix test
|
||
|
|
make: Nothing to be done for `build'.
|
||
|
|
Generated cc_precompiler_example app
|
||
|
|
.
|
||
|
|
|
||
|
|
Finished in 0.00 seconds (0.00s async, 0.00s sync)
|
||
|
|
1 test, 0 failures
|
||
|
|
|
||
|
|
Randomized with seed 102464
|
||
|
|
```
|
||
|
|
|
||
|
|
### Precompile for available targets
|
||
|
|
|
||
|
|
It's possible to either setup a CI task to do the precompilation job or precompile on a local machine and upload the precompiled artefacts.
|
||
|
|
|
||
|
|
To precompile for all targets on a local machine:
|
||
|
|
|
||
|
|
```shell
|
||
|
|
MIX_ENV=prod mix elixir_make.precompile
|
||
|
|
```
|
||
|
|
|
||
|
|
Environment variable `ELIXIR_MAKE_CACHE_DIR` can be used to set the cache dir for the precompiled artefacts, for instance, to output precompiled artefacts in the cache directory of the current working directory, `export ELIXIR_MAKE_CACHE_DIR="$(pwd)/cache"`.
|
||
|
|
|
||
|
|
To setup a CI task such as GitHub Actions, the following workflow file can be used for reference:
|
||
|
|
|
||
|
|
```yml
|
||
|
|
name: precompile
|
||
|
|
|
||
|
|
on:
|
||
|
|
push:
|
||
|
|
tags:
|
||
|
|
- 'v*'
|
||
|
|
|
||
|
|
jobs:
|
||
|
|
linux:
|
||
|
|
runs-on: ubuntu-latest
|
||
|
|
env:
|
||
|
|
MIX_ENV: "prod"
|
||
|
|
steps:
|
||
|
|
- uses: actions/checkout@v3
|
||
|
|
|
||
|
|
- uses: erlef/setup-beam@v1
|
||
|
|
with:
|
||
|
|
otp-version: "25.1"
|
||
|
|
elixir-version: "1.14"
|
||
|
|
|
||
|
|
- name: Install system dependencies
|
||
|
|
run: |
|
||
|
|
sudo apt-get update
|
||
|
|
sudo apt-get install -y build-essential automake autoconf pkg-config bc m4 unzip zip \
|
||
|
|
gcc g++ \
|
||
|
|
gcc-i686-linux-gnu g++-i686-linux-gnu \
|
||
|
|
gcc-aarch64-linux-gnu g++-aarch64-linux-gnu \
|
||
|
|
gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf \
|
||
|
|
gcc-riscv64-linux-gnu g++-riscv64-linux-gnu \
|
||
|
|
gcc-powerpc64le-linux-gnu g++-powerpc64le-linux-gnu \
|
||
|
|
gcc-s390x-linux-gnu g++-s390x-linux-gnu
|
||
|
|
|
||
|
|
- name: Get musl cross-compilers (Optional, use this if you have musl targets to compile)
|
||
|
|
run: |
|
||
|
|
for musl_arch in x86_64 aarch64 riscv64
|
||
|
|
do
|
||
|
|
wget "https://musl.cc/${musl_arch}-linux-musl-cross.tgz" -O "${musl_arch}-linux-musl-cross.tgz"
|
||
|
|
tar -xf "${musl_arch}-linux-musl-cross.tgz"
|
||
|
|
done
|
||
|
|
|
||
|
|
- name: Mix Test
|
||
|
|
run: |
|
||
|
|
# Optional, use this if you have musl targets to compile
|
||
|
|
for musl_arch in x86_64 aarch64 riscv64
|
||
|
|
do
|
||
|
|
export PATH="$(pwd)/${musl_arch}-linux-musl-cross/bin:${PATH}"
|
||
|
|
done
|
||
|
|
|
||
|
|
mix deps.get
|
||
|
|
MIX_ENV=test mix test
|
||
|
|
|
||
|
|
- name: Create precompiled library
|
||
|
|
run: |
|
||
|
|
export ELIXIR_MAKE_CACHE_DIR=$(pwd)/cache
|
||
|
|
mkdir -p "${ELIXIR_MAKE_CACHE_DIR}"
|
||
|
|
mix elixir_make.precompile
|
||
|
|
|
||
|
|
- uses: softprops/action-gh-release@v1
|
||
|
|
if: startsWith(github.ref, 'refs/tags/')
|
||
|
|
with:
|
||
|
|
files: |
|
||
|
|
cache/*.tar.gz
|
||
|
|
|
||
|
|
macos:
|
||
|
|
runs-on: macos-11
|
||
|
|
env:
|
||
|
|
MIX_ENV: "prod"
|
||
|
|
|
||
|
|
steps:
|
||
|
|
- uses: actions/checkout@v3
|
||
|
|
|
||
|
|
- name: Install erlang and elixir
|
||
|
|
run: |
|
||
|
|
brew install erlang elixir
|
||
|
|
mix local.hex --force
|
||
|
|
mix local.rebar --force
|
||
|
|
|
||
|
|
- name: Mix Test
|
||
|
|
run: |
|
||
|
|
mix deps.get
|
||
|
|
MIX_ENV=test mix test
|
||
|
|
|
||
|
|
- name: Create precompiled library
|
||
|
|
run: |
|
||
|
|
export ELIXIR_MAKE_CACHE_DIR=$(pwd)/cache
|
||
|
|
mkdir -p "${ELIXIR_MAKE_CACHE_DIR}"
|
||
|
|
mix elixir_make.precompile
|
||
|
|
|
||
|
|
- uses: softprops/action-gh-release@v1
|
||
|
|
if: startsWith(github.ref, 'refs/tags/')
|
||
|
|
with:
|
||
|
|
files: |
|
||
|
|
cache/*.tar.gz
|
||
|
|
```
|
||
|
|
|
||
|
|
### Generate checksum file
|
||
|
|
After CI has finished, you can fetch the precompiled binaries from GitHub.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
$ MIX_ENV=prod mix elixir_make.checksum --all --ignore-unavailable
|
||
|
|
```
|
||
|
|
|
||
|
|
Meanwhile, a checksum file will be generated. In this example, the checksum file will be named as `checksum.exs` in current working directory.
|
||
|
|
|
||
|
|
This checksum file is extremely important in the scenario where you need to release a Hex package using precompiled NIFs. It's **MANDATORY** to include this file in your Hex package (by updating the `files` field in the `mix.exs`). Otherwise your package **won't work**.
|
||
|
|
|
||
|
|
```elixir
|
||
|
|
defp package do
|
||
|
|
[
|
||
|
|
files: [
|
||
|
|
# ...
|
||
|
|
"checksum.exs",
|
||
|
|
# ...
|
||
|
|
],
|
||
|
|
# ...
|
||
|
|
]
|
||
|
|
end
|
||
|
|
```
|
||
|
|
|
||
|
|
However, there is no need to track the checksum file in your version control system (git or other).
|
||
|
|
|
||
|
|
### (Optional) Test fetched artefacts can work locally
|
||
|
|
```shell
|
||
|
|
# delete previously built binaries so that
|
||
|
|
# elixir_make will try to restore the NIF library
|
||
|
|
# from the downloaded tarball file
|
||
|
|
$ rm -rf _build/prod/lib/cc_precompiler_example
|
||
|
|
# set to prod env and test everything
|
||
|
|
$ MIX_ENV=prod mix test
|
||
|
|
==> castore
|
||
|
|
Compiling 1 file (.ex)
|
||
|
|
Generated castore app
|
||
|
|
==> elixir_make
|
||
|
|
Compiling 5 files (.ex)
|
||
|
|
Generated elixir_make app
|
||
|
|
==> cc_precompiler
|
||
|
|
Compiling 1 file (.ex)
|
||
|
|
Generated cc_precompiler app
|
||
|
|
|
||
|
|
20:47:42.262 [debug] Restore NIF for current node from: /Users/cocoa/Library/Caches/cc_precompiler_example-nif-2.16-aarch64-apple-darwin-0.1.0.tar.gz
|
||
|
|
==> cc_precompiler_example
|
||
|
|
Compiling 1 file (.ex)
|
||
|
|
Generated cc_precompiler_example app
|
||
|
|
.
|
||
|
|
|
||
|
|
Finished in 0.01 seconds (0.00s async, 0.01s sync)
|
||
|
|
1 test, 0 failures
|
||
|
|
|
||
|
|
Randomized with seed 539590
|
||
|
|
```
|
||
|
|
|
||
|
|
## Recommended flow
|
||
|
|
To recap, the suggested flow is the following:
|
||
|
|
|
||
|
|
1. Choose an appropriate precompiler for your NIF library and set all necessary options in the `mix.exs`.
|
||
|
|
2. (Optional) Test if your NIF library compiles locally.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
mix compile
|
||
|
|
mix test
|
||
|
|
```
|
||
|
|
|
||
|
|
3. (Optional) Test if your NIF library can precompile to all specified targets locally.
|
||
|
|
```shell
|
||
|
|
MIX_ENV=prod mix elixir_make.precompile
|
||
|
|
```
|
||
|
|
|
||
|
|
4. Precompile your library on CI or locally.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
# locally
|
||
|
|
MIX_ENV=prod mix elixir_make.precompile
|
||
|
|
# CI
|
||
|
|
# please see the docs above
|
||
|
|
```
|
||
|
|
|
||
|
|
5. Fetch precompiled binaries from GitHub.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
# only fetch artefact for current host
|
||
|
|
MIX_ENV=prod mix elixir_make.checksum --only-local --print
|
||
|
|
# fetch all
|
||
|
|
MIX_ENV=prod mix elixir_make.checksum --all --print
|
||
|
|
# to fetch all available artefacts at the moment
|
||
|
|
MIX_ENV=prod mix elixir_make.checksum --all --print --ignore-unavailable
|
||
|
|
```
|
||
|
|
|
||
|
|
6. (Optional) Test if the downloaded artefacts works as expected.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
rm -rf _build/prod/lib/NIF_LIBRARY_NAME
|
||
|
|
MIX_ENV=prod mix test
|
||
|
|
```
|
||
|
|
|
||
|
|
6. Update Hex package to include the checksum file.
|
||
|
|
7. Release the package to Hex.pm (make sure your release includes the correct files).
|