# Precompilation guide

This guide has two sections, the first one is intended for precompiler module developers. It covers a minimal example of creating a precompiler module. The second section is intended for library developers who want their library to be able to use precompiled artefacts in a simple way.

## Library Developer

This guide assumes you have already added `elixir_make` to your library and you have written a `Makefile` that compiles the native code in your project. Once your native code compile and works as expected, you are now ready to precompile it.

A full demo project is available on [cocoa-xu/cc_precompiler_example](https://github.com/cocoa-xu/cc_precompiler_example).

### Setup mix.exs

To use a precompiler module such as the `CCPrecompiler` example above, we first add the precompiler (`:cc_precompiler` here) and `:elixir_make` to `deps`.

```elixir
def deps do
[
    # ...
    {:elixir_make, "~> 0.6", runtime: false},
    {:cc_precompiler, "~> 0.1", runtime: false, github: "cocoa-xu/cc_precompiler"}
    # ...
]
end
```

Then add `:elixir_make` to the `compilers` list, and set `CCPrecompile` as the value for `make_precompiler`.

```elixir
@version "0.1.0"
def project do
  [
    # ...
    compilers: [:elixir_make] ++ Mix.compilers(),
    # elixir_make specific config
    make_precompiler: {:nif, CCPrecompiler},
    make_precompiler_url: "https://github.com/cocoa-xu/cc_precompiler_example/releases/download/v#{@version}/@{artefact_filename}",
    make_precompiler_filename: "nif",
    make_precompiler_priv_paths: ["nif.*"],
    make_precompiler_unavailable_target: :compile,
    # ...
  ]
end
```

Another required field is `make_precompiled_url`. It is a URL template to the artefact file.

`@{artefact_filename}` in the URL template string will be replaced by corresponding artefact filenames when fetching them. For example, `cc_precompiler_example-nif-2.16-x86_64-linux-gnu-0.1.0.tar.gz`.

Note that there is an optional config key for elixir_make, `make_precompiler_filename`. If the name (file extension does not count) of the shared library is different from your app's name, then `make_precompiler_filename` should be set. For example, if the app name is `"cc_precompiler_example"` while the name shared library is `"nif.so"` (or `"nif.dll"` on windows), then `make_precompiler_filename` should be set as `"nif"`.

Another optional config key is `make_precompiler_priv_paths`. For example, say the `priv` directory is organised as follows in Linux, macOS and Windows respectively,

Also, you can specify how to recover from unavailable targets using the `make_precompiler_unavailable_target` config key. Allowed values are `:compile` and `:ignore`. Defaults to `:compile`.

It is also possible to pass in a 2-arity function to `make_precompiler_unavailable_target`: the first argument is the triplet of the unavailable target, and the second argument is a list that contains all available targets given by the precompiler.

```
# Linux
.
├── assets
│   ├── model.onnx
│   └── data.json
├── lib
│   ├── libpriv1.so
│   ├── libpriv2.so
│   └── libpriv3.so
└── nif.so

# macOS
.
├── assets
│   ├── model.onnx
│   └── data.json
├── lib
│   ├── libpriv1.dylib
│   ├── libpriv2.dylib
│   └── libpriv3.dylib
└── nif.so

# Windows
.
├── assets
│   ├── model.onnx
│   └── data.json
├── lib
│   ├── libpriv1.dll
│   ├── libpriv2.dll
│   └── libpriv3.dll
└── nif.dll
```

By default, everything in `priv` will be included in the precompiled tar file. However, files in `assets` can be very large or platform-independent, therefore, we would like to only include the `nif.so` (`nif.dll`) file and everything in the `lib` directory in the precompiled tar file to reduce the footprint. In this case, we can set `make_precompiler_priv_paths` to `["nif.so", "nif.dll", "lib"]`.

Of course, wildcards (`?`, `**`, `*`) are supported when specifying files. For example, `["nif.*", "lib/*.so", "lib/*.dll", "lib/*.dylib"]` will include `nif.so` (Linux/macOS) or `nif.dll` (Windows), and `.so` or `.dll` files in the `lib` directory. 

Directory structures and symbolic links are preserved.

### (Optional) Test the NIF code locally

To test the NIF code locally, you can either set `force_build` to `true` or append `"-dev"` to your NIF library's version string.

```elixir
@version "0.1.0-dev"

def project do
  [
    # either append `"-dev"` to your NIF library's version string
    version: @version,
    # or set force_build to true
    force_build: true,
    # ...
  ]
end
```

Doing so will ask `elixir_make` to only compile for the current host instead of building for all available targets.

```shell
$ mix compile
cc -shared -std=c11 -O3 -fPIC -I"/usr/local/lib/erlang/erts-13.0.3/include" -undefined dynamic_lookup -flat_namespace -undefined suppress "/Users/cocoa/git/cc_precompiler_example/c_src/cc_precompiler_example.c" -o "/Users/cocoa/Git/cc_precompiler_example/_build/dev/lib/cc_precompiler_example/priv/nif.so"
$ mix test
make: Nothing to be done for `build'.
Generated cc_precompiler_example app
.

Finished in 0.00 seconds (0.00s async, 0.00s sync)
1 test, 0 failures

Randomized with seed 102464
```

### Precompile for available targets

It's possible to either setup a CI task to do the precompilation job or precompile on a local machine and upload the precompiled artefacts.

To precompile for all targets on a local machine:

```shell
MIX_ENV=prod mix elixir_make.precompile
```

Environment variable `ELIXIR_MAKE_CACHE_DIR` can be used to set the cache dir for the precompiled artefacts, for instance, to output precompiled artefacts in the cache directory of the current working directory, `export ELIXIR_MAKE_CACHE_DIR="$(pwd)/cache"`.

To setup a CI task such as GitHub Actions, the following workflow file can be used for reference:

```yml
name: precompile

on:
  push:
    tags:
      - 'v*'

jobs:
  linux:
    runs-on: ubuntu-latest
    env:
      MIX_ENV: "prod"
    steps:
      - uses: actions/checkout@v3

      - uses: erlef/setup-beam@v1
        with:
          otp-version: "25.1"
          elixir-version: "1.14"

      - name: Install system dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y build-essential automake autoconf pkg-config bc m4 unzip zip \
            gcc g++ \
            gcc-i686-linux-gnu g++-i686-linux-gnu \
            gcc-aarch64-linux-gnu g++-aarch64-linux-gnu \
            gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf \
            gcc-riscv64-linux-gnu g++-riscv64-linux-gnu \
            gcc-powerpc64le-linux-gnu g++-powerpc64le-linux-gnu \
            gcc-s390x-linux-gnu g++-s390x-linux-gnu

      - name: Get musl cross-compilers (Optional, use this if you have musl targets to compile)
        run: |
          for musl_arch in x86_64 aarch64 riscv64
          do
            wget "https://musl.cc/${musl_arch}-linux-musl-cross.tgz" -O "${musl_arch}-linux-musl-cross.tgz"
            tar -xf "${musl_arch}-linux-musl-cross.tgz"
          done

      - name: Mix Test
        run: |
          # Optional, use this if you have musl targets to compile
          for musl_arch in x86_64 aarch64 riscv64
          do
            export PATH="$(pwd)/${musl_arch}-linux-musl-cross/bin:${PATH}"
          done

          mix deps.get
          MIX_ENV=test mix test

      - name: Create precompiled library
        run: |
          export ELIXIR_MAKE_CACHE_DIR=$(pwd)/cache
          mkdir -p "${ELIXIR_MAKE_CACHE_DIR}"
          mix elixir_make.precompile

      - uses: softprops/action-gh-release@v1
        if: startsWith(github.ref, 'refs/tags/')
        with:
          files: |
            cache/*.tar.gz

  macos:
    runs-on: macos-11
    env:
      MIX_ENV: "prod"

    steps:
      - uses: actions/checkout@v3

      - name: Install erlang and elixir
        run: |
          brew install erlang elixir
          mix local.hex --force
          mix local.rebar --force

      - name: Mix Test
        run: |
          mix deps.get
          MIX_ENV=test mix test

      - name: Create precompiled library
        run: |
          export ELIXIR_MAKE_CACHE_DIR=$(pwd)/cache
          mkdir -p "${ELIXIR_MAKE_CACHE_DIR}"
          mix elixir_make.precompile

      - uses: softprops/action-gh-release@v1
        if: startsWith(github.ref, 'refs/tags/')
        with:
          files: |
            cache/*.tar.gz
```

### Generate checksum file
After CI has finished, you can fetch the precompiled binaries from GitHub.

```shell
$ MIX_ENV=prod mix elixir_make.checksum --all --ignore-unavailable
```

Meanwhile, a checksum file will be generated. In this example, the checksum file will be named as `checksum.exs` in current working directory.

This checksum file is extremely important in the scenario where you need to release a Hex package using precompiled NIFs. It's **MANDATORY** to include this file in your Hex package (by updating the `files` field in the `mix.exs`). Otherwise your package **won't work**.

```elixir
defp package do
  [
    files: [
      # ...
      "checksum.exs",
      # ...
    ],
    # ...
  ]
end
```

However, there is no need to track the checksum file in your version control system (git or other).

### (Optional) Test fetched artefacts can work locally
```shell
# delete previously built binaries so that
# elixir_make will try to restore the NIF library
# from the downloaded tarball file
$ rm -rf _build/prod/lib/cc_precompiler_example
# set to prod env and test everything
$ MIX_ENV=prod mix test
==> castore
Compiling 1 file (.ex)
Generated castore app
==> elixir_make
Compiling 5 files (.ex)
Generated elixir_make app
==> cc_precompiler
Compiling 1 file (.ex)
Generated cc_precompiler app

20:47:42.262 [debug] Restore NIF for current node from: /Users/cocoa/Library/Caches/cc_precompiler_example-nif-2.16-aarch64-apple-darwin-0.1.0.tar.gz
==> cc_precompiler_example
Compiling 1 file (.ex)
Generated cc_precompiler_example app
.

Finished in 0.01 seconds (0.00s async, 0.01s sync)
1 test, 0 failures

Randomized with seed 539590
```

## Recommended flow
To recap, the suggested flow is the following:

1. Choose an appropriate precompiler for your NIF library and set all necessary options in the `mix.exs`.
2. (Optional) Test if your NIF library compiles locally.

  ```shell
  mix compile
  mix test
  ```

3. (Optional) Test if your NIF library can precompile to all specified targets locally.
  ```shell
  MIX_ENV=prod mix elixir_make.precompile
  ```

4. Precompile your library on CI or locally.

  ```shell
  # locally
  MIX_ENV=prod mix elixir_make.precompile
  # CI
  # please see the docs above
  ```

5. Fetch precompiled binaries from GitHub.

  ```shell
  # only fetch artefact for current host
  MIX_ENV=prod mix elixir_make.checksum --only-local --print
  # fetch all
  MIX_ENV=prod mix elixir_make.checksum --all --print
  # to fetch all available artefacts at the moment
  MIX_ENV=prod mix elixir_make.checksum --all --print --ignore-unavailable
  ```

6. (Optional) Test if the downloaded artefacts works as expected.

  ```shell
  rm -rf _build/prod/lib/NIF_LIBRARY_NAME
  MIX_ENV=prod mix test
  ```

6. Update Hex package to include the checksum file.
7. Release the package to Hex.pm (make sure your release includes the correct files).