.. _buildfile: Build File ========== The build file is a YAML file that describes the build process. There are two types of build files: - :ref:`Continuous Integration (CI) Build File ` - :ref:`Deploy Build File ` .. _ci_build_file: Continuous Integration (CI) Build File -------------------------------------- The CI build file is a YAML file that describes the build process for a continuous integration (CI) system. The CI build file is divided into the following sections: - **py_ci**: The version of the build file. This is a required field. - **type**: The type of the build file. This is a required field. - **options**: The options for the build file. - **venv**: The virtual environment options. - **interpreters**: The interpreters to use. - **stages**: The stages of the build process. Type ~~~~ The type of the build file is always ``ci`` and the version is the version of py-ci, currently only version ``1.0`` is available. .. code-block:: yaml :caption: :ref:`ci_build_file.yaml ` type: ci version: "1.0" Options ~~~~~~~ The options for the build file are as follows: - **interpreters**: The interpreters to use. This is a list of strings for the interpreter versions. At least one is required. .. code-block:: yaml :caption: :ref:`ci_build_file.yaml ` `continued` interpreters: - python3.11 - **venv**: The virtual environment options. - **path**: The path to the virtual environment. This will become the ``{venv_path}`` variable and can use the ``{interpreter}`` variable. - **scripts**: The path to the scripts directory in the virtual environment. This will become the ``{venv_scripts}`` variable and can use the ``{venv_path}`` and ``{interpreter}`` variables. .. code-block:: yaml :caption: :ref:`ci_build_file.yaml ` `continued` options: venv: path: ./venv/{interpreter} scripts: ./{venv_path}/bin .. _ci_build_file_stages: Stages ~~~~~~ The stages of the build process are arbitrary and can be named anything. Each stage consists of a list of commands to execute on the build system shell. The environment variables on the user system are available to the build file, as well as the following variables for each interpreter specified earlier: - **{venv_scripts}**: The path to the scripts directory in the virtual environment. - **{venv_path}**: The path to the virtual environment. - **{interpreter}**: The interpreter version. (e.g., ``"python3.11"``) .. code-block:: yaml :caption: :ref:`ci_build_file.yaml ` `continued` stages: before_install: - ./{venv_scripts}/python -m pip install -U pip wheel setuptools - ./{venv_scripts}/pip install poetry install: - ./{venv_scripts}/poetry install quality_checks: - ./{venv_scripts}/poetry run pytest publish: - mkdir -p ./build/{interpreter} - cp ./coverage.xml ./build/{interpreter}/coverage.xml - cp ./junit.xml ./build/{interpreter}/junit.xml - cp ./.coverage ./build/{interpreter}/.coverage - cp -R ./htmlcov ./build/{interpreter}/htmlcov cleanup: - rm -R ./{venv_path} .. _ci_build_file_full_example: Full Example ~~~~~~~~~~~~ .. code-block:: yaml :caption: ci_build_file.yaml py_ci: "1.0" type: ci options: venv: path: ./venv/{interpreter} scripts: ./{venv_path}/bin interpreters: - python3.11 stages: before_install: - ./{venv_scripts}/python -m pip install -U pip wheel setuptools - ./{venv_scripts}/pip install poetry install: - ./{venv_scripts}/poetry install quality_checks: - ./{venv_scripts}/poetry run pytest cleanup: - rm -R ./{venv_path} .. _deploy_build_file: Deploy Build File ----------------- The deploy build file is a YAML file that describes the build process for deploying an application in a continuous development (CD) system. The deploy build file is divided into the following sections: - **py_ci**: The version of the build file. This is a required field. - **type**: The type of the build file. This is a required field. - **connections**: The connections to use. - **variables**: The variables to use. - **stages**: The stages of the build process. Type ~~~~ The type of the build file is always ``deploy`` and the version is the version of py-ci, currently only version ``1.0`` is available. .. code-block:: yaml :caption: :ref:`deploy_build_file.yaml ` type: deploy version: "1.0" Connections ~~~~~~~~~~~ The connection section is a list of connections to deploy to. Each connection has the following options: - **type**: The type of connection to use. Currently, only ``ssh`` is supported. - **host**: The host to connect to, in the format of ``user@host[:port]``. This can use environment variables. - **client**: The client options. - **key_type**: The type of key to use. RSA, DSSKey, ECDSAKey, and Ed25519 are supported. - **key**: The key to use. This can use environment variables. (Must provide this or the **keyfile** option.) - **keyfile**: The key file to use. This can use environment variables. (Must provide this or the **key** option.) - **passphrase**: The passphrase to use. This can use environment variables. This can be omitted if the key does not have a passphrase. .. code-block:: yaml :caption: :ref:`deploy_build_file.yaml ` `continued` connections: - type: ssh host: "{DEPLOY_HOST}" client: key_type: RSA # In this example, the RSA key is stored in the RSA_KEY environment variable. key: "{RSA_KEY}" passphrase: "{RSA_PASSPHRASE}" # Can be omitted if the key does not have a passphrase. Variables ~~~~~~~~~ The variables section is to define variables to use in the build file. These variables can be used in the stages section. This section can use environment variables, as well as other variables defined before it from within the ``variables`` section. .. code-block:: yaml :caption: :ref:`deploy_build_file.yaml ` `continued` variables: host_venv: "{DEPLOY_TO}/venv" # Here, DEPLOY_TO is an environment variable. deploy_bin: "{host_venv}/bin" # We can use the host_venv variable here. dist_dir: ./dist wheel_file: "*.whl" Stages ~~~~~~ As with the CI build file, the stages of the build process are arbitrary and can be named anything. Each stage consists of a list of commands to execute, either locally (the default) or remotely. The environment variables on the user system are available to the build file, as well as the variables defined in the ``variables`` section. .. code-block:: yaml :caption: :ref:`deploy_build_file.yaml ` `continued` stages: clean: - rm -R {dist_dir}/* # Local command build: - poetry build # Local command Remote Commands ^^^^^^^^^^^^^^^ To run commands on the remote host, use the ``!target remote`` directive. To change back to running commands locally, use the ``!target local`` directive. .. code-block:: yaml :caption: :ref:`deploy_build_file.yaml ` `stages section, continued` prepare_host: # Set the run target to remote - !target remote - python3 -m virtualenv "{host_venv}" # The host_venv variable from the variables section is used here. # Change it back to local - !target local - echo "Host prepared" This could also be done using the ``!local`` and ``!remote`` directives, regardless of any previous ``!target`` directive. .. code-block:: yaml :caption: **Alternative** :ref:`deploy_build_file.yaml ` `stages section, continued, using the !local and !remote directives` prepare_host: - !remote python3 -m virtualenv "{host_venv}" - !local echo "Host prepared" .. note:: Variables accessible using the ``{}`` syntax are available in the remote commands, including environment variables from the local system, but not including environment variables from the remote system. Transfer Files ^^^^^^^^^^^^^^ To transfer files to the remote host, use the ``!put`` directive. This directive has the following arguments: - **source**: The source file or directory to transfer. This supports a glob pattern to put multiple files. .. warning:: The glob pattern must start with a ``*`` or ``**`` to work correctly. This will be fixed in a future release. - **dest**: The destination directory on the remote host. All arguments can use variables from the ``variables`` section and environment variables from the local system. .. code-block:: yaml :caption: :ref:`deploy_build_file.yaml ` `stages section, continued` put_to_host: - !put source: "{dist_dir}/{wheel_file}" dest: "{DEPLOY_TO}" Full Example ~~~~~~~~~~~~ .. code-block:: yaml :caption: deploy_build_file.yaml :name: deploy_build_file_full_example # Expects DEPLOY_HOST, RSA_KEY, RSA_PASSPHRASE, and DEPLOY_TO environment variables # to be set in the CI/CD system. py_ci: "1.0" type: deploy connections: - type: ssh host: "{DEPLOY_HOST}" client: key_type: RSA # In this example, the RSA key is stored in the RSA_KEY environment variable. key: "{RSA_KEY}" passphrase: "{RSA_PASSPHRASE}" variables: host_venv: "{DEPLOY_TO}/venv" # Here, DEPLOY_TO is an environment variable. deploy_bin: "{host_venv}/bin" # We can use the host_venv variable here. dist_dir: ./dist wheel_file: "*.whl" stages: clean: - rm -R {dist_dir}/* # Local command build: - poetry build # Local command prepare_host: # Set the run target to remote - !target remote - python3 -m virtualenv "{host_venv}" # Change it back to local - !target local - echo "Host prepared" # prepare_host: # - !remote python3 -m virtualenv "{host_venv}" # - !local echo "Host prepared" put_to_host: - !put source: "{dist_dir}/{wheel_file}" dest: "{DEPLOY_TO}" install: - !target remote - "{deploy_bin}/pip install --upgrade --force-reinstall $(ls -1v {DEPLOY_TO}/{wheel_file} | tail -n 1)" clean_host: - !target remote - rm -R {DEPLOY_TO}/{wheel_file}