.. _installation-production-manual:

Manual installation
===================

The described installation steps are supposed to be run on a freshly installed (virtual)
machine.

.. note::
    A new system user called ``kadi`` will be created as part of the installation.
    Please make sure that there are no existing users with this name on the machine
    where the installation is being performed.

Installing the dependencies
---------------------------

.. include:: ../snippets/dependencies.rst

Elasticsearch
~~~~~~~~~~~~~

.. include:: ../snippets/elasticsearch.rst

Apache
~~~~~~

The `Apache HTTP Server <https://httpd.apache.org>`__ is used as a reverse proxy server
in front of the application server (which will be installed later), handling the actual
HTTP requests. Any up to date version **>=2.4** should work, which can be installed like
this, including all additionally required modules:

.. code-block:: bash

    sudo apt install apache2 libapache2-mod-proxy-uwsgi libapache2-mod-xsendfile

Installing |kadi|
-----------------

Before installing the application, it is best to create a dedicated ``kadi`` user (and
corresponding group) under which the application and all required services will run:

.. code-block:: bash

    sudo adduser kadi --system --group --home /opt/kadi --shell /bin/bash

To ensure that Apache can access and deliver local files created by the ``kadi`` user
(via X-Sendfile), the ``www-data`` user needs to be added to the ``kadi`` group:

.. code-block:: bash

    sudo usermod -a -G kadi www-data

As some later steps require root privileges again, all commands that require the newly
created user are prefixed with ``(kadi)``. Therefore it is best to switch to the new
user using a separate terminal window:

.. code-block:: bash

    sudo su - kadi

To create and activate a new virtual environment for the application, the following
commands can be used:

.. code-block:: bash

    (kadi) virtualenv -p python3 ${HOME}/venv
    (kadi) source ${HOME}/venv/bin/activate

.. include:: ../snippets/venv.rst

This will create and activate a new virtual environment named *venv* using Python 3 as
interpreter. **For all following steps requiring the newly created** ``kadi`` **user,
the virtual environment is assumed to be active** (by sourcing the ``activate`` script,
which we will automate later on).

Afterwards, the application can be installed like this:

.. code-block:: bash

    (kadi) pip install kadi

Configuration
-------------

PostgreSQL
~~~~~~~~~~

To set up PostgreSQL, a user and a database belonging to that user have to be created.
Note that creating the user will prompt for a password, and an appropriately secure
value should be chosen. **This password will be required again later when configuring
Kadi4Mat.**

.. code-block:: bash

    sudo -Hiu postgres createuser -P kadi
    sudo -Hiu postgres createdb -O kadi -E utf-8 -T template0 kadi

.. _installation-production-manual-configuration-kadi4mat:

|kadi|
~~~~~~

While most of the application configuration values have usable defaults configured, some
values need to be specified explicitly when using a production environment. For this, a
separate configuration file has to be created:

.. code-block:: bash

    (kadi) mkdir ${HOME}/config
    (kadi) touch ${HOME}/config/kadi.py
    (kadi) chmod 600 ${HOME}/config/kadi.py

This configuration file is important for all services that need access to the
application's configuration, but also for using the Kadi command line interface (CLI).
The Kadi CLI offers some useful tools and utility functions running in the context of
the application. As such, it also needs access to the configuration file, which can be
done by setting the ``KADI_CONFIG_FILE`` environment variable:

.. code-block:: bash

    (kadi) export KADI_CONFIG_FILE=${HOME}/config/kadi.py

The above line should also be added to ``~/.profile`` or a similar configuration file
for convenience, so it is always executed when switching to the ``kadi`` user, together
with automatically activating the virtual environment:

.. code-block:: bash

    (kadi) echo 'export KADI_CONFIG_FILE=${HOME}/config/kadi.py' >> ~/.profile
    (kadi) echo 'test -z "${VIRTUAL_ENV}" && source ${HOME}/venv/bin/activate' >> ~/.profile

.. _installation-production-manual-configuration-kadi4mat-config:

Shown below is an example of such a configuration file, listing the most important
configuration items. This example can be used as a starting point after adjusting some
of the sample values. Please see the :ref:`Configuration <installation-configuration>`
section for an explanation of all configuration items as well as general information
about the configuration file.

.. code-block:: python3

    AUTH_PROVIDERS = [{"type": "local"}]
    SERVER_NAME = "kadi4mat.example.edu"
    SECRET_KEY = "<secret_key>"
    SQLALCHEMY_DATABASE_URI = "postgresql://kadi:<password>@localhost/kadi"
    STORAGE_PATH = "/opt/kadi/storage"
    MISC_UPLOADS_PATH = "/opt/kadi/uploads"
    SMTP_HOST = "localhost"
    MAIL_NO_REPLY = "no-reply@kadi4mat.example.edu"

.. warning::
    Make sure to have a valid :ref:`configuration file <installation-configuration>`
    before continuing with the installation, as some later steps rely on the values
    specified in this file.

uWSGI
~~~~~

`uWSGI <https://uwsgi-docs.readthedocs.io>`__ is the application server used to serve
the actual Python application and should already be installed as part of the application
package. To generate a basic configuration for uWSGI, the Kadi CLI can be used:

.. code-block:: bash

    (kadi) kadi utils uwsgi --out ${HOME}/kadi-uwsgi.ini

The generated configuration should be rechecked as further customization may be
necessary. Once the configuration is suitable, it should be moved to a proper location:

.. code-block:: bash

    sudo mv /opt/kadi/kadi-uwsgi.ini /etc/
    sudo chown root:root /etc/kadi-uwsgi.ini

To set up uWSGI as a systemd service, a corresponding unit file needs to be created as
well:

.. code-block:: bash

    (kadi) kadi utils uwsgi-service --out ${HOME}/kadi-uwsgi.service

Again, the generated configuration should be rechecked as further customization may be
necessary. Once the configuration is suitable, it needs to be moved to the correct
location:

.. code-block:: bash

    sudo mv /opt/kadi/kadi-uwsgi.service /etc/systemd/system/
    sudo chown root:root /etc/systemd/system/kadi-uwsgi.service

To let systemd know about the new service and also configure it to start automatically
when the system boots, the following commands can be used:

.. code-block:: bash

    sudo systemctl daemon-reload
    sudo systemctl enable kadi-uwsgi

Additionally, a logrotate configuration file can be created to make sure that any log
files created by uWSGI, found at ``/var/log/uwsgi``, are rotated and compressed once a
week:

.. code-block:: bash

    echo -e "/var/log/uwsgi/*.log {\n  copytruncate\n  compress\n  delaycompress\n  missingok\n  notifempty\n  rotate 10\n  weekly\n}" | sudo tee /etc/logrotate.d/uwsgi

As uWSGI serves the actual |kadi| application, potential errors will most likely end up
in these log files, unless they occur within a background task run via Celery (as
described later).

.. _installation-production-manual-apache:

Apache
~~~~~~

To generate a basic configuration for Apache, the Kadi CLI can be used:

.. code-block:: bash

    (kadi) kadi utils apache --out ${HOME}/kadi.conf

The command will ask, among others, for a certificate and a key file, used to encrypt
the HTTP traffic using SSL/TLS (HTTPS). For internal use, a self-signed certificate may
be used, which can be generated like this (note that the ``<server_name>`` placeholder
has to be substituted with the actual name or IP of the host that was also configured
via :confval:`SERVER_NAME` in |kadi|):

.. code-block:: bash

    sudo apt install openssl
    sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout /etc/ssl/private/kadi.key -out /etc/ssl/certs/kadi.crt -subj "/CN=<server_name>" -addext "subjectAltName=DNS:<server_name>"

.. note::
    Web browsers and other clients may issue warnings when using self-signed
    certificates, however, the traffic will still be encrypted. When switching to a
    proper SSL/TLS certificate, the ``SSLCertificate*`` directives in the Apache
    configuration file will have to be adapted. Alternatively, a new Apache
    configuration file may be generated using the instructions above. This is for
    example also the case when obtaining a free certificate from `Let's Encrypt
    <https://letsencrypt.org>`__, which is easiest done by using their `Certbot
    <https://certbot.eff.org>`__ utility.

The generated configuration should be rechecked as further customization may be
necessary. Once the configuration is suitable, it needs to be moved to the correct
location and enabled:

.. code-block:: bash

    sudo mv /opt/kadi/kadi.conf /etc/apache2/sites-available/
    sudo chown root:root /etc/apache2/sites-available/kadi.conf
    sudo a2dissite 000-default
    sudo a2ensite kadi

Finally, all required Apache modules have to be activated, which should already be the
case for most of them:

.. code-block:: bash

    sudo a2enmod deflate headers http2 proxy_uwsgi socache_shmcb ssl xsendfile

Note that any potential errors regarding Apache can be found inside the log files at
``/var/log/apache2``.

Celery
~~~~~~

To run Celery as a background service, it is recommended to set it up as a systemd
service. To generate the necessary unit file, the Kadi CLI can be used again:

.. code-block:: bash

    (kadi) kadi utils celery --out ${HOME}/kadi-celery.service

Celery Beat, used to execute periodic tasks, needs its own unit file as well:

.. code-block:: bash

    (kadi) kadi utils celerybeat --out ${HOME}/kadi-celerybeat.service

The generated configurations should be rechecked as further customization may be
necessary. Once the configurations are suitable, they need to be moved to the correct
location:

.. code-block:: bash

    sudo mv /opt/kadi/kadi-celery.service /etc/systemd/system/
    sudo chown root:root /etc/systemd/system/kadi-celery.service

    sudo mv /opt/kadi/kadi-celerybeat.service /etc/systemd/system/
    sudo chown root:root /etc/systemd/system/kadi-celerybeat.service

To let systemd know about the new services and also configure them to start
automatically when the system boots, the following commands can be used:

.. code-block:: bash

    sudo systemctl daemon-reload
    sudo systemctl enable kadi-celery kadi-celerybeat

Additionally, a logrotate configuration file can be created to make sure that any log
files created by Celery, found at ``/var/log/celery``, are rotated and compressed once a
week:

.. code-block:: bash

    echo -e "/var/log/celery/*.log {\n  copytruncate\n  compress\n  delaycompress\n  missingok\n  notifempty\n  rotate 10\n  weekly\n}" | sudo tee /etc/logrotate.d/celery

Initializing the application
----------------------------

Before the application can be used, some initialization steps have to be performed using
the Kadi CLI again:

.. code-block:: bash

    (kadi) kadi db init     # Initialize the database
    (kadi) kadi search init # Initialize the search indices

Finally, all new and modified services have to be restarted:

.. code-block:: bash

    sudo systemctl restart apache2 kadi-uwsgi kadi-celery kadi-celerybeat

Accessing |kadi|
----------------

|kadi| should now be reachable at ``https://<server_name>`` via any web client. Again,
the ``<server_name>`` placeholder has to be substituted with the actual name or IP of
the host that was also configured via :confval:`SERVER_NAME` in |kadi|.

To be able to access |kadi| from a different machine, make sure that any firewall that
may run on the server does not block access to ports 80 and 443, which are the default
HTTP(S) ports.