Manual installation

The described installation steps are supposed to be run on a freshly installed (virtual) machine.

Note

A new system user called kadi will be created as part of the installation. Please make sure that there are no existing users with this name on the machine where the installation is being performed.

Installing the dependencies

Python and Virtualenv

As the backend code of the application is based on the Flask web framework and multiple other Python libraries, Python 3 needs to be installed. This should generally be the case already, otherwise it can be installed using:

sudo apt install python3

Note

Note that a Python version >=3.8 and <3.13 is required. The currently installed version can be checked using:

python3 --version

If the currently installed Python version is not suitable, please see the instructions on how to install and use a different Python version for production or development installations.

To create an isolated environment for the application and its dependencies, Virtualenv is used, which can be installed like this:

sudo apt install virtualenv

Libraries

Some external libraries and tools are required as additional dependencies, which can be installed using:

sudo apt install libmagic1 build-essential python3-dev python3-venv libpq-dev libpcre3-dev

PostgreSQL

The RDBMS used in the application is PostgreSQL. Any up to date version >=13 should work, which can be installed like this:

sudo apt install postgresql

Redis

Redis is an in-memory data structure that can be used for different purposes. Currently it is used as cache for request rate limiting and as a message broker for running asynchronous tasks with Celery, a distributed task queue. Any up to date version >=6 should work, which can be installed like this:

sudo apt install redis-server

Elasticsearch

Elasticsearch is the full-text search engine used in the application. Currently, only version 8 is supported, which can be installed like this:

sudo apt install wget apt-transport-https gnupg
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update && sudo apt install elasticsearch

Some settings have to be changed after installation by using the following command, which will overwrite the default configuration file. Among others, these settings configure Elasticsearch to use a single-node cluster and disable the basic security features included in free Elasticsearch installations, which are not needed in this simple setup.

echo -e "path.data: /var/lib/elasticsearch\npath.logs: /var/log/elasticsearch\ndiscovery.type: single-node\nxpack.security.enabled: false\nxpack.security.transport.ssl.enabled: false\nxpack.security.http.ssl.enabled: false" | sudo tee /etc/elasticsearch/elasticsearch.yml

To start Elasticsearch and also configure it to start automatically when the system boots, the following commands can be used:

sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Apache

The Apache HTTP Server is used as a reverse proxy server in front of the application server (which will be installed later), handling the actual HTTP requests. Any up to date version >=2.4 should work, which can be installed like this, including all additionally required modules:

sudo apt install apache2 libapache2-mod-proxy-uwsgi libapache2-mod-xsendfile

Installing Kadi4Mat

Before installing the application, it is best to create a dedicated kadi user that the application and all required services will run under:

sudo adduser kadi --system --home /opt/kadi --ingroup www-data --shell /bin/bash

As some later steps require root privileges again, all commands that require the newly created user are prefixed with (kadi). Therefore it is best to switch to the new user using a separate terminal window:

sudo su - kadi

To create and activate a new virtual environment for the application, the following commands can be used:

(kadi) virtualenv -p python3 ${HOME}/venv
(kadi) source ${HOME}/venv/bin/activate

Note

If not using the default Python version installed via APT, virtual environments need to be created in a slightly different way, since the virtualenv version installed via APT may not work properly with the newly installed Python version. Instead, the built-in venv module can be used:

python3.x -m venv <my_venv>   # Instead of "virtualenv -p python3 <my_venv>"
source <my_venv>/bin/activate
pip install wheel
pip install -U pip

Note that the wheel package needs to be installed separately after activating the virtual environment for the first time, while updating pip ensures that its most recent version is used, both of which virtualenv normally would have done automatically.

This will create and activate a new virtual environment named venv using Python 3 as interpreter. For all following steps requiring the newly created kadi user, the virtual environment is assumed to be active (by sourcing the activate script, which we will automate later on).

Afterwards, the application can be installed like this:

(kadi) pip install kadi

Configuration

PostgreSQL

To set up PostgreSQL, a user and a database belonging to that user have to be created. Note that creating the user will prompt for a password, and an appropriately secure value should be chosen. This password will be required again later when configuring Kadi4Mat.

sudo -Hiu postgres createuser -P kadi
sudo -Hiu postgres createdb -O kadi -E utf-8 -T template0 kadi

Kadi4Mat

While most of the application configuration values have usable defaults configured, some values need to be specified explicitly when using a production environment. For this, a separate configuration file has to be created, for example:

(kadi) mkdir ${HOME}/config
(kadi) touch ${HOME}/config/kadi.py
(kadi) chmod 640 ${HOME}/config/kadi.py

This configuration file is important for all services that need access to the application’s configuration, but also for using the Kadi command line interface (CLI). The Kadi CLI offers some useful tools and utility functions running in the context of the application. As such, it also needs access to the configuration file, which can be done by setting the KADI_CONFIG_FILE environment variable:

(kadi) export KADI_CONFIG_FILE=${HOME}/config/kadi.py

The above line should also be added to ~/.profile or a similar configuration file for convenience, so it is always executed when switching to the kadi user, together with automatically activating the virtual environment:

(kadi) echo 'export KADI_CONFIG_FILE=${HOME}/config/kadi.py' >> ~/.profile
(kadi) echo 'test -z "${VIRTUAL_ENV}" && source ${HOME}/venv/bin/activate' >> ~/.profile

Shown below is an example of such a configuration file, listing the most important configuration options. This example can be used as a starting point after adjusting some of the sample values. Please see the Configuration section for an explanation of all these options and more general information about the configuration file.

AUTH_PROVIDERS = [{"type": "local"}]
SERVER_NAME = "kadi4mat.example.edu"
SECRET_KEY = "<secret_key>"
SQLALCHEMY_DATABASE_URI = "postgresql://kadi:<password>@localhost/kadi"
STORAGE_PATH = "/opt/kadi/storage"
MISC_UPLOADS_PATH = "/opt/kadi/uploads"
SMTP_HOST = "localhost"
SMTP_PORT = 25
SMTP_USERNAME = ""
SMTP_PASSWORD = ""
MAIL_NO_REPLY = "no-reply@kadi4mat.example.edu"

Warning

Make sure to have a valid configuration file before continuing with the installation, as some later steps rely on the values specified in this file.

uWSGI

uWSGI is the application server used to serve the actual Python application and should already be installed as part of the application package. To generate a basic configuration for uWSGI, the Kadi CLI can be used:

(kadi) kadi utils uwsgi --out ${HOME}/kadi-uwsgi.ini

The generated configuration should be rechecked as further customization may be necessary. Once the configuration is suitable, it should be moved to a suitable place:

sudo mv /opt/kadi/kadi-uwsgi.ini /etc/

To set up uWSGI as a systemd service, a corresponding unit file needs to be created as well:

(kadi) kadi utils uwsgi-service --out ${HOME}/kadi-uwsgi.service

Again, the generated configuration should be rechecked as further customization may be necessary. Once the configuration is suitable, it can be enabled like this:

sudo mv /opt/kadi/kadi-uwsgi.service /etc/systemd/system/

To let systemd know about the new service and also configure it to start automatically when the system boots, the following commands can be used:

sudo systemctl daemon-reload
sudo systemctl enable kadi-uwsgi

Additionally, a logrotate configuration file can be created to make sure that any log files created by uWSGI, found at /var/log/uwsgi, are rotated and compressed once a week:

echo -e "/var/log/uwsgi/*.log {\n  copytruncate\n  compress\n  delaycompress\n  missingok\n  notifempty\n  rotate 10\n  weekly\n}" | sudo tee /etc/logrotate.d/uwsgi

As uWSGI serves the actual Kadi4Mat application, potential errors will most likely end up in these log files, unless they occur within a background task run via Celery (as described later).

Apache

To generate a basic configuration for Apache, the Kadi CLI can be used:

(kadi) kadi utils apache --out ${HOME}/kadi.conf

The command will ask, among others, for a certificate and a key file, used to encrypt the HTTP traffic using SSL/TLS (HTTPS). For internal use, a self-signed certificate may be used, which can be generated like this (note that the <server_name> placeholder has to be substituted with the actual name or IP of the host that was also configured via SERVER_NAME in Kadi4Mat):

sudo apt install openssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout /etc/ssl/private/kadi.key -out /etc/ssl/certs/kadi.crt -subj "/CN=<server_name>" -addext "subjectAltName=DNS:<server_name>"

Note

Web browsers and other clients may issue warnings when using self-signed certificates. However, note that the traffic will still be encrypted.

Tip

When switching to a proper SSL/TLS certificate, the SSLCertificateChainFile directive might need to be added to the Apache configuration file in order to deploy the full certificate chain to clients. Alternatively, a new Apache configuration file may be generated using the instructions above. This is for example also the case when obtaining a free certificate from Let’s Encrypt, which is easiest done by using their Certbot utility. For this to work however, the server should be reachable externally on port 80, once the installation process is completed.

The generated configuration should be rechecked as further customization may be necessary. Once the configuration is suitable, it can be enabled like this:

sudo mv /opt/kadi/kadi.conf /etc/apache2/sites-available/
sudo a2dissite 000-default
sudo a2ensite kadi

Finally, all required Apache modules have to be activated, which should already be the case for most of them:

sudo a2enmod deflate headers proxy_uwsgi socache_shmcb ssl xsendfile

Note that any potential errors regarding Apache can be found inside the log files at /var/log/apache2.

Celery

To run Celery as a background service, it is recommended to set it up as a systemd service. To generate the necessary unit file, the Kadi CLI can be used again:

(kadi) kadi utils celery --out ${HOME}/kadi-celery.service

Celery Beat, used to execute periodic tasks, needs its own unit file as well:

(kadi) kadi utils celerybeat --out ${HOME}/kadi-celerybeat.service

The generated configurations should be rechecked as further customization may be necessary. Once both configurations are suitable, they can be enabled like this:

sudo mv /opt/kadi/kadi-celery.service /etc/systemd/system/
sudo mv /opt/kadi/kadi-celerybeat.service /etc/systemd/system/

To let systemd know about the new services and also configure them to start automatically when the system boots, the following commands can be used:

sudo systemctl daemon-reload
sudo systemctl enable kadi-celery kadi-celerybeat

Additionally, a logrotate configuration file can be created to make sure that any log files created by Celery, found at /var/log/celery, are rotated and compressed once a week:

echo -e "/var/log/celery/*.log {\n  copytruncate\n  compress\n  delaycompress\n  missingok\n  notifempty\n  rotate 10\n  weekly\n}" | sudo tee /etc/logrotate.d/celery

Initializing the application

Before the application can be used, some initialization steps have to be performed using the Kadi CLI again:

(kadi) kadi db init     # Initialize the database
(kadi) kadi search init # Initialize the search indices

Finally, all new and modified services have to be restarted:

sudo systemctl restart apache2 kadi-uwsgi kadi-celery kadi-celerybeat

Accessing Kadi4Mat

Kadi4Mat should now be reachable at https://<server_name> via any web client. Again, the <server_name> placeholder has to be substituted with the actual name or IP of the host that was also configured via SERVER_NAME in Kadi4Mat.

To be able to access Kadi4Mat from a different machine, make sure that any firewall that may run on the server does not block access to ports 80 and 443, which are the default HTTP(S) ports.