Manual installation
The described installation steps are supposed to be run on a freshly installed (virtual) machine.
Note
A new system user called kadi will be created as part of the installation.
Please make sure that there are no existing users with this name on the machine
where the installation is being performed.
Installing the dependencies
Python and Virtualenv
As the backend code of the application is based on the Flask web framework and multiple other Python libraries, Python 3 needs to be installed. This should generally be the case already, otherwise it can be installed using:
sudo apt install python3
Note
Note that a Python version >=3.9 and <3.14 is required. The currently installed version can be checked using:
python3 --version
If the currently installed Python version is not suitable, please see the instructions on how to install and use a different Python version for production or development installations.
To create an isolated environment for the application and its dependencies, Virtualenv is used, which can be installed like this:
sudo apt install virtualenv
Libraries
Some external libraries and tools are required as additional dependencies, which can be installed using:
sudo apt install libmagic1 build-essential python3-dev python3-venv libpq-dev libpcre3-dev
PostgreSQL
The RDBMS used in the application is PostgreSQL. Any up to date version >=13 should work, which can be installed like this:
sudo apt install postgresql
Redis
Redis is an in-memory data structure that can be used for different purposes. Currently it is used as cache for request rate limiting and as a message broker for running asynchronous tasks with Celery, a distributed task queue. Any up to date version >=6 should work, which can be installed like this:
sudo apt install redis-server
Elasticsearch
Elasticsearch is the full-text search engine used in the application. Currently, only version 8 is supported, which can be installed like this:
sudo apt install wget apt-transport-https gnupg
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update && sudo apt install elasticsearch
Some settings have to be changed after installation by using the following command, which will overwrite the default configuration file. Among others, these settings configure Elasticsearch to use a single-node cluster and disable the basic security features included in free Elasticsearch installations, which are not needed in this simple setup.
echo -e "path.data: /var/lib/elasticsearch\npath.logs: /var/log/elasticsearch\ndiscovery.type: single-node\nxpack.security.enabled: false\nxpack.security.transport.ssl.enabled: false\nxpack.security.http.ssl.enabled: false" | sudo tee /etc/elasticsearch/elasticsearch.yml
To start Elasticsearch and also configure it to start automatically when the system boots, the following commands can be used:
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
Apache
The Apache HTTP Server is used as a reverse proxy server in front of the application server (which will be installed later), handling the actual HTTP requests. Any up to date version >=2.4 should work, which can be installed like this, including all additionally required modules:
sudo apt install apache2 libapache2-mod-proxy-uwsgi libapache2-mod-xsendfile
Installing Kadi4Mat
Before installing the application, it is best to create a dedicated kadi user (and
corresponding group) under which the application and all required services will run:
sudo adduser kadi --system --group --home /opt/kadi --shell /bin/bash
To ensure that Apache can access and deliver local files created by the kadi user
(via X-Sendfile), the www-data user needs to be added to the kadi group:
sudo usermod -a -G kadi www-data
As some later steps require root privileges again, all commands that require the newly
created user are prefixed with (kadi). Therefore it is best to switch to the new
user using a separate terminal window:
sudo su - kadi
To create and activate a new virtual environment for the application, the following commands can be used:
(kadi) virtualenv -p python3 ${HOME}/venv
(kadi) source ${HOME}/venv/bin/activate
Note
If not using the default Python version installed via APT, virtual environments need
to be created in a slightly different way, since the virtualenv version installed
via APT may not work properly with the newly installed Python version. Instead, the
built-in venv module can be used:
python3.x -m venv <my_venv> # Instead of "virtualenv -p python3 <my_venv>"
source <my_venv>/bin/activate
pip install wheel
pip install -U pip
Note that the wheel package needs to be installed separately after activating
the virtual environment for the first time, while updating pip ensures that its most
recent version is used, both of which virtualenv normally would have done
automatically.
This will create and activate a new virtual environment named venv using Python 3 as
interpreter. For all following steps requiring the newly created kadi user,
the virtual environment is assumed to be active (by sourcing the activate script,
which we will automate later on).
Afterwards, the application can be installed like this:
(kadi) pip install kadi
Configuration
PostgreSQL
To set up PostgreSQL, a user and a database belonging to that user have to be created. Note that creating the user will prompt for a password, and an appropriately secure value should be chosen. This password will be required again later when configuring Kadi4Mat.
sudo -Hiu postgres createuser -P kadi
sudo -Hiu postgres createdb -O kadi -E utf-8 -T template0 kadi
Kadi4Mat
While most of the application configuration values have usable defaults configured, some values need to be specified explicitly when using a production environment. For this, a separate configuration file has to be created:
(kadi) mkdir ${HOME}/config
(kadi) touch ${HOME}/config/kadi.py
(kadi) chmod 600 ${HOME}/config/kadi.py
This configuration file is important for all services that need access to the
application’s configuration, but also for using the Kadi command line interface (CLI).
The Kadi CLI offers some useful tools and utility functions running in the context of
the application. As such, it also needs access to the configuration file, which can be
done by setting the KADI_CONFIG_FILE environment variable:
(kadi) export KADI_CONFIG_FILE=${HOME}/config/kadi.py
The above line should also be added to ~/.profile or a similar configuration file
for convenience, so it is always executed when switching to the kadi user, together
with automatically activating the virtual environment:
(kadi) echo 'export KADI_CONFIG_FILE=${HOME}/config/kadi.py' >> ~/.profile
(kadi) echo 'test -z "${VIRTUAL_ENV}" && source ${HOME}/venv/bin/activate' >> ~/.profile
Shown below is an example of such a configuration file, listing the most important configuration options. This example can be used as a starting point after adjusting some of the sample values. Please see the Configuration section for an explanation of all these options and more general information about the configuration file.
AUTH_PROVIDERS = [{"type": "local"}]
SERVER_NAME = "kadi4mat.example.edu"
SECRET_KEY = "<secret_key>"
SQLALCHEMY_DATABASE_URI = "postgresql://kadi:<password>@localhost/kadi"
STORAGE_PATH = "/opt/kadi/storage"
MISC_UPLOADS_PATH = "/opt/kadi/uploads"
SMTP_HOST = "localhost"
SMTP_PORT = 25
SMTP_USERNAME = ""
SMTP_PASSWORD = ""
MAIL_NO_REPLY = "no-reply@kadi4mat.example.edu"
Warning
Make sure to have a valid configuration file before continuing with the installation, as some later steps rely on the values specified in this file.
uWSGI
uWSGI is the application server used to serve the actual Python application and should already be installed as part of the application package. To generate a basic configuration for uWSGI, the Kadi CLI can be used:
(kadi) kadi utils uwsgi --out ${HOME}/kadi-uwsgi.ini
The generated configuration should be rechecked as further customization may be necessary. Once the configuration is suitable, it should be moved to a proper location:
sudo mv /opt/kadi/kadi-uwsgi.ini /etc/
sudo chown root:root /etc/kadi-uwsgi.ini
To set up uWSGI as a systemd service, a corresponding unit file needs to be created as well:
(kadi) kadi utils uwsgi-service --out ${HOME}/kadi-uwsgi.service
Again, the generated configuration should be rechecked as further customization may be necessary. Once the configuration is suitable, it needs to be moved to the correct location:
sudo mv /opt/kadi/kadi-uwsgi.service /etc/systemd/system/
sudo chown root:root /etc/systemd/system/kadi-uwsgi.service
To let systemd know about the new service and also configure it to start automatically when the system boots, the following commands can be used:
sudo systemctl daemon-reload
sudo systemctl enable kadi-uwsgi
Additionally, a logrotate configuration file can be created to make sure that any log
files created by uWSGI, found at /var/log/uwsgi, are rotated and compressed once a
week:
echo -e "/var/log/uwsgi/*.log {\n copytruncate\n compress\n delaycompress\n missingok\n notifempty\n rotate 10\n weekly\n}" | sudo tee /etc/logrotate.d/uwsgi
As uWSGI serves the actual Kadi4Mat application, potential errors will most likely end up in these log files, unless they occur within a background task run via Celery (as described later).
Apache
To generate a basic configuration for Apache, the Kadi CLI can be used:
(kadi) kadi utils apache --out ${HOME}/kadi.conf
The command will ask, among others, for a certificate and a key file, used to encrypt
the HTTP traffic using SSL/TLS (HTTPS). For internal use, a self-signed certificate may
be used, which can be generated like this (note that the <server_name> placeholder
has to be substituted with the actual name or IP of the host that was also configured
via SERVER_NAME in Kadi4Mat):
sudo apt install openssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout /etc/ssl/private/kadi.key -out /etc/ssl/certs/kadi.crt -subj "/CN=<server_name>" -addext "subjectAltName=DNS:<server_name>"
Note
Web browsers and other clients may issue warnings when using self-signed certificates. However, note that the traffic will still be encrypted.
Tip
When switching to a proper SSL/TLS certificate, the SSLCertificateChainFile
directive might need to be added to the Apache configuration file in order to deploy
the full certificate chain to clients. Alternatively, a new Apache configuration
file may be generated using the instructions above. This is for example also the
case when obtaining a free certificate from Let’s Encrypt, which is easiest done by using their Certbot utility. For this to work however, the server should be
reachable externally on port 80, once the installation process is completed.
The generated configuration should be rechecked as further customization may be necessary. Once the configuration is suitable, it needs to be moved to the correct location and enabled:
sudo mv /opt/kadi/kadi.conf /etc/apache2/sites-available/
sudo chown root:root /etc/apache2/sites-available/kadi.conf
sudo a2dissite 000-default
sudo a2ensite kadi
Finally, all required Apache modules have to be activated, which should already be the case for most of them:
sudo a2enmod deflate headers http2 proxy_uwsgi socache_shmcb ssl xsendfile
Note that any potential errors regarding Apache can be found inside the log files at
/var/log/apache2.
Celery
To run Celery as a background service, it is recommended to set it up as a systemd service. To generate the necessary unit file, the Kadi CLI can be used again:
(kadi) kadi utils celery --out ${HOME}/kadi-celery.service
Celery Beat, used to execute periodic tasks, needs its own unit file as well:
(kadi) kadi utils celerybeat --out ${HOME}/kadi-celerybeat.service
The generated configurations should be rechecked as further customization may be necessary. Once the configurations are suitable, they need to be moved to the correct location:
sudo mv /opt/kadi/kadi-celery.service /etc/systemd/system/
sudo chown root:root /etc/systemd/system/kadi-celery.service
sudo mv /opt/kadi/kadi-celerybeat.service /etc/systemd/system/
sudo chown root:root /etc/systemd/system/kadi-celerybeat.service
To let systemd know about the new services and also configure them to start automatically when the system boots, the following commands can be used:
sudo systemctl daemon-reload
sudo systemctl enable kadi-celery kadi-celerybeat
Additionally, a logrotate configuration file can be created to make sure that any log
files created by Celery, found at /var/log/celery, are rotated and compressed once a
week:
echo -e "/var/log/celery/*.log {\n copytruncate\n compress\n delaycompress\n missingok\n notifempty\n rotate 10\n weekly\n}" | sudo tee /etc/logrotate.d/celery
Initializing the application
Before the application can be used, some initialization steps have to be performed using the Kadi CLI again:
(kadi) kadi db init # Initialize the database
(kadi) kadi search init # Initialize the search indices
Finally, all new and modified services have to be restarted:
sudo systemctl restart apache2 kadi-uwsgi kadi-celery kadi-celerybeat
Accessing Kadi4Mat
Kadi4Mat should now be reachable at https://<server_name> via any web client. Again,
the <server_name> placeholder has to be substituted with the actual name or IP of
the host that was also configured via SERVER_NAME in Kadi4Mat.
To be able to access Kadi4Mat from a different machine, make sure that any firewall that may run on the server does not block access to ports 80 and 443, which are the default HTTP(S) ports.