Manual installation

Manual installation is currently the recommended installation method, as it offers more flexibility in terms of how the development environment is set up. If you are interested in running some of the listed dependencies in Docker containers, without losing too much flexibility, please refer to the hybrid installation as well.

Installing the dependencies

Python and Virtualenv

As the backend code of the application is based on Flask and multiple other Python libraries, Python 3 needs to be installed (note that version 3.6 or greater is required). This should generally be the case already, otherwise it can be installed using:

sudo apt install python3

To create an isolated environment for the application and its dependencies, it is highly recommended to use Virtualenv, which can be installed like this:

sudo apt install virtualenv

The rest of the documentation assumes Virtualenv is installed.

Libraries

Some external libraries and tools are required as dependencies for some Python packages. The following command installs all of them:

sudo apt install libmagic1 build-essential python3-dev libpq-dev

PostgreSQL

The RDBMS used in the application is PostgreSQL, which can be installed like this:

sudo apt install postgresql

Redis

Redis is an in-memory data structure that can be used for different purposes. Currently it is used as cache for rate limits and as a message broker for running asynchronous tasks with celery, a distributed task queue. It can be installed like this:

sudo apt install redis-server

Node.js

Node.js, including npm, are used for managing the frontend dependencies and building/compiling the asset bundles. They can be installed using:

sudo apt install curl
curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -
sudo apt install nodejs

Elasticsearch

Note

As Elasticsearch requires quite a few system resources, especially memory, installation is optional for development. Note that some of the search features, mainly related to records, collections and groups, may be limited or disabled if not installed.

If Elasticsearch is needed, especially when developing related search functionality, it can also be installed later using the instructions below. Remember to initialize the search indices after the installation using the Kadi CLI (as also explained later in more detail):

kadi search init

Alternatively, if installed, Elasticsearch can be stopped and optionally disabled at startup until needed using:

sudo systemctl disable elasticsearch
sudo systemctl stop elasticsearch

In both cases, the Kadi CLI may need to be used to reindex existing data if the search indices went out of sync with the database (which happens when creating or updating resources while Elasticsearch is not running):

kadi search reindex

Elasticsearch is the full-text search engine used in the application. Currently, only version 7 is supported, which can be installed like this:

sudo apt install wget apt-transport-https gnupg
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt update && sudo apt install elasticsearch

Some configuration values have to be set manually after installation by using the following command. These settings configure Elasticsearch to use a single-node cluster and disable the basic security features included in free Elasticsearch installations, which are not necessarily needed in this simple setup.

echo -e "discovery.type: single-node\nxpack.security.enabled: false" | sudo tee -a /etc/elasticsearch/elasticsearch.yml

To start Elasticsearch and also configure it to start automatically when the system boots, the following commands can be used:

sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Note

If the Elasticsearch service has trouble starting in time, try increasing the start timeout from 90 seconds to 180 seconds:

sudo mkdir /etc/systemd/system/elasticsearch.service.d
echo -e "[Service]\nTimeoutStartSec=180" | sudo tee /etc/systemd/system/elasticsearch.service.d/startup-timeout.conf
sudo systemctl daemon-reload
sudo systemctl restart elasticsearch.service

Installing Kadi4Mat

To create and activate a new virtual environment for the application, the following commands can be used:

virtualenv -p python3 ${HOME}/.venvs/kadi
source ${HOME}/.venvs/kadi/bin/activate

This will create and activate a new virtual environment named kadi using Python 3 as interpreter. The environment is stored inside the .venvs directory in the current user’s home directory. This directory can of course be changed freely. For an easier way to manage virtual environments, a tool like virtualenvwrapper can also be helpful. For all following steps, the virtual environment is assumed to always be active.

Afterwards, the application can be installed using the previously checked out source code:

pip install -e ${HOME}/workspace/kadi[dev]

This will install the application in editable mode, which simply creates a link to the sources so all changes are reflected in the installed package immediately. By specifying the [dev] modifier, all development dependencies listed in setup.py are installed as well.

At this point, it is also recommended to install the pre-commit hooks already by running:

pre-commit install

Configuration

PostgreSQL

To set up PostgreSQL, a user and a database belonging to that user have to be created. When prompted for a password, use kadi. This way, the default configuration of the application does not need to be changed later on.

sudo -u postgres createuser -P kadi
sudo -u postgres createdb -O kadi kadi -E utf-8

Kadi4Mat

Some environment variables need to be set in order to use the application and the correct configuration. Those can be set either directly on the command line each time or, more conveniently, in a .env file, which is read each time the application is started.

This file should reside in the project’s root directory, i.e. ${HOME}/workspace/kadi/.env when following along with the example directory structure. After creating it, the following content needs to be added, again, assuming the example directory structure:

FLASK_APP=${HOME}/workspace/kadi/kadi/wsgi.py
FLASK_ENV=development

More details about this file can also be found in the Flask Documentation.

For the development environment, all configuration values, e.g. regarding the database, have default values set, which correspond to the configuration described in this documentation, so no further changes should be necessary.

If necessary, customizing any configuration values is best done using the KADI_CONFIG_FILE environment variable. This variable needs to point to a valid configuration file, which also needs to be created first, in which the desired values can be specified. As before, the .env file can be used for convenience, e.g. assuming the config file resides in config/config.py relative to the project’s example root directory:

KADI_CONFIG_FILE=${HOME}/workspace/kadi/config/config.py

Please see Configuration for more information about the configuration file itself and its values. Also, check out the manual production installation for an example and kadi/config.py to see all default configuration values for the different environments (grouped via different classes).

Setting up the application

Before the application can be used, some initialization steps have to be done using the Kadi command line interface (CLI). The Kadi CLI offers some useful tools and utility functions running in the context of the application (see also Command line interfaces). As such, it also needs access to the correct configuration specified by the environment variables explained in the previous section. When using the .env file, this means that the commands have to be run inside the directory that contains this file.

kadi assets dev   # Install the frontend dependencies and build the assets
kadi db init      # Initialize the database
kadi i18n compile # Compile the backend translations

If Elasticsearch was installed, the following command has to be run additionally to initialize the search indices:

kadi search init

Another useful command when setting up the application for the first time is the following one, which can be used to set up some initial local dummy users and resources:

kadi db test-data

The three main dummy users useful for testing are:

Username

Password

System Role

admin

admin123

Admin (Can manage any resource)

member

member123

Member (Can create new resources)

guest

guest123

Guest (Read only access)

Running the application

Flask includes a lightweight development server, which can be run using the Flask CLI (see also Command line interfaces):

flask run

When using the .env file and the application cannot be found, make sure to run the command in the correct directory, i.e. the project’s root directory. Afterwards the application should run locally on port 5000:

firefox http://localhost:5000

To be able to run asynchronous background tasks with celery (needed for example when uploading files or sending emails), the following command can be run in a separate terminal:

kadi celery worker -B --loglevel=INFO

This will start the normal celery worker as well as celery beat, which is used for periodic tasks (e.g. for deleting expired uploads) in a single process, which is convenient for development.

To simulate sending emails without using an actual SMTP server, the following can be used to simply print the emails on the terminal instead (this debugging server is already configured as default when developing):

python -m smtpd -n -c DebuggingServer localhost:8025

Updating the application

After updating the application code via git, the following commands may have to be run again:

pip install -e ${HOME}/workspace/kadi[dev] # Install any new backend dependencies
kadi assets dev                            # Install any new frontend dependencies and rebuild the assets
kadi db upgrade                            # Upgrade the database schema
kadi i18n compile                          # Recompile any new backend translations

While mainly intended for productive installations, it is also recommended to take a look at the update notes, since some of the listed changes may also be relevant for development environments.