General

This section contains some general tips and considerations about developing or contributing to Kadi4Mat.

Useful tools

Kadi CLI

The Kadi command line interface (CLI) contains various useful tools and utilities as part of mulitple subcommands and is available automatically after installing the package. An overview over all commands can be obtained by simply running:

kadi

The Kadi CLI ensures that each subcommand runs inside the context of the application, which is why it always needs access to the correct Kadi environment and, if applicable, configuration file (see also the manual development installation). For this reason, some subcommands are simply wrappers over existing ones provided by other libraries, making their use in certain scenarios or environments easier. See also cli.

virtualenvwrapper

virtualenvwrapper is an extension to the Virtualenv tool and can be used to manage and switch between multiple virtual environments more easily. The tool can be installed globally via pip while not having any virtual environment currently active:

pip3 install virtualenvwrapper

Afterwards, some environment variables have to be set. Generally, a suitable place for them is the ~/.bashrc file. An example could look like the following:

export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export WORKON_HOME=${HOME}/.venvs
source ${HOME}/.local/bin/virtualenvwrapper.sh

Please refer to the official documentation about their meaning as well as other possible variables that can be used, as their values differ by system and personal preference.

EditorConfig

For general editor settings related to indentation, maximum line length and line endings, the settings in the .editorconfig file can be applied. This file can be used in combination with a text editor or IDE that supports it. For more information, take a look at the EditorConfig documentation.

pre-commit

pre-commit is a framework for managing and maintaining multi-language pre-commit hooks, which get executed each time git commit is run. The tool itself should be installed already. The hooks listed in .pre-commit-config.yaml can be installed by simply running:

pre-commit install

The hooks can also be run manually on all versioned and indexed files using:

pre-commit run -a

black

black is a code formatter which is used throughout all Python code in the project. The tool itself should be installed already and can be applied on one or multiple files using:

black <path>

Besides running black on the command line, there are also various integrations available for different text editors and IDEs. black is also part of the pre-commit hooks. As such, it will run automatically on each commit or when running the pre-commit hooks manually.

Pylint

Pylint is a static code analysis tool for Python and should already be installed as well. It can be used on the command line to aid with detecting some common programming or style mistakes, even if not using an IDE that already does that. It can be used for the whole kadi package by running the following:

pylint kadi

Pylint will automatically use the configuration specified in the .pylintrc file in the application’s root directory. Sometimes, there is also code that should never be checked for certain things. Using specific comments, one can instruct Pylint to skip such code, e.g. the following line will not raise a message for an unused import statement:

import something # pylint: disable=unused-import

ESLint

ESLint is a linter and basic code formatter which is used for all JavaScript code throughout the project, including any code snippets inside script tags and Vue.js components. It should be already installed and can be applied on the whole kadi folder using the eslint script exposed by the npm command. Note that npm needs access to the package.json file, see also Managing frontend dependencies.

npm run eslint kadi

The configuration of ESlint can be found inside .eslintrc.js. Besides running ESlint on the command line, there are also various integrations available for different text editors and IDEs. Some files also contain code that should never be checked for certain things. Using specific comments again, one can instruct ESLint to skip such code, e.g. the following will suppress errors for unused variables in the specified function:

/* eslint-disable no-unused-vars */
function foo(a) {}
/* eslint-enable no-unused-vars */
// eslint-disable-next-line no-unused-vars
function foo(a) {}

ESLint is also part of the pre-commit hooks. As such, it will run automatically on each commit or when running the pre-commit hooks manually.

Backend development

Adjusting or adding database models

For managing incremental changes to the database schema, potentially including existing data, Alembic is used via Flask-Migrate. These tools enable migration scripts to be generated, each script corresponding to the necessary database revisions, allowing another developer or system administrator to run the script and get the same changes in their database.

When adding a new model or adjusting an existing one, a new migration script has to be created to perform the necessary upgrades (see also migrations). To automatically generate such a script, the following command can be used:

kadi migrations migrate -m "Add some new table"

Note

When adding a new model, it needs to be imported somewhere in order for the tools to find it, as otherwise no changes might be detected. Furthermore, the current database schema always needs to be up to date in accordance to the latest migration script for the command to work.

The resulting code of the migration script should be checked and adjusted accordingly, as not all changes to the models may be detected automatically, such as new check constraints. Additionally, further steps may be necessary to migrate any existing data as well when adjusting existing models. Afterwards, the database can be upgraded by running the following command:

kadi migrations upgrade

When making further changes to a model after applying a newly generated migration script during development, it is usually best to recreate the script rather than creating another one. However, before deleting the old script, make sure to downgrade the database, as otherwise it may end up in an inconsistent state in regards to the revisions:

kadi migrations downgrade

Managing dependencies

All Python dependencies are currently specified via the requirements*.txt files, which list all direct dependencies used in the project. All package versions are pinned in order to ensure installations that are mostly deterministic. In order to check for new package versions, the following helper script that is included in the Kadi4Mat source code can be used:

${HOME}/workspace/kadi/bin/check_requirements.py

Especially in case of major updates, any updated dependencies should always be checked for potentially breaking changes beforehand and for any issues that may arise in the application after the update.

Frontend development

Writing frontend code

To process and package all frontend assets, including Vue.js components, into individual JavaScript bundles runnable in a browser context, webpack is used. The main webpack configuration can be found in webpack.config.js, while a few other relevant global settings can be found in kadi/assets/scripts/main.js, which defines the main entry point for webpack. When writing frontend code, it is necessary to run the following in a separate terminal:

kadi assets watch

This way, changes to existing files will be detected and the resulting bundles in kadi/static/dist will be rebuilt automatically. When adding new files, the command might have to be restarted to pick them up, depending on which directory the files reside in. See also assets and static.

Managing dependencies

All frontend dependencies are managed using npm, the package manager of Node.js. The corresponding npm command uses the dependencies and configuration options as specified in the package.json file, so it must always be run somewhere inside the application’s root directory. Additionally, a package-lock.json file is generated automatically each time the package.json file is updated by npm to ensure deterministic installations. In order to install a new dependency, the following command can be used.

npm install <package>

This will add the new dependency to package.json automatically. In order to check all existing dependencies for updates, the following command can be used:

npm outdated

The outdated packages may be shown in different colors, depending on how each package is specified in package.json and on the magnitude of the update in accordance with Semantic Versioning. To apply any updates, one of the following commands can be used:

npm update                  # Automatically update all packages with compatible versions
npm install <package>@x.y.z # Install version x.y.z (e.g. a new major version) of a package

Especially in case of major updates, any updated dependencies should always be checked for potentially breaking changes beforehand and for any issues that may arise in the application after the update.

Security audits

npm offers built-in functionality to check all installed packages and subpackages for known vulnerabilities. As sometimes certain vulnerabilities may not be relevant for various reasons, e.g. false positives or the affected package is only used during development, it is possible to ignore certain vulnerabilities using audit-ci via the audit-ci.json file found inside the application’s root directory.

The modified security audit can then be run using the following command. Note that npm needs access to the package.json file, see also Managing frontend dependencies.

npm run audit

Common issues

The Flask dev server and/or the webpack watcher are not working properly

Both the Flask development server and webpack use inotify to efficiently watch for any file changes to automatically restart the server/rebuild the asset bundles. The number of file watches that inotify can use may be limited by the operating system by default. In that case, the limit can be increased permanently by running:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

This amount should be fine in most cases. However, note that this might use more (unswappable) kernel memory than before.

Code running inside a background task is not updated

When modifying Python code that is run as part of a background task via celery and the celery worker itself is already running, it needs to be restarted to pick up any new changes.

There is currently no built-in way to automatically restart the worker on code changes. However, the code that is actually executed in the task should generally be runnable outside of a task as well. Doing so may be more convenient at first before testing the celery integration.

Can’t locate revision identified by <identifier>

This can happen when working on multiple branches with differing commits while trying to run some database related commands. In this case, one of those branches is missing one or more database migration scripts/revisions that were already applied previously. At least one of those scripts should be associated with the identifier printed in the error message.

The best way to fix it is to simply trying to bring both branches up to date. If this is not an option, the database can simply be downgraded again to the lowest common revision:

kadi migrations downgrade <revision>

Note that one needs to be on the branch that actually has the missing migration scripts to be able to run them. Also, this can potentially erase some data from the database, depending on the content of the migration scripts.