General

Command line interfaces

There are two useful command line interfaces (CLI) after installing Kadi4Mat for running different tools or utility functions. These can be used by running the following commands:

flask # Flask CLI
kadi  # Kadi CLI

Running them will return a list of all available subcommands. The first CLI comes from Flask itself (see Command Line Interface and also Working with the Shell) and also includes commands added by Flask extensions, while the Kadi CLI contains all application-specific commands. See also cli.

The Kadi CLI also ensures that each command runs inside the context of the application, which is why it always needs access to the Kadi configuration file. The configuration can be modified as usual by overriding its default values, as explained in the manual development installation. For this reason, some (sub)commands are simply wrappers over existing ones, making their use easier in certain scenarios or environments.

Useful tools

EditorConfig

For general editor settings related to indentation, maximum line length and line endings, the settings in the .editorconfig file can be applied. This file can be used in combination with a text editor or IDE that supports it. For more information, take a look at the EditorConfig documentation.

pre-commit

pre-commit is a framework for managing and maintaining multi-language pre-commit hooks, which get executed each time git commit is run. The tool itself should be installed already. The hooks listed in .pre-commit-config.yaml can be installed by simply running:

pre-commit install

The hooks can also be run manually on all versioned and indexed files using:

pre-commit run -a

black

black is a code formatter which is used throughout all Python code in the project. The tool itself should be installed already and can be applied on one or multiple files using:

black <path>

Besides running black on the command line, there are also various integrations available for different text editors and IDEs. black is also part of the pre-commit hooks. As such, it will run automatically on each commit or when running the pre-commit hooks manually.

Pylint

Pylint is a static code analysis tool for Python and should already be installed as well. It can be used on the command line to aid with detecting some common programming or style mistakes, even if not using an IDE that already does that. It can be used for the whole kadi package by running the following:

pylint kadi

Pylint will automatically use the configuration specified in the .pylintrc file in the application’s root directory. Sometimes, there is also code that should never be checked for certain things. Using specific comments, one can instruct Pylint to skip such code, e.g. the following line will not raise a message for an unused import statement:

import something # pylint: disable=unused-import

ESLint

ESLint is a linter and basic code formatter which is used for all JavaScript code throughout the project, including any code snippets inside script tags and Vue.js components. It should be already installed and can be applied on the whole kadi folder using the npm command. Note that npm needs access to the package.json file, so the command needs to be run inside the application’s root directory (see also Managing frontend dependencies).

npm run eslint kadi

The configuration of ESlint can be found inside .eslintrc.js. Besides running ESlint on the command line, there are also various integrations available for different text editors and IDEs. Some files also contain code that should never be checked for certain things. Using specific comments again, one can instruct ESLint to skip such code, e.g. the following will suppress errors for unused variables in the specified function:

/* eslint-disable no-unused-vars */
function foo(a) {}
/* eslint-enable no-unused-vars */
// eslint-disable-next-line no-unused-vars
function foo(a) {}

ESLint is also part of the pre-commit hooks. As such, it will run automatically on each commit or when running the pre-commit hooks manually.

virtualenvwrapper

virtualenvwrapper is an extension to the Virtualenv tool and can be used to manage and switch between multiple virtual environments more easily. The tool can be installed globally via pip while not having any virtual environment currently active:

pip3 install virtualenvwrapper

Afterwards, some environment variables have to be set. Generally, a suitable place for them is the .bashrc file. An example could look like the following:

export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export WORKON_HOME=${HOME}/.venvs
source ${HOME}/.local/bin/virtualenvwrapper.sh

Please refer to the official documentation about their meaning as well as other possible variables that can be used, as their values differ by system and personal preferences.

Backend development

Managing dependencies

All Python dependencies are currently specified via the files requirements.txt and requirements.dev.txt (for development dependencies), which list all direct dependencies used in the project. All package versions are pinned in order to ensure installations that are mostly deterministic. In order to check for new package versions, the following helper script that is included in the Kadi4Mat source code can be used:

${HOME}/workspace/kadi/bin/check_requirements.py

Especially in case of major updates, any updated dependencies should always be checked for potentially breaking changes beforehand and for any issues that may arise in the application after the update.

Adjusting or adding database models

For managing incremental database schema revisions (and potentially data) and generating corresponding migration scripts, Alembic is used. This way, each change to the database can be checked into version control in form of the migration script, which also allows another developer or system administrator to run the script and get the same incremental updates in their database.

When adding a new model or adjusting an existing one, a new migration script has to be created to perform the necessary upgrades (see also migrations). To automatically generate such a script, the following command can be used:

flask db migrate -m "Add some new table"

The resulting code of the migration script should be checked and adjusted accordingly, especially when using custom column types. Afterwards, the database can be upgraded by running the following command:

flask db upgrade

Note that when adjusting existing models further steps may be necessary to migrate any existing data as well.

Frontend development

Writing frontend code

To process and package all frontend assets into individual JavaScript bundles runnable in a browser context, webpack is used. The main webpack configuration file can be found in webpack.config.js, while a few other relevant global settings can be found in kadi/assets/scripts/main.js. When writing frontend code, it is necessary to run the following in a separate terminal:

kadi assets watch

This way, changes to existing files will be detected and the resulting bundles in kadi/static/dist will be rebuilt automatically. When adding new files, the command might have to be restarted to pick them up, depending on which directory the files reside in. See also assets.

Managing dependencies

All frontend dependencies are managed using npm, the package manager of Node.js. The corresponding npm command uses the dependencies and configuration options as specified in the package.json file, so it must always be run inside the application’s root directory where this file resides. Additionally, a package-lock.json file is generated automatically each time the package.json file is updated by npm to ensure deterministic installations. In order to check all dependencies for any updates, the following command can be used:

npm outdated

The outdated packages may be shown in different colors, depending on how each package is specified in package.json and on the magnitude of the update in accordance with Semantic Versioning. To apply any updates, one of the following commands can be used:

npm update                  # Automatically update all packages with compatible versions
npm install <package>@x.y.z # Force the update of a package, e.g. for major version updates

Especially in case of major updates, any updated dependencies should always be checked for potentially breaking changes beforehand and for any issues that may arise in the application after the update.

Common issues

The Flask dev server and/or the webpack watcher are not working properly

Both the Flask development server and webpack use inotify to efficiently watch for any file changes to automatically restart the server/rebuild the asset bundles. The number of file watches that inotify can use may be limited by the operating system by default. In that case, the limit can be increased permanently by running:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

This amount should be fine in most cases. However, note that this might use more (unswappable) kernel memory than before.

Code running inside a background task is not updated

When modifying Python code that is run as part of a background task via celery and the celery worker itself is already running, it needs to be restarted to pick up any new changes.

There is currently no builtin way to automatically restart the worker on code changes. However, the code that is actually executed in the task should generally be runnable outside of a task as well. Doing so may be more convenient at first before testing the celery integration.

Can’t locate revision identified by <identifier>

This can happen when working on multiple branches with differing commits while trying to run some database related commands. In this case, one of those branches is missing one or more database migration scripts/revisions that were already applied previously. At least one of those scripts should be associated with the identifier printed in the error message.

The best way to fix it is to simply bring both branches up to date. If this is not an option for some reason, the database can simply be downgraded again to the lowest common revision:

flask db downgrade <revision>

Note that we need to be on the branch that actually has the missing migration scripts to be able to run them. Also, this can potentially erase some data from the database, depending on the content of the migration scripts.