> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-mintlify-8c05c8a2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Overview of the ClickHouse continuous integration system

# Continuous Integration (CI)

When you submit a pull request, some automated checks are ran for your code by the ClickHouse [continuous integration (CI) system](/resources/develop-contribute/contribute/tests#test-automation).
This happens after a repository maintainer (someone from ClickHouse team) has screened your code and added the `can be tested` label to your pull request.
The results of the checks are listed on the GitHub pull request page as described in the [GitHub checks documentation](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-status-checks).
If a check is failing, you might be required to fix it.
This page gives an overview of checks you may encounter, and what you can do to fix them.

If it looks like the check failure is not related to your changes, it may be some transient failure or an infrastructure problem.
Push an empty commit to the pull request to restart the CI checks:

```shell theme={null}
git commit --allow-empty
git push
```

If you are not sure what to do, ask a maintainer for help.

<h2 id="merge-with-master">
  Merge with master
</h2>

Verifies that the PR can be merged to master.
If not, it will fail with a message `Cannot fetch mergecommit`.
To fix this check, resolve the conflict as described in the [GitHub documentation](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/resolving-a-merge-conflict-on-github), or merge the `master` branch to your pull request branch using git.

<h2 id="docs-check">
  Docs check
</h2>

Tries to build the ClickHouse documentation website.
It can fail if you changed something in the documentation.
Most probable reason is that some cross-link in the documentation is wrong.
Go to the check report and look for `ERROR` and `WARNING` messages.

<h2 id="description-check">
  Description check
</h2>

Check that the description of your pull request conforms to the template [PULL\_REQUEST\_TEMPLATE.md](https://github.com/ClickHouse/ClickHouse/blob/master/.github/PULL_REQUEST_TEMPLATE.md).
You have to specify a changelog category for your change (e.g., Bug Fix), and write a user-readable message describing the change for [CHANGELOG.md](/resources/changelogs/oss/2026)

<h2 id="docker-image">
  Docker image
</h2>

Builds the ClickHouse server and keeper Docker images to verify that they build correctly.

<h3 id="official-docker-library-tests">
  Official docker library tests
</h3>

Runs the tests from the [official Docker library](https://github.com/docker-library/official-images/tree/master/test#alternate-config-files) to verify that the `clickhouse/clickhouse-server` Docker image works correctly.

To add new tests, create a directory `ci/jobs/scripts/docker_server/tests/$test_name` and the script `run.sh` there.

Additional details about the tests can be found in the [CI jobs scripts documentation](https://github.com/ClickHouse/ClickHouse/tree/master/ci/jobs/scripts/docker_server).

<h2 id="marker-check">
  Marker check
</h2>

This check means that the CI system started to process the pull request.
When it has 'pending' status, it means that not all checks have been started yet.
After all checks have been started, it changes status to 'success'.

<h2 id="style-check">
  Style check
</h2>

Performs various style checks on the code base. Each sub-check below corresponds to a `testname` in [`ci/jobs/check_style.py`](https://github.com/ClickHouse/ClickHouse/blob/master/ci/jobs/check_style.py) and can be run individually with `--test <name>` (see below).

<h5 id="cpp">
  cpp
</h5>

Regex-based C++ style checks via [`check_cpp.sh`](https://github.com/ClickHouse/ClickHouse/blob/master/ci/jobs/scripts/check_style/check_cpp.sh). If it fails, fix the issues according to the [code style guide](/resources/develop-contribute/contribute/style).

<h5 id="whitespace-check">
  whitespace\_check
</h5>

Flags double spaces after commas in C++ that are not part of column alignment.

<h5 id="catch-all">
  catch\_all
</h5>

Forbids `catch (...)` outside of destructors, `main`, and fuzzer entry points where swallowing an unknown exception is unsafe.

<h5 id="yamllint">
  yamllint
</h5>

Lints YAML workflow files under `.github/` using `.yamllint`.

<h5 id="xmllint">
  xmllint
</h5>

Validates XML files under `tests/` and `programs/`.

<h5 id="functional-tests-check">
  functional\_tests\_check
</h5>

Checks stateless tests: queries filtering on `event_date` must use `>= yesterday()` rather than `today()` (to avoid flakiness around midnight), and test file names must not contain `fail`.

<h5 id="test-numbers-check">
  test\_numbers\_check
</h5>

Flags large gaps in stateless test numbering (`tests/queries/0_stateless/<NNNNN>_*`).

<h5 id="symlinks">
  symlinks
</h5>

Detects broken symlinks in the repository.

<h5 id="various">
  various
</h5>

Miscellaneous repository checks via [`various_checks.sh`](https://github.com/ClickHouse/ClickHouse/blob/master/ci/jobs/scripts/check_style/various_checks.sh): queries on `system.query_log` / `system.parts` / etc. must filter by `currentDatabase`, `Replicated*MergeTree` ZooKeeper paths must include a per-test prefix, integration test directories must have `__init__.py`, no UTF BOMs, no executable bits on source/data files, no `:latest` tags on third-party docker-compose images, and more.

<h3 id="running-style-check-locally">
  Running the style check job locally
</h3>

The entire *Style Check* job can be run locally in a Docker container with:

```sh theme={null}
python -m ci.praktika run "Style check"
```

To run a specific check (e.g., *cpp* check):

```sh theme={null}
python -m ci.praktika run "Style check" --test cpp
```

These commands pull the `clickhouse/style-test` Docker image and run the job in a containerized environment.
No dependencies other than Python 3 and Docker are required.

<h2 id="running-stateless-tests">
  Running stateless tests
</h2>

A locally installed ClickHouse with default settings may work for specific test cases, but cannot run all test queries correctly. In CI, each job installs a specific ClickHouse configuration (e.g., S3 storage, Parallel Replicas) which can be cumbersome to reproduce manually. To avoid this, you can reproduce any CI job locally using the same orchestration as CI — no manual configuration needed.

<h4 id="ci-prerequisites">
  Prerequisites
</h4>

* Python 3 (standard library only)
* Docker

Install Docker on Ubuntu if needed and re-login:

```sh theme={null}
sudo apt-get update
sudo apt-get install docker.io
sudo usermod -aG docker "$USER"
sudo tee /etc/docker/daemon.json <<'EOF'
{
  "ipv6": true,
  "ip6tables": true
}
EOF
sudo systemctl restart docker
```

<h4 id="run-ci-job-locally">
  Run a CI Job Locally
</h4>

Pick any job name from a CI report and run it locally:

```bash theme={null}
python -m ci.praktika run "<JOB_NAME>"
```

* Always quote the job name exactly as it appears in the CI report (it may contain spaces and commas), e.g.: `"Stateless tests (amd_debug, parallel)"`. This sets up the same ClickHouse configuration and runs the same tests as in CI.
* The architecture and build type in the job name (e.g., `amd_debug`) are CI-specific labels. When running locally, they have no effect — the job will use whatever binary you provide, on whatever architecture you are running. The job name only determines the ClickHouse configuration and the test set (unless overridden with `--test`).
* In CI, functional tests are split into batches for better resource utilization. For example, `"Stateless tests (amd_debug, parallel)"` and `"Stateless tests (amd_debug, sequential)"` together cover the entire scope: parallel-safe tests run concurrently, and the rest run sequentially. The split reduces total CI time by maximizing parallelism where possible. To reproduce the full test scope locally, run both batches.
* There is also a `"Fast test"` CI job that runs a limited scope of functional tests to verify basic ClickHouse functionality — it uses a build without all optional modules and is the quickest way to catch regressions. You can run it locally the same way. Place your ClickHouse binary in one of the default search paths (`./ci/tmp/clickhouse`, `./build/programs/clickhouse`, or `./clickhouse`) — otherwise the job will attempt to build ClickHouse first:
  ```bash theme={null}
  python -m ci.praktika run "Fast test"
  ```

<h4 id="run-specific-tests-within-ci-job">
  Run Specific Tests Within a CI Job
</h4>

With `--test`, the job prepares an identical ClickHouse setup as used in CI but runs only the selected tests:

```bash theme={null}
python -m ci.praktika run "Stateless tests (amd_debug, parallel)" \
  --test 00001_select1
```

* You can pass multiple test names:
  ```bash theme={null}
  python -m ci.praktika run "Stateless tests (amd_debug, parallel)" \
    --test 00001_select1 00002_log_and_exception_messages_formatting
  ```
* Tip: If any ClickHouse configuration is acceptable and you just need to run specific tests, use the alias `functional` instead of the full job name:
  ```bash theme={null}
  python -m ci.praktika run functional --test 00001_select1
  ```

<h4 id="additional-customization-options">
  Additional Customization Options
</h4>

* `--path PATH` — custom path to the ClickHouse binary. By default, the runner searches in order: `./ci/tmp/clickhouse`, `./build/programs/clickhouse`, `./clickhouse`.
* `--count N` — repeat each test N times.
* `--workers N` — override the automatic calculation of parallel workers derived from machine capacity.

<h2 id="build-check">
  Build check
</h2>

Builds ClickHouse in various configurations for use in further steps.

<h3 id="running-builds-locally">
  Running Builds Locally
</h3>

The build can be run locally in a CI-like environment using:

```bash theme={null}
python -m ci.praktika run "<BUILD_JOB_NAME>"
```

No dependencies other than Python 3 and Docker are required.

<h4 id="available-build-jobs">
  Available Build Jobs
</h4>

The build job names are exactly as they appear in the CI Report:

**AMD64 Builds:**

* `Build (amd_debug)` - Debug build with symbols
* `Build (amd_release)` - Optimized release build
* `Build (amd_asan)` - Address Sanitizer build
* `Build (amd_tsan)` - Thread Sanitizer build
* `Build (amd_msan)` - Memory Sanitizer build
* `Build (amd_ubsan)` - Undefined Behavior Sanitizer build
* `Build (amd_binary)` - Quick release build without Thin LTO
* `Build (amd_compat)` - Compatibility build for older systems
* `Build (amd_musl)` - Build with musl libc
* `Build (amd_darwin)` - macOS build
* `Build (amd_freebsd)` - FreeBSD build

**ARM64 Builds:**

* `Build (arm_release)` - ARM64 optimized release build
* `Build (arm_asan)` - ARM64 Address Sanitizer build
* `Build (arm_coverage)` - ARM64 build with coverage instrumentation
* `Build (arm_binary)` - ARM64 Quick release build without Thin LTO
* `Build (arm_darwin)` - macOS ARM64 build
* `Build (arm_v80compat)` - ARMv8.0 compatibility build

**Other Architectures:**

* `Build (ppc64le)` - PowerPC 64-bit Little Endian
* `Build (riscv64)` - RISC-V 64-bit
* `Build (s390x)` - IBM System/390 64-bit
* `Build (loongarch64)` - LoongArch 64-bit

If the job succeeds, build results will be available in the `<repo_root>/ci/tmp/build` directory.

**Note:** For builds not in the "Other Architectures" category (which use cross-compilation), your local machine architecture must match the build type to produce the build as requested by `BUILD_JOB_NAME`.

<h4 id="example-run-local">
  Example
</h4>

To run a local debug build:

```bash theme={null}
python -m ci.praktika run "Build (amd_debug)"
```

If the above approach does not work for you, use the cmake options from the build log and follow the [general build process](/resources/develop-contribute/build/build).

<h2 id="functional-stateless-tests">
  Functional stateless tests
</h2>

Runs [stateless functional tests](/resources/develop-contribute/contribute/tests#functional-tests) for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc.
Look at the report to see which tests fail, then reproduce the failure locally as described [here](/resources/develop-contribute/contribute/tests#functional-tests).
Note that you have to use the correct build configuration to reproduce -- a test might fail under AddressSanitizer but pass in Debug.
Download the binary from [CI build checks page](/get-started/setup/self-managed/advanced), or build it locally.

<h2 id="integration-tests">
  Integration tests
</h2>

Runs [integration tests](/resources/develop-contribute/contribute/tests#integration-tests).

<h2 id="bugfix-validate-check">
  Bugfix validate check
</h2>

Checks that either a new test (functional or integration) or there some changed tests that fail with the binary built on master branch.
This check is triggered when pull request has "pr-bugfix" label.

<h2 id="stress-test">
  Stress test
</h2>

Runs stateless functional tests concurrently from several clients to detect concurrency-related errors. If it fails:

* Fix all other test failures first;
* Look at the report to find the server logs and check them for possible causes
  of error.

<h2 id="compatibility-check">
  Compatibility check
</h2>

Checks that `clickhouse` binary runs on distributions with old libc versions.
If it fails, ask a maintainer for help.

<h2 id="ast-fuzzer">
  AST fuzzer
</h2>

Runs randomly generated queries to catch program errors.
If it fails, ask a maintainer for help.

<h2 id="performance-tests">
  Performance tests
</h2>

Measure changes in query performance.
This is the longest check that takes just below 6 hours to run.
The performance test report is described in detail [here](https://github.com/ClickHouse/ClickHouse/blob/master/tests/performance/scripts/README.md#how-to-read-the-report).