Skip to content

User Guide

Modules

diqu CLI is built in the modular approach with 3 main ones:

  1. Source Modules: Build Data Source connections (e.g. Snowflake, csv file)
  2. Package Modules: Manage the Query of the issues captured (e.g. dq-tools's query)
  3. Alert Modules: Define how to alert/notify the issue to which platforms (e.g. Jira, Slack)

👉 See the next pages for more information on how to configure the modules

How it works

Generally, diqu executes the following steps:

  1. Takes in dbt test results from your Source Module of choice, let refer to this as the test_results table.

    • Depending on the Package Module you are using, the schema of this table might vary.
    • If you are using CSV as the Source Module, the schema of test_results should be defined by the .csv.

  2. A Query specified for this test_results's schema is executed to produce the common test metadata for Alert Module

  3. The Alert Module of choice takes the test metadata in the previous steps and sends it to the specified destinations

Issue Statuses & Issue Deprecation

Besides the basic dbt test statuses (pass, warn, error), in the built-in query for dq-tools, we introduced the new status of deprecate.

deprecate means no longer valid, and we don't need to take immediate action (erm ... hooray 🙌?).

A test is marked deprecate if it's not executed & recorded in a specified number of days.

This can defined by the ISSUE_DEPRECATED_WINDOW_IN_DAYS variable (see Query Variables Config)

Our built-in query also labels each status with its corresponding "traffic lights" 🚥 icon for easier identification (who doesn't love colorful icons in their alerts? 🎉)

  • pass(🟢)
  • warn(🟡)
  • failed(🔴)
  • deprecate(⚫)

The statuses of each executed dbt test are crucial for issue management, hence it is highly recommended to include in your custom CSV or custom Query if you're building one yourself.