学习数仓工具 dbt

DBT 是一个有趣的工具，它通过一种结构化的方式定义了数仓中各种表、视图的构建和填充方式。

dbt 面相的对象是数据开发团队，提供了如下几个最有价值的能力：

  • 支持多种数据库
  • 通过 select 来定义数据，无需编写 DML
  • 构建数据时，自动分析数据依赖，按照依赖顺序生成模型
  • 版本化数据定义，支持团队 review
  • 支持数据质量测试

任何数据库都可以按照规范实现 DBT-XXX 适配器，使得你的数据库能够使用 DBT 来管理。

因为 OB 和 TiDB 都是兼容 MySQL 的，所以我先用 TiDB 的适配器来跑一下 OB，看看效果。因为 OB 对 information_schema.tables 表中的 table_type 字段内容做了一些扩展，除了 “VIEW” 和 “TABLE” 之外还引入了 “SYSTEM TABLE”, “SYSTEM VIEW”：

OceanBase(admin@test)>select distinct table_type from information_schema.tables;
+--------------+
| table_type   |
+--------------+
| BASE TABLE   |
| VIEW         |
| SYSTEM TABLE |
| SYSTEM VIEW  |
+--------------+
4 rows in set (0.008 sec)

这需要适配一下。改一下adapters.sql 即可。

/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/dbt/include/tidb/macros/adapters.sql

本文的 dbt 脚本见 https://github.com/dbt-labs/jaffle_shop

已经定义好 dbt 脚本后，数据开发工程师每次需要更新数仓中的数据时，只需要调用 dbt run 即可让所有数据更新，提升工作效率。可以反复执行 dbt run， dbt run背后会建立一些临时表，如：

customers__dbt_backup
stg_customers__dbt_tmp

运行(https://cn.pingcap.com/blog/when-tidb-meets-dbt/
) 里的例子，内部日志如下。仔细阅读，就能理解其工作原理了：

============================== 22:08:55.622663 | f43f4f98-1c20-4751-8101-18f6577098a1 ==============================
22:08:55.622663 [info ] [MainThread]: Running with dbt=1.6.9
22:08:55.622992 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'fail_fast': 'False', 'debug': 'False', 'log_path': '/Users/raywill/code/raywill/jaffle_shop/logs', 'version_check': 'True', 'profiles_dir': '/Users/raywill/.dbt', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'introspect': 'True', 'static_parser': 'True', 'log_format': 'default', 'target_path': 'None', 'invocation_command': 'dbt run', 'send_anonymous_usage_stats': 'True'}
22:08:55.662363 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10afdad10>]}
22:08:55.669168 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10a9cd050>]}
22:08:55.669586 [info ] [MainThread]: Registered adapter: tidb=1.6.4
22:08:55.678545 [debug] [MainThread]: checksum: f795ff6322dabffafa5fc1f59b34557dea99c8bef60412d1b41cabcb4e97dcce, vars: {}, profile: , target: , version: 1.6.9
22:08:55.693348 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
22:08:55.693590 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing
22:08:55.697272 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10767aa50>]}
22:08:55.703811 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10b01b950>]}
22:08:55.704055 [info ] [MainThread]: Found 5 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 349 macros, 0 groups, 0 semantic models
22:08:55.704224 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10b008050>]}
22:08:55.705247 [info ] [MainThread]:
22:08:55.705615 [debug] [MainThread]: Acquiring new tidb connection 'master'
22:08:55.706190 [debug] [ThreadPool]: Acquiring new tidb connection 'list_schemas'
22:08:55.711040 [debug] [ThreadPool]: Using tidb connection "list_schemas"
22:08:55.711272 [debug] [ThreadPool]: On list_schemas: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "connection_name": "list_schemas"} */
select distinct schema_name
        from information_schema.schemata
22:08:55.711413 [debug] [ThreadPool]: Opening a new connection, currently in state init
22:08:55.791747 [debug] [ThreadPool]: SQL status: SUCCESS 7 in 0.0 seconds
22:08:55.793001 [debug] [ThreadPool]: On list_schemas: Close
22:08:55.794166 [debug] [ThreadPool]: Re-using an available connection from the pool (formerly list_schemas, now list_None_oceanbase)
22:08:55.797532 [debug] [ThreadPool]: Using tidb connection "list_None_oceanbase"
22:08:55.797766 [debug] [ThreadPool]: On list_None_oceanbase: BEGIN
22:08:55.797922 [debug] [ThreadPool]: Opening a new connection, currently in state closed
22:08:55.872273 [debug] [ThreadPool]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:55.872627 [debug] [ThreadPool]: Using tidb connection "list_None_oceanbase"
22:08:55.872860 [debug] [ThreadPool]: On list_None_oceanbase: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "connection_name": "list_None_oceanbase"} */
select
      null as "database",
      table_name as name,
      table_schema as "schema",
      case when table_type = 'BASE TABLE' then 'table'
           when table_type = 'SYSTEM TABLE' then 'table'
           when table_type = 'SYSTEM VIEW' then 'view'
           when table_type = 'VIEW' then 'view'
           else table_type
      end as table_type
    from information_schema.tables
    where table_schema = 'oceanbase'

22:08:55.982141 [debug] [ThreadPool]: SQL status: SUCCESS 1067 in 0.0 seconds
22:08:55.991267 [debug] [ThreadPool]: On list_None_oceanbase: ROLLBACK
22:08:55.999188 [debug] [ThreadPool]: On list_None_oceanbase: Close
22:08:56.050320 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10b26a3d0>]}
22:08:56.050653 [debug] [MainThread]: Using tidb connection "master"
22:08:56.050791 [debug] [MainThread]: On master: BEGIN
22:08:56.050913 [debug] [MainThread]: Opening a new connection, currently in state init
22:08:56.112223 [debug] [MainThread]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.112550 [debug] [MainThread]: On master: COMMIT
22:08:56.112700 [debug] [MainThread]: Using tidb connection "master"
22:08:56.112843 [debug] [MainThread]: On master: COMMIT
22:08:56.127682 [debug] [MainThread]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.127872 [debug] [MainThread]: On master: Close
22:08:56.128367 [info ] [MainThread]: Concurrency: 1 threads (target='dev')
22:08:56.128558 [info ] [MainThread]:
22:08:56.129995 [debug] [Thread-1 (]: Began running node model.jaffle_shop.stg_customers
22:08:56.130305 [info ] [Thread-1 (]: 1 of 5 START sql view model oceanbase.stg_customers ............................ [RUN]
22:08:56.130725 [debug] [Thread-1 (]: Re-using an available connection from the pool (formerly list_None_oceanbase, now model.jaffle_shop.stg_customers)
22:08:56.130937 [debug] [Thread-1 (]: Began compiling node model.jaffle_shop.stg_customers
22:08:56.132859 [debug] [Thread-1 (]: Writing injected SQL for node "model.jaffle_shop.stg_customers"
22:08:56.133337 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.stg_customers (compile): 22:08:56.131077 => 22:08:56.133228
22:08:56.133526 [debug] [Thread-1 (]: Began executing node model.jaffle_shop.stg_customers
22:08:56.158597 [debug] [Thread-1 (]: Writing runtime sql for node "model.jaffle_shop.stg_customers"
22:08:56.159066 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_customers"
22:08:56.159227 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: BEGIN
22:08:56.159372 [debug] [Thread-1 (]: Opening a new connection, currently in state closed
22:08:56.217134 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.217351 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_customers"
22:08:56.217555 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_customers"} */

  create view `oceanbase`.`stg_customers__dbt_tmp`


  as (
    with source as (
    select * from `oceanbase`.`raw_customers`

),

renamed as (

    select
        id as customer_id,
        first_name,
        last_name

    from source

)

select * from renamed
  );
22:08:56.329009 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.364055 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_customers"
22:08:56.364679 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_customers"} */

    drop view if exists `oceanbase`.`stg_customers__dbt_backup` cascade

22:08:56.388533 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.390391 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_customers"
22:08:56.390649 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_customers"} */

    rename table `oceanbase`.`stg_customers` to `oceanbase`.`stg_customers__dbt_backup`

22:08:56.482864 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.489215 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_customers"
22:08:56.489815 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_customers"} */

    drop view if exists `oceanbase`.`stg_customers` cascade

22:08:56.510550 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.513581 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_customers"
22:08:56.514166 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_customers"} */

    rename table `oceanbase`.`stg_customers__dbt_tmp` to `oceanbase`.`stg_customers`

22:08:56.600139 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.620564 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: COMMIT
22:08:56.621050 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_customers"
22:08:56.621314 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: COMMIT
22:08:56.636866 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.642164 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_customers"
22:08:56.642473 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_customers"} */
drop view if exists `oceanbase`.`stg_customers__dbt_backup`
22:08:56.719959 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.724716 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.stg_customers (execute): 22:08:56.133648 => 22:08:56.724313
22:08:56.725436 [debug] [Thread-1 (]: On model.jaffle_shop.stg_customers: Close
22:08:56.727114 [debug] [Thread-1 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10adad4d0>]}
22:08:56.728088 [info ] [Thread-1 (]: 1 of 5 OK created sql view model oceanbase.stg_customers ....................... [SUCCESS 0 in 0.60s]
22:08:56.728829 [debug] [Thread-1 (]: Finished running node model.jaffle_shop.stg_customers
22:08:56.729372 [debug] [Thread-1 (]: Began running node model.jaffle_shop.stg_orders
22:08:56.729887 [info ] [Thread-1 (]: 2 of 5 START sql view model oceanbase.stg_orders ............................... [RUN]
22:08:56.730682 [debug] [Thread-1 (]: Re-using an available connection from the pool (formerly model.jaffle_shop.stg_customers, now model.jaffle_shop.stg_orders)
22:08:56.731039 [debug] [Thread-1 (]: Began compiling node model.jaffle_shop.stg_orders
22:08:56.734512 [debug] [Thread-1 (]: Writing injected SQL for node "model.jaffle_shop.stg_orders"
22:08:56.735763 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.stg_orders (compile): 22:08:56.731390 => 22:08:56.735555
22:08:56.736135 [debug] [Thread-1 (]: Began executing node model.jaffle_shop.stg_orders
22:08:56.752144 [debug] [Thread-1 (]: Writing runtime sql for node "model.jaffle_shop.stg_orders"
22:08:56.752639 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_orders"
22:08:56.752858 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: BEGIN
22:08:56.753063 [debug] [Thread-1 (]: Opening a new connection, currently in state closed
22:08:56.817242 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.817668 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_orders"
22:08:56.818016 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_orders"} */

  create view `oceanbase`.`stg_orders__dbt_tmp`


  as (
    with source as (
    select * from `oceanbase`.`raw_orders`

),

renamed as (

    select
        id as order_id,
        user_id as customer_id,
        order_date,
        status

    from source

)

select * from renamed
  );
22:08:56.896459 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.903640 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_orders"
22:08:56.903963 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_orders"} */

    drop view if exists `oceanbase`.`stg_orders__dbt_backup` cascade

22:08:56.925139 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.926137 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_orders"
22:08:56.926412 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_orders"} */

    rename table `oceanbase`.`stg_orders` to `oceanbase`.`stg_orders__dbt_backup`

22:08:56.994692 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:56.997962 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_orders"
22:08:56.998295 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_orders"} */

    drop view if exists `oceanbase`.`stg_orders` cascade

22:08:57.016278 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.017341 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_orders"
22:08:57.017666 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_orders"} */

    rename table `oceanbase`.`stg_orders__dbt_tmp` to `oceanbase`.`stg_orders`

22:08:57.091437 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.097185 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: COMMIT
22:08:57.097653 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_orders"
22:08:57.097973 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: COMMIT
22:08:57.113205 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.119160 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_orders"
22:08:57.119605 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_orders"} */
drop view if exists `oceanbase`.`stg_orders__dbt_backup`
22:08:57.191539 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.196838 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.stg_orders (execute): 22:08:56.736369 => 22:08:57.196496
22:08:57.197465 [debug] [Thread-1 (]: On model.jaffle_shop.stg_orders: Close
22:08:57.198946 [debug] [Thread-1 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10b00e090>]}
22:08:57.199806 [info ] [Thread-1 (]: 2 of 5 OK created sql view model oceanbase.stg_orders .......................... [SUCCESS 0 in 0.47s]
22:08:57.200556 [debug] [Thread-1 (]: Finished running node model.jaffle_shop.stg_orders
22:08:57.201094 [debug] [Thread-1 (]: Began running node model.jaffle_shop.stg_payments
22:08:57.201612 [info ] [Thread-1 (]: 3 of 5 START sql view model oceanbase.stg_payments ............................. [RUN]
22:08:57.202546 [debug] [Thread-1 (]: Re-using an available connection from the pool (formerly model.jaffle_shop.stg_orders, now model.jaffle_shop.stg_payments)
22:08:57.202993 [debug] [Thread-1 (]: Began compiling node model.jaffle_shop.stg_payments
22:08:57.206589 [debug] [Thread-1 (]: Writing injected SQL for node "model.jaffle_shop.stg_payments"
22:08:57.207501 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.stg_payments (compile): 22:08:57.203264 => 22:08:57.207308
22:08:57.207855 [debug] [Thread-1 (]: Began executing node model.jaffle_shop.stg_payments
22:08:57.224227 [debug] [Thread-1 (]: Writing runtime sql for node "model.jaffle_shop.stg_payments"
22:08:57.224759 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_payments"
22:08:57.224987 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: BEGIN
22:08:57.225194 [debug] [Thread-1 (]: Opening a new connection, currently in state closed
22:08:57.286954 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.287364 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_payments"
22:08:57.287725 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_payments"} */

  create view `oceanbase`.`stg_payments__dbt_tmp`


  as (
    with source as (
    select * from `oceanbase`.`raw_payments`

),

renamed as (

    select
        id as payment_id,
        order_id,
        payment_method,

        -- `amount` is currently stored in cents, so we convert it to dollars
        amount / 100 as amount

    from source

)

select * from renamed
  );
22:08:57.369772 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.381916 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_payments"
22:08:57.382347 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_payments"} */

    drop view if exists `oceanbase`.`stg_payments__dbt_backup` cascade

22:08:57.404650 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.406090 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_payments"
22:08:57.406480 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_payments"} */

    rename table `oceanbase`.`stg_payments` to `oceanbase`.`stg_payments__dbt_backup`

22:08:57.476087 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.482015 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_payments"
22:08:57.482508 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_payments"} */

    drop view if exists `oceanbase`.`stg_payments` cascade

22:08:57.504008 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.507457 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_payments"
22:08:57.508634 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_payments"} */

    rename table `oceanbase`.`stg_payments__dbt_tmp` to `oceanbase`.`stg_payments`

22:08:57.584623 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.589099 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: COMMIT
22:08:57.589588 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_payments"
22:08:57.589984 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: COMMIT
22:08:57.607837 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.614888 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.stg_payments"
22:08:57.615358 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.stg_payments"} */
drop view if exists `oceanbase`.`stg_payments__dbt_backup`
22:08:57.690624 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.695481 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.stg_payments (execute): 22:08:57.208062 => 22:08:57.695113
22:08:57.696561 [debug] [Thread-1 (]: On model.jaffle_shop.stg_payments: Close
22:08:57.697780 [debug] [Thread-1 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10b02da10>]}
22:08:57.698487 [info ] [Thread-1 (]: 3 of 5 OK created sql view model oceanbase.stg_payments ........................ [SUCCESS 0 in 0.50s]
22:08:57.699066 [debug] [Thread-1 (]: Finished running node model.jaffle_shop.stg_payments
22:08:57.700156 [debug] [Thread-1 (]: Began running node model.jaffle_shop.customers
22:08:57.700680 [info ] [Thread-1 (]: 4 of 5 START sql table model oceanbase.customers ............................... [RUN]
22:08:57.701414 [debug] [Thread-1 (]: Re-using an available connection from the pool (formerly model.jaffle_shop.stg_payments, now model.jaffle_shop.customers)
22:08:57.701787 [debug] [Thread-1 (]: Began compiling node model.jaffle_shop.customers
22:08:57.707287 [debug] [Thread-1 (]: Writing injected SQL for node "model.jaffle_shop.customers"
22:08:57.708262 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.customers (compile): 22:08:57.702033 => 22:08:57.708100
22:08:57.708561 [debug] [Thread-1 (]: Began executing node model.jaffle_shop.customers
22:08:57.737060 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:57.737355 [debug] [Thread-1 (]: On model.jaffle_shop.customers: BEGIN
22:08:57.737543 [debug] [Thread-1 (]: Opening a new connection, currently in state closed
22:08:57.794172 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.794493 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:57.794774 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */


        create view
            oceanbase.temp_view
        as (
            with customers as (

    select * from `oceanbase`.`stg_customers`

),

orders as (

    select * from `oceanbase`.`stg_orders`

),

payments as (

    select * from `oceanbase`.`stg_payments`

),

customer_orders as (

        select
        customer_id,

        min(order_date) as first_order,
        max(order_date) as most_recent_order,
        count(order_id) as number_of_orders
    from orders

    group by customer_id

),

customer_payments as (

    select
        orders.customer_id,
        sum(amount) as total_amount

    from payments

    left join orders on
         payments.order_id = orders.order_id

    group by orders.customer_id

),

final as (

    select
        customers.customer_id,
        customers.first_name,
        customers.last_name,
        customer_orders.first_order,
        customer_orders.most_recent_order,
        customer_orders.number_of_orders,
        customer_payments.total_amount as customer_lifetime_value

    from customers

    inner join customer_orders
        on customers.customer_id = customer_orders.customer_id

    inner join customer_payments
        on  customers.customer_id = customer_payments.customer_id

)

select * from final
        )

22:08:57.895098 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:57.898817 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:57.899487 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */

        show columns from oceanbase.temp_view

22:08:57.924479 [debug] [Thread-1 (]: SQL status: SUCCESS 7 in 0.0 seconds
22:08:57.926493 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:57.926786 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */

        create   table if not exists
        `oceanbase`.`customers__dbt_tmp`
        (

                `customer_id`
                int(11)
                 not null
                ,

                `first_name`
                text
                 not null
                ,

                `last_name`
                text
                 not null
                ,

                `first_order`
                date
                 not null
                ,

                `most_recent_order`
                date
                 not null
                ,

                `number_of_orders`
                bigint(20)
                 not null
                ,

                `customer_lifetime_value`
                decimal(37,4)
                 not null


        )

22:08:58.097120 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.101002 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:58.101631 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */

        insert into `oceanbase`.`customers__dbt_tmp` select * from oceanbase.temp_view

22:08:58.153432 [debug] [Thread-1 (]: SQL status: SUCCESS 62 in 0.0 seconds
22:08:58.156432 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:58.157239 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */

        drop view if exists oceanbase.temp_view

22:08:58.227587 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.229403 [debug] [Thread-1 (]: Writing runtime sql for node "model.jaffle_shop.customers"
22:08:58.230580 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:58.231106 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */





















22:08:58.245210 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.252855 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:58.253059 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */

    drop table if exists `oceanbase`.`customers__dbt_backup` cascade

22:08:58.274641 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.276792 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:58.277412 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */

    rename table `oceanbase`.`customers` to `oceanbase`.`customers__dbt_backup`

22:08:58.348851 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.354702 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:58.355026 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */

    drop table if exists `oceanbase`.`customers` cascade

22:08:58.375974 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.378890 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:58.379742 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */

    rename table `oceanbase`.`customers__dbt_tmp` to `oceanbase`.`customers`

22:08:58.449860 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.461118 [debug] [Thread-1 (]: On model.jaffle_shop.customers: COMMIT
22:08:58.461648 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:58.461999 [debug] [Thread-1 (]: On model.jaffle_shop.customers: COMMIT
22:08:58.476014 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.479457 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.customers"
22:08:58.479875 [debug] [Thread-1 (]: On model.jaffle_shop.customers: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.customers"} */
drop table if exists `oceanbase`.`customers__dbt_backup`
22:08:58.583995 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.585602 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.customers (execute): 22:08:57.708757 => 22:08:58.585436
22:08:58.585951 [debug] [Thread-1 (]: On model.jaffle_shop.customers: Close
22:08:58.586683 [debug] [Thread-1 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10c884f10>]}
22:08:58.587183 [info ] [Thread-1 (]: 4 of 5 OK created sql table model oceanbase.customers .......................... [SUCCESS 0 in 0.89s]
22:08:58.587656 [debug] [Thread-1 (]: Finished running node model.jaffle_shop.customers
22:08:58.587999 [debug] [Thread-1 (]: Began running node model.jaffle_shop.orders
22:08:58.588335 [info ] [Thread-1 (]: 5 of 5 START sql table model oceanbase.orders .................................. [RUN]
22:08:58.588989 [debug] [Thread-1 (]: Re-using an available connection from the pool (formerly model.jaffle_shop.customers, now model.jaffle_shop.orders)
22:08:58.589298 [debug] [Thread-1 (]: Began compiling node model.jaffle_shop.orders
22:08:58.592665 [debug] [Thread-1 (]: Writing injected SQL for node "model.jaffle_shop.orders"
22:08:58.593351 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.orders (compile): 22:08:58.589484 => 22:08:58.593212
22:08:58.593607 [debug] [Thread-1 (]: Began executing node model.jaffle_shop.orders
22:08:58.607013 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:58.607297 [debug] [Thread-1 (]: On model.jaffle_shop.orders: BEGIN
22:08:58.607492 [debug] [Thread-1 (]: Opening a new connection, currently in state closed
22:08:58.660110 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.660481 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:58.660790 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */


        create view
            oceanbase.temp_view
        as (


with orders as (

    select * from `oceanbase`.`stg_orders`

),

payments as (

    select * from `oceanbase`.`stg_payments`

),

order_payments as (

    select
        order_id,

        sum(case when payment_method = 'credit_card' then amount else 0 end) as credit_card_amount,
        sum(case when payment_method = 'coupon' then amount else 0 end) as coupon_amount,
        sum(case when payment_method = 'bank_transfer' then amount else 0 end) as bank_transfer_amount,
        sum(case when payment_method = 'gift_card' then amount else 0 end) as gift_card_amount,
        sum(amount) as total_amount

    from payments

    group by order_id

),

final as (

    select
        orders.order_id,
        orders.customer_id,
        orders.order_date,
        orders.status,

        order_payments.credit_card_amount,

        order_payments.coupon_amount,

        order_payments.bank_transfer_amount,

        order_payments.gift_card_amount,

        order_payments.total_amount as amount

    from orders


    left join order_payments
        on orders.order_id = order_payments.order_id

)

select * from final
        )

22:08:58.746833 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.749953 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:58.750953 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */

        show columns from oceanbase.temp_view

22:08:58.774376 [debug] [Thread-1 (]: SQL status: SUCCESS 9 in 0.0 seconds
22:08:58.777285 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:58.778144 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */

        create   table if not exists
        `oceanbase`.`orders__dbt_tmp`
        (

                `order_id`
                int(11)
                 not null
                ,

                `customer_id`
                int(11)
                 not null
                ,

                `order_date`
                date
                 not null
                ,

                `status`
                text
                 not null
                ,

                `credit_card_amount`
                decimal(37,4)
                 not null
                ,

                `coupon_amount`
                decimal(37,4)
                 not null
                ,

                `bank_transfer_amount`
                decimal(37,4)
                 not null
                ,

                `gift_card_amount`
                decimal(37,4)
                 not null
                ,

                `amount`
                decimal(37,4)
                 not null


        )

22:08:58.901266 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:58.904156 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:58.904774 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */

        insert into `oceanbase`.`orders__dbt_tmp` select * from oceanbase.temp_view

22:08:58.942316 [debug] [Thread-1 (]: SQL status: SUCCESS 99 in 0.0 seconds
22:08:58.944322 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:58.944929 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */

        drop view if exists oceanbase.temp_view

22:08:59.018315 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.021627 [debug] [Thread-1 (]: Writing runtime sql for node "model.jaffle_shop.orders"
22:08:59.023735 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:59.024275 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */





















22:08:59.039661 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.052185 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:59.052598 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */

    drop table if exists `oceanbase`.`orders__dbt_backup` cascade

22:08:59.071808 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.073143 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:59.073565 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */

    rename table `oceanbase`.`orders` to `oceanbase`.`orders__dbt_backup`

22:08:59.150592 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.156193 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:59.156664 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */

    drop table if exists `oceanbase`.`orders` cascade

22:08:59.176531 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.179595 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:59.180285 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */

    rename table `oceanbase`.`orders__dbt_tmp` to `oceanbase`.`orders`

22:08:59.248180 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.253036 [debug] [Thread-1 (]: On model.jaffle_shop.orders: COMMIT
22:08:59.253797 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:59.254205 [debug] [Thread-1 (]: On model.jaffle_shop.orders: COMMIT
22:08:59.268802 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.275387 [debug] [Thread-1 (]: Using tidb connection "model.jaffle_shop.orders"
22:08:59.275828 [debug] [Thread-1 (]: On model.jaffle_shop.orders: /* {"app": "dbt", "dbt_version": "1.6.9", "profile_name": "jaffle_shop", "target_name": "dev", "node_id": "model.jaffle_shop.orders"} */
drop table if exists `oceanbase`.`orders__dbt_backup`
22:08:59.460776 [debug] [Thread-1 (]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.465491 [debug] [Thread-1 (]: Timing info for model.jaffle_shop.orders (execute): 22:08:58.593775 => 22:08:59.465044
22:08:59.466131 [debug] [Thread-1 (]: On model.jaffle_shop.orders: Close
22:08:59.467397 [debug] [Thread-1 (]: Sending event: {'category': 'dbt', 'action': 'run_model', 'label': 'f43f4f98-1c20-4751-8101-18f6577098a1', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10c885950>]}
22:08:59.468191 [info ] [Thread-1 (]: 5 of 5 OK created sql table model oceanbase.orders ............................. [SUCCESS 0 in 0.88s]
22:08:59.468896 [debug] [Thread-1 (]: Finished running node model.jaffle_shop.orders
22:08:59.471036 [debug] [MainThread]: Using tidb connection "master"
22:08:59.471461 [debug] [MainThread]: On master: BEGIN
22:08:59.471741 [debug] [MainThread]: Opening a new connection, currently in state closed
22:08:59.556370 [debug] [MainThread]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.557436 [debug] [MainThread]: On master: COMMIT
22:08:59.557989 [debug] [MainThread]: Using tidb connection "master"
22:08:59.558425 [debug] [MainThread]: On master: COMMIT
22:08:59.574008 [debug] [MainThread]: SQL status: SUCCESS 0 in 0.0 seconds
22:08:59.574684 [debug] [MainThread]: On master: Close
22:08:59.576154 [debug] [MainThread]: Connection 'master' was properly closed.
22:08:59.576798 [debug] [MainThread]: Connection 'model.jaffle_shop.orders' was properly closed.
22:08:59.577387 [info ] [MainThread]:
22:08:59.578056 [info ] [MainThread]: Finished running 3 view models, 2 table models in 0 hours 0 minutes and 3.87 seconds (3.87s).
22:08:59.579442 [debug] [MainThread]: Command end result
22:08:59.591150 [info ] [MainThread]:
22:08:59.591525 [info ] [MainThread]: Completed successfully
22:08:59.591792 [info ] [MainThread]:
22:08:59.592064 [info ] [MainThread]: Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5
22:08:59.592567 [debug] [MainThread]: Command `dbt run` succeeded at 22:08:59.592489 after 4.01 seconds
22:08:59.592899 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10a4eff10>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1046a7f50>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1046b4dd0>]}
22:08:59.593216 [debug] [MainThread]: Flushing usage events

对于 view stg_customers，dbt 的处理过程如下：

create view `oceanbase`.`stg_customers__dbt_tmp` as ...;
drop view if exists `oceanbase`.`stg_customers__dbt_backup` cascade;
rename table `oceanbase`.`stg_customers` to `oceanbase`.`stg_customers__dbt_backup`;
drop view if exists `oceanbase`.`stg_customers` cascade;
rename table `oceanbase`.`stg_customers__dbt_tmp` to `oceanbase`.`stg_customers`;

可见，它依靠创建临时 view，操作无误后通过 rename 的方式保证“原子性”。

对于 table orders，dbt 的处理过程如下：

create view   oceanbase.temp_view      as ( ... select * from final  );
create   table if not exists  `oceanbase`.`orders__dbt_tmp`  (...);
insert into `oceanbase`.`orders__dbt_tmp` select * from oceanbase.temp_view 
drop view if exists oceanbase.temp_view;
drop table if exists `oceanbase`.`orders__dbt_backup` cascade;
rename table `oceanbase`.`orders` to `oceanbase`.`orders__dbt_backup`;
drop table if exists `oceanbase`.`orders` cascade;
rename table `oceanbase`.`orders__dbt_tmp` to `oceanbase`.`orders`;

这里面，它创建了一个临时 temp_view，然后把 temp_view 的内容插入到 orders 表里。为了保证原子性，也使用了类似 view 的手法，创建了一些临时表。

作为一个完整的产品，dbt 还提供了一些贴心功能给数据使用者：
dbt docs generate 能够给数仓表生成文档，dbt docs serve 能启动一个服务用于浏览这些文档。真的很贴心。
在这里插入图片描述
在这里插入图片描述

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：/a/401761.html

如若内容造成侵权/违法违规/事实不符，请联系我们进行投诉反馈qq邮箱809451989@qq.com，一经查实，立即删除！

相关文章

色彩搭配：打造视觉吸引力与用户体验的关键

色彩搭配：打造视觉吸引力与用户体验的关键

title: 色彩搭配&#xff1a;打造视觉吸引力与用户体验的关键 date: 2024/2/22 12:01:11 updated: 2024/2/22 12:01:11 tags: 网站色彩搭配视觉吸引力品牌形象用户体验设计色彩心理学配色技巧色轮互补 在当今数字化时代&#xff0c;网站已经成为了人们获取信息、进行交流和进行…
阅读更多...
嵌入式学习之Linux入门篇——使用VMware创建Unbuntu虚拟机

嵌入式学习之Linux入门篇——使用VMware创建Unbuntu虚拟机

目录 主机硬件要求 VMware 安装 安装Unbuntu 18.04.6 LTS 新建虚拟机 进入Unbuntu安装环节 主机硬件要求 内存最少16G 硬盘最好分出一个单独的盘&#xff0c;而且最少预留200G&#xff0c;可以使用移动固态操作系统win7/10/11 VMware 安装 版本&#xff1a;VMware Works…
阅读更多...
Jmeter内置变量 vars 和props的使用详解

Jmeter内置变量 vars 和props的使用详解

JMeter是一个功能强大的负载测试工具&#xff0c;它提供了许多有用的内置变量来支持测试过程。其中最常用的变量是 vars 和 props。 vars 变量 vars 变量是线程本地变量&#xff0c;它们只能在同一线程组内的所有线程中使用&#xff08;线程组内不同线程之间变量不共享&#…
阅读更多...
机器学习——正规方程

机器学习——正规方程

正规方程的基本介绍 之前我们使用梯度下降算法求代价函数J(θ)的最小值&#xff0c;而梯度下降算法是通过一步步不断地迭代来收敛到全局最小值&#xff0c;如下 而正规方程则是另一种求解J(θ)最小值的方法&#xff0c;并且正规方程不需要通过迭代&#xff0c;而是一次性得到θ…
阅读更多...
体育网站的比分、赛事数据一般从哪里获取？

体育网站的比分、赛事数据一般从哪里获取？

像一般的体育类门户网站&#xff0c;或者是APP产品&#xff0c;换句话说&#xff0c;不是专业做数据的公司&#xff0c;基本上都是购买付费的api接口&#xff0c;越是大公司越是依靠从大的服务商处购买的。比如说whoscored这样的网站&#xff0c;以及像曼城、利物浦这样的俱乐部…
阅读更多...
跨境电商本土化运营：深度融合本地市场，提升用户体验与市场份额

跨境电商本土化运营：深度融合本地市场，提升用户体验与市场份额

随着全球经济的不断发展&#xff0c;跨境电商在国际贸易中扮演着越来越重要的角色。然而&#xff0c;单一地面对全球市场可能并不足以满足用户的多样化需求&#xff0c;因此&#xff0c;跨境电商需要与本地市场深度融合&#xff0c;实现本土化运营。本文Nox聚星将和大家探讨跨境…
阅读更多...
软件兼容性测试要考虑什么？

软件兼容性测试要考虑什么？

1、向前兼容和向后兼容。向前兼容是指可以使用软件的未来版本&#xff0c;向后兼容是指可以使用软件的以前版本。并非所有的软件都要求向前兼容和向后兼容&#xff0c;这是软件设计者需要决定的产品特性。 2、不同版本之间的兼容。不同版本之间的兼容指要实现测试平台和应用软…
阅读更多...
【elasticsearch实战】知识库文件系统检索工具FSCrawler

【elasticsearch实战】知识库文件系统检索工具FSCrawler

需求背景 最近有一个需求需要建设一个知识库文档检索系统&#xff0c;这些知识库物料附件的文档居多&#xff0c;有较多文档格式如&#xff1a;PDF, Open Office, MS Office等&#xff0c;需要将这些格式的文件转化成文本格式&#xff0c;写入elasticsearch 的全文检索索引&am…
阅读更多...
解决app中以webview的方式嵌入h5网页，h5网页加载不出来

解决app中以webview的方式嵌入h5网页，h5网页加载不出来

问题描述&#xff1a;我的h5网页在web端和手机浏览器都能正常渲染展示&#xff0c;但是嵌入到客户的webview中&#xff0c;渲染加载不出来&#xff0c;仔细检查代码之后并没有任何代码错误和后台报错。抓耳挠腮查找两天之后发现&#xff0c;原因为整个h5网页的最外层高度设置成…
阅读更多...
六、回归与聚类算法 - 线性回归

六、回归与聚类算法 - 线性回归

目录 1、线性回归的原理 1.1 应用场景 1.2 什么是线性回归 1.2.1 定义 1.2.2 线性回归的特征与目标的关系分析 2、线性回归的损失和优化原理 2.1 损失函数 2.2 优化算法 2.2.1 正规方程 2.2.2 梯度下降 3、线性回归API 4、回归性能评估 5、波士顿房价预测 5.1 流…
阅读更多...
打造纯Lua组件化开发模式：Unity xLua框架详解

打造纯Lua组件化开发模式：Unity xLua框架详解

在传统的Unity开发中&#xff0c;通常会使用C#来编写游戏逻辑和组件。但是&#xff0c;随着Lua在游戏开发中的应用越来越广泛&#xff0c;我们可以将游戏逻辑和组件完全用Lua来实现&#xff0c;实现纯Lua的组件化开发模式。这样做的好处是可以更加灵活地修改游戏逻辑&#xff0…
阅读更多...
Python: argparse基本用法

Python: argparse基本用法

Python: argparse基本用法 &#x1f308; 个人主页&#xff1a;高斯小哥 &#x1f525; 高质量专栏&#xff1a;【Matplotlib之旅&#xff1a;零基础精通数据可视化】 &#x1f4a1; 创作高质量博文&#xff0c;分享更多关于深度学习、PyTorch、Python领域的优质内容&#xff0…
阅读更多...
小程序--引入vant组件库

小程序--引入vant组件库

一、npm初始化 在微信开发者工具中打开终端&#xff0c;输入npm init&#xff0c;对npm进行初始化 二、安装vant组件库 npm install vant/weapp 三、修改app.json 修改 app.json&#xff0c;移除全局配置 "style": "v2"&#xff0c;否则 Vant 组件的样式…
阅读更多...
Vectorlabs橙黄网胞盘菌凝集素（Aleuria Aurantia Lectin）

Vectorlabs橙黄网胞盘菌凝集素（Aleuria Aurantia Lectin）

与荆豆凝集素和莲藕凝集素偏好&#xff08;α-1,2&#xff09;连接的岩藻糖残基不同&#xff0c;AAL由2个相同36kDa大小的亚基组成的二聚体&#xff0c;与fucose linked (α -1,6) to N-acetylglucosamine或fucose linked (α -1,3) to N-acetyllactosamine相关结构结合。AAL也…
阅读更多...
Linux多线程服务端编程：使用muduo C++网络库 学习笔记 第十二章 C++经验谈（二）

Linux多线程服务端编程：使用muduo C++网络库 学习笔记 第十二章 C++经验谈（二）

12.8.4 用partition()实现“重排数组&#xff0c;让奇数位于偶数前面” std::partition()的作用是把符合条件的元素放到区间首部&#xff0c;不符合条件的元素放到区间后部&#xff0c;我们只需把“符合条件”定义为“元素是奇数”就能解决这道题。复杂度是O(N)时间和O(1)空间…
阅读更多...
Web 3超入门—踏上Web 3.0的征程：超入门探索指南【文末送书-21】

Web 3超入门—踏上Web 3.0的征程：超入门探索指南【文末送书-21】

文章目录 Web 3超入门—踏上Web 3.0的征程&#xff1a;超入门探索指南1. 什么是Web 3.0&#xff1f;2. 区块链技术3. 去中心化应用&#xff08;DApps&#xff09;4. 数字身份和隐私5. 通证经济6. Web 3.0的应用领域Web 3超入门【文末送书-21】 Web 3超入门—踏上Web 3.0的征程&…
阅读更多...
TSINGSEE青犀AI智能分析网关V4初始配置与算法相关配置介绍

TSINGSEE青犀AI智能分析网关V4初始配置与算法相关配置介绍

TSINGSEE青犀AI智能分析网关V4内置了近40种AI算法模型&#xff0c;支持对接入的视频图像进行人、车、物、行为等实时检测分析&#xff0c;上报识别结果&#xff0c;并能进行语音告警播放。硬件管理平台支持RTSP、GB28181协议、以及厂家私有协议接入&#xff0c;可兼容市面上常见…
阅读更多...
OM6650AM 一款低功耗车规级蓝牙5.1SoC芯片

OM6650AM 一款低功耗车规级蓝牙5.1SoC芯片

OM6650AM是一款超低功耗、同时支持蓝牙5.1协议栈与2.4GHz私有协议的双模无线连接SoC芯片&#xff0c;采用4.0 mm x 4.0 mm QFN32封装&#xff0c;具有丰富的资源&#xff0c;极低的功耗&#xff0c;优异的射频性能&#xff0c;可广泛应用于车载数字钥匙模组、胎压检测、PKE钥匙…
阅读更多...
机器视觉技术的演进：YOLO系列与Halcon的深度对比

机器视觉技术的演进：YOLO系列与Halcon的深度对比

YOLO系列的发展历程 YOLO&#xff0c;作为一种流行的实时目标检测算法&#xff0c;自2015年首次被提出以来&#xff0c;经历了多个版本的迭代。最初的YOLOv1因其独特的单次检测框架而备受关注&#xff0c;它将图像分割成网格&#xff0c;并对每个网格预测多个边界框和类别概率&…
阅读更多...
vue-利用属性（v-if）控制表单（el-form-item）显示/隐藏

vue-利用属性（v-if）控制表单（el-form-item）显示/隐藏

表单控制属性 v-if 示例&#xff1a; 通过switch组件作为开关&#xff0c;控制表单的显示与隐藏 <el-form-item label"创建数据集"><el-switch v-model"selectFormVisible"></el-switch></el-form-item><el-form-item label&…
阅读更多...
最新文章

Copyright @ 2022~2023