dbt 0.13.0 newly added characteristic sources trial

dbt 0.13 adds a new feature sourcesfor me can be used to do the following

• Selecting data from the source table in the basis of the model
• Hypothesis test for the source data
• Source data calculatedfreshness

source operating

• Definition of source template format

  Note that for other types of pg, if the schema contains additional parameters may need to be configured, or by convention schema

# This example defines a source called `source_1` containing one table

# called `table_1`. This is a minimal example of a source definition.

​

version: 2

​

sources:

  - name: source_1

    tables:

      - name: table_1

      - name: table_2

​

  - name: source_2

    tables:

      - name: table_1

 

 

• data source schema format configuration

# This source entry describes the table:

# "raw"."public"."Orders_"

#

# It can be referenced with:

# {{ source('ecommerce', 'orders') }}

​

version: 2

​

sources:

  - name: ecommerce

    database: raw # Tell dbt to look for the source in the "raw" database

    schema: public # You wouldn't put your source data in public, would you?

    tables:

      - name: orders

        identifier: Orders_ # To alias table names to account for strange casing or naming of tables

 

 

A simple example

I configured model directly in the source folder can refer https://github.com/rongfengliang/dbt-source-demo , on table data structure
can also refer to this project

• Preparing the Environment (using python venv management)

python3 -m venv venv 

source venv/bin/activate

pip install dbt

• Test database preparation (using a docker-compose)

version: '3.6'

services:

  postgres:

    image: postgres:9.6.11

    ports: 

    - "5432:5432"

    environment:

    - "POSTGRES_PASSWORD:dalong"

  graphql-engine:

    image: hasura/graphql-engine:v1.0.0-beta.2

    ports:

    - "8080:8080"

    depends_on:

    - "postgres"

    environment:

    - "HASURA_GRAPHQL_DATABASE_URL=postgres://postgres:dalong@postgres:5432/postgres"

    - "HASURA_GRAPHQL_ENABLE_CONSOLE=true"

    - "HASURA_GRAPHQL_ENABLE_ALLOWLIST=true"

• model source configuration

models

├── apps

│ ├── app_summary.sql

│ └── sources. YML

└── users

    ├── sources. YML

    ├── user_summary.sql

    └── user_summary2.sql

• source content

  Very simple, it is to configure table

version: 2

sources:

  - name: apps

    schema: public

    tables:

      - name: apps

• running result

dbt run

effect

Running with dbt=0.13.1

Found 3 models, 0 tests, 0 archives, 0 analyses, 94 macros, 0 operations, 0 seed files, 2 sources

​

17:43:42 | Concurrency: 3 threads (target='dev')

17:43:42 | 

17:43:42 | 1 of 3 START view model public.app_summary........................... [RUN]

17:43:42 | 2 of 3 START view model public.user_summary.......................... [RUN]

17:43:42 | 3 of 3 START table model public.user_summary2........................ [RUN]

17:43:44 | 2 of 3 OK created view model public.user_summary..................... [CREATE VIEW in 0.26s]

17:43:45 | 1 of 3 OK created view model public.app_summary...................... [CREATE VIEW in 0.27s]

17:43:46 | 3 of 3 OK created table model public.user_summary2................... [SELECT 2 in 0.27s]

17:43:46 | 

17:43:46 | Finished running 2 view models, 1 table models in 4.46s.

​

Completed successfully

​

Done. PASS=3 ERROR=0 SKIP=0 TOTAL=3

Reference material

https://github.com/rongfengliang/dbt-source-demo