You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: metadata-ingestion/docs/sources/dbt/dbt.md
+13-2Lines changed: 13 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,6 +35,11 @@ meta_mapping:
35
35
operation: "add_term"
36
36
config:
37
37
term: "Finance_test"
38
+
terms_list:
39
+
match: ".*"
40
+
operation: "add_terms"
41
+
config:
42
+
separator: ","
38
43
```
39
44
</TabItem>
40
45
<TabItem value="json" label="JSON">
@@ -66,6 +71,11 @@ meta_mapping:
66
71
"operation": "add_term",
67
72
"config": {"term": "Finance_test"},
68
73
},
74
+
"terms_list": {
75
+
"match": ".*",
76
+
"operation": "add_terms",
77
+
"config": {"separator": ","},
78
+
},
69
79
}
70
80
```
71
81
</TabItem>
@@ -74,10 +84,11 @@ meta_mapping:
74
84
We support the following operations:
75
85
1. add_tag - Requires ```tag``` property in config.
76
86
2. add_term - Requires ```term``` property in config.
77
-
3. add_owner - Requires ```owner_type``` property in config which can be either user or group. Optionally accepts the ```owner_category``` config property which you can set to one of ```['TECHNICAL_OWNER', 'BUSINESS_OWNER', 'DATA_STEWARD', 'DATAOWNER'``` (defaults to `DATAOWNER`).
87
+
3. add_terms - Accepts an optional ```separator``` property in config.
88
+
4. add_owner - Requires ```owner_type``` property in config which can be either user or group. Optionally accepts the ```owner_category``` config property which you can set to one of ```['TECHNICAL_OWNER', 'BUSINESS_OWNER', 'DATA_STEWARD', 'DATAOWNER'``` (defaults to `DATAOWNER`).
78
89
79
90
Note:
80
-
1.Currently, dbt meta mapping is only supported for meta elements defined at the model level (not supported for columns).
91
+
1.The dbt `meta_mapping` config works at the model level, while the `column_meta_mapping` config works at the column level. The `add_owner` operation is not supported at the column level.
81
92
2. For string meta properties we support regex matching.
82
93
83
94
With regex matching, you can also use the matched value to customize how you populate the tag, term or owner fields. Here are a few advanced examples:
If you have deployed DataHub using our official [helm charts](https://github.com/acryldata/datahub-helm) you can use the
4
+
datahub ingestion cron subchart to schedule ingestions.
5
+
6
+
Here is an example of what that configuration would look like in your **values.yaml**:
7
+
8
+
```yaml
9
+
datahub-ingestion-cron:
10
+
enabled: true
11
+
crons:
12
+
mysql:
13
+
schedule: "0 * * * *"# Every hour
14
+
recipe:
15
+
configmapName: recipe-config
16
+
fileName: mysql_recipe.yml
17
+
```
18
+
19
+
This assumes the pre-existence of a Kubernetes ConfigMap which holds all recipes being scheduled in the same namespace as
20
+
where the cron jobs will be running.
21
+
22
+
An example could be:
23
+
```yaml
24
+
apiVersion: v1
25
+
kind: ConfigMap
26
+
metadata:
27
+
name: recipe-config
28
+
data:
29
+
mysql_recipe.yml: |-
30
+
source:
31
+
type: mysql
32
+
config:
33
+
# Coordinates
34
+
host_port: <MYSQL HOST>:3306
35
+
database: dbname
36
+
37
+
# Credentials
38
+
username: root
39
+
password: example
40
+
41
+
sink:
42
+
type: datahub-rest
43
+
config:
44
+
server: http://<GMS_HOST>:8080
45
+
```
46
+
47
+
For more information, please see the [documentation](https://github.com/acryldata/datahub-helm/tree/master/charts/datahub/subcharts/datahub-ingestion-cron) of this sub-chart.
0 commit comments