refractor(snowflake): move snowflake-beta to certified snowflake source

Author: mayurinehate

docs/cli.md

@@ -94,7 +94,6 @@ We use a plugin architecture so that you can install only the dependencies you a
 | [redshift](./generated/ingestion/sources/redshift.md)                           | `pip install 'acryl-datahub[redshift]'`                    | Redshift source                     |
 | [sagemaker](./generated/ingestion/sources/sagemaker.md)                         | `pip install 'acryl-datahub[sagemaker]'`                   | AWS SageMaker source                |
 | [snowflake](./generated/ingestion/sources/snowflake.md)                         | `pip install 'acryl-datahub[snowflake]'`                   | Snowflake source                    |
-| [snowflake-usage](./generated/ingestion/sources/snowflake.md#module-snowflake-usage)                   | `pip install 'acryl-datahub[snowflake-usage]'`             | Snowflake usage statistics source   |
 | [sqlalchemy](./generated/ingestion/sources/sqlalchemy.md)                       | `pip install 'acryl-datahub[sqlalchemy]'`                  | Generic SQLAlchemy source           |
 | [superset](./generated/ingestion/sources/superset.md)                           | `pip install 'acryl-datahub[superset]'`                    | Superset source                     |
 | [tableau](./generated/ingestion/sources/tableau.md)                             | `pip install 'acryl-datahub[tableau]'`                     | Tableau source                      |

docs/how/updating-datahub.md

@@ -9,6 +9,7 @@ This file documents any backwards-incompatible changes in DataHub and assists pe
 - Browse Paths have been upgraded to a new format to align more closely with the intention of the feature. 
   Learn more about the changes, including steps on upgrading, here: https://datahubproject.io/docs/advanced/browse-paths-upgrade
 - The dbt ingestion source's `disable_dbt_node_creation` and `load_schema` options have been removed. They were no longer necessary due to the recently added sibling entities functionality.
+- The `snowflake` source now uses newer faster implementation (earlier `snowflake-beta`). Config properties `provision_role` and `check_role_grants` are not supported. Older `snowflake` and `snowflake-usage` are available as `snowflake-legacy` and `snowflake-usage-legacy` sources respectively.
 
 ### Potential Downtime
 

metadata-ingestion/docs/dev_guides/stateful.md

@@ -105,7 +105,7 @@ Note that a `.` is used to denote nested fields in the YAML recipe.
 #### Sample Configuration
 ```yaml
 source:
-  type: "snowflake-usage"
+  type: "snowflake-usage-legacy"
   config:
     username: <user_name>
     password: <password>

metadata-ingestion/docs/sources/snowflake/README.md

@@ -1,9 +1,7 @@
-Ingesting metadata from Snowflake requires either using the **snowflake-beta** module with just one recipe (recommended) or the two separate modules **snowflake** and **snowflake-usage** (soon to be deprecated) with two separate recipes. 
+Ingesting metadata from Snowflake requires either using the **snowflake** module with just one recipe (recommended) or the two separate modules **snowflake-legacy** and **snowflake-usage-legacy** (soon to be deprecated) with two separate recipes. 
 
 All three modules are described on this page. 
 
-We encourage you to try out the new **snowflake-beta** plugin as alternative to running both **snowflake** and **snowflake-usage** plugins and share feedback. `snowflake-beta` is much faster than `snowflake` for extracting metadata.
-
 ## Snowflake Ingestion through the UI
 
 The following video shows you how to ingest Snowflake metadata through the UI.

metadata-ingestion/docs/sources/snowflake/snowflake.md renamed to metadata-ingestion/docs/sources/snowflake/snowflake-legacy.md

@@ -1,11 +1,20 @@
 source:
-  type: snowflake-beta
+  type: snowflake-legacy
   config:
+
+    check_role_grants: True
+    provision_role: # Optional
+      enabled: false
+      dry_run: true
+      run_ingestion: false
+      admin_username: "${SNOWFLAKE_ADMIN_USER}"
+      admin_password: "${SNOWFLAKE_ADMIN_PASS}"
+
     # This option is recommended to be used for the first time to ingest all lineage
     ignore_start_time_lineage: true
     # This is an alternative option to specify the start_time for lineage
     # if you don't want to look back since beginning
-    start_time: "2022-03-01T00:00:00Z"
+    start_time: '2022-03-01T00:00:00Z'
 
     # Coordinates
     account_id: "abc48144"
@@ -16,12 +25,14 @@ source:
     password: "${SNOWFLAKE_PASS}"
     role: "datahub_role"
 
-    # Change these as per your database names. Remove to get all databases
+    # Change these as per your database names. Remove to all all databases
     database_pattern:
       allow:
         - "^ACCOUNTING_DB$"
         - "^MARKETING_DB$"
-
+    schema_pattern:
+      deny:
+        - "information_schema.*"
     table_pattern:
       allow:
         # If you want to ingest only few tables with name revenue and sales
@@ -31,10 +42,12 @@ source:
     profiling:
       # Change to false to disable profiling
       enabled: true
-      profile_table_level_only: true
     profile_pattern:
       allow:
-        - "ACCOUNTING_DB.*.*"
-        - "MARKETING_DB.*.*"
-# Default sink is datahub-rest and doesn't need to be configured
-# See https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub for customization options
+        - 'ACCOUNTING_DB.*.*'
+        - 'MARKETING_DB.*.*'
+      deny:
+        - '.*information_schema.*'
+
+sink:
+# sink configs

metadata-ingestion/docs/sources/snowflake/snowflake-beta_recipe.yml renamed to metadata-ingestion/docs/sources/snowflake/snowflake-legacy_recipe.yml

@@ -1,6 +1,6 @@
 ### Prerequisites
 
-In order to execute the `snowflake-usage` source, your Snowflake user will need to have specific privileges granted to it. Specifically, you'll need to grant access to the [Account Usage](https://docs.snowflake.com/en/sql-reference/account-usage.html) system tables, using which the DataHub source extracts information. Assuming you've followed the steps outlined in `snowflake` plugin to create a DataHub-specific User & Role, you'll simply need to execute the following commands in Snowflake. This will require a user with the `ACCOUNTADMIN` role (or a role granted the IMPORT SHARES global privilege). Please see [Snowflake docs for more details](https://docs.snowflake.com/en/user-guide/data-share-consumers.html).
+In order to execute the `snowflake-usage-legacy` source, your Snowflake user will need to have specific privileges granted to it. Specifically, you'll need to grant access to the [Account Usage](https://docs.snowflake.com/en/sql-reference/account-usage.html) system tables, using which the DataHub source extracts information. Assuming you've followed the steps outlined in `snowflake` plugin to create a DataHub-specific User & Role, you'll simply need to execute the following commands in Snowflake. This will require a user with the `ACCOUNTADMIN` role (or a role granted the IMPORT SHARES global privilege). Please see [Snowflake docs for more details](https://docs.snowflake.com/en/user-guide/data-share-consumers.html).
 
 ```sql
 grant imported privileges on database snowflake to role datahub_role;
@@ -16,7 +16,7 @@ This plugin extracts the following:
 
 :::note
 
-This source only does usage statistics. To get the tables, views, and schemas in your Snowflake warehouse, ingest using the `snowflake` source described above.
+This source only does usage statistics. To get the tables, views, and schemas in your Snowflake warehouse, ingest using the `snowflake-legacy` source described above.
 
 :::
 

metadata-ingestion/docs/sources/snowflake/snowflake-usage_pre.md renamed to metadata-ingestion/docs/sources/snowflake/snowflake-usage-legacy_pre.md

@@ -1,5 +1,5 @@
 source:
-  type: snowflake-usage
+  type: snowflake-usage-legacy
   config:
     # Coordinates
     account_id: account_name

metadata-ingestion/docs/sources/snowflake/snowflake-usage_recipe.yml renamed to metadata-ingestion/docs/sources/snowflake/snowflake-usage-legacy_recipe.yml

@@ -1,20 +1,11 @@
 source:
   type: snowflake
   config:
-
-    check_role_grants: True
-    provision_role: # Optional
-      enabled: false
-      dry_run: true
-      run_ingestion: false
-      admin_username: "${SNOWFLAKE_ADMIN_USER}"
-      admin_password: "${SNOWFLAKE_ADMIN_PASS}"
-
     # This option is recommended to be used for the first time to ingest all lineage
     ignore_start_time_lineage: true
     # This is an alternative option to specify the start_time for lineage
     # if you don't want to look back since beginning
-    start_time: '2022-03-01T00:00:00Z'
+    start_time: "2022-03-01T00:00:00Z"
 
     # Coordinates
     account_id: "abc48144"
@@ -25,14 +16,12 @@ source:
     password: "${SNOWFLAKE_PASS}"
     role: "datahub_role"
 
-    # Change these as per your database names. Remove to all all databases
+    # Change these as per your database names. Remove to get all databases
     database_pattern:
       allow:
         - "^ACCOUNTING_DB$"
         - "^MARKETING_DB$"
-    schema_pattern:
-      deny:
-        - "information_schema.*"
+
     table_pattern:
       allow:
         # If you want to ingest only few tables with name revenue and sales
@@ -42,12 +31,10 @@ source:
     profiling:
       # Change to false to disable profiling
       enabled: true
+      turn_off_expensive_profiling_metrics: true
     profile_pattern:
       allow:
-        - 'ACCOUNTING_DB.*.*'
-        - 'MARKETING_DB.*.*'
-      deny:
-        - '.*information_schema.*'
-
-sink:
-# sink configs
+        - "ACCOUNTING_DB.*.*"
+        - "MARKETING_DB.*.*"
+# Default sink is datahub-rest and doesn't need to be configured
+# See https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub for customization options