--- title: Database Connector App description: How to use the database connector app keywords: [snowflake, native app, database connector app] --- # Database Connector App :::info Early Access This application is currently in early access. If you’d like to request trial access, please fill out the form here: https://dlthub.com/contact ::: The dltHub Database Connector App is a Snowflake Native App that lets you move data from your source database into Snowflake using a simple UI. You can: - connect an external [SQL database](../../dlt-ecosystem/verified-sources/sql_database) to Snowflake - create one or more [pipelines](../../general-usage/pipeline) (each pipeline defines what to load and where) - run pipelines on demand - schedule pipelines to run automatically - monitor runs and inspect logs without leaving Snowflake. Under the hood, the app uses Snowflake-managed compute and [dlt](https://dlthub.com/product/dlt) to [extract](../../reference/explainers/how-dlt-works#extract), [normalize](../../reference/explainers/how-dlt-works#normalize), and [load](../../reference/explainers/how-dlt-works#load) data into your destination database, while keeping credentials stored securely in your account. This documentation explains how to set up sources, create and manage pipelines, monitor runs, and troubleshoot common issues. --- # How to use the Snowflake Native App ## Prerequisites Before creating your first pipeline, make sure you have: 1. A destination database in Snowflake where the loaded data should land 2. Role with permissions to create references requested by the app (External Access Integrations) 3. Connection details for your source database, including: - host + port - database name / schema - username + password 4. (Optional) An S3 bucket if you plan to stage data externally ## Install and open the app Use the link we provided to you or download the app via the [Snowflake Marketplace](https://app.snowflake.com/marketplace). ## Create a pipeline 1. Go to the pipeline tab in the UI 2. Enter a pipeline name 3. Click the **+** button to create the pipeline 4. Fill in the configurations ![add a pipeline](https://storage.googleapis.com/dlt-blog-images/sna_add_pipeline.png) ## Fill in the configuration - **[Pipeline](../../general-usage/pipeline) name** A unique ID for this pipeline. It is used to store configuration and to identify runs. ### Source settings Source settings define **what to ingest** from your external SQL database and **how to connect** to it. These values are used to configure the `dlt.sources.sql_database` pipeline. #### Required fields - **Database type** Select the type of [SQL database](../../dlt-ecosystem/verified-sources/sql_database) you want to ingest from. The selected type determines the driver and default connection behavior. The following database types are currently supported: - MySQL - PostgreSQL - MSSQL If you need support for a different database type, please reach out via our [contact form](https://dlthub.com/contact). - **Host and port** Enter the hostname and port of your source database. If you leave the port empty, the app will use the database type’s default port. - **Database name** The name of the source database to connect to. - **Network rule** Allow external access to the source database endpoint. This allows the app to reach your database through the configured External Access Integration. - **Database credentials** Enter the username and password for your source database. Credentials are stored in your Snowflake account as secrets. #### Optional fields - **Schema name** Specify a [schema](../../general-usage/schema) to ingest from. Leave empty to use the default schema of the database/user. - **Chunk size** Number of rows fetched per batch during extraction. Larger values reduce round trips but increase memory usage. - **Table backend** Choose the backend used to introspect and read tables. If unsure, keep the default **SQLAlchemy**. See the backend options in the [verified source docs](../../dlt-ecosystem/verified-sources/sql_database/configuration#configuring-the-backend). - **JSON format configuration (advanced)** Provide additional `sql_database` source options not exposed in the UI. Values are passed as keyword arguments to dlt’s `sql_database` source [function](../../api_reference/dlt/sources/sql_database/__init__) and override UI values if the same option is set in both places. Example: ```json { "chunk_size": 10000, "reflection_level": "minimal" } ``` ### Table Settings Table settings control **which tables are ingested** and allow **per-table configuration**. ![tables setting](https://storage.googleapis.com/dlt-blog-images/sna_table_settings.png) #### Tables to ingest Choose whether to ingest: - **All** tables from the selected schema, or - a **Subset** of tables. If you choose **Subset**, add the tables you want to ingest. If you choose **All**, you may still list specific tables below to customize their settings.Any tables not listed will be ingested using default configuration. #### Optional fields (per configured table) - **[Write disposition](../../general-usage/incremental-loading#choosing-a-write-disposition)** Choose how dlt writes data to the destination (e.g., append, replace, merge) depending on what the UI offers. - **Primary key** Column (or list of columns) that uniquely identifies rows in the table. Used for deduplication and merge-based loading. - **Max chunks** Maximum number of chunks to extract. `max_chunks × chunk_size` defines the maximum rows extracted. Set to **0** for no limit. Passed to dlt as `max_items` via `add_limit()`. - **[Incremental loading](../../general-usage/incremental-loading)** Enable cursor-based incremental loading for the table. Only new or changed rows since the last run are extracted. - **Table JSON configuration (advanced)** Provide per-table hints not available in the UI. Values override UI settings when the same option is set in both places. These are passed as keyword arguments to dlt’s `apply_hints()` [function](../../api_reference/dlt/extract/hints#apply_hints) for the table resource. Example: ```json { "merge_key": "id", "columns": { "created_at": { "dedup_sort": "desc" }, "deleted_flag": { "hard_delete": true } } } ``` ### Destination Settings Destination settings define where in Snowflake the data is loaded and allow advanced destination tuning. - **Destination database**: Name of the Snowflake database to load into. - **Check privileges**: Validates that the app has access to the destination database. If permissions are missing, the app will request/grant the required access. ![destination settings](https://storage.googleapis.com/dlt-blog-images/sna_destination_settings.png) - **Destination JSON configuration**: Provide additional destination options not exposed in the UI. Values override UI settings when the same option is set in both places. These are passed as keyword arguments to dlt’s Snowflake destination [factory](../../api_reference/dlt/destinations/impl/snowflake/factory). Example: ```json { "create_indexes": true, "replace_strategy": "staging-optimized" } ``` ### Pipeline Settings This section contains optional pipeline-level defaults that affect how and where data is written. - **Dataset name**: Destination schema name for the pipeline output. If not set, the app derives it from the pipeline name as `_dataset`. - **[Dev mode](../../general-usage/pipeline#do-experiments-with-dev-mode)**: Enables dlt dev mode. When enabled, dlt appends a datetime-based suffix to the dataset name, producing a fresh schema on each run. This is intended for testing and avoids overwriting existing datasets. - **[Staging](../../dlt-ecosystem/destinations/snowflake#staging-support)** Staging location used during loading: - **Internal staging** in Snowflake, or - **External staging** via a configured external stage (if supported). ### Compute settings Compute settings control the Snowflake resources used to run and load the pipeline. Fields may vary depending on what the app exposes in your environment. - **Compute pool** The compute pool runs the job service responsible for [extracting](../../reference/explainers/how-dlt-works#extract) data from the source database and [normalizing](../../reference/explainers/how-dlt-works#normalize) it with dlt. - **Warehouse** The warehouse is used for [loading](../../reference/explainers/how-dlt-works#load) the extracted/normalized data into the destination database. - **[Instance family](https://docs.snowflake.com/en/sql-reference/sql/create-compute-pool)** The instance family of the compute pool the ingestion job runs on. This determines the CPU/memory profile available to the job container. - **Auto-suspend (compute pool)** Number of seconds of inactivity after which the compute pool is automatically suspended. Set to **0** to disable auto-suspend. ## Edit a pipeline 1. Open pipeline tab 2. Click **edit** button 3. Fill in the configurations 4. Save ## Monitor runs and logs Go to the **Runs** tab to see a history list of the Jobs. Click the **view details** button to view: - pipeline name - Job ID - triggered by - triggered at - started at - ended at - status Statuses typically move through: `STARTING → RUNNING → SUCCESS or FAILED`. Click the **view logs** tab to see the logs of the job: - When each pipeline stage (extract, normalize, load) starts and finishes - Progress information for each stage - Performance metrics (processing time, memory usage, CPU, ...) ![schedule a pipeline](https://storage.googleapis.com/dlt-blog-images/sna_runs_button.png) ## Schedule a pipeline The app uses Snowflake [Tasks](https://docs.snowflake.com/en/user-guide/tasks-intro) to run pipelines on a schedule. To schedule a pipeline: 1. Open the pipeline’s tab. 2. Click the **scheduling** button 3. Enable scheduling 4. Enter a schedule string 5. Click Save ![schedule a pipeline](https://storage.googleapis.com/dlt-blog-images/sna_scheduling_button.png) This operation creates or updates a Snowflake task that triggers the pipeline according to the specified schedule. To stop the task, set its status to `SUSPENDED`. You can do this either via SQL: ```sql ALTER TASK [ IF EXISTS ] RESUME | SUSPEND ``` or directly in the Snowflake UI by changing the task’s status: ![change task status](https://storage.googleapis.com/dlt-blog-images/sna_suspend.png)