How to Create a Databasin Pipeline

This guide provides step-by-step instructions to create pipelines in Databasin. You will learn how to add and configure a new pipeline to automate the ingestion various data sources.

Prerequisites

• You should have an active account on databasin.wustl.edu.
• Basic understanding of database connections and parameters.

Steps

1. Login to Databasin
   1. Navigate to databasin.wustl.edu and log in with your credentials.
2. Select a Project
   1. Once logged in, select the project where you want to manage connectors.
3. Navigate to the Pipelines Page
   1. In your project, go to the Pipelines page.
4. Create a New Pipeline
   1. Click the Create New Pipline button.
      
5. Name the Pipeline
   1. Enter the name of the pipeline in the dialog box.
   2. Click the Create button.
      
6. Source Connector
   1. Find the source connector you would like to use for the pipeline.
   2. Click on the row with the appropriate connector.
   3. Click the Select Connector button.
      
7. Destination Connector
   1. Find the destination connector you would like to use for the pipeline.
   2. Click the row with the appropriate connector.
   3. Click the Select Connector button.
      
8. Source Catalogs
   1. Find the source catalog (or schema) that you would like to ingest data from.
   2. Click on the appropriate catalog.
      
9. Source Tables
   1. Click on the tables you would like to ingest.
   2. Click the Continue button.
      
10. Source Columns
    1. Select the columns you would like to ingest for all of the selected tables.
    2. All columns are select by default. You may toggle all columns on or off by using the All Selected/Select All button.
    3. Once the appropriate columns are selected, click the Continue button.
       
11. Data Ingestion Options
    1. This screen allows you to modify various details of the ingestion process for each table. We will leave the default settings for this guide. However, additional information can be found in the Managing Ingestion Options document.
    2. Once the options are configured, click on the All my configurations are set. Continue... button.
       
12. Final Configuration
    This screen allows you to modify various details of the ingestion process, such as schedule, visibility etc. We will leave most of the details as the default settings for this guide. One important configuration option that should be entered for each pipeline is the Tags. Tags are used to ensure the appropriate team is charged for the usage incurred by the pipeline. Additional information can be found in the Managing Ingestion Options document.
    1. Add billing tags
       1. Enter WusmDepartment as the key
       2. Enter the appropriate department name as the value.
       3. Click Add new Tag
          Note: Ensure there are no empty tags in the list before clicking Finish. Use the X button to remove the empty tag if needed.
          
    2. Adjust Additional Settings*
       1. Modify other configuration options as appropriate for the pipeline. Refer to the Managing Ingestion Options document for more information.
    3. Finish Configuration
       1. Once the options are configured, click on the Finish button.