Skip to main content

Managing DataHub Metadata

Overview​

DataHub metadata has several owners. Repository-managed definitions live under yaml_config/ in dap-datahub-tools; source connectors and Airflow own the technical assets they generate; approved users maintain permitted metadata and associations in the DataHub UI.

Metadata Ownership and Permissions​

This table records the current implementation:

Metadata or entityCurrent ownership and permitted change path
DomainsDefinitions are managed through repository YAML.
Data ProductsDefinitions are managed through repository YAML.
Domain and Data Product associationsApproved Asset Admins attach assets through the DataHub UI. Associations are not managed in repository YAML.
Glue, Qlik Cloud, and Tableau assetsTechnical assets are created and managed primarily by their ingestion sources.
Data PlatformsDisplay definitions are currently managed through repository YAML.
Web maps, Earthlight, and Qlik Sense Enterprise datasetsDataset metadata YAML is generated by Airflow and stored only in the runtime S3 prefix.
Lake Formation-derived GroupsMembers can edit dataset and container descriptions only where the asset's LF-tags match their Group scope. This currently applies primarily to Glue Catalog assets.
dap-datahub-asset-adminsApproved members can edit the configured asset types, including datasets, dashboards, charts, Domains, and Data Products. This permission is not currently restricted to Glue.

Configuration File Structure​

yaml_config/
├── authorization/                # DataHub permission profiles
├── ingestion/                    # Source ingestion recipes
│   ├── glue.yaml                 # AWS Glue Catalog ingestion
│   ├── qlik_cloud.yaml           # Qlik Cloud ingestion
│   └── tableau_cloud.yaml        # Tableau Cloud ingestion
├── metadata/
│   ├── domains/                  # Domain definitions
│   ├── dataplatforms/            # Platform definitions
│   │   └── dataplatforms.yaml
│   └── dataproducts/             # Data product definitions (grouped by domain)
│       ├── child-fam-services/
│       │   └── *.yaml
│       └── housing/
│           └── *.yaml

How to Add a New Domain​

File: metadata/domains/domains.yaml

Add a new entry to the list:

- id: your-domain-id
  display_name: Your Domain Display Name
  description: Description of what this domain covers

Fields:

  • id: Unique identifier (kebab-case)
  • display_name: Name shown in DataHub UI
  • description: What the domain covers

How to Add a New Data Platform​

File: metadata/dataplatforms/dataplatforms.yaml

Add a new platform to the platforms list:

platforms:
  - id: glue
    display_name: Data Analytics Platform
    description: Platform description
    logo: https://url-to-logo.png

Fields:

  • id: Must match the source type used in the ingestion configuration (e.g., glue, qlik-sense, athena) or a custom platform ID if you are adding metadata for a non-ingested source
  • display_name: Custom name shown in DataHub UI
  • logo: URL to logo image (optional)

Note: after adding a new platform, without attaching a dataset to it, the platform will not show up in the DataHub UI.

How to Add a Data Product​

  1. Navigate to the correct domain folder: metadata/dataproducts/{domain-name}/
  2. Create a new YAML file: e.g., my-product.yaml
  3. Add the data product definition:
id: domain-name-product-name
domain: urn:li:domain:domain-name
display_name: Product Display Name
description: Business description of what this data product provides.

Fields:

  • id: Stable Data Product identifier
  • domain: Must match an existing domain
  • display_name: Name shown in the DataHub UI
  • description: Business description of the Data Product

After creating the Data Product, approved Asset Admins attach datasets to it through the DataHub UI. Dataset associations are not managed in the YAML definition.

How to Maintain Dataset Metadata​

Datasets are created by ingestion sources. Do not add dataset YAML under dap-datahub-tools/yaml_config/metadata/datasets/; that local directory is no longer part of the repository.

Approved Asset Admins maintain permitted human-authored dataset metadata in the DataHub UI:

  1. Find the dataset through search or its Data Platform page.
  2. Edit its description and ownership on the dataset page.
  3. Use the UI to attach it to the correct Domain and Data Product.

The separate Airflow asset-generation pipeline still publishes machine- generated metadata for web maps, Earthlight layers, and Qlik Sense Enterprise dashboards directly to the runtime S3 prefix. This is an automated system contract, not a manual YAML-authoring route.