Skip to main content

Amundsen deployment

Objective​

Confirm that the data catalog meets the minimum expectations for infrastructure and security so that it can be maintained long term

Considerations​

The following criteria were to be investigated

  • Docker/container setup (via compose, ECS, Fargate etc)
  • SSO
  • Role Based Access Control
  • Notifications/Alerts

Findings​

Docker/container setup (via compose, ECS, Fargate etc)​

Deployed a non-production amundsen on AWS ECS using aws-cli.

Amundsen also has some limited documentation on Kubernetes deployment, we have not investigated this.

SSO​

You can enable OIDC authentication in React application (amundsen frontend).

This was not further investigated in this spike due to setup complications.

Summary​

Limitations​

  • Unable to load the data though the frontend. This is despite the fact that the data was successfully loaded to the Neo4j backend and the data was queryable though elasticsearch
  • The Amundsen documentation assumes a level of experience with the tool
  • There are no obvious solutions for notifications or alerts
  • Role bases access control was not reviewed as part of the spike due to time constraints and deployment issues
  • Notifications/alerts was not reviewed as part of the spike due to time constraints and deployment issues

Suggestions​

Helpful Resources/Documentation​

  • To deploy amundsen, first install the ECS CLI according to the AWS documentation
  • Then follow Amundsen documentation
  • If you are using aws-vault, you can follow the commands below

Note that you will need to set up a key pair

cd amundsen/docs/installation-aws-ecs

aws-vault exec hackney-dataplatform-development -- ecs-cli configure --cluster amundsen --region eu-west-2 --default-launch-type EC2 --config-name amundsen

aws-vault exec hackney-dataplatform-development -- ecs-cli configure profile --profile-name amundsen

aws-vault exec hackney-dataplatform-development --  ecs-cli up --keypair test-amundsen --extra-user-data userData.sh --capability-iam --size 1 --instance-type t2.large --cluster-config amundsen --verbose --force

aws-vault exec hackney-dataplatform-development --  ecs-cli compose --cluster-config amundsen --file docker-ecs-amundsen.yml up --create-log-groups

aws-vault exec hackney-dataplatform-development -- ecs-cli ps