RDC and WUSM Infrastructure Repositories

This document contains notes on the repositories that are used to manage the RDC and WUSM infrastructure.

Repositories

i2rdc-packer

Description: Packer templates for building RDC images. Images are stored in the Azure compute gallery i2rdc3devimagegallery.

Currently builds 2 images:

• AlmaLinux8CISL2
  • Used as the base image for RDC VMs
• AlmaLinux8DevOpsAgent
  • Built on top of AlmaLinux8CISL2, used as the image for the Azure DevOps agent VMs, i2rdc3-dev-devopsagent.

Deployment:

• Each image folder (base and devops-agent) has a YAML file defining it's Azure DevOps pipeline. The pipelines are set to trigger manually, with the only option passed in is SemVer Major number, which corresponds to the version of vanilla Alma Linux that it is sourced from. It then puts together the full semantic version using SemVer Major number, the yyyymmdd as the minor version, and the hhmmss as the patch version. This was done when SemVer Major number had a different meaning, and now isn't really necessary to be a passed in value. A TODO would be to hardcode this in and change it at the same time the vars in the pkrvars file are updated.

Provisioned VMs that use these images:

i2rdc-terraform

Description: Terraform project to manage RDC infrastructure. Refer to the repository README for usage details. Terraform creates and manages everything in the following resource groups:

• i2rdc3-dev-rg-backup
• i2rdc3-dev-rg-images
• i2rdc3-dev-rg-k8s
• i2rdc3-dev-rg-main
• i2rdc3-prod-rg-backup
• i2rdc3-prod-rg-k8s
• i2rdc3-prod-rg-main
• i2rdc3-qa-rg-backup
• i2rdc3-qa-rg-k8s
• i2rdc3-qa-rg-main

Of main concern in these resource groups are:

• Nifi VMs
• AKS cluster that supports the Nifi VMs
• RDC Postgres VMs
• DNS VMs
• Azure compute gallery for the previously mentioned Packer images
• All of the Key Vaults, Networking, and Storage Accounts to support the above

Deployment:

• There is an Azure DevOps pipeline that is triggered on a push to the main branch. It then runs the terraform init and terraform plan commands, waits for user approval, and then runs the terraform apply command.

i2rdc-ansible

Description: Ansible playbooks that are used to configure the RDC VMs created by Terraform. The top level directory has 4 playbooks that define the tasks and roles for each use case:

• playbook_dns.yml - Configures the DNS VMs
• playbook_nifi.yml - Configures the Nifi VMs
• playbook_postgres.yml - Configures the RDC Postgres VMs
• playbook_patching.yml - Runs a yum update and reboots the VM if necessary

The inventories directory has a directory for each environment (dev, prod, qa) wherein there is a group_vars directory that contains a variables file for each playbook, and an all.yml with variables that are common to all playbooks. Also in each environment directory is a hosts file that defines which hosts the playbooks will be run on.

Deployment:

• There is an Azure DevOps pipeline that can be manually triggered. It is configured to allow the user to select which playbook to run. It has never been used, as all deployments were initially done manually.

i2rdc-kubernetes

Description: Contains the kubernetes manifest files for the RDC AKS cluster. It defines the configuration for the following services across dev, qa, and prod:

• Zookeeper - Used to manage coordination between Nifi VM nodes
• Nifi - Does not contain any deployments, only used to create self-signed certificates for the Nifi nodes to use
• Nifi Registry - Deployed in Production only, used to support Nifi
• Selenium - Deployed in Dev only, used in Databricks for some processing that requires a browser
• Philter - Deployed in Dev only
• Proxy - Deployed in Dev only, creates a HTTP and SOCKS5 proxy to allow access to on-prem network
• ExternalDNS - Used to allow the above services to register their DNS records in Azure DNS

Deployment:

• All manifests are deployed manually, using kubectl. Each service has a README.md that describes how to configure and deploy it.

wusm-terraform

Description: Terraform project to manage WUSM infrastructure. Refer to the repository README for usage details. Terraform creates and manages everything in the following resource groups:

• wusm-dev-rg-adb
• wusm-dev-rg-main
• wusm-prod-rg-adb
• wusm-prod-rg-gic
• wusm-prod-rg-main

Of main concern in these resource groups are:

• Databricks
• Databasin (Only the networking, not the actual install)
• GIC
• All of the Key Vaults, Networking, and Storage Accounts to support the above

Deployment:

• There is an Azure DevOps pipeline yaml that is configured in same way as the i2rdc-terraform pipeline, however it has not been setup in Azure DevOps, and all deployments have been done manually.