Azure Server Setup & Administration Details

Important URLs and Addresses

Production Server (gic01)

Test Server (gic02)

SSH Public Key File

Go to Azure's Portal "Key Vaults > wusm-prod-kv > Secrets > vm-admin-ssh-key-prod ". Place the secret value into a file. Afterwards, ensure that the file is chmod 0400.

ssh shell command

ssh -i ~/.ssh/azure-configs/gic-server -l azureuser 10.25.47.138

# or via the azure cli (this is preferred)

az ssh vm --ip 10.25.47.138

Jenkins

Administration

Application Configuration

Production Server (gic01)

Production Server (gic02)

Password

See /var/jenkins_home/secrets/initialAdminPassword on the relevant host.

Azure Details

Relevant Inter-Institutional Variables

These were variables that we needed to share with Danielle Pillion (Danielle_Pillion@hms.harvard.edu) and Anil Degala (AnilKumar.Degala@childrens.harvard.edu) of Harvard Medical School / Boston Children's Hospital.

Production Server (gic01)

Test Server (gic02)

Setup Process

Cloud Setup

In March 2023, we moved this server setup from an Internal WashU hosted virtual machine to Azure Cloud. It resides along the same virtual network as WashU's Azure Databricks Data Lake.

At a high level, the server setup currently consists of

• 2 cloud servers (wusm-prod-gic01 and wusm-prod-gic02)
  • wusm-prod-gic01 serves the the "primary" server
  • wusm-prod-gic02 serves as the backup data update instance
• 1 application gateway : this the public facing load-balancer and has the public IP address
• 1 resource group: wusm-prod-rg-gic -- for billing purposes
• 1 public IP address, so that other institutions can access the site
• 1 dns address: https://gic.wustl.edu -- obtained with WashU IT's assistance
• 1 shared ntfs File Share -- shared between the 2 cloud servers so that we can perform data updates

See [[GIC-Server-Setup#Azure Details]] for more Azure Setup Details. See Azure Devops > wustl-i2 > RDC > wusm_terraform > azure_gic.tf for the terraform setup details.

Certificates

Step 1: Create the Initial Certificate Request for the frontend

In Terraform, in the "Frontend GIC cert" section, we use a "azurerm_key_vault_certificate" resource that provisions the CSR in Key Vault. Once provisioned, there is a new entry in the Key Vault Certificate objects, and the option to download the CSR as a file for use in the next step.

Step 2: WashU IT ServiceNow Request

Had to make a WashU IT Service Now "Certificate Signing Request (CSR)" and attach the certificate file created in Step 1. WashU IT eventually returned back a signed certificate file that was then merged onto the original CSR in Azure: "Key Vaults" > "wusm-prod-kv | Certificates" > gic-wustl-edu

This cert is then used on the Application Gateway, configured by the Terraform resource "azurerm_application_gateway".

Step 3: Placing the certifications into the PIC-SURE application

Do this step after all the appropriate steps in hms-dbmi/pic-sure-gic-institution and hms-dbmi/pic-sure-all-in-one have been performed.

for gic01:

cp -v /var/lib/waagent/B2224CEAC2002F14CF2F971008EF8CB72077B83F.crt /usr/local/docker-config/httpd/cert/server.chain
cp -v /var/lib/waagent/6A822E2F054BE8D0F38048205B4446D6576B8FC8.crt /usr/local/docker-config/httpd/cert/server.crt
cp -v /var/lib/waagent/6A822E2F054BE8D0F38048205B4446D6576B8FC8.prv /usr/local/docker-config/httpd/cert/server.key

for gic02:

cp -v /var/lib/waagent/B2224CEAC2002F14CF2F971008EF8CB72077B83F.crt /usr/local/docker-config/httpd/cert/server.chain
cp -v /var/lib/waagent/8588106B77B17A7DFDFFE5F440757F7B8F3770EB.crt /usr/local/docker-config/httpd/cert/server.crt
cp -v /var/lib/waagent/8588106B77B17A7DFDFFE5F440757F7B8F3770EB.prv /usr/local/docker-config/httpd/cert/server.key

The IDs on the .crt and .prv files should match the items in the azure certificates page mentioned in Step 2. See also the "Jenkins Setup" down below.

File Share Setup

• Install CIFS

  sudo dnf install cifs-utils

• Create the directories for credentials and mount points

  sudo mkdir -p /etc/smbcredentials/ /mnt/wusmprodgic/wusmprodgicshare/

• Create file /etc/smbcredentials/wusmprodgic.cred with the relevant credentials (username/password).
• Added the relevant mount to /etc/fstab

  //wusmprodgic.file.core.windows.net/wusmprodgicshare /mnt/wusmprodgic/wusmprodgicshare cifs nofail,credentials=/etc/smbcredentials/wusmprodgic.cred,serverino,nosharesock,actimeo=30

• Finally mount the shared file system:

  $ sudo mount /mnt/wusmprodgic/wusmprodgicshare

  You may get an error like so:

  # mount: (hint) your fstab has been modified, but systemd still uses
  #        the old version; use 'systemctl daemon-reload' to reload.

  In that case, simply run the systemctl daemon-reload command as the root user, and try the above mount command again. See How to Mount CIFS Share in Linux with Examples for additional help.

Firewall Setup

• Open ports on the VM:

  sudo firewall-cmd --permanent --zone=public --add-port=80/tcp
  sudo firewall-cmd --permanent --zone=public --add-port=8080/tcp
  sudo firewall-cmd --reload

Initial Application Code Setup

Followed the instructions based on the README in hms-dbmi/pic-sure-gic-institution and hms-dbmi/pic-sure-all-in-one

Part 1: Initial bash-based terminal setup

ssh -i ~/.ssh/azure-configs/gic-server -l azureuser 10.25.47.138
sudo bash

git clone https://github.com/hms-dbmi/pic-sure-all-in-one
cd pic-sure-all-in-one/initial-configuration

# if we are cleaning up from a prior installation
bash uninstall.sh

bash install-dependencies.sh

Then browse to https://10.25.47.138:8080 to setup Jenkins. Use the values mentioned in the [[GIC-Server-Setup#Jenkins]] section.

[Part 2: Jenkins Setup][jenkins-setup]

In Jenkins,

• Manage Jenkins button on the left, then select Configure System. In the Global Properties section, change the value of the release_control_branch to */main.
• On the dashboard, run Configure Institution Node. Used the [[GIC-Server-Setup#Jenkins#Application Configuration]] details to configure the build.

Part 3: Additional alterations

• We had to alter the permissions on the additional files to ensure that the web server could access all the relevant files on the disk:

  chmod 0644 /usr/local/docker-config/httpd/psamaui_settings.json
  chmod 0644 /usr/local/docker-config/httpd/picsureui_settings.json

• We symlinked /usr/local/docker-config/ to the File Share:

  mv /usr/local/docker-config /usr/local/docker-config-bak
  ln -s /mnt/wusmprodgic/wusmprodgicshare/docker-config /usr/local/

• We replaced the auto-generated certs with our own previously created self-signed certs:

  cp /var/lib/waagent/<CA-CERT-THUMBPRINT-HASH-FROM-KV>.crt /usr/local/docker-config/httpd/cert/server.chain
  cp /var/lib/waagent/<VM-CERT-THUMBPRINT-HASH-FROM-KV>.crt /usr/local/docker-config/httpd/cert/server.crt
  cp /var/lib/waagent/<VM-CERT-THUMBPRINT-HASH-FROM-KV>.prv /usr/local/docker-config/httpd/cert/server.key

• We ran the Jenkins "Create Admin User" again with the following build parameters:

  EMAIL: indraniel@gmail.com
  CONNECTION_LABEL: Google

  This allows for another user (idas@wustl.edu) to be a PIC-SURE Top Admin, and log into the local PIC-SURE site.

  (Please include both indraniel@gmail.com and kaushalmadhurima@gmail.com as admin users!)

Part 3: TrustStore Setup

As a final step to the installation process, login to Jenkins and run the “Import Server Cert to TrustStore” job in the configuration section.

1. Enter the following parameters in the Jenikins job build:
   • SERVERNAME: avillachlab.auth0.com
   • CERTALIAS: AUTH0
2. run ./stop-picsure.sh and ./start-picsure.sh to restart the wildfly server.

Data

From the original WashU VM server,wuit-s-12475.accounts.ad.wustl.edu, we copied over the main medical data file of interest:

1. encryption_key
2. columnMeta.json
3. allObservationsStore.javabin

The last file, allObservationsStore.javabin, contains the main data that's loaded into memory on the web server.

cp -v /mnt/wusmprodgic/wusmprodgicshare/docker-config/hpds/encryption_key /usr/local/docker-config/hpds/encryption_key
cp -v /mnt/wusmprodgic/wusmprodgicshare/docker-config/hpds/columnMeta.json /usr/local/docker-config/hpds/columnMeta.json
cp -v /mnt/wusmprodgic/wusmprodgicshare/docker-config/hpds/allObservationsStore.javabin /usr/local/docker-config/hpds/allObservationsStore.javabin

The medical data resides on /usr/local/docker-config/hpds.

Note, hpds stands for "High Performance Data Store".

Important Changes

• Switched from CentOS 7 to AlmaLinux 8 (needed by WashU IT Security Group to pass and have a WashU DNS name attached to it)
• there's a shared file system, /mnt/wusmprodgic/wusmprodgicshare/docker-config, shared between the two cloud servers: wusm-prod-gic01 and wusm-prod-gic02. We hope to use this shared mount point to keep and important files and data across all GIC related machines. We're no longer symlinking the directory to /usr/local/docker-config as that might be problematic with the code Harvard/BCH is producing for the PIC-SURE system.

Troubleshooting

Sometimes a user cannot log into the local institution PIC-SURE node. When this happens, try investigating the logs during the login process, by running the following docker command:

docker container logs -f wildfly

If you see the following kind of error messages:

05:19:15,568 ERROR [edu.harvard.hms.dbmi.avillach.auth.service.auth.FENCEAuthenticationService] (default task-1) fence_mapping.json not found at /opt/jboss/wildfly/standalone/configuration/emailTemplates/
05:19:15,680 ERROR [edu.harvard.dbmi.avillach.util.HttpClientUtil] (default task-1) InputStream simpleGet() cannot get response by GET from url: https://avillachlab.auth0.com//userinfo - PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
05:19:15,680 WARN  [edu.harvard.hms.dbmi.avillach.auth.service.auth.AuthenticationService] (default task-1) Failed to authenticate.  Retrying
05:19:15,797 ERROR [edu.harvard.dbmi.avillach.util.HttpClientUtil] (default task-1) InputStream simpleGet() cannot get response by GET from url: https://avillachlab.auth0.com//userinfo - PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
05:19:15,798 WARN  [edu.harvard.hms.dbmi.avillach.auth.service.auth.AuthenticationService] (default task-1) Failed to authenticate.  Retrying
05:19:15,914 ERROR [edu.harvard.dbmi.avillach.util.HttpClientUtil] (default task-1) InputStream simpleGet() cannot get response by GET from url: https://avillachlab.auth0.com//userinfo - PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
05:19:15,914 ERROR [edu.harvard.hms.dbmi.avillach.auth.service.auth.AuthenticationService] (default task-1) Failed to authenticate.  Giving up!

or simply just see an error message with PKIX in it, simply re-run the Jenkins job: "Import Server Cert to Trust Store" with the same parameters as mentioned above and restart pic-sure.

Other Issues

Please see the Common errors and triaging solutions BcH confluence page for other newer types of technical issues.

Useful commands

Need to be root to run these commands:

docker check ups

Examine running and exited containers

docker container ps -a

Inspect logs for a given container

docker container logs -f wildfly

uninstall everything

cd /home/azureuser/pic-sure-all-in-one/initial-configuration
bash uninstall.sh

reinstall everything

cd /home/azureuser/pic-sure-all-in-one/initial-configuration
bash install-dependencies.sh

start/stop jenkins

cd /home/azureuser/pic-sure-all-in-one
./start-jenkins.sh
./stop-jenkins.sh

docker container ps

start/stop pic-sure

cd /home/azureuser/pic-sure-all-in-one
./start-picsure.sh
./stop-picsure.sh

docker container ps

View Jenkins Job Code

See:

/home/azureuser/pic-sure-all-in-one/initial-configuration/jenkins/jenkins-docker/jobs

MySQL database exploration session for the PIC-SURE installation

The docker command is:

docker run -it -v /root/.my.cnf:/root/.my.cnf --network=host mysql bash

Once in the container bash session simply type:

mysql -u root

For the auth database, where we modified users, please see:

USE auth
SHOW TABLES;
select * from users;
select * fron user_roles;
select * from application;
select * from connection;