REDCap PaaS Migration Notes

This is a working document to contain notes on migrating the REDCap infrastructure from the current virtual machine to an Azure App Service. This section provides an overview and goal for the migration. The resources section will provide a short summary and link to the various Azure environments, code repositories, and other related information. The sections following Resources will be used to provide details on the various aspects of the project.

The goal of the migration is to remove the existing virtual machine and replace it with an app service. The app service should be able to support horizontal scaling and WUSTL Key authentication for both WashU users and guest accounts. A CI/CD pipeline should also be created to allow automated deployment and testing in various REDCap environments.

Once the migration is complete, a clean-up of all REDCap environments should be done to remove deprecated assets and reduce costs. We should also consider archiving deprecated code repositories to clear away any noise it may cause.

Resources

• REDCap Dev Env - contains the Dev instances of the REDCap resources
• REDCap QA Env - contains QA instances of the REDCap resources
• REDCap Prod Env - contains Prod instances of the REDCap resources
• REDCap Terraform Repository - contains the infrastructure code for the Azure environments
• REDCap Application Repository - contains the REDCap application code
• REDCap Automated Tests Repository - contains code used to run automated REDCap tests

deprecated resources

I am providing this links for additional reference but these should be removed once the migration is complete. The ansible repository may be the most useful in this list if you are curious about the current server configuration.

• REDCap Ansible - virtual machine configuration code, will not be needed post migration
• REDCap Utils - scripts to assist in managing the virtual machine and other misc stuff
• Previous REDCap Automated Test Repository - this has been replaced with the repo above

Azure Environments

The current Azure REDCap environments contain several resources that comprise the REDCap solution. Each environment is made of up of an application gateway, a virtual machine, a MySQL service, and a storage account. There are other ancillary components to support these resources as well. However, these comprise the backbone of the environment.

We tried to keep the naming convention of these resources in this pattern: i2-redcap-{env}-{service} However, the virtual machines include a numeric suffix in case we needed to bring others online and storage accounts do not allow special characters.

For example, below are the primary resources in the REDCap Dev environment.

• i2-redcap-dev-gateway
  • redcapdev.wustl.edu
  • contains configuration to route traffic to the VM and/or app service
• i2-redcap-dev-vm-01
  • virtual machine currently hosting the dev instance of redcap 14
• i2-redcap-dev-flex-mysql
  • PaaS mysql that hosts the dev instances of the REDCap database(s)
• i2redcapdevstorage
  • contains the logs and file uploads in distinct file shares

Application Configuration

During my initial investigation into the migration I was able to spin up a working POC in the dev environment. In order to make REDCap run on the AppService, I had to reorganize the application code and add configuration files as well as a startup script. The configuration files can be found in the config folder under the feature/paas branch in the application repository. Here is a link.

These files are used by the app service to configure nginx and PHP. The startup.sh file in the root of the repository handles copying the configuration to the correct location on the app service.

Other configuration values are stored in environment variables that can be configured in the app service configuration. These include the DB connection information, the PHP INI directory, and the environment label. You can see examples from the POC here.

Action Items

WUSTL Key Integration

1. Validate Authentication
   • Ensure authentication works for both WashU accounts and guest accounts.
2. Reconcile REDCap Usernames
   • Query the database and EntraId to compare usernames.
   • Compare userPrincipleId and onPremSamAccount.
   • Pull local AD object for users and evaluate account name fields.

Storage Configuration

1. File Upload Verification
   • Ensure that file upload functionality is working as expected.
   • Will need PaaS configured to use dev db and have dev files mounted.

Dependencies Management

1. Composer and ImageMagick
   • Ensure composer is working as expected.
   • Ensure ImageMagick is installed and functioning correctly.
2. Startup Script Updates
   • Update apt install and composer install in startup.sh if needed.
   • Verify user accounts for sendmail and any other necessary user accounts.

NGNIX Configuration

1. Configuration Comparison
   • Compare the VM and AppSvc configurations to ensure all necessary settings are accounted for in the AppSvc configuration.

Logging Configuration

1. Logging Verification
   • Verify that PHP/REDCap logs to stdout/stderr are available for event hub.
   • Ensure REDCap logging setting is configured appropriately.

Testing

1. Core and Module Testing
   • Complete all core and module testing in the dev environment using the current dev database.
   • Verify both WUSTLKey and WUSTL Guest Key accounts are working.
2. Functionality Testing
   • Test file upload functionality.
     • Ensure that file upload functionality is working as expected.
     • Will need PaaS configured to use dev db and have dev files mounted.
   • Test SMTP sending.
     • Test using port 2525
   • Verify CRON jobs are working.
   • Ensure useful logs are captured.
   • Conduct EPIC/FHIR integration testing.
3. Custom Code Testing
   • Test hooks and other custom code via curl.
   • Reference test cases in DevOps. Test Plans
   • Verify authentication path exceptions in nginx configuration (API, Hooks, Surveys).
   • Reference: Current Apache Config
4. End-to-End Testing
   • Conduct thorough end-to-end testing of the application.
5. Load Testing and Scaling
   • Stress test the app service to ensure it performs as expected and scales to handle increased traffic.
6. Logging and Debugging
   • Ensure logs are accessible for troubleshooting.
   • Set up alerts for the gateway and app service, and verify necessary alerts are working as expected.

Timeline

• Week 1-2: Complete WUSTL Key Integration and Storage Configuration.
• Week 3-4: Testing in Dev environment.
• Week 5-6: Deployment to QA, and user testing.
• Week 7: Finalize Documentation and Prepare for Production deployment.

Deliverables

• Validated authentication for WashU and guest accounts.
• Reconciled REDCap usernames.
• Ensured dependencies are installed and working.
• Verified AppSvc configurations.
• Completed testing and documentation.
• Finalized system ready for deployment.