Platform Engineering 2025 Roadmap

Website
Databasin
REDCap
MDClone

Q1

AI/ML API Hosting

• Test Databricks model serving

Atlas Hosting

• Start pilot with small group of users

CBDS Migration

• Finalize CBDS Migration
  • Pending SAS from RIS team

Data Lake / Databasin

• ICS Teams begins formal Databricks training
  • Find cost for certs, look into free training (https://www.databricks.com/training/catalog)
• Data Warehousing team transition to manage ingestion requests
• Team management entirely within Databasin
• Ensure data sources have appropriate security groups
  • Place users in appropriate data source groups
  • Ensure all users belong to proper team groups
• Announce DL Open Office
  • Add new users to OO meeting
  • Add new users to WUSM Data Lake team
• Data lake public website
  • Deliver content to marketing team
• Databricks examples
  • Using ArcGIS Data
  • Using LLM Inference in Databricks
• Improve Databasin documentation
• Radiology DICOM header data

GIC

• Transition data engineering support to DW team

Genomic Data

• Richard Head's team collaboration

I2DB Support

• Documentation Website Enhancements
  • Dynamic menus
• Team Onboarding Documentation
• Platform Support Teams Channels
• Full cloud inventory and standardization of best practices
• Source control audit
  • Clean up legacy git repositories
  • Identify and migrate any code not properly managed
• Identify solutions that need to be updated
  • Solutions on legacy platforms
  • Solutions lacking deployment process

MDClone

• Support MDClone migration to Databricks
• Infrastructure upgrades

RDC Support

• RDC Infrastructure migration
  • Inventory, document, and transition support for existing RDC/Data lake IaC repositories from TPI to PE
• Begin OMOP migration to Databricks
• Identify all other pipelines/processes that should move to Databricks

REDCap PaaS

• QA testing
• Load testing / scaling
• Production migration
• Release new e-consent template and SOPs
• Develop training materials for both
• Migrate users from Multi-Signature Consent external module

Reporting

• Data lake access
• User reports
  • Per department
  • Timeline
• Data source reports
  • Timeline
• BYOB auditing
• Expiring Access
• REDCap reports
  • Siteman reports
  • ICTS reports

Q2

AI/ML Support

• Improve management of AI API endpoints to allow for easy team management including billing
• Train PE (ICS resource) on ML lifecycles in Databricks

Atlas Hosting

• Rollout to DL community

Data Lake / Databasin

• Data Warehousing team transition to manage pipelines and connectors
• Update documentation around team management
  • How to use Databasin
  • How the security layer is built. ie. team groups vs data source groups
• Improve WUSM Data Lake documentation
  • Provide guides on schemas
  • Data / asset lifecycles. ie. working in sandbox vs curated
• Databricks examples
  • OMOP Queries
  • Clarity Queries - for Clarity users only(?)

GIC

• GIC rollout to DL community

I2DB Support

• Documentation Website Enhancements
  • I2DB Documentation Chatbot
• Complete I2DB documentation platform onboarding documentation
• Finalize tech stack for deployment automation
  • TF vs Bash vs Bicep etc

OHIDS Migration

• OHIDS Migration kick-off

RDC Support

• Identify legacy solutions currently supported by DW and RIS
• Create a plan to migrate support of these solutions to PE
• Begin migration of other pipelines to Databricks

Reporting

• Leadership report
• PE Ticketing

Q3

AI/ML Support

• Chatbot as a Service
• Self-service inference endpoints

Data Lake / Databasin

• Begin pilot of Databasin with BYOBs and other select teams
• ICS Teams begins formal Databricks training
• Begin monthly training on new Databasin features with I2DB and BYOBs
• Data Warehousing team transition to manage teams/access requests
• Data lake yearly audit
  • Ensure legacy users/teams are updated for management in Databasin

I2DB Support

• Shared resources available in both dev and prod environments to support PE projects
  • PSQL Server
  • AppService Plan
  • vNet
  • Storage
  • etc
• Azure logging audit
  • All resources have appropriate logging in place

REDCap PaaS

• Automated deployment pipelines
  • Integrated automated tests in the development environment
• Dedicated REDCap environments

RDC Support

• Finalize OMOP Databricks migration
• Finalize additional pipeline Databricks migrations

Q4

AI/ML

• Self-service Custom Chatbot enablement

Data Lake / Databasin

• Roll out Databasin to additional teams

I2DB Support

• Yearly cloud audit
  • Ensure resources comply with our patterns and standards
  • Identify any resources that need to be updated
  • Remove unused resources

OHIDS Migration

• OHIDS completed

RDC Support

• Retire NiFi infrastructure