MEETING NOTES:

We have started digging into cleaning up the access to OMOP and clarity data within the data lake. During the discussion, we had a few questions come up that we would like to get your input on.

  1. Do we want to expose both the cleansed.epic_clarity and cleansed.epic_clarity_orgfilter schemas to users approved for clarity access in the data lake?
    1. Prefer epic_clarity_orgfilter going forward
    2. Internally will need time to migrate away from long table names
    3. MDClone has some code pointing to longer table names
    4. Need a report to get a list of active users of the schema
  2. Do we want to move the "standardized" schema to curated, like we did for omop? Ie. cleansed.epic_clarity_orgfilter becomes curated.epic_clarity_orgfilter
    1. Will need migration for existing to move catalog name
  3. Does anyone outside of ICS (and maybe BYOBs) need access to cleasend.omop after we migrate users to curated.omop?
    1. Users get curated, ICS get cleansed
    2. If they need extensions or crosswalks will need to work with ICS
    3. cleansed.omop can be retired in the future, moving to dedicated catalog for OMOP ETL process
    4. report of usage on cleansed.omop within last 2 months

More broadly, do you think it is good for us to have a policy that any data sources (schemas) that are intended to be shared with multiple groups, be located in the curated schema? Obviously there will be exceptions, but the idea is for anything that we promote as an available data source would be shared from curated and only ICS would have access to the cleansed version. I feel like this will make it easier to manage and also improve user experience by allowing them to focus only on the curated and sandbox schemas.

  • cleansed inventory
  • determine what should be in curated based on common access requests
  • can brokers assist with migrate to curated
  • how to bill, Sherry

Please let me know your thoughts on this when you get a chance. Once we finalize the details, we can put together a plan to send out notifications to affected users and select dates to remove access to the "retired" data sources. In the meantime, the PE team is preparing to do some work behind the scenes to clean up the groups/permissions for both clarity and omop. Part of this process will be ensuring users get access to the "new" data source (ie. curated.omop) to allow them to begin migrating their code.

Table of Contents


Updated on August 7, 2025