Review of the flags on metadata records for Data

[NicoledeGreef]

Based on service design activity notes, we currently have three boolean flags included in the metadata content model :

• Critical information
• Authoritative info
• High value info

According to the DMP v1.0.1

High value data: Data that holds significant business value, enabling decision-making in support of programs, services, evaluation, and accountability requirements benefiting a high number of users. This includes data that supports critical services or data of public interest or permanent value (e.g., critical information and authoritative data).

3.4 Ministries must catalogue high value data in their control in accordance with the Core Administrative and Descriptive Metadata Standard. This includes critical information and authoritative data.

Authoritative Data as per the DMP v1.0.1
Authoritative data: Data located in any repository system that is recognized through
governance, law, or common acceptance as the source of origin for accurate data on a
specific topic. Authoritative data may be stored in a data register.

Authoritative data is critical for government decision making and service delivery. The
corporate-level data governance (e.g., the Deputy Minister Committee on Digital and Data)
establishes the processes to govern authoritative data.

2.4 Ministries must follow the corporate processes for identifying and establishing
authoritative data.

Deputy ministers have the responsibility to:
• Provide strategic direction concerning data in their ministry’s custody and/or
control to increase data discovery, access, sharing, and use
• Identify their ministry’s authoritative data, and follow any corporate standards
related to authoritative data
• Oversee the development and implementation of ministry-specific policies,
processes, and procedures to support data management
• Promote the importance of data by advocating for and supporting data knowledge
and skills development in their ministry
• Ensure that ministry-specific data management resources are in place to guide and
support adherence to corporate policies, standards, processes, and procedures
• Establish and maintain data governance, typically fulfilled by delegation to senior level point(s) of contact who will:
o Oversee data in their ministry’s custody and/or control,
o Ensure alignment with corporate-level data governance (e.g., the Deputy
Minister Committee on Digital & Data), and
o Ensure that the ministry’s data is managed in accordance with applicable
legislation and corporate policies, standards, processes, and procedures

The Government Chief Information Officer, or delegate, has the responsibility to:
• Provide strategic corporate direction related to government data
• Collaborate with ministries to develop clear and adequate data management
policies, standards, processes, and procedures
• Provide expert advice and services to help ministries meet their data management
obligations
• Promote the importance of data by advocating for data knowledge and skills for
government
• Administer government’s corporate data catalogue (i.e., the BC Data Catalogue)
• Develop and maintain the process for identifying, reviewing and approving
authoritative government data
• Oversee the approval of authoritative government data and maintain a publicly
accessible authoritative government data directory
• Administer the government’s data governance model

---

Inference: if a dataset is deemed critical or authoritative, it could be considered high value.

Questions:

• Should the Ministry be cataloguing anything that is not high value? We should be assessing ROT (Redundancy, Obsolescence, Triviality) upfront during the inventory stage and avoid unnecessary cataloguing effort by eliminating the data where ROT applies._
• Do we truly need 3 flags, or just one or two?

[NicoledeGreef]

Idea:
We determine criteria for Critical and Authoritative within Ministry of Finance and create scorecard resources to assist domain reps and consultants in making those assessments (see #196 , #194 )

If one or more of those applies we can automatically apply a High Value badge to the metadata record so no need for an explicit High Value field to flag it in the content model.

Also, make the update interval a mandatory field and validate to a maximum number of months (12?)
If a metadata record goes beyond the max and is overdue, it automatically loses the High Value badge until a review is done and then it is automatically re-applied.

[mjmcclung]

I like your idea to make update interval mandatory and agree that 12 months should be the max. I don't think this is unreasonable.

I don't agree on losing the badge as this introduces risk of someone not knowing it has High Value. I suggest perhaps some other rating be affected instead (data quality? accuracy? trust score? <-- Oooo, I like this one as I think 'trusted data' will be a key objective of any data mgmt work!) Link to related article

[mjmcclung]

I think we should explore the linkage between these 3 flags. You may be correct in that applying one means that High Value automatically gets applied, but if so, what is the value of having this third flag in policy? Since DMP is a sub-policy to MGIP maybe only 2 flags are needed? I think this will get flushed out as part of the discussion. Questions like these might also be good to direct back at BC Data Service to get their thoughts. Perhaps there was already discussion around this and we just don't have the insight into why High Value was included as it's own item. EDIT - I will send in this question to them.

[mjmcclung]

MGIP v1.1:
Critical Information
1.7 Ministries must identify critical information in their custody or control.
1.8 Ministries must ensure that critical information in their custody or control is protected in a manner that will allow the information to retain its integrity and remain reliable, usable, accessible, and secure for as long as needed.

Critical information: The records and data essential to the operations of a government business area. This includes information that supports business continuity by documenting and supporting core programs, functions, responsibilities and commitments (e.g., security and risk mitigation information, records needed to meet financial and legal requirements). Critical information also includes information of public interest and permanent value.

Critical Information Records Mgmt Guide: https://www2.gov.bc.ca/assets/gov/british-columbians-our-governments/services-policies-for-government/information-management-technology/records-management/guides/critical-info.pdf

[mjmcclung]

In response to your question:

Should the Ministry be cataloguing anything that is not high value? We should be assessing ROT (Redundancy, Obsolescence, Triviality) upfront during the inventory stage and avoid unnecessary cataloguing effort by eliminating the data where ROT applies

The answer is perhaps. I like the idea of cataloguing everything at first, as we will probably be well down the discussion path before determining if it is or isn't high value. There is also still value in knowing what we have, whether it is high value or not. This can prove useful later as we mature and look at improving (like the ROT you mentioned).

For now, I would suggest not looking at ROT. It can lead to exploding scope and may take a long time to detangle all the things, which might lead to long timelines and reduced effort available for actual cataloguing. Right now let's just get a handle on what we have. We can look at improving and streamlining later as part of data quality and authoritative work.

[NicoledeGreef]

From the Data Management Policy page:

• This policy establishes the BC Data Catalogue as government’s corporate data catalogue, where ministries and central agencies are expected to list their high-value data resources for discovery and access.

• This policy defines “high value data” as data with significant business value, enabling decision-making for programs, services, evaluation and accountability benefiting many users. This includes critical data, authoritative data, data of public interest or data having permanent value.

So, if a data set is any one of critical, authoritative, of public interest, or having permanent value it is considered "high value" OR
is "high value" our Ministry's interpretation of whether it is or not based on gauged demand (statistically or anecdotally)?

High Value data: data that has a lot of value for research, solving problems and creating services (pretty vague); if data has demand, it has value.

[NicoledeGreef]

For MVP, I propose we implement High Value Data, Authoritative Data, and Critical Information as non-mandatory values in order to save a metadata record. Each value will be set by a metadata author once adequate consideration of the nature of the metadata has occurred (we need to develop guidance for this process as in Scorecards). Scorecards could even be built into the app eventually.

When values are set for these, they will appear as visual badges at the top of a metadata record.
We will not reset the High Value badge programmatically for now as proposed above.

[CraigClark]

@NicoledeGreef that is potentially a lot of badges. We already have personal info and media ready (or whatever it is). This would add 3 more. If you have 5 badges, it's going to be busy.

I'd suggest High value as a badge, the other two in the sidebar.

[mjmcclung]

Suggest dropping Authoritative Data from the MVP for now. Ministries must follow the corporate process for identifying and establishing authoritative data, but this has not been developed yet. This is the responsibility of the GCIO (or delegate) and I've not heard any rumblings on when this might happen.

EDIT: reached out to my contact at BC Data Service. Sounds like the process might follow the same data policy and standard approval process which, if recent policy and standards are anything to go by, could take a year or more. My contact also offered this interesting comment, which is worth discussing at some future date: "Rather than a boolean flag, I would add an attribute with a values like "candidate", "approved", and maybe also distinguish generic from "data register"."

[NicoledeGreef]

Thank you @mjmcclung. Since it is not part of the Core Metadata Standard we can drop Authoritative from the MVP. It's the DMP statement that trips me up:

Ministries must catalogue high value data in their control in accordance with the Core Administrative and Descriptive Metadata Standard. This includes critical information and authoritative data.

In the spirit of not trying to boil the ocean, however, the Data Catalogue is our initial inventory and as you've suggested elsewhere in this thread we shouldn't attempt to filter what goes into the Catalogue; once we have the complete inventory captured we can look at ROT and what is considered Authoritative, Critical and High Value data.

[mjmcclung]

Perspective is everything! I read that DMP statement as: if you are cataloging high value data, that data must include all the elements outlined in the metadata standard. With that in mind, as long as we include the metadata standard elements in our catalogue design, we should be well on the path of being able to catalogue high value data appropriately.

[mjmcclung]

Idea: could we combine High Value and Critical Info into one badge (if the dataset is either, or both it gets this badge?) Something like an 'important data' icon would cover both of these bases. Is the purpose to provide the user with an up-front visual indicator of the importance (and possible implications/dependencies associated with it)? If so, than a single badge for these two indicators should do the trick.

[mjmcclung]

Thinking more about this overnight... I've often come across the term "Critical Data Element" (CDE). Perhaps this is an approach we want to look at for the Catalogue? I.e. having a CDE badge and then more information in the sidebar that gives insight into what about the data is critical (continuity importance, accountability requirements, value/high use, need for protection, etc).