[ENH][BEP028] Specification update for BEP028 BIDS-Prov #2099

bclenet · 2025-04-10T15:04:58Z

This is a work in progress PR proposing a specification update for BEP028 BIDS-Prov.

BIDS specification changes
- [ ] being proofread
BIDS schema changes
- [ ] validator error : /prov/* NOT_INCLUDED
- [ ] validator error : /prov/*.json SIDECAR_WITHOUT_DATAFILE
- [ ] validator error : derivative files are listed as NOT_INCLUDED / ALL_FILENAME_RULES_HAVE_ISSUES /FILENAME_MISMATCH / ENTITY_WITH_NO_LABEL
BIDS examples

for more information, see https://pre-commit.ci

src/schema/rules/files/common/tables.yaml

yarikoptic

some initial notes

yarikoptic · 2025-09-02T17:08:48Z

src/modality-agnostic-files/provenance.md

+!!! bug
+    TODO: Environment not currently defined in the context


what to be done about that?

should be added.

Is there an IRI from PROV-O or from any other ontology we should map with Enviroment ?

Environment is a prov:Entity

I think we may want to subtype prov:Entity or at least give an easy solution to differentiate Environment from Input files (which are also represented as entities)...

if we do this, i think we go back to the ontology development work that i think we did not end up finishing. perhaps reuse the nidm classes and properties?

I can look into NIDM but don't think we had a term for that unfortunately. If we stick to prov:Entity for Environement, then maybe we should have a different signature of REQUIRED fields in input files compared to environment. I really think it should be possible for a machine to automatically know if something is an input file versus an environment...

Just realized now that actually in BIDS-Prov, it's easy to differentiate as "Environments" and "Entities" arrays as they are defined using different keywords (although in RDF they both map to prov:Entity) so I think we can keep as is.

Sorry for the noise!

src/modality-agnostic-files/provenance.md

yarikoptic · 2025-09-02T17:23:06Z

src/modality-agnostic-files/provenance.md

+   }
+) }}
+
+The `dataset_description.json` file of a BIDS-Derivatives dataset MUST include the following key to describe provenance:


this is from the "common-principles", so better be referred to it?

Do you suggest to remove all metadata tables from the Provenance at dataset level section because these metadata are already listed in src/modality-agnostic-files/dataset-description.json ?

This is not duplication, the suggestion is to move all the provenance-related info (including how to write out GeneratedBy) from common-principles into Provenance. See #2099 (comment)

satra

looks very nice @bclenet - left some comments which should be very lightweight to address.

src/modality-agnostic-files/dataset-description.md

src/modality-agnostic-files/provenance.md

satra · 2025-09-03T00:37:28Z

src/modality-agnostic-files/provenance.md

+            {
+                "Id": "bids::sub-01/func/sub-01_task-tonecounting_bold.nii",
+                "Label": "sub-01_task-tonecounting_bold.nii",
+                "AtLocation": "sub-01/func/sub-01_task-tonecounting_bold.nii",


any reason not to use the same uri as Id here?

For the Label, I guess we want to use the name of the file (without directory path) so that is more human readable.
About AtLocation, we describe this metadata as:

For input files, this is the relative path to the file on disk.

Should we be more specific for this definition ?

i meant to use the bids uri here as well.

Discussed w/ Boris today. We could use uri and change the spec to say "this is the path to the file on disk." (instead of "relative path").

But... Now that the id is in fact the path to the file in bids -- maybe we should consider removing AtLocation from BIDS-Prov as this only creates duplication of information. What do you think?

the unfortunate bit here is that the bids uri is not persistent. we do need to add something to the id for uniqueness. otherwise two datasets loaded into a graph could have the same id for a same named file (e.g. sub-01/fmri/sub-01_task-rest_bold.nii.gz could be in many datasets). i don't think their id's should be the same. so i would suggest keeping the bids uri for atlocation and then consider how to get uniqueness to the ids here and elsewhere.

for a different project we have been constructing the id using a function of the checksum of the metadata associated with the node. this allows us to generate a graph without creating new nodes each time we run the process. and if the metadata changes (whether in keys and/or their values) a new node id is generated.

@satra - could we get uniqueness using the dataset name in the BIDS URI? e.g. bids:ds109:sub-01/fmri/sub-01_task-rest_bold.nii.gz is unique). I think this may be a BIDS issue more than BIDS-Prov.

Note: we might need another online meeting to discuss those remaining bits? Let me know how you'd like to proceed.

unfortunately, the dataset name is technically a key from the dataset_description (see: https://bids-specification.readthedocs.io/en/stable/common-principles.html#examples_1), so not guaranteed to be unique across datasets. bids intentionally decoupled uniqueness or persistence from the key name and in that section on bids uri notes that while certain aspects of a scheme uri are not being used, it could be used in the future (see: https://bids-specification.readthedocs.io/en/stable/common-principles.html#future-statement).

my suggestion would be to perhaps do what is being done for uniqueness of other ids.

src/modality-agnostic-files/provenance.md

satra · 2025-09-03T00:45:45Z

src/modality-agnostic-files/provenance.md

+
+## Minimal example
+
+Here is a comprehensive example that considers the following dataset:


for these examples, do consider the changes about study-level organization and what that means for prov.

Hi @satra! I discussed this with Boris. Can you give us some more insights on what you mean by "study-level organization". Is this a change in the BIDS spec? Or do you mean a re-organization of the files within a BIDS folder for a given study?

here is the relevant doc: https://bids-specification.readthedocs.io/en/stable/common-principles.html#study-dataset this was merged with the recent release.

This PR will allow to describe provenance of any BIDS dataset (may it be in the dataet_description.json, in the prov directory, or in a sidecar). This applies to a study dataset as well as to its nested BIDS datasets. Therefore, @cmaumet and I think there is nothing specific to add about study-level inside the Provenance section of the specification.

just to clarify (and perhaps to make clear):

so one could add any provenance to the study level dataset_description.json.

any use of bids uris would of course have to reflect cross-dataset links (https://bids-specification.readthedocs.io/en/stable/common-principles.html#examples_1)

the cross-dataset link for bids-uri could come up in individual datasets as well, where a dataset is used as input and the outputs are some derivatives in a different dataset. in this scenario, would we recommend that a study level dataset_description is required?

src/modality-agnostic-files/provenance.md

yarikoptic

Next batch of comments and recommendations

src/common-principles.md

yarikoptic · 2025-09-03T12:59:37Z

src/modality-agnostic-files/dataset-description.md

- https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md
-->
-{{ MACROS___make_subobject_table("metadata.GeneratedBy.items") }}
-


All these removals is text and headings leave examples "hanging". If all of this is to move to the provenance section, surgery might be needed to become more thorough and have explicit reference to that section.

There is a link to the Provenance section in the description of GeneratedBy to cover the fact that this table was moved to the Provenance section.

Discussed w/ @bclenet today. The description of GeneratedBy was moved from "Modality-agnostic files/Dataset description" into "Modality-agnostic files/Provenance/Provenance at dataset level". The example itself is a dataset_description.json file and as such it makes sense to keep it in "Modality-agnostic files/Dataset description" where the overall structure of that file is described. So we are not sure why you mention the examples are hanging. Let us know if that answer your comment or if further updates are needed.

yarikoptic · 2025-09-03T13:03:49Z

src/modality-agnostic-files/provenance.md

+
+### Principles for encoding provenance in BIDS
+
+-   Provenance information SHOULD be included in a BIDS dataset when possible.


I believe that GeneratedBy is just MAY, so here it then should also be such to be consistent

Suggested change

- Provenance information SHOULD be included in a BIDS dataset when possible.

- Provenance information MAY be included in a BIDS dataset when possible.

GeneratedBy is RECOMMENDED in BIDS datasets and REQUIRED in BIDS derivative datasets.

Adding the sources in BIDS-Spec:

for BIDS datasets: https://bids-specification.readthedocs.io/en/stable/modality-agnostic-files/dataset-description.html#dataset_descriptionjson

for derivatives: https://bids-specification.readthedocs.io/en/stable/modality-agnostic-files/dataset-description.html#derived-dataset-and-pipeline-description

So this seems like we can keep SHOULD?

src/modality-agnostic-files/provenance.md

yarikoptic · 2025-09-03T13:45:30Z

src/modality-agnostic-files/provenance.md

+## Provenance at dataset level
+
+Provenance metadata MAY be stored inside the `dataset_description.json` of any BIDS dataset (or BIDS-Derivatives dataset) it applies to.
+This metadata describes the provenance of the whole dataset.


Similarly to SidecarGeneratedBy vs GeneratedBy, we have ambiguity as to define the scope, ie - does it apply to all files? What if file had it's own records? We could instruct the same need for a copy but it might be too hard to keep consent etc...

We should spell it out one way or another here

I'm not sure to fully understand what you mean.

But we can leave Provenance at dataset level as a (minimal) option to describe the provenance of the dataset as an Entity.
Of course this most probably contains partial information, but this does not prevent more precise descriptions from being included inside sidecars.

bclenet · 2025-09-11T09:46:17Z

Hi @effigies ,

Here are quick questions about the schema modifications in this PR.
I tested the BIDS examples listed in the description of this PR against a BIDS validator using the modified schema, but I still encounter errors.

Provenance files (.json) inside the prov directory are considered as sidecars, leading to SIDECAR_WITHOUT_DATAFILE error files
Provenance files (.json) inside the prov directory are NOT_INCLUDED according to the validator, although

I think this is due to the fact that the src/schema/rules/files/common/modality_agnostic.yaml file I've added has no impact for now.

Also, files inside derivative datasets lead to NOT_INCLUDED / ALL_FILENAME_RULES_HAVE_ISSUES / FILENAME_MISMATCH / ENTITY_WITH_NO_LABEL errors because their names do not conform to BIDS naming. It seems to be contradictory to this part of the spec

Could you please give your thoughts about these ? Let me know if I should post these anywhere else.

Thanks,

src/images/provenance_definitions_graph.png

src/modality-agnostic-files/provenance.md

cmaumet · 2025-09-16T14:14:14Z

@satra @yarikoptic: a point on which we'd love your input w/ @bclenet. It seems to us a bit problematic to use the word "Entity" in the Provenance spec for BIDS since entity in BIDS is already defined (and is different from the meaning we give in BIDS-Prov). We could easily use a different word (that would still be mapped to prov:Entity in the context file), but which word...

Just found out about "Transput" which seems to be a blanket term for both Inputs and Outputs, would that work for you both? Or any other suggestions?

Thanks!

bclenet and others added 30 commits March 18, 2025 11:06

Adding dir at dataset level

3b64a2b

Adding dir at any level

45f6af4

Adding rule for dir at dataset level

4c315ce

Adding objects: JSONLD extension + top level dir

b490c91

Adding object : prov entity

f2a67a6

Add suffixes in object.suffixes

d9bb7b0

GeneratedByProv and SidecarGeneratedBy metadata

64e2c65

prov entitiy and provenance json/jsonld files

acc6af0

provenance json/jsonld files inside new rules file

34fafe1

Merge branch 'master' into BEP028_schema

f73b668

[SCHEMA] removing provenance subdir level description

a1fbb8a

Yaml lint

6622d08

[SPEC] first draft for BEP028 BIDS-Prov

b05b950

[pre-commit.ci] auto fixes from pre-commit.com hooks

9ae1649

for more information, see https://pre-commit.ci

Adapting markdown to mkdocs

f572174

[pre-commit.ci] auto fixes from pre-commit.com hooks

ee0d30d

for more information, see https://pre-commit.ci

Broken paragraph links

b619de1

Citing BIRS Provenance + removing 'BIDS-Prov' from spec

74722b1

Into JSON only provenance files

34411ba

[pre-commit.ci] auto fixes from pre-commit.com hooks

248d2b8

for more information, see https://pre-commit.ci

Into JSON only provenance files

99faa51

Into JSON only provenance files

ac7aff4

Merge branch 'master' into BEP028_schema

6289722

Merge branch 'spec-refac' into BEP028_spec

3fef3dc

Flags for markdown code blocks

03026d1

Merge branch 'master' into BEP028_spec

a3dd018

Typo in provenance.md

0202806

Remark check

58055b7

Removing non latin content

a3cef66

Merge branch 'master' into BEP028_spec

c9a65b7

bclenet added 5 commits July 22, 2025 15:29

Correcting typos

535705c

Links from snippets to bids-examples

8cf521f

Links from snippets to bids-examples

2c43698

Provenance graph for definitions

d978907

Provenance example graph with neuro context

d4dabb4

effigies added this to the 1.11.0 milestone Aug 27, 2025

yarikoptic reviewed Sep 2, 2025

View reviewed changes

src/schema/rules/files/common/tables.yaml Outdated Show resolved Hide resolved

yarikoptic requested changes Sep 2, 2025

View reviewed changes

satra reviewed Sep 3, 2025

View reviewed changes

bclenet added 6 commits September 3, 2025 10:11

Merge branch 'master' into BEP028_spec

5eb1ec3

Remove unrelated changes

9381bf4

Typo and replace CAN by MAY

5c83813

Examples at the begining of subsections

891730b

Uniqueness of ids

5181dd0

Minimal example > consistency of Ids

c20912c

yarikoptic requested changes Sep 3, 2025

View reviewed changes

bclenet added 6 commits September 3, 2025 16:15

Issue with dataset naming

ff4746a

Simplification in definitions

9f6eeae

Deidentification

2f05124

Merge branch 'master' into BEP028_spec

ca61869

Removing prefix all

7345757

Remove unrelated changes

a3f5238

effigies added the BEP label Sep 4, 2025

Sidecargeneratedby precision

242062c

satra reviewed Sep 14, 2025

View reviewed changes

src/images/provenance_definitions_graph.png Outdated Show resolved Hide resolved

satra reviewed Sep 14, 2025

View reviewed changes

src/modality-agnostic-files/provenance.md Show resolved Hide resolved

bclenet added 2 commits September 16, 2025 16:41

Using markdown mermaid graph for provenance

4840a3e

Scopes of provenance description

575c7c3

		!!! bug
		TODO: Environment not currently defined in the context


		## Minimal example

		Here is a comprehensive example that considers the following dataset:


		### Principles for encoding provenance in BIDS

		- Provenance information SHOULD be included in a BIDS dataset when possible.

	- Provenance information SHOULD be included in a BIDS dataset when possible.
	- Provenance information MAY be included in a BIDS dataset when possible.

[ENH][BEP028] Specification update for BEP028 BIDS-Prov #2099

Are you sure you want to change the base?

[ENH][BEP028] Specification update for BEP028 BIDS-Prov #2099

Uh oh!

Conversation

bclenet commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmaumet Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmaumet Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

satra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bclenet Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmaumet Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

bclenet commented Apr 10, 2025 •

edited

Loading

cmaumet Sep 16, 2025 •

edited

Loading

cmaumet Sep 16, 2025 •

edited

Loading

bclenet Sep 3, 2025 •

edited

Loading

cmaumet Sep 9, 2025 •

edited

Loading

cmaumet Sep 16, 2025 •

edited

Loading