[FIX] clarify no blank and duplicated headers in TSV (#1116)

* FIX: clarify no blank and duplicated headers in TSV

* also add a note to physio/stim

* restructure and clarify physio section

* fix md

Author: sappelhoff

src/02-common-principles.md

@@ -477,10 +477,12 @@ files where commas are replaced by tabs. Tabs MUST be true tab characters and
 MUST NOT be a series of space characters. Each TSV file MUST start with a header
 line listing the names of all columns (with the exception of
 [physiological and other continuous recordings](04-modality-specific-files/06-physiological-and-other-continuous-recordings.md)).
-Names MUST be separated with tabs.
 It is RECOMMENDED that the column names in the header of the TSV file are
 written in [`snake_case`](https://en.wikipedia.org/wiki/Snake_case) with the
 first letter in lower case (for example, `variable_name`, not `Variable_name`).
+As for all other data in the TSV files, column names MUST be separated with tabs.
+Furthermore, column names MUST NOT be blank (that is, an empty string) and MUST NOT
+be duplicated within a single TSV file.
 String values containing tabs MUST be escaped using double
 quotes. Missing and non-applicable values MUST be coded as `n/a`. Numerical
 values MUST employ the dot (`.`) as decimal separator and MAY be specified

src/04-modality-specific-files/06-physiological-and-other-continuous-recordings.md

@@ -1,5 +1,14 @@
 # Physiological and other continuous recordings
 
+Physiological recordings such as cardiac and respiratory signals and other
+continuous measures (such as parameters of a film or audio stimuli) MAY be
+specified using two files:
+
+1.  a [gzip](https://datatracker.ietf.org/doc/html/rfc1952)
+    compressed TSV file with data (without header line)
+
+1.  a JSON file for storing metadata fields (see below)
+
 [Example datasets](https://github.com/bids-standard/bids-examples)
 with physiological data have been formatted using this specification
 and can be used for practical guidance when curating a new dataset:
@@ -18,8 +27,6 @@ sub-<label>/[ses-<label>/]
         <matches>[_recording-<label>]_stim.json
 ```
 
-Optional: Yes
-
 For the template directory name, `<datatype>` can correspond to any data
 recording modality, for example `func`, `anat`, `dwi`, `meg`, `eeg`, `ieeg`,
 or `beh`.
@@ -29,8 +36,11 @@ before the suffix.
 For example for the file `sub-control01_task-nback_run-1_bold.nii.gz`,
 `<matches>` would correspond to `sub-control01_task-nback_run-1`.
 
-The [`recording-<label>`](../99-appendices/09-entities.md#recording) entity can be used to distinguish between several
-recording files.
+Note that when supplying a `*_<physio|stim>.tsv.gz` file, an accompanying
+`*_<physio|stim>.json` MUST be supplied as well.
+
+The [`recording-<label>`](../99-appendices/09-entities.md#recording)
+entity MAY be used to distinguish between several recording files.
 For example `sub-01_task-bart_recording-eyetracking_physio.tsv.gz` to contain
 the eyetracking data in a certain sampling frequency, and
 `sub-01_task-bart_recording-breathing_physio.tsv.gz` to contain respiratory
@@ -39,14 +49,7 @@ measurements in a different sampling frequency.
 Physiological recordings (including eyetracking) SHOULD use the `_physio`
 suffix, and signals related to the stimulus SHOULD use `_stim` suffix.
 
-Physiological recordings such as cardiac and respiratory signals and other
-continuous measures (such as parameters of a film or audio stimuli) can be
-specified using two files: a [gzip](https://datatracker.ietf.org/doc/html/rfc1952)
-compressed TSV file with data (without header line)
-and a JSON file for storing the following metadata fields.
-
-Note that when supplying a `*_<physio|stim>.tsv.gz` file, an accompanying
-`*_<physio|stim>.json` MUST be supplied as well.
+The following table specifies metadata fields for the `*_<physio|stim>.json` file.
 
 <!-- This block generates a metadata table.
 The definitions of these fields can be found in
@@ -59,6 +62,10 @@ and a guide for using macros can be found at
       "SamplingFrequency": "REQUIRED",
       "StartTime": "REQUIRED",
       "Columns": "REQUIRED",
+      "Manufacturer": "RECOMMENDED",
+      "ManufacturersModelName": "RECOMMENDED",
+      "SoftwareVersions": "RECOMMENDED",
+      "DeviceSerialNumber": "RECOMMENDED",
    }
 ) }}
 
@@ -68,9 +75,12 @@ example, the units of the recorded time series.
 Please note that, in contrast to other TSV files in BIDS, the TSV files specified
 for physiological and other continuous recordings *do not* include a header
 line.
-Instead the name of columns are specified in the JSON file.
+Instead the name of columns are specified in the JSON file (see `Columns` field).
 This is to improve compatibility with existing software (for example, FSL, PNM)
 as well as to make support for other file formats possible in the future.
+As in any TSV file, column names MUST NOT be blank (that is, an empty string),
+and MUST NOT be duplicated within a single JSON file describing a headerless
+TSV file.
 
 Example `*_physio.tsv.gz`:
 
@@ -114,15 +124,32 @@ A guide for using macros can be found at
 
 ```JSON
 {
-   "SamplingFrequency": 100.0,
-   "StartTime": -22.345,
-   "Columns": ["cardiac", "respiratory", "trigger"],
-   "cardiac": {
-       "Units": "mV"
-   }
+    "SamplingFrequency": 100.0,
+    "StartTime": -22.345,
+    "Columns": ["cardiac", "respiratory"],
+    "Manufacturer": "Brain Research Equipment ltd.",
+    "cardiac": {
+        "Description": "continuous pulse measurement",
+        "Units": "mV"
+        },
+    "respiratory": {
+        "Description": "continuous measurements by respiration belt",
+        "Units": "mV"
+        }
 }
 ```
 
+Note how apart from the general metadata fields like `SamplingFrequency`, `StartTime`, `Columns`,
+and `Manufacturer`,
+each individual column in the TSV file may be documented as its own field in the JSON file
+(identical to the practice in other TSV+JSON file pairs).
+Here, only the `Description` and `Units` fields are shown, but you may use any other of the
+[defined fields](../02-common-principles.md#tabular-files) such as `TermURL`, `LongName`, and so on.
+In this example, the `"cardiac"` and `"respiratory"` time series are produced by devices from
+the same manufacturer and follow the same sampling frequency.
+To specify different sampling frequencies or manufacturers, the time series would have to be split
+into separate files like `*_recording-cardiac_physio.<tsv.gz|json>` and `*_recording-respiratory_physio.<tsv.gz|json>`.
+
 ## Recommendations for specific use cases
 
 To store pulse or breathing measurements, or the scanner trigger signal, the
@@ -146,7 +173,9 @@ For any other data to be specified in columns, the column names can be chosen
 as deemed appropriate by the researcher.
 
 Recordings with different sampling frequencies or starting times should be
-stored in separate files.
+stored in separate files
+(and the [`recording-<label>`](../99-appendices/09-entities.md#recording)
+entity MAY be used to distinguish these files).
 
 If the same continuous recording has been used for all subjects (for example in
 the case where they all watched the same movie), one file MAY be used and
@@ -176,25 +205,3 @@ A guide for using macros can be found at
       },
    }
 ) }}
-
-### Other RECOMMENDED metadata for physiological data
-
-The following RECOMMENDED metadata can also be added in the side-car JSON files
-of any `*_<physio>.tsv.gz` file.
-
-<!-- This block generates a metadata table.
-The definitions of these fields can be found in
-  src/schema/objects/metadata.yaml
-and a guide for using macros can be found at
- https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md
--->
-{{ MACROS___make_metadata_table(
-   {
-      "Manufacturer": "RECOMMENDED",
-      "ManufacturersModelName": "RECOMMENDED",
-      "SoftwareVersions": "RECOMMENDED",
-      "DeviceSerialNumber": "RECOMMENDED",
-   }
-) }}
-
-<!-- Link Definitions -->