Document View for JSON

[falkreon]

When I say "Document" I don't mean org.w3c.dom.Document, but it's on my mind. Preserving ordering and comments are a pillar of this library and honestly some of the edge-case behavior is still unacceptable.

Currently, comments are attributed to a node in the node-list (usually a key-value pair). But what happens in this case?

{
  foo: //this is a key
  10 //this is a value
}

The best thing we could do after a roundtrip right now is this

{
  /* this is a key
  this is a value */
  foo: 10
}

And I'm really not sure we're even doing that. Worse, what do we do here?

/* this object blah blah blah
...
blah. */
{
  foo: 10
}

So every object needs to also carry around a "top-level comment" field?

What I want to do, and this is going to be a big project, is to create a structural view of the file. That is, an object is a sub-file and contains individual nodes for keys, objects, multiline comments, end-of-line comments, etc, and can be used to more or less recreate the original file, without moving or merging comments. We will fix indentation and any errors we find along the way, and in some ways regularize the file. This is important for jankson's usefulness as a preprocessor. But this would unhook the comments from the key-value pairs and give us a foothold up to handling all the edge cases.

Does such an object continue to support random access and structural modifications? Do we replace the implementation for JsonObject, or hold these two views into the file as distinct - one document view and one semantic view, and let the user pick which one they want to work with for any given task?

This is a hard design problem, and I'm looking for input on how we could get it done cleanly and clearly.

[Barteks2x]

I think the most reasonable way to deal with it would be for comments to be their own type of entry inside any json object or array, and also have a top level json "container" for the entire file. That would naturally preserve the order and allow to keep any extra information.

It would mean they are no longer associated with any given elements, but there could be a way to access an element with comment context around it, and then it would be on the user of that library to decide how to interpret it when manipulating the data.

In json object it wouldn't be associated with any key, but could be accessible in a collection of all entries.

Edit: I didn't think about comment being in the middle of a key-value pair. At that point it would also require putting an optional comment field in each a json object entry.

Alternatively, comments could be be considered more as separate things in each json object and array, with information about which elements are before and after it (possibly including information whether it's on the same line or a different line?).

Moving, removing and adding elements would need an option to specify how to deal with comments around it.

But at least for my use case, I think the most natural way to represent it would be for json objects to just expose a collection of some sort of "json object components" which could be: comment, key, value (possibly as a linked list type of structure, where each element would just have a reference to previous and next). And also allow constructing json objects from such elements. But my use case is quite unusual and probably wouldn't work for everyone.

[Barteks2x]

The reason I thought line breaks matter is that this:

{
  "key": "value", // comment
  "key2": "valye2"
}

is in user's view very different from

{
  "key": "value", 
  // comment
  "key2": "valye2"
}

In the first case, the comment obviously belongs to the first key-value pair (which is why I came up with the distinction for end of line comments)
In the second case it seems like it should be attributed to key2.

Dealing with them as completely separate without attribution and not preserving type solves that. Then you would only have separate after key and before value and after value distinction for placement around a separator. And not keeping the type of comment would be fine, I just wasn't sure if you wanted to do that.

Then we would have the following:

{
  // comment1
  "key" /* comment2 */: /* comment3 */ "value" /* comment4 */, // comment5
  // comment6
}

comment1 and comment5\ncomment6 would be standalone comments. comment2 would be jsonObjectKey.commentAfter(), comment3 would be jsonObjectValue.commentBefore() and comment4 would be jsonObjectValue.commentAfter().

Object representation could compared to your above example look like this:

documentRoot {
  objectElement {
    standaloneComment: "comment1"
    objectEntry {
      afterKeyComment: "comment2"
      key: "key"
      beforeValueComment: "comment3"
      afterValueComment: "comment4"
      value: primitiveElement { "value" }
    }
    standaloneComment: "comment5\ncomment6"
  }
}

(2 comments could be kept as separate instead of with newlines)
And it could get reformatted as:

{
  /* comment1 */
  "key" /* comment2 */: /* comment3 */ "value" /* comment4 */, 
  /* comment5
  comment6 */
}

or alternatively with an option to prefer // style comments when they are on their own line:

{
  // comment1
  "key" /* comment2 */: /* comment3 */ "value" /* comment4 */, 
  // comment5
  // comment6
}

Or possibly with an option to force // comments:

{
  // comment1
  "key" // comment2
  : // comment3
  "value" // comment4
  , 
  // comment5
  // comment6
}

In json arrays, the only possible distinction may have to be "after value" comment to distibguish between

[
  "value", // comment
  "value"
]

and

[
  "value" /* comment */,
  "value"
]

[falkreon]

[ "value" /*comment*/ "value" ]

this is a legal construct in jankson. I don't think commentAfter is particularly useful, but if it exists, then it's possible for the commentAfter to be the commentBefore of the next element.

[falkreon]

As I get further into the nuts and bolts of implementing such a thing, it seems like we want to make a "KeyValuePair" intermediate entry which itself holds a list of entries associated with it. This would help clean up the semantics for what commentBefore and commentAfter might eventually mean:

//things
/* stuff */
{
    //thing
    //other thing
    "status": /* foo */
    [ 4 ] // number chosen by a fair diceroll; guaranteed to be random
    //bar
} // a
//b

Possible object tree:

documentRoot {
    commentEntry "things" // document preamble
    objectEntry { //not synthetically generated for "bare root objects". if the root was bare, it's bare in the doc.
        commentBefore: commentEntry "stuff" //multiple consecutive single-line comments would be folded into a multiline here
        commentAfter: commentEntry "a" //only because it's a line-end comment on the same line
        ---
        keyValueEntry {
            commentBefore: commentEntry "thing\nother thing" //folded
            commentAfter: commentEntry "number chosen by a fair diceroll; guaranteed to be random"
            ---
            primitiveValueEntry "status" //key
            commentEntry "foo" //not commentBefore arrayEntry because it's not "bare"
            arrayEntry {
                commentBefore: null
                commentAfter: null //consumed by keyValueEntry? same as keyValueEntry? TBD
                ---
                primitiveValueEntry 4
            }
        }
        commentEntry "bar"
    }
    commentEntry "b" //Not on the same line as the closing brace, so this is a document footer, along with all following comments.
}

When reserializing part of a document, the commentBefore is serialized, then the object itself (including any sub-objects), then commentAfter, then the final newline.

I'm still unsure of whether there should be a distinct KeyEntry type, and how to handle the naming. The current prototype uses JsonDocument and CommentDocumentEntry but these are cumbersome, and honestly if we wind up parsing and serializing in different formats like hocon, that language will become wrong.

[falkreon]

This should improve the readability of any accesses too, because you could potentially grab an iterator for only the keyValueEntries in an object, which would include any associated comment information based on our comment association rules, and not touch any of the user fluff around or in between the important parts. It would also help the internal implementation avoid scanning too far, and help keep validation of a document simple.

[falkreon]

continuing to tweak as I implement - if I hardcode a little bit more of the KeyValueDocumentEntry, such as:

keyValueEntry {
    key: "status"
    value: (reference hard equal (==) to the "arrayEntry" below)
    commentBefore: commentEntry "thing\nother thing" //folded
    commentAfter: commentEntry "number chosen by a fair diceroll; guaranteed to be random"
    contents: [
      commentEntry "foo"
      arrayEntry: (...)
    ]
}

The serialization flow for this would be to write commentBefore, then the key, then all entries, then commentAfter, followed by the newline. I'm not convinced that we need the ability to serialize comments before the colon, and in this scenario any comments deserialized before the colon, would be reserialized after it (but still before the key). The colon is "part of the key" to me, and that's important to the readability of the file, it's like fixing indents.

Speaking of which, we probably want to detect the indent style of the whole resource and stash it in the Document. Just count up the leading tabs and spaces as we scan, compare them to the indent levels, and get heuristic measurements:

• average number of leading spaces per indent level for spaces indents
• average number of leading tabs per indent level for tabs indents
• number of leading spaces and tabs to select whether spaces or tabs are used
  And use this to assemble a String representation of a single indent level which gets stashed on Document. In the future, perhaps we can measure some other heuristics like brace style and stash those too.

Thinking I actually do want newlines tagged in the document, so that an array like

[ 1, 2, 3, /* foo */ 4,
  5, 6, 7, /* bar */ 8 ]

can pretty much survive unmangled in a full de/serialize cycle. Line-end comments require a newline, so they'd imply one. But an empty line-end comment is not the same thing as a raw line break, so trailing // would survive too, which can be important for multi-line-end comments that make up a larger block.

[unascribed]

[falkreon]

I'm not convinced that adding an asterisk to the front of a comment will make it clear to a non-technical end-user what the comment is for. I think if they see a document formatted like that, they'll just assume that's how you do comments - and then be confused when their comment is completely clobbered.

Yes, introspecting into documents can be difficult. Yes, things would be much easier if we could, instead of dealing with comments everywhere, specify that comments, or a certain type of comments, only go before keys. If we can have a hot path for dealing with the common case (such as observing that the document only has comments before keys, therefore comments may be attributed cleanly to keys), we should make that easy. But this does not help us preserve comments stuffed in odd places.

[falkreon]

really the attribution problem is more like

/* a */
{
  /* b */
  foo /* c */ : /*d*/ bar //e
}

What you seem to be suggesting is that b should be the only reasonable attribution position (and a for the object), but should we really make e undetectable?

[unascribed]

I'm not convinced that adding an asterisk to the front of a comment will make it clear to a non-technical end-user what the comment is for.

I feel like it's the responsibility of the program creating a file that makes use of doc comments that is expected to be used by people who don't know what a doc comment is to briefly explain the concept in a header comment.

But this does not help us preserve comments stuffed in odd places.

No. It's a suggestion for making a certain kind of special comment easier to introspect, while still using the document approach.

What you seem to be suggesting is that b should be the only reasonable attribution position (and a for the object), but should we really make e undetectable?

All of the comments would be detectable, but only in the document view as explicit comment nodes. Only a and b would be available as doc comments if the API user chose to use them, and if they started with /** rather than /*.

[falkreon]

mostly on board, but this is somewhat at odds with the pillar of embracing the weirdness humans create in their files. Why does it have to be a doc comment to be detected?

[falkreon]

Starting to implement the Document view in mainline. How should we relate the semantic objects to the structural ones? What level of caching or defensive copying should occur? Does, say, a structural change in the document affect any derived semantic objects, and vise versa?