Request: option to replace duplicate frames with interpolated ones instead of dropping them

[awebeer256]

Usually, duplicate frames in the input video do not represent an actual ceasing of motion in the canonical events that the video is portraying. Instead, they are typically an artifact of either the video having been reformatted from 24 or 25fps to 30, or an animation style that duplicates frames to cut costs. In both of these cases, an interpolated video will be smoother if the interpolation software recognizes these duplicate frames as placeholders. For example, suppose your input video has the following frames: A, B, C, C, D, etc.

If I understand correctly, the current default behaviour (doubling the framerate) will cause the following output:

In  Out
--------
A   A
    A-B
B   B
    B-C
C   C
    C
C   C
    C-D
D   D
etc.

And using the --skip flag will produce this instead:

In  Out
-------------
A   A
    A-B
B   B
    B-C
C   C
    C-D
C   D
etc.

That is, with --skip, duplicate frames are effectively cut from the input video—which shortens the video—before interpolation. This strikes me as...let's call it "extremely counter-intuitive"...from a user experience standpoint. I'm guessing that this flag was meant to only be used in tech demos, but the fact that it's the only even remotely intelligent way of handling duplicate frames gives the opposite impression. The intuitive behaviour would be to keep the duration the same but interpolate through the pair of duplicate frames, so the output would look like this:

In  Out
-------------
A   A
    A-B
B   B
    B-C
C   C
    C-(C-D)
C   C-D
    (C-D)-D
D   D
etc.

In other words, instead of the duplicate frame being cut, it is replaced with an interpolated frame, which is then interpolated with its neighbours as normal.

What I'm proposing is changing the default behaviour such that whenever a sequence of duplicate frames in the input video, with a length of exactly 2, is found, the second one is replaced with an interpolated frame before the main, framerate-increasing interpolation is done. This would cover the two main use cases that I mentioned up top. False-positives would be both rare and minimally harmful, but a user could disable the feature through command-line options. Perhaps an option called --dupes, with the possible values of keep (the current default), absorb (the new default that I'm proposing), and skip (equivalent to the --skip flag, which would in turn be deprecated).

[FelixKainz]

I amsolutely second this.

[hzwer]

@awebeer256 I realize this new function, it looks amazing.

[talosh]

We have it in Flame version and it is one of the most useful things

[bubbaprog]

Just to be clear, the removal of --skip breaks interpolation when there really are static frames and when paired with an fps change results in video that freezes on every static frame.

[mirh]

Isn't --exp=0 supposed to handle that?
#143

[WKolodziejski]

As a complement to this option, wouldn't be possible to add an option to replace predefined frames? Example: An anime scene where the background is cgi and moves every frame, but the foreground is animated at half fps. The user could replace the static frames with some very distinct predefined image, and then use that image as an input parameter for the algorithm. This way, the algorithm would recognize the image among the frames and replace it with a new predicted frame.

[crowbardoctor]

An FPS selector for the video input would be useful and solve the problem. This would allow users to choose the desired FPS for their videos, such as 15 FPS, before the video is interpolated by Flowframes. This would streamline the video editing and post-production process.