Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for preferred video role flags #9402

Closed
soldierinwhite opened this issue Sep 8, 2021 · 7 comments
Closed

Add support for preferred video role flags #9402

soldierinwhite opened this issue Sep 8, 2021 · 7 comments
Assignees

Comments

@soldierinwhite
Copy link

soldierinwhite commented Sep 8, 2021

Use case description

Some manifests can include alternative video tracks for example for sign language interpretation of the same video. The manifest parser is able to parse the roles included on these representations and is accessible via format.roleFlags. A client should be able to give a preferred role flag for video from those defined in C.java to the track selector in the same way preferred language or audio role flags are catered for. Right now this is possible with setting a custom selection override, but that also entails including all other filters such as bitrate and viewport width filtering in a custom override rather than using the default trackselector's built-in support.

Proposed solution

DefaultTrackSelector.ParametersBuilder would have a setPreferredVideoRoleFlags(int flags) that would accept single or combined flags for instance setPreferredVideoRoleFlags(ROLE_FLAG_ALTERNATE|ROLE_FLAG_SIGN). In the case of a single flag the track selector would select tracks that may have more flags than those specified, but includes at least all of the specified flags, so in the example above, it would be able to select a track with flags=ROLE_FLAG_SIGN|ROLE_FLAG_ALTERNATE|ROLE_FLAG_SUPPLEMENTARY but not one with flags=ROLE_FLAG_ALTERNATE|ROLE_FLAG_EASY_TO_READ. If the specified flags are not available on any video track, revert to ROLE_FLAG_MAIN or 0, which is the default flag if none is specified.

Alternatives considered

The order of filtering related to other parameters, such as maxVideoBitrate, might be a point that needs some thought. Should flags be filtered on after bitrate filters are already applied or rather bitrate filter be applied after role flags are already filtered. In my opinion the latter is preferable, as that would be most compatible with user needs.

@ojw28
Copy link
Contributor

ojw28 commented Sep 8, 2021

The track selector would select tracks that may have more flags than those specified, but include at least all of the specified flags

Would you expect this logic for audio and text as well, because we don't enforce the part in bold for those at the moment. We should probably make sure all three work in the same way, else it's going to be pretty confusing for someone who's using more than one of them.

@soldierinwhite
Copy link
Author

soldierinwhite commented Sep 9, 2021

Would you expect this logic for audio and text as well, because we don't enforce the part in bold for those at the moment. We should probably make sure all three work in the same way, else it's going to be pretty confusing for someone who's using more than one of them.

I didn't know that, but in my opinion that would be best. Since exoplayer parses different tags in DASH adaptation sets to one scheme of flags, you will often see adaptation sets with an accessibility tag that gets one flag, and a role tag that gets another flag. An exoplayer client might just be interested in the accessibility for instance, say ROLE_FLAG_SIGN, so then a role flag like ROLE_FLAG_ALTERNATE shouldn't have to be specified as well, since the client would be equally interested in having that track selected if it had ROLE_FLAG_MAIN or no role tag whatsoever. Imagine a service where you have dual medium content (default video translated to sign language), but also content specifically made for accessibility (only sign language exists). One user setting to choose sign language should easily be able to cover both cases.

If the preferred flags had to be a perfect match, it would mean that such clients where the flags can come in a variety of combinations would still have to implement custom selection overrides that implements this type of behavior.

Edit: If your meaning was that any of the preferred tags added would be filtered on, that might be less problematic. I think it would be acceptable that that would be the case if that is the case for audio right now.

@ojw28
Copy link
Contributor

ojw28 commented Sep 9, 2021

To clarify, our current logic for audio and text does not require perfect matching. It does a bit-wise AND between the requested flags and the flags defined by the media track, and counts the number of set bits in the result. A track with a higher count is preferred to a track with a lower one. Suppose the requested flags are X and Y:

  • If there are tracks that declare X and Y, they will be preferred over other tracks.
  • There is no penalty for tracks that also declare additional flags. A track that declares X, Y and Z will be scored the same as one that declares X and Y exactly.
  • A track that only declares one of X and Y will be preferred over a track that declares neither of them.

The third bullet point was the distinction I was drawing between our current logic and your description. In your description, you stated that a track should "include at least all of the specified flags" to be chosen, where-as we'll prefer to choose a partial match over a track that declares none of the flags.

@soldierinwhite
Copy link
Author

soldierinwhite commented Sep 9, 2021

Thanks for the clarification. In that case I prefer the current logic, you should keep the same for video as well. That is much better than what I envisaged. I guess the only use case this does not cater for is when a specific track I am looking for is identified by more than one flag, but I am not interested in other tracks that only have some of the flags. But that seems like an edge case that is hard to cater for.

@ojw28
Copy link
Contributor

ojw28 commented Sep 9, 2021

Thanks! There is one small change I'd propose for the existing logic, which is to boost the "exact match" case so that tracks whose flags match exactly are preferred over tracks that also declare additional flags, in the case that both are present.

@soldierinwhite
Copy link
Author

That sounds like a great improvement.

@ojw28 ojw28 assigned tonihei and unassigned ojw28 Nov 3, 2021
kim-vde pushed a commit that referenced this issue Nov 26, 2021
And also tweak existing role flag logic to strictly prefer perfect
matches over partial matches.

Caveat: Video role flags only supported for fixed track selections
(same issue as Issue: #9519).

Issue: #9402
PiperOrigin-RevId: 412292835
@tonihei
Copy link
Collaborator

tonihei commented Nov 29, 2021

Added support for preferred video role flags now and update existing role flag logic as described above (=prefer perfect matches over partial matches). Note that preferred video flags are only applied for fixed video track selections and will not be used for adaptive playbacks. This is due to the same problem as #9519 and we will use #9519 to track fixing this for both cases.

@tonihei tonihei closed this as completed Nov 29, 2021
tonihei added a commit to androidx/media that referenced this issue Dec 2, 2021
And also tweak existing role flag logic to strictly prefer perfect
matches over partial matches.

Caveat: Video role flags only supported for fixed track selections
(same issue as Issue: google/ExoPlayer#9519).

Issue: google/ExoPlayer#9402
PiperOrigin-RevId: 412292835
@google google locked and limited conversation to collaborators Jan 29, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants