Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HLS muxed audio missing format label #9608

Open
szaboa opened this issue Oct 26, 2021 · 8 comments
Open

HLS muxed audio missing format label #9608

szaboa opened this issue Oct 26, 2021 · 8 comments
Assignees
Labels

Comments

@szaboa
Copy link
Contributor

szaboa commented Oct 26, 2021

We have the following HLS (live) master playlist:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-STREAM-INF:BANDWIDTH=130780,CODECS="mp4a.40.2",AUDIO="aac-primary"
../../<path>/chunklist_ao.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=130780,CODECS="mp4a.40.2",AUDIO="aac-backup"
../../<path>/chunklist_ao.m3u8
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-primary",NAME="Home",LANGUAGE="eng",DEFAULT=YES,AUTOSELECT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-backup",NAME="Home",LANGUAGE="eng",DEFAULT=YES,AUTOSELECT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-primary",NAME="Away",LANGUAGE="eng",URI="../../<path>/chunklist_ao.m3u8",DEFAULT=NO,AUTOSELECT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-backup",NAME="Away",LANGUAGE="eng",URI="../../<path>/chunklist_ao.m3u8",DEFAULT=NO,AUTOSELECT=YES,CHANNELS="2"

Based on the HlsPlaylistParser (line 514), if an audio track has no URI defined, then it will be stored in the muxedAudioFormat. However in this case, whenever we query the audio tracks, the muxed one has no id/label in the Format.

See the event logger's output:

MediaCodecAudioRenderer [
  Group:0, adaptive_supported=YES_NOT_SEAMLESS [
    [ ] Track:0, id=0, mimeType=audio/mp4a-latm, bitrate=130909, codecs=mp4a.40.2, channels=2, sample_rate=48000, supported=YES
    [ ] Track:1, id=1, mimeType=audio/mp4a-latm, bitrate=130909, codecs=mp4a.40.2, channels=2, sample_rate=48000, supported=YES
  ]
  Group:1, adaptive_supported=YES_NOT_SEAMLESS [
    [X] Track:0, id=aac-primary:Away, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=en, label=Away, supported=YES
    [ ] Track:1, id=aac-backup:Away, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=en, label=Away, supported=YES
  ]

This is causing a problem for us, as we would need to select the tracks based on the labels. The question is whether the missing labels of muxed audio tracks is intentional or if it's a bug? How could we workaround this issue on our side (without changing the manifest itself)?

Thanks in advance.

@ojw28
Copy link
Contributor

ojw28 commented Oct 27, 2021

Do you see this with 2.15.1 (i.e., the latest release)? I took a quick look at the code, and it seems to be written so that the label should be picked up. If that's not happening then I'm probably missing something, and it's probably a bug.

If you can reproduce with 2.15.1, could you provide a complete HLS sample that we can use to easily reproduce the issue, either as a link to an accessible test stream or as a zip file containing the necessary files? The master playlist alone isn't sufficient. Thanks!

@szaboa
Copy link
Contributor Author

szaboa commented Oct 29, 2021

@ojw28 We can't really provide a clear sample stream easily, however we have found a solution that we can apply on our forked version. You can see the change in the linked pull request, feel free to give us a feedback on that.

@ojw28
Copy link
Contributor

ojw28 commented Nov 1, 2021

I'm not sure, but that commit doesn't seem to make much sense to me. I don't understand why you'd want to set primaryExtractorTrackType = C.TRACK_TYPE_VIDEO for an audio only stream.

We can't really provide a clear sample stream easily

Can you provide a DRM protected stream easily, even without the corresponding DRM server information? I'm not sure, but I suspect that the tracks are determined before the player needs the DRM keys, so that might be sufficient to be able to reproduce. Thanks!

@szaboa
Copy link
Contributor Author

szaboa commented Nov 2, 2021

The primaryExtractorTrackType is set to C.TRACK_TYPE_VIDEO because otherwise the muxedAudioFormat won't be used at all, see the logic a few lines below the change in the PR (line 1371-1381).

@ojw28
Copy link
Contributor

ojw28 commented Nov 3, 2021

Ah, I see. Could you try applying this change locally, to see whether it also resolves the issue correctly: 6b78164

Thanks!

@guoen21
Copy link

guoen21 commented Nov 8, 2021

I apply and test this commit locally, seems it also works for me, thanks very much!

Now we get two following muxed tracks, but the id is same, because we use one muxedAudioFormat, but from our master m3u8 it included two tracks which has no URI defined.

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:0, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
  [X] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

expected

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:0, id=aac-primary:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
  [X] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

@guoen21
Copy link

guoen21 commented Nov 18, 2021

I also add comment for commit 45857b5,
line 1398, playlistFormat = playlistFormat.withManifestFormatInfo(muxedAudioFormat);
In here the language attribute of muxedAudioFormat may be miss, because of the playlistFormat.sampleMimeType is null.

For above audio-only case, it will output the expected result

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:0, id=aac-primary:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
  [ ] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

But for normal case which has video tracks, muxed audio tracks, and additional audio tracks, the output has only one muxed audio track as following:

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

In my commit I enhance it to support multiple muxed audio tracks - change the single Format muxedAudioFormat to List<Format> muxedAudioFormats. I also merge this 45857b5 commit. Now I can get the expected output

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:0, id=aac-primary:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
  [ ] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

tonihei pushed a commit that referenced this issue Nov 18, 2021
@ojw28
Copy link
Contributor

ojw28 commented Nov 19, 2021

In here the language attribute of muxedAudioFormat may be miss, because of the playlistFormat.sampleMimeType is null.

This is a good observation. We'll take a look; thanks!

In my commit I enhance it to support multiple muxed audio tracks

There is a TODO about this here. If you've implemented it, please feel free to send us a pull request. If you've implemented it in a different way then that might also be interesting, or you could try and adapt your approach to match what's written in the TODO.

icbaker pushed a commit to androidx/media that referenced this issue Nov 19, 2021
Issue: google/ExoPlayer#9608
#minor-release
PiperOrigin-RevId: 410236626
@ojw28 ojw28 assigned christosts and unassigned ojw28 Jul 5, 2022
@tianyif tianyif self-assigned this Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants