Improve Android package management for the NDK #916
Comments
|
Could you post a link to AAR? A simple google search for "aar package manager" doesn't reveal anything. |
|
https://developer.android.com/studio/projects/android-library tl;dr it's like a jar, but has Android specific pieces. It's distributed via maven like a jar would be. |
|
@DanAlbert, I believe that the androidNativeBundle gradle plugin already implements this for AAR, at least partially. |
|
So this is basically an Android only solution? It's not meant to provide packages targeting Linux, Mac or Windows? I hoped that AAR were something different than Android Archive (AAR) that I didn't know about. |
|
tl;dr: No, it isn't Android specific (see that "Non-Android targets" section of the original post). To elaborate and hopefully clear up the confusion, it depends on what you mean by "this". There are two separate pieces here:
There are a large number of good distribution mechanisms out there, so I figure we're better of leaning on those to do what they do best. Build system integration in the form that we want it, however, doesn't have many good options. Requirements 2, 4, and 10 in particular ruled out the existing options. This bug describes a new tool (Prefab) that generates build scripts for a set of prebuilt libraries and headers. It plays no part in dependency resolution or distribution. So, no, Prefab is not Android specific. As the original post states, Android support will be the first implemented, and I personally don't have plans to add other platforms (though Linux might be added just so I can prove to myself that what I have is easily extended to support additional platforms), but we'll accept patches for other platforms (within reason; if you want VAX support you can maintain your own fork). |
|
Considering native builds, did you have a look to conan ? |
|
From a quick read, that seems to check must of the boxes. I'll have a more thorough look later today to see if that can work for us. My initial concern is that it doesn't support multi language packages, but that may be something we can add. Thanks for the pointer. Hadn't seen that one before. |
|
Huh. Having now read further, this actually looks a lot like what I'd designed, but even includes some of the stretch goals I haven't discussed here (source distribution, namely). The only requirement we have that I don't think this covers is the ability to play well with mixed Java/C++ packages, but at a glance that might still be doable? Conan looks incredibly well thought out and executed, so I'm going to play with this a bit and see if it can be made to fit all our requirements. |
|
It looks like Conan can also be made to package a jar quite easily. Seems like the only thing we would need to do would be to build a gradle plugin that can handle Conan (it might be a part of the Android gradle plugin, if that integrates better). |
|
On my side, I started the development of native dependencies management some years ago, and started to define coherent compile/link rules for cross-platform development, before conan gets stable. Nowadays, the system (recently renamed remaken) has evolved (a release will come soon - the main tool is in the finalization step) to a meta-packaging system. I integrated conan, vcpkg, but also system's dependent packagers (apt, yum, brew, chocolatey) and the binary format defined at the beginning of the project. Conan was easy to integrate to remaken, and nowadays remaken offers the versatility to choose dependencies among the various binaries available, and depending on the project specific needs. If you 're interested, I will provide the link to the (future) github repository hosting the tool. For the moment, the qmake build rules are available from https://github.com/b-com-software-basis/builddefs-qmake. The remaken tool roadmap will include cmake compile and link flags generation, remaken format packager from a build folder ... |
|
Whatever solution you decide on, please, don't require external dependencies - like Python, Java... |
|
The top two choices at this time will require one of those. The NDK/SDK already depend on and include Java and Python, so this isn't anything new. |
|
Wanted to give a quick update since it's been a while: Spent some time evaluating other options and came to the conclusion that the original plan (discussed in the first post of this thread) was going to offer the best UX for Android. The tooling to make this work and the gradle plugin changes to make it a transparent part of an AAR have been written. There's still a fair amount of polish that needs to be done on both before it's shippable, but we have a working proof of concept. The current hope is that this will be available alongside the Studio 3.7 release cycle, which will likely go to stable in the first half of next year (I'm not sure when the canaries will first be available, or which stage will be the first one with these features available). Hoping to publish the source for the tooling far ahead of that, which will include design info and usage docs. |
|
What I find worrisome about creating a completely new delivery system is who is going to add and maintain new packages? I doubt that the audience of NDK is that huge. Would google add a few popular libraries and that would be it. How will this plan fit with this: #1011 |
That's covered in the original post. Also #916 (comment)
Orthogonal. |
Gradle needs to be able to specify a dependency import path for Prefab modules. Add NDK_GRADLE_INJECTED_IMPORT_PATH to support this. This could instead be a more generic NDK_ADDITIONAL_IMPORT_PATHS, but that's probably of limited use since Gradle will clobber the user's flags with its own. Test: ./run_tests.py --rebuild Bug: android/ndk#916 Change-Id: I18f9ff2830846d9ea63130d6949c9d199b72db60
|
This is very good news, as all dependencies in a CMake-based project should always use ˋfind_packageˋ for resolving dependencies. I have written about this topic before as well and demonstrated a straight forward approach for Android- as well as external dependencies here in case you are interested. [1] [1] http://kai-wolf.me/cpp/android/cmake/gradle/2019/02/18/working-with-cpp-in-android-effectively/ |
|
Many commonly-used libraries is very need to use (cURL, openSSL, boringSSl, etc,... ). But hard to build for use in ndk. |
|
BoringSSL is particularly annoying (you need a go toolchain to build it...) It's not particularly useful for apps either, so that's not one we intend to publish (but anyone that wants to do so can of course do so themselves). cURL and OpenSSL are ones that we've already prepped for this, along with jsoncpp, since it was easy and helped me make a demo with the other two. |
|
The underlying implementation of this is now up on GitHub: https://github.com/google/prefab (expect a lot of broken URLs; I only just published this so I haven't setup the matching github.io page yet). While it's a big step closer to finishing off this bug, it's just one piece. We still need to:
Should still be on track to have this available in AGP 4.0. |
|
Out of curiosity, which are the reasons that made you discard Conan in the end? |
|
Is there a wishlist/roadmap of which libraries you (Google) will support? As a part of the NDK users are multimedia developers I'd expect that at least the most popular multimedia/graphics/audio libraries to be included: freetype2, SDL2, OpenAL Soft, opus, vorbis, ffmpeg, some VP9, av1 libraries, etc |
Seamless integration with Gradle was one reason. The largest concern though is that with conan the user's machine has to build the packages, not just fetch them. The building part is one of the things we're trying to avoid doing. It adds unnecessary time to the user's build and almost certainly will result in "works on my machine". With how much code out there still uses autoconf and similar tools, that would likely be a large problem for Windows users. I don't think this has been much of a problem with Conan so far because primarily the people building things on Windows are also the people building things for Windows. For Android development there would likely be many Linux-only packages that would not build on Windows. We also are able to do a bit better with ABI matching than Conan can. Conan requires an exact match of the build settings across modules, so the common case for Android where a library is built for android-16 but the app is targeting 21 would not work. With Prefab we can currently match that, and by the end of the week it should be improved to allow finding a best match if there's more than one valid match (google/prefab#8).
I've already taken care of OpenSSL, cURL, and jsoncpp. The cURL/OpenSSL use case was our benchmark for "have we done this right at all", and jsoncpp tied those together into a presentable sample. Someone I spoke with at ADS requested LuaJIT, and @alexcohn has a big list for us over at #456 (comment). The Prefab bug tracker is going to be the home for package requests. I haven't filed any of those I just mentioned yet, but feel free to start populating it: https://github.com/google/prefab/issues/new/choose (use the "Package request" template). We're going to have to think about which packages we commit to, since each one will have a recurring cost that comes with it. We fortunately do have some tooling that helps us keep things up to date in AOSP that I'm hoping we can extend to cover this use case, but I'm particularly worried about testing these. That is to say, please file those requests, but be patient while we figure out what we can and can't afford to support right away. Based on what I've seen people struggle with on SO, I think that from your list SDL2 and ffmpeg are definitely two that we want to support. |
|
A quick update on where this stands:
I've just sent a PR adding a sample: android/ndk-samples#678. You won't be able to build it yet since the AGP change hasn't shipped yet, but that should be happening soon (I don't know the exact date, but early January is likely). Once that's done, you'll also need to build the AARs that it imports yourself because those also need to be published, which will happen later in January. |
|
FYI the Gradle portion of this was included in AGP 4.0.0-alpha09. An initial set of packages, docs, and a blog post are still on the way. |
|
The curl, openssl, and jsoncpp packages got pushed to maven today. The sample is still pending (android/ndk-samples#678), but should now be a usable starting point for anyone wanting to poke at this. Bugs should go to https://issuetracker.google.com/issues/new?component=803099 (if you're sure it's a prefab bug and not a bug in the gradle plugin, can also file at https://github.com/google/prefab/issues, but prefer Issue Tracker if you're not sure). I'm just waiting for final approval on the docs and the blog post so those should be live within a week. |
|
I am waiting for the docs to try out this feature. And maybe contribute a couple of packages if possible. https://github.com/google/prefab/blob/master/docs/index.md
I don't understand are only binaries distributed by the system or source code or both? |
|
@Zingam the sample is live now: https://github.com/android/ndk-samples/tree/master/prefab/curl-ssl. The docs provide some explanations for things, but that might be enough to get you started playing with it. I'm hoping to have the docs live before the end of the day, by we're still polishing them up a bit. As for building your own packages, how easy that is depends on what package you want to ship. https://android.googlesource.com/platform/tools/ndkports/ is what we're using the build curl, openssl, and jsoncpp. It can technically handle any project that you have the source for, but it would be annoying to use for iterative development. Later on (4.1, I'm hoping) I'll get the gradle plugin to do this for regular android library modules, and that'll be the thing that really enables people to build these themselves.
It's binary (and headers) only. https://google.github.io/prefab/ explains some of the motivations, and the blog post will elaborate. The simplest answer is that Windows exists and it's really hard to build autotools projects on an arbitrary Windows machine, but that's not the only reason. |
|
btw, for anyone trying to use this with canary 9, apparently canary 9 defaults to prefab-1.0.0-alpha3 which is lacking google/prefab#47 and google/prefab#49, so you'll want to add the following to your gradle.properties alongside the property to enable prefab: # Will need this until 4.1 or later
android.enablePrefab=true
# Will need this until 4.0 beta 2
android.prefabVersion=1.0.0-alpha5 |
|
While this feature was under development, support for GitHub Packages has been announced – a solution that helps to distribute built projects, especially the ones hosted on GitHub. They already have gradle integration, and probably the distance from full support for Prefab should be minimal. |
|
For Android projects, no extra work needed. Prefab packages are already distributed as AARs through Maven, and it looks like GitHub packages already supports that. Definitely an option for folks to consider. Using it for non-Android projects is a different story :) The format and tooling are amenable to it, at least, it just isn't fleshed out. |
|
@DanAlbert Thank you very much for taking your time to explain me and I'd like to apologize for the stupid questions. I was confused about the workings of these features. What I did for my current project is a standard gradle library module with a special directory where I drop the third party C++ source code. I include the library in my main project and whenever I need to add a new library I just copy that standard library module as necessary. |
Not all in one piece, no. We will be tackling each of those problems individually, but as far as distribution goes I think distributing binaries and headers is going to be a much better fit for our users given the wide variety of build environments (and everyone's favorite: faster builds).
You don't actually need to build the libraries to debug them; you just need debug info and the sources that match the binaries. Filed google/prefab#65. It's not something I'll be able to look at any time soon, but it's a good idea. Thanks for the suggestion! |
|
Docs are live: https://developer.android.com/studio/preview/features#native-dependencies They'll be migrated into their own page once 4.0 is out of preview. |
|
@DanAlbert Is there an index/an register (planned) where a developer might check for what prefabs are available? |
|
Can find the ones we (the NDK team) own by searching for "ndkports" in the Maven index that's linked in the blog post. For everything else, the project will probably publish their own links, so they'll be easy to find (Oboe and eventually androidx, for example). We can also keep a list in the prefab wiki. |
|
One proposal that should have come to my mind earlier. It would be nice if the same AAR could be used both by C++ and Java developers. I believe that the compiled classes can be inserted into a prefab archive at no cost, but the expected locations of native libraries that should be packaged into the resulting APK or AAB, are different. |
|
That's already covered, and was actually one of the reasons we ended up choosing this over a different option. GTestJni needs that behavior, and its requirements drove that aspect of the design. If the Java code in the AAR depends on the same libraries that prefab is trying to export, I think that should Just Work, but I haven't tried it. The library will end up duplicated into both the jni and prefab directories, but the size of the AAR won't affect the app at all. |
I think that's a good idea. |
I thought this duplication could be avoided. It looks like |
With some effort it could be done, but I'm not sure it would be worth it.
Created https://github.com/google/prefab/wiki/Package-List. Just send a PR if you have a package to add to the list. |
The list should at least link to https://maven.google.com/web/index.html; other packages that will eventually be not on Google's Maven repository, will need more explanations (e.g. GitHub Packages require access token to be imported). |
Unfortunately, this duplication is bad. Not only because it make the AAR almost twice as big (nobody cares about development file sizes these days), not only because maintaining the binaries in two locations to be identical is and additional burden on the developer who chooses to produce the AAR, but mainly because when it is consumed, AGP v.4.0.0-beta04 shows an alarm: I tried to use prefab for ndk-samples/hello-libs, but to avoid this warning I generate two separate AAR files for gen-libs module (one of which is a throwaway). |
|
Unfortunately, it seems that the cause of warning is not that the binaries are duplicated inside the AAR, but that the two subsystems in AGP push them to the output bundle after all kinds of transforms having been applied. Note that this happens before packaging, so the standard |
|
Recently I gave a presentation on prefab: https://youtu.be/-QbnbrSUaGA, unfortunately under the limitations of the coronavirus. |
|
(nice! here's a direct link to alexcohn's slides for that video: https://corecppil.github.io/Meetups/2020-03-37_Core_C++_in_Cyberspace_Online/Package_Management_for_Android_C++_Alex.pdf ) |
Test: SHELL=/bin/bash tools/base/bazel/test_studio.sh Bug: http://b/111405525 Bug: android/ndk#916 Change-Id: I13ab1a05b6043ca97a211a721e4b2cd290a7ad23
This bug tracks the Easier access to common open-source libraries item on our roadmap.
Some questions about this came up on another bug, specifically in what we were planning to use and why, so I wanted to publish the requirements that we had for this, show what we'd considered, and explain the current plan.
Note that this isn't the full design document. I've spared the nitty gritty details since I don't think they're particularly interesting at this time, but if anyone has questions just ask and I can elaborate :)
Requirements
Options
With these requirements, none of the existing options we know of will work as-is. We considered cdep, AAR (Android packages via Maven), vcpkg, and hunter. cdep is C/C++ only, which means packages like GTestJNI can't be distributed with it. AAR doesn't support C/C++ well (it supports C/C++ implementation, but the interface to the library must be Java), which obviously won't work for the NDK. vcpkg doesn't seem to support more complex build system features such as exporting required flags to dependents. Hunter is CMake only.
Of these options, AAR seems like our best bet. It doesn't meet the single most obvious need (being able to actually expose a C++ interface), but unlike the other options it's the only one that could be reasonably modified to fit these requirements. It also is the extant method for distributing Java dependencies for Android, and reusing existing infrastructure is a big plus.
Design
The plan is to build some low level tooling (which I'm currently calling Prefab) for generating build system integration for prebuilt packages and some metadata describing usage requirements and then extending gradle and AAR to make use of that.
When I say low level, what I mean is that Prefab only solves the build system integration part of the problem. For an Android gradle project, gradle is still responsible for resolving and fetching dependencies. When gradle fetches an AAR that contains Prefab artifacts, it extracts those to a directory and calls prefab to have it generate build system integration. The generated output is then passed to ndk-build or CMake to be used as you would any other dependency. This means that if your organization uses some other mechanism for distributing dependencies, Prefab artifacts can be distributed that way as well (including something as simple as a git submodule).
For example, your build.gradle would contain something like the following:
and your CMakeLists.txt would contain:
And that's it. Gradle will fetch the dependencies, call Prefab to generate the CMake package definitions, and
find_packagewill find the dependencies.Build system integration
Build system integration is provided by plugins for Prefab, and we'll include CMake and ndk-build plugins by default. If you're using a custom build system, you can provide a path to a plugin jar on the command line to teach Prefab how to generate what you need.
Non-Android targets
Support for non-Android targets will be simple to add (basically just a class recognizing and perhaps disambiguating libraries for the platform, such as Android needs to do with multiple ABIs, minSdkVersion targets, etc). We're only planning on implementing support for Android, but patches adding other platforms are welcome.
So, that's the rough outline. If I've missed any major requirements that make this approach not viable for your project, please speak up. We're still in the very early phases of this so it's not too late for large changes.
The text was updated successfully, but these errors were encountered: