Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebAssembly platform #2

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

tshort
Copy link
Contributor

@tshort tshort commented Dec 18, 2018

This adds a platform for WebAssembly using an Emscripten shard. Object files produces are LLVM IR with the wasm32-unknown-emscripten triplet. Compiled libraries can be used to produce WebAssembly files.

Here is a builder converted to target WebAssembly.

Two things are still kludgy about this. One is that emcc has to be run once in the builder to initialize it. Emscripten needs to write set up information in ~/.emscripten. I don't know how to pre-populate that, and I don't see a way to tell emcc to look somewhere else for this. Another issue is that Emscripten depends on nodejs. For now, I just used apk add in the builder. It needs a builder, or it needs to be added as a tool. I don't know which is best. It looks a bit tough to build (I gave it a quick try but had an early failure).

PR's incoming to BinaryBuilder and BinaryProvider to support this.

A BinaryBuilder issue I ran into is that the sources for different components of Emscripten have the same base name (1.38.20.tar.gz). I didn't know how to handle this, so I used wget to pull in some sources manually.

My main goal is to try to support WebAssembly compilation of Julia code. This especially includes LLVM and some BLAS.

Emscripten has two main approaches for generating WebAssembly. This uses the oldest, most stable and most robust approach. It includes a custom version of LLVM/Clang with a special JavaScript backend. The newer approach uses LLVM's native WebAssembly backend, and it produces a wasm32-unknown-unknown target. Once LLVM v8 comes out, that would be a good time to switch (especially when Julia switches to LLVM v8).

@staticfloat
Copy link
Member

Wow, this is super interesting. I'm not aware of any platform that has attempted this, but I'm willing to give it a shot. To be clear, I'm expecting that you and other wonderful people like you are going to be supporting most of the Emscripten issues, but I'm happy to help you get things integrated properly. Let's compile the world. :)

A BinaryBuilder issue I ran into is that the sources for different components of Emscripten have the same base name (1.38.20.tar.gz).

This is unfortunate; I have half-converted things to prepend hashes before downloaded filenames (so, e.g. it should download it as <sha256>-1.38.20.tar.gz but there may be some code paths I missed. Give me a reproducer and I'll fix it.

My main goal is to try to support WebAssembly compilation of Julia code. This especially includes LLVM and some BLAS.

You want to compile .jl files to emscripten? How does this work help with that? I would think that something like FluxJS.jl would be more up your alley.

One is that emcc has to be run once in the builder to initialize it. Emscripten needs to write set up information in ~/.emscripten. I don't know how to pre-populate that, and I don't see a way to tell emcc to look somewhere else for this.

I wrote up instructions for how to hack this in, then wrote "ideally this should be set through an environment variable instead", then thought to myself "why don't I google around a bit and see if that's how this already works" and lo and behold: you should set EM_CONFIG to the path of your .emscripten file.

@tshort
Copy link
Contributor Author

tshort commented Dec 20, 2018

I do want to compile .jl files to WebAssembly using Emscripten. This work helps by (hopefully) helping to get base Julia working in the browser. @Keno and @SimonDanisch have compiled libjulia far enough so that it fails when it needs LLVM/codegen (the kf/wasm branch). This new platform may help get LLVM compiled, so that may help support codegen in the browser. I already have a version that compiles LLVM's basics using this platform and your LLVMBuilder. It'll also help by getting libraries like BLAS compiled to WebAssembly (it might have to be a basic BLAS).

Thanks for the EM_CONFIG option. I'll work on that. That'll help clean this up.

Understood on supporting the Emscripten issues. This will be experimental for a while, and depending on my time availability, things may stall at times.

Another issue I've run into is cmake builders. Emscripten comes with its own toolchain file, so it's in a different place. I can't think of a way to make it fit in the same place as others. Right now, this shard maps to /opt/x86_64-linux-gnu/. I tried mapping it to /opt/wasm32-unknown-emscripten, so I could copy the toolchain file to the same place as other platforms. But then the cache didn't work because it was read only. I didn't dig deeper.

@staticfloat
Copy link
Member

I tried mapping it to /opt/wasm32-unknown-emscripten, so I could copy the toolchain file to the same place as other platforms. But then the cache didn't work because it was read only. I didn't dig deeper.

I'm not sure what you mean by "the cache didn't work". Which cache?

@tshort
Copy link
Contributor Author

tshort commented Dec 20, 2018

Emscripten’s cache which was in /opt/wasm32-unknown-emscripten/lib/emscripten-cache.

@staticfloat
Copy link
Member

Ah, I see. You need a cache similar to the cache we provide for ccache; some persistent scratch space?

If so, you're going to need to add logic to mount a read-writable directory along with that shard. ccache is not associated with any particular shard, and so its logic is separate, but you basically want something like:

if platform_is_webasm
    push!(workspaces, webasm_cache_dir() => "/opt/wasm32-unknown-emscripten/lib/emscripten-cache"
end

Where webasm_cache_dir() can just be defined similarly to how we define ccache_dir(), and that will cause an appropriate persistent, read-write mapping to be mounted.

If you don't want the cache directory to be persistent, then we can just mount a tmpfs there within the bash setup script. Something like:

if [[ ${target} == *-emscripten* ]]; then
    mount -t tmpfs tmpfs /opt/${target}/lib/emscripten-cache
fi

You just need to make sure that directory exists first. ;)

@tshort
Copy link
Contributor Author

tshort commented Dec 21, 2018

Wow. Pretty cool. It is meant to be a persistent cache that's prepopulated.

- Move location to /opt/wasm32-unknown-emscripten
- Add .emscripten
- Install NodeJS
- Copy the Emscripten CMake toolchain to the right spot
@tshort
Copy link
Contributor Author

tshort commented Dec 31, 2018

I removed most of the clunkiness for the user.

  • It has a .emscripten, so the user doesn't need to run emcc in a build script.
  • The user doesn't need to install nodejs.
  • The CMake toolchain is in the right place.

The internal items that are still clunky are:

  • I crudely installed nodejs by hand from downloaded APK's (link). It'd be better to install this with the Rootfs.

  • Emscripten tries to write a file next to the .emscripten file for "sanity checks". That fails because it's read only. I don't know that the sanity checks are that important. I couldn't figure out why the /opt/x86_64-linux-gnu/ directory is writable, but the /opt/wasm32-unknown-emscripten/ directory is read only.

  • I have to use wget to pull in some sources because they have the same base name (here). When the first one is added to the downloads directory, it thinks it has the next one. For some files, the SHA is added to the front of the name, but it's only for some.

I played around with adding a cache directory to workspaces, but I ended up leaving it out.

@staticfloat staticfloat mentioned this pull request Nov 30, 2019
@vchuravy
Copy link
Member

vchuravy commented Dec 2, 2019

It would be great to have this, one issue is that we need a fairly new LLVM version as a baseline. 8 might be barely new enough and things are enough in flux that I work of ToT.

Basically it would be great to start having the Julia dependencies available for a WASM build, e.g. BINARYBUILDER_TRIPLET=wasm32-unknown-emscripten

@tshort
Copy link
Contributor Author

tshort commented Dec 2, 2019

Agreed. I've been holding off on waiting for the dust to settle on the LLVM side and with Emscripten. Another piece changing is the WebAssembly System Interface (WASI and wasi-libc). That might be used with or without Emscripten.

Sidenote: it took me a bit to google ToT (tip of tree).

@vchuravy
Copy link
Member

vchuravy commented Dec 3, 2019

Sidenote: it took me a bit to google ToT (tip of tree).

Sorry LLVM-speak. Yeah WASI is rather interesting since that might simplify the bootstrapping issue quite a bit.

@Keno
Copy link
Contributor

Keno commented Dec 3, 2019

The problem with wasi is that I do think we want to bootstrap in a js environment, so we can do things like use the js native regex and BigInt rather than shipping gmp and pcre. Wasi and jsvm may just be different targets for us.

mkitti referenced this pull request in mkitti/Yggdrasil May 20, 2021
Blosc2: Patch armv7l code for proper neon hardware detection
@giordano giordano added the long shot 🏹 This is going to be fun label Jul 22, 2021
vchuravy added a commit that referenced this pull request Dec 9, 2021
vchuravy added a commit that referenced this pull request Dec 9, 2021
[skip build]
simeonschaub pushed a commit to simeonschaub/Yggdrasil that referenced this pull request Feb 23, 2022
@PallHaraldsson
Copy link
Contributor

Python has dropped support for Emscripten, i.e.:

Name: wasm32-unknown-emscripten
Unsupported in: Python 3.13

switching to tier2 support for wasm32-unknown-wasi, so is that something Julia should also rather do?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
long shot 🏹 This is going to be fun
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants