-
Notifications
You must be signed in to change notification settings - Fork 717
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: a zlib plugin #2244
Comments
As newbie to |
Of course! You do need to have a good understanding of C++. Please read this doc first: https://wasmedge.org/book/en/plugin.html |
Sorry, I thought the plugin was written in rust; as I have no experience with this in C++. Please others continue to contribute. |
@juntao ,I do have a good understanding of C++.Can you provide me resources to complete this issue as I am newbie to Wasmedge . |
Sure. See the following SDKs for creating WasmEdge plugins: C++: https://wasmedge.org/book/en/plugin.html Rust: https://github.com/second-state/wasmedge_plugin_rust_sdk |
@juntao Hi, i'd like to work on this using the Rust SDK. IIUC the new plugin would be in |
Hi @littledivy Would love to see your contribution! Yes, we would like this to be an "official" plugin. So, let's put it in the official repo like other plugins. |
Hey @juntao I was able to setup a Rust plugin and some zlib functions work in this POC: https://github.com/WasmEdge/WasmEdge/compare/master...littledivy:WasmEdge:zlib_plugin?expand=1 I have a question, are we planning to expose the raw zlib C API to WASM? If so, how are pointers supposed to be handled across the WASM <-> Host memory boundary? and ofcourse the struct layout of the 64-bit host is different than 32-bit WASM code. My POC does some very hacky/unsafe WASM to host struct layout copy-conversion...which I think isn't ideal. A workaround is to not expose the exact zlib API and write wrapper plugin (something like Node's |
Hi @littledivy Thanks! I think we need a rust crate for zlib. When it is compiled into Wasm, the bytecode will call your host functions. Maybe similar to this https://crates.io/crates/libz-sys An example is the WasmEdge WASI socket crate, which provides Wasm access to the socket-related host functions. |
@juntao Yup understood, I'm using the zlib_sys crate. However, I realised my question wasn't clear, there's a problem with this approach: // WASM bytecode
let z_stream = unsafe { zeroed() };
// Calls into Host function
deflateInit_(&mut z_stream, ...); // Host function
fn deflateInit_(
frame: CallingFrame,
inputs: Vec<WasmValue>,
) -> Result<Vec<WasmValue>, HostFuncError> {
let strm = get_frame_pointer(&frame, inputs[0]);
zlib_sys::deflateInit_(strm, ...);
// ...
} The host's WASI sockets are designed to not involve sharing raw pointers between host and wasm bytecode, hence this is not a problem there. |
How about separating a 64-bit type into two 32-bit types? Concatenate them in the host function side and then do the decode. |
@hydai I'm not sure I understand. How would that look like for the deflateInit example? It's also the layout of the struct |
Hi @littledivy Could you please explain the details of
|
I have setup a minimal but complete zlib deflate and inflate test program and ran it successfully with WasmEdge C SDK with necessary host functions, and after further design decisions we can transition into an official plugin for the same. Test Host & Module Implementation Now, I will discuss what important factors, assumption and design decision needs to be discussed about. My approach / plan:
static void
RegisterHostFunction(const std::string &_function_name,
WasmEdge_HostFunc_t _func_pointer,
std::vector<WasmEdge_ValType> _params_list,
std::vector<WasmEdge_ValType> _return_list, Util *_util,
WasmEdge_ModuleInstanceContext *_module_context) {
WasmEdge_String HostFuncName =
WasmEdge_StringCreateByCString(_function_name.c_str());
WasmEdge_FunctionTypeContext *HostFType =
WasmEdge_FunctionTypeCreate(_params_list.data(), _params_list.size(),
_return_list.data(), _return_list.size());
WasmEdge_FunctionInstanceContext *HostFunc =
WasmEdge_FunctionInstanceCreate(HostFType, _func_pointer, _util, 0);
WasmEdge_ModuleInstanceAddFunction(_module_context, HostFuncName, HostFunc);
WasmEdge_FunctionTypeDelete(HostFType);
WasmEdge_StringDelete(HostFuncName);
}
WasmEdge_Result
WasmEdge_deflateInit_(void *Data,
const WasmEdge_CallingFrameContext *CallFrameCxt,
const WasmEdge_Value *In, WasmEdge_Value *Out) {
uint32_t wasm_z_stream_ptr = (uint32_t)WasmEdge_ValueGetI32(In[0]);
int32_t wasm_level = WasmEdge_ValueGetI32(In[1]);
uint32_t wasm_version_ptr = (uint32_t)WasmEdge_ValueGetI32(In[2]);
int32_t wasm_stream_size = WasmEdge_ValueGetI32(In[3]);
ValidateWasmZStream(CallFrameCxt, wasm_z_stream_ptr, wasm_version_ptr,
wasm_stream_size);
auto stream = GetInitHostZStream(CallFrameCxt, wasm_z_stream_ptr);
const auto z_res =
deflateInit_(stream, wasm_level, ZLIB_VERSION, sizeof(z_stream));
Out[0] = WasmEdge_ValueGenI32(z_res);
reinterpret_cast<Util *>(Data)->stream_map.insert(
{wasm_z_stream_ptr, stream});
return WasmEdge_Result_Success;
}
void ValidateWasmZStream(const WasmEdge_CallingFrameContext *CallFrameCxt,
uint32_t _wasm_z_stream_ptr,
uint32_t _wasm_version_ptr,
int32_t _wasm_stream_size) {
WasmEdge_MemoryInstanceContext *MemCxt =
WasmEdge_CallingFrameGetMemoryInstance(CallFrameCxt, 0);
wasm_z_stream *wasm_stream =
reinterpret_cast<wasm_z_stream *>(WasmEdge_MemoryInstanceGetPointer(
MemCxt, _wasm_z_stream_ptr, sizeof(wasm_z_stream)));
const char *wasm_ZLIB_VERSION = reinterpret_cast<const char *>(
WasmEdge_MemoryInstanceGetPointer(MemCxt, _wasm_version_ptr, 1));
// Check major version of zlib and assert sizeof z_stream == 56
if (wasm_ZLIB_VERSION[0] != ZLIB_VERSION[0])
throw std::runtime_error(std::string("Host(") + wasm_ZLIB_VERSION[0] +
") and Wasm Modue(" + ZLIB_VERSION[0] +
") zlib Major Version does not match!");
if (_wasm_stream_size != 56)
throw std::runtime_error(std::string("WASM sizeof(z_stream) != 56 but ") +
std::to_string(_wasm_stream_size));
}
template <auto &Func>
WasmEdge_Result WasmEdge_algo1(void *Data,
const WasmEdge_CallingFrameContext *CallFrameCxt,
const WasmEdge_Value *In, WasmEdge_Value *Out) {
uint32_t wasm_z_stream_ptr = (uint32_t)WasmEdge_ValueGetI32(In[0]);
int32_t wasm_flush = WasmEdge_ValueGetI32(In[1]);
auto stream_map_it =
reinterpret_cast<Util *>(Data)->stream_map.find(wasm_z_stream_ptr);
if (stream_map_it == reinterpret_cast<Util *>(Out)->stream_map.end())
throw std::runtime_error("ZStream not found in map");
auto stream = stream_map_it->second;
WasmEdge_MemoryInstanceContext *MemCxt =
WasmEdge_CallingFrameGetMemoryInstance(CallFrameCxt, 0);
uint8_t *wasm_mem =
WasmEdge_MemoryInstanceGetPointer(MemCxt, 0, 128 * 1024 * 1024);
wasm_z_stream *wasm_stream =
reinterpret_cast<wasm_z_stream *>(wasm_mem + wasm_z_stream_ptr);
stream->avail_in = wasm_stream->avail_in;
stream->avail_out = wasm_stream->avail_out;
stream->next_in = wasm_mem + wasm_stream->next_in;
stream->next_out = wasm_mem + wasm_stream->next_out;
const auto z_res = Func(stream, wasm_flush);
// now write it to wasm memory
wasm_stream->avail_in = stream->avail_in;
wasm_stream->avail_out = stream->avail_out;
wasm_stream->next_in = stream->next_in - wasm_mem;
wasm_stream->next_out = stream->next_out - wasm_mem;
Out[0] = WasmEdge_ValueGenI32(z_res);
return WasmEdge_Result_Success;
}
template <auto &Func>
WasmEdge_Result
WasmEdge_ZlibEnd(void *Data, const WasmEdge_CallingFrameContext *CallFrameCxt,
const WasmEdge_Value *In, WasmEdge_Value *Out) {
uint32_t wasm_z_stream_ptr = (uint32_t)WasmEdge_ValueGetI32(In[0]);
WasmEdge_MemoryInstanceContext *MemCxt =
WasmEdge_CallingFrameGetMemoryInstance(CallFrameCxt, 0);
wasm_z_stream *wasm_stream =
reinterpret_cast<wasm_z_stream *>(WasmEdge_MemoryInstanceGetPointer(
MemCxt, wasm_z_stream_ptr, sizeof(wasm_z_stream)));
auto stream_map_it =
reinterpret_cast<Util *>(Data)->stream_map.find(wasm_z_stream_ptr);
if (stream_map_it == reinterpret_cast<Util *>(Data)->stream_map.end())
throw std::runtime_error("ZStream not found in map");
const auto z_res = Func(stream_map_it->second);
Out[0] = WasmEdge_ValueGenI32(z_res);
reinterpret_cast<Util *>(Data)->stream_map.erase(stream_map_it);
return WasmEdge_Result_Success;
}
fathomless@vividecstasy:~/repo/wasmedge-zlib/src$ em++ module.cpp -O2 -o module.wasm -sSTANDALONE_WASM -sWARN_ON_UNDEFINED_SYMBOLS=0 -sIMPORTED_MEMORY -sINITIAL_MEMORY=128MB -sALLOW_MEMORY_GROWTH=0 && g++ -O2 host.cpp -o host -lz -lwasmedge && ./host
Test Result : Success Any inputs and further guidance will be much appreciated. |
@hydai @juntao If my approach is right, I will start cleaning the codebase and start with implementing the plugin and create a PR. I am a 3rd year student, so I am planning to participate in GSOC 2023, tomorrow 4th April is the last date of proposal submission. I had been looking into this issue/feature request for well over a week or two. I faced quite a lot of obstacles some of which I mentioned and some of which I left out and will add as constructive criticism to improve the docs and help new newcomers. I was focused on first creating a running Proof Of Concept before trying to talk or initiate a conversation. It would be very kind of you, if you can review it. I am also writing the GSOC proposal draft, which will contain a more detailed overview of myself, the expected project timeline and code explaination. Thank you. |
Since the GSoC 2023 declined this proposal, we will move it to LFX mentorship. |
Okay sure. I will apply on the LFX site with an improved version of the proposal. |
Motivation
The zlib is required for compiling and running many existing C / C++ / Rust apps in Wasm. Most noticeably, it is required for the Python port to Wasm. The VMWare Wasm Labs team is using a zlib port from Singlestore in their Python Wasm runtime.
In WasmEdge, we could support the zlib host functions through our plug-in system. This way, any existing zlib apps can be compiled to Wasm and runs inside WasmEdge. The immediate benefits of this approach are three folds:
Since we are supporting zlib from within WasmEdge, we are not introducing another host app to wrap around WasmEdge. WasmEdge remains the "container" of the app. The zlib apps would be able to seamlessly run in WasmEdge embedded in Docker Desktop, Kubernetes, containerd, OpenShift, and other container tools.
Details
Create a plug-in for zlib host functions
Apply here
https://mentorship.lfx.linuxfoundation.org/project/74cecdf7-e886-4830-8bb0-7814f0d1aa2d
The text was updated successfully, but these errors were encountered: