Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release pipeline #2620

Merged
merged 11 commits into from
Jul 2, 2023
224 changes: 224 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,224 @@
name: CI
on:
push:
branches:
- master
tags:
- "v*"
pull_request:
owenthereal marked this conversation as resolved.
Show resolved Hide resolved
jobs:
linux:
strategy:
fail-fast: false
matrix:
compiler: [gcc, clang]
image: [ubuntu-20.04, ubuntu-22.04]
include:
- compiler: gcc
image: ubuntu-20.04
configure_flag: ''
- compiler: gcc
image: ubuntu-22.04
configure_flag: --enable-static --enable-all-static
- compiler: clang
image: ubuntu-20.04
configure_flag: ''
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh so only ubuntu-20.04 and clang fails using static? other ubuntus are fine? 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah no also gcc, misread above

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can reproduce with ubuntu docker image:

$ docker run -ti -v $PWD:$PWD -w $PWD ubuntu:20.04
... <configure, build etc>...
root@83a347734b03:/Users/wader/src/jq# gdb --args ./jq
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
....
(gdb) r
Starting program: /Users/wader/src/jq/jq
warning: Error disabling address space randomization: Operation not permitted

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00000000004ec78c in __register_frame_info_bases.part.0 ()
#2  0x0000000000401ec1 in frame_dummy ()
#3  0x0000000000000001 in ?? ()
#4  0x00000000004edd3c in __libc_csu_init ()
#5  0x00000000004ed48e in __libc_start_main ()
#6  0x0000000000401dde in _start ()
(gdb)

Dugg a bit, i suspect it might be something like https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95989 some version of glibc pthreads don't like static linking. Usually you can make it work by providing a lots of link flags to tell it to resolve weak symbols and whatnot, but it's a mess.

Also got me thinking: do the jq cli need threads? looking at the code it seems to be about making libjq possible to use concurrently? but it's probably no easy task to make it optional

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also see that jq do call some libc functions like getpwuid that i suspect might trigger NSS usage that i think might trigger dynamic loading and might crash hmm

In my experience if one really want static binaries linking with musl is the "safest"... but that might have some other issues.... but again in my experience with modern musl versions there are few issues

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw alpine, that uses musl, already package jq and i haven't had any issues with it and looking at their bug tracker for jq issues https://gitlab.alpinelinux.org/alpine/aports/-/issues/?search=jq&sort=created_date&state=closed&first_page_size=20 i don't see anything that looks like reported random strange crashes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a test using CC=musl-gcc (needs package musl musl-dev musl-tools) and it seems to work fine and make check passes

root@83a347734b03:/Users/wader/src/jq# file jq
jq: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
root@83a347734b03:/Users/wader/src/jq# ls -lh jq
-rwxr-xr-x 1 root root 1.8M Jul  2 14:04 jq
root@83a347734b03:/Users/wader/src/jq# strip jq
root@83a347734b03:/Users/wader/src/jq# file jq
jq: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
root@83a347734b03:/Users/wader/src/jq# ls -lh jq
-rwxr-xr-x 1 root root 1.1M Jul  2 14:04 jq
root@83a347734b03:/Users/wader/src/jq# ./jq --version
jq-1.6-215-gf88c4e5-dirty
root@83a347734b03:/Users/wader/src/jq# ./jq -n 1+2
3

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find! I'm going to merge this PR so that you have something to base off.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 sounds good

Yeap i think alpine for static binaries could be a good alternative, then i think we could truely have one binary that works on any linux kernel. @itchyny seemed a bit concerned about it but maybe can be convinced :) the alternative think is find a good distro and glibc version that produce known "good" static binaries, i guess that is was the previous releases did?

Out of curiosity had a look at the current jq 1.6 linux64 binary:

$ ldd jq-linux64
	not a dynamic executable
$ file jq-linux64
jq-linux64: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=c159e8a134ee222019dfbac0a1af4bc94b302eb0, with debug_info, not stripped

but couldn't find what libc variant/version is it, glibc something?

also notice that jq 1.6 had 32bit builds

- compiler: clang
image: ubuntu-22.04
configure_flag: --enable-static --enable-all-static
runs-on: ${{ matrix.image }}
env:
CC: ${{ matrix.compiler }}
SUFFIX: linux-${{ matrix.image}}-${{ matrix.compiler }}
steps:
- name: Clone repository
uses: actions/checkout@v3
with:
submodules: true
- name: Install packages
run: |
sudo apt-get update -qq
sudo apt-get install -y \
automake \
autoconf \
bison \
flex \
gdb \
python3
- name: Build
run: |
autoreconf -fi
./configure --disable-dependency-tracking \
--disable-silent-rules \
--disable-maintainer-mode \
--disable-valgrind \
--with-oniguruma=builtin \
${{ matrix.configure_flag }} \
YACC="$(which bison) -y"
make
- name: Test
run: |
make check
- name: Upload Test Logs
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: test-logs-${{ env.SUFFIX }}
retention-days: 7
path: |
test-suite.log
tests/*.log
- name: Upload Artifacts
uses: actions/upload-artifact@v3
with:
name: jq-${{ env.SUFFIX }}
if-no-files-found: error
retention-days: 7
path: |
jq
macos:
strategy:
fail-fast: false
matrix:
compiler: [gcc, clang]
image: [macos-11, macos-12, macos-13]
runs-on: ${{ matrix.image }}
env:
CC: ${{ matrix.compiler }}
SUFFIX: macos-${{ matrix.image}}-${{ matrix.compiler }}
steps:
- name: Clone repository
uses: actions/checkout@v3
with:
submodules: true
- name: Install packages
run: |
# brew update sometimes fails with "Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask failed!"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed this on some CI builds but for some reason (randomly?) only on macOS 12

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, that's what I saw. Perhaps the GH Actions macos-12 image has some issue? The update-reset fallback helps with that, as you see.

brew update || brew update-reset
brew install \
autoconf \
automake \
libtool \
flex \
bison
- name: Build
run: |
autoreconf -fi
./configure --disable-dependency-tracking \
--disable-silent-rules \
--disable-maintainer-mode \
--disable-valgrind \
--with-oniguruma=builtin \
--enable-static \
--enable-all-static \
YACC="$(brew --prefix)/opt/bison/bin/bison -y"
make
- name: Test
run: |
make check
- name: Upload Test Logs
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: test-logs-${{ env.SUFFIX }}
retention-days: 7
path: |
test-suite.log
tests/*.log
- name: Upload Artifacts
uses: actions/upload-artifact@v3
with:
name: jq-${{ env.SUFFIX }}
if-no-files-found: error
retention-days: 7
path: |
jq
windows:
strategy:
fail-fast: false
matrix:
compiler: [gcc]
image: [windows-2019, windows-2022]
runs-on: ${{ matrix.image }}
env:
CC: ${{ matrix.compiler }}
SUFFIX: windows-${{ matrix.image}}-${{ matrix.compiler }}
steps:
- name: Clone repository
uses: actions/checkout@v3
with:
submodules: true
- uses: msys2/setup-msys2@v2
with:
update: true
install: >-
base-devel
git
clang
autoconf
automake
libtool
bison
flex
- name: Build
shell: msys2 {0}
run: |
autoreconf -fi
./configure --disable-dependency-tracking \
--disable-silent-rules \
--disable-maintainer-mode \
--disable-valgrind \
--with-oniguruma=builtin \
--disable-shared \
--enable-static \
--enable-all-static \
YACC="$(which bison) -y"
make
- name: Test
shell: msys2 {0}
run: |
make check
- name: Upload Test Logs
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
name: test-logs-${{ env.SUFFIX }}
retention-days: 7
path: |
test-suite.log
tests/*.log
- name: Upload Artifacts
uses: actions/upload-artifact@v3
with:
name: jq-${{ env.SUFFIX }}
if-no-files-found: error
retention-days: 7
path: |
jq.exe
release:
runs-on: ubuntu-latest
permissions:
contents: write
needs: [linux, macos, windows]
if: startsWith(github.event.ref, 'refs/tags/v')
steps:
- name: Clone repository
uses: actions/checkout@v3
with:
submodules: true
- name: Merge built artifacts
uses: actions/download-artifact@v3
with:
path: artifacts
- name: Upload release
env:
TAG_NAME: ${{ github.ref_name }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
mkdir release
cp artifacts/jq-linux-ubuntu-22.04-gcc/jq release/jq-linux-amd64
cp artifacts/jq-macos-macos-13-gcc/jq release/jq-macos-amd64
cp artifacts/jq-windows-windows-2022-gcc/jq.exe release/jq-windows-amd64.exe

gh release create $TAG_NAME --draft --title "jq ${TAG_NAME#v}" --generate-notes
gh release upload $TAG_NAME --clobber release/jq-*
70 changes: 0 additions & 70 deletions .github/workflows/linux.yml

This file was deleted.

62 changes: 0 additions & 62 deletions .github/workflows/macos.yml

This file was deleted.