Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add a sparse-checkout workflow #55286

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add a sparse-checkout workflow #55286

wants to merge 1 commit into from

Conversation

raisedadead
Copy link
Member

@raisedadead raisedadead commented Jun 22, 2024 •
edited
Loading

This PR aims to be the most non-intrusive approach to manage getting the files that one needs for working on the platform.

The strategy is as follows:

  • We have close to 90K files related to curriculum.

  • A fresh clone, of the repo (before this PR lands) has these stats:

    ❯ git clone https://github.com/freeCodeCamp/freeCodeCamp.git
    ❯ git count-objects -vH
  count: 0
  size: 0 bytes
  in-pack: 551871
  packs: 2
  size-pack: 438.05 MiB
  prune-packable: 0
  garbage: 0
  size-garbage: 0 bytes

  • We could instead use --sparse and --filter=blob:none to get only git related information.

      ❯ git clone --filter=blob:none --sparse https://github.com/freeCodeCamp/freeCodeCamp.git
      ❯ git count-objects -vH
  count: 0
  size: 0 bytes
  in-pack: 249102
  packs: 3
  size-pack: 59.60 MiB
  prune-packable: 0
  garbage: 0
  size-garbage: 0 bytes

  • Now to get the actual files in the filesystem we would do something along the lines of

    git sparse-checkout init
git sparse add api
git sparse add client
...
git checkout main

  • However doing the above manually is tedious and error prone. Instead this PR adds a bash script to help. There are detailed commands and steps are in the comments in the file.

    Critically, downloading the needed directories:

      ❯ ./bootstrap.sh --checkout english
   :> Checking out language folders: english...
   :> Initializing sparse-checkout with base configuration...
  remote: Enumerating objects: 229, done.
  remote: Counting objects: 100% (113/113), done.
  remote: Compressing objects: 100% (48/48), done.
  remote: Total 229 (delta 87), reused 65 (delta 65), pack-reused 116
  Receiving objects: 100% (229/229), 170.02 KiB | 1.16 MiB/s, done.
  Resolving deltas: 100% (133/133), done.
   :> Sparse-checkout initialized.
  remote: Enumerating objects: 7288, done.
  remote: Counting objects: 100% (5740/5740), done.
  remote: Compressing objects: 100% (4784/4784), done.
  remote: Total 7288 (delta 1310), reused 956 (delta 956), pack-reused 1548
  Receiving objects: 100% (7288/7288), 6.21 MiB | 7.00 MiB/s, done.
  Resolving deltas: 100% (1853/1853), done.
  Updating files: 100% (7288/7288), done.
   :> Checkout of specified languages completed.

    reduces the repos size on the local machine:

      ❯ git count-objects -vH
  count: 0
  size: 0 bytes
  in-pack: 256619
  packs: 5
  size-pack: 66.18 MiB
  prune-packable: 0
  garbage: 0
  size-garbage: 0 bytes

    and the count of files, for example:

      ❯ cd curriculum/challenges/english
  ❯ find . -type f | wc -l
      7286

Note

This PR aims to have minimal impact.

  • This PR should will not affect any existing flows that we have like CI & CD.
  • The idea is to add instructions for using this script and workflows to our documentation. - Contributors can simply opt-in to follow this workflow to grab only the needed parts of the repository.
  • Follow-up PRs may optimize some automations like CI, etc. but that is left to discretion of the dev-team.

@raisedadead raisedadead force-pushed the feat/sparse-checkout branch 2 times, most recently from f8da6e4 to 8a2be7b Compare June 22, 2024 09:47
@raisedadead raisedadead marked this pull request as ready for review June 22, 2024 10:17
@raisedadead raisedadead requested a review from a team as a code owner June 22, 2024 10:17
@raisedadead
Copy link
Member Author

This is more like a demo PR.

I will have to adjust some nuances around files and their location. Example there is a .markdownlint.yaml buried in the challenges folder for some reason and so on.

@raisedadead
Copy link
Member Author

raisedadead commented Jun 22, 2024 •
edited
Loading

For reviewers:

Follow the steps in the bootstrap.sh file, replacing my fork https://github.com/raisedadead/freeCodeCamp.git for testing.

Ideally you want to clone from scratch in an isolated location to avoid overwriting any existing setups you have.

moT01
moT01 reviewed Jun 25, 2024
View reviewed changes
Copy link
Member

@moT01 moT01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty neat. I had to checkout the _meta folder to be able to build - git sparse-checkout add curriculum/challenges/_meta. Perhaps that could be added to the script.

ojeytonwilliams
ojeytonwilliams reviewed Jun 27, 2024
View reviewed changes
Copy link
Contributor

@ojeytonwilliams ojeytonwilliams left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we do something like nvm and ask contributors to curl the script? That way the user has to copy something like

curl https://raw.githubusercontent.com/freeCodeCamp/freeCodeCamp/main/bootstrap.sh | bash

for the initial setup. If we fold

git clone --filter=blob:none --sparse https://github.com/freeCodeCamp/freeCodeCamp.git
cd freeCodeCamp
git sparse-checkout init --no-cone

into the script and default to checking out english, then the whole process is nice and simple.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ojeytonwilliams ojeytonwilliams ojeytonwilliams left review comments

@moT01 moT01 moT01 left review comments

@ShaunSHamilton ShaunSHamilton ShaunSHamilton left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@raisedadead @ojeytonwilliams @moT01 @ShaunSHamilton