This repository is designed to contain the following implementations towards TensorFlow Extended(TFX) and Weights and Biases(W&B) integrations

• Experiment Tracking: TFX and KerasTuner can be combined naturally, so different sets of hyper-parameter tunings from KerasTuner could be logged in W&B.

• Model Registry: Trained & blessed (qualified) models from TFX pipeline could be pushed to the W&B Model Registry. Optionally push prototype application on Hugging Face Space Hub with the right model version.

In order to keep the implementation simple, this repository uses only the following standard TFX components. However, note that you could easily integrate more of TFX components.

• ImportExampleGen: to bring prepared dataset into the TFX pipeline.
• Trainer: to train the model with the best found hyper-parameters from Tuner
• Tuner: to search the best hyper-parameters. Experiment Tracking feature of W&B will be integrated into this component.

Additionally, this project will present a custom TFX component.

• WandBPusher: to push a trained model from Trainer to the Model Registry in W&B. This component could be integrated with Evaluator as well.

Again, in order to simplify and focus on the purpose of this project, we are going to use a simple model and dataset as the following.

• ViT from HuggingFace's transformers
• beans dataset which will be prepared as TFRecord format from HuggingFace datasets library.

• write a notebook to prepare TFRecord dataset
• write a notebook to run the basic chain of TFX components
• extend the notebook to add Experiment Tracking feature
• extend the notebook to leverage KerasTuner's HyperModel and Tuner
• port the notebook to standalone TFX pipeline w/o notebook
• integrate GCP's Vertex AI with GitHub Action
• write a notebook for the new custom component (WandBPusher)
• write a notebook to integrate WandBPusher component into the existing TFX pipeline
• port WandBPusher into the existing standadone TFX pipeline
• modify WandBPusher to push prototype application to Hugging Face Space Hub