beam-ml

ML sample notebooks

Starting with the Apache Beam SDK version 2.40, users have access to a RunInference transform.

This transform allows you to make predictions and inference on data with machine learning (ML) models. The model handler abstracts the user from the configuration needed for specific frameworks, such as Tensorflow, PyTorch, and others. For a full list of supported frameworks, see the About Beam ML page.

Use the notebooks

These notebooks illustrate ways to use Apache Beam's RunInference transforms, as well as different use cases for ModelHandler implementations. Beam comes with multiple ModelHandler implementations.

Load the notebooks

To get started quickly with notebooks, use Colab.
In Colab, open the notebook from GitHub using the notebook URL, for example:

https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

To run most notebooks, you need to change the Google Cloud project and bucket to your project and bucket.

Notebooks

This section contains the following example notebooks.

Data processing

Data enrichment

Prediction and inference with pretrained models

Custom inference

Machine Learning Use Cases

Automatic Model Refresh

Update your model in production

Multi-model pipelines

Model Evaluation

Use TFMA to evaluate and compare model performance

Name		Name	Last commit message	Last commit date
parent directory ..
data_preprocessing		data_preprocessing
README.md		README.md
automatic_model_refresh.ipynb		automatic_model_refresh.ipynb
bigtable_enrichment_transform.ipynb		bigtable_enrichment_transform.ipynb
custom_remote_inference.ipynb		custom_remote_inference.ipynb
dataframe_api_preprocessing.ipynb		dataframe_api_preprocessing.ipynb
image_processing_tensorflow.ipynb		image_processing_tensorflow.ipynb
mltransform_basic.ipynb		mltransform_basic.ipynb
nlp_tensorflow_streaming.ipynb		nlp_tensorflow_streaming.ipynb
per_key_models.ipynb		per_key_models.ipynb
run_custom_inference.ipynb		run_custom_inference.ipynb
run_inference_gemma.ipynb		run_inference_gemma.ipynb
run_inference_generative_ai.ipynb		run_inference_generative_ai.ipynb
run_inference_huggingface.ipynb		run_inference_huggingface.ipynb
run_inference_multi_model.ipynb		run_inference_multi_model.ipynb
run_inference_pytorch.ipynb		run_inference_pytorch.ipynb
run_inference_pytorch_tensorflow_sklearn.ipynb		run_inference_pytorch_tensorflow_sklearn.ipynb
run_inference_sklearn.ipynb		run_inference_sklearn.ipynb
run_inference_tensorflow.ipynb		run_inference_tensorflow.ipynb
run_inference_tensorflow_with_tfx.ipynb		run_inference_tensorflow_with_tfx.ipynb
run_inference_vertex_ai.ipynb		run_inference_vertex_ai.ipynb
run_inference_windowing.ipynb		run_inference_windowing.ipynb
run_inference_with_tensorflow_hub.ipynb		run_inference_with_tensorflow_hub.ipynb
run_inference_xgboost.ipynb		run_inference_xgboost.ipynb
speech_emotion_tensorflow.ipynb		speech_emotion_tensorflow.ipynb
tfma_beam.ipynb		tfma_beam.ipynb
vertex_ai_feature_store_enrichment.ipynb		vertex_ai_feature_store_enrichment.ipynb

Files

beam-ml

Directory actions

More options

Directory actions

More options

Latest commit

History

beam-ml

Folders and files

parent directory

ML sample notebooks

Use the notebooks

Load the notebooks

Notebooks

Data processing

Data enrichment

Prediction and inference with pretrained models

Custom inference

Machine Learning Use Cases

Automatic Model Refresh

Multi-model pipelines

Model Evaluation