Pixels to the people!
By Nicholas Clinton, Earth Engine Developer Relations
Earth Engine is Google’s Cloud-based service for geospatial processing. It has a rich API, loaded with algorithms for image processing, spatial analysis, visualization and machine learning. For years, scientists, researchers and professionals have been implementing new and exciting things in Earth Engine. When it’s been time to harvest the fruits of their labor, many developers have turned to Export, getDownloadUrl or maybe the REST API to get image data out of Earth Engine. To make this even easier, we are pleased to announce the release of the Data Extraction API comprising two new client methods: getPixels and computePixels. Using these methods, you can get image data in a variety of formats from an Earth Engine client integrated with your Python or Javascript workflows, to enhance just about anything with spatial awareness.
Although you can still use Export for getting data out of Earth Engine, try the new methods and you might find that you prefer them. For very small patches of data, the backend overhead associated with starting an Export means that you probably need to wait longer for results. This makes getting the pixels with getPixels or computePixels preferable since you can get your data faster. For very large volumes of data, Export may result in scaling issues (e.g. “Computation timed out” or the dreaded “User memory limit exceeded”). Making many smaller requests can be preferable, depending on the shape of the problem. For example, if you are generating a dataset of image patches (or “chips”), for training a machine learning model, you’re in luck! That’s the kind of problem these new methods were designed to help tackle. So if your data extraction job can be run in many, small independent pieces, then getPixels and/or computePixels might be just the thing.
When you make a request, you choose the scale, projection, region, visualization and format of the response. In the following example, image is a computed ee.Image for example a cloud-free composite, and grid is where you specify the region, scale and projection (Python):
request = {
'expression': image,
'fileFormat': 'PNG',
'bandIds': [...],
'grid': {...},
'visualizationOptions': {...},
}
image_png = ee.data.computePixels(request)These methods work by taking many bite-sized pieces out of the problem. To get more throughput, you can make more requests in parallel. To facilitate this, Earth Engine provides the high-volume endpoint, which is set up to handle a lot of requests, but at higher latency. You get 40 concurrent requests by default, but those can be increased for both commercial and non-commercial users. Although it’s a bit more work to get everything up and running, you keep visibility and control over the entire process. For example, you can retry requests automatically and monitor errors, number of requests, latency, response codes, etc. You can use your library of choice to parallelize, for example Python’s built-in concurrency library or Apache Beam.
We’ve got a boatload of docs, examples and demos to help you get started. Check out the computePixels (notebook) or getPixels (link) examples in the Earth Engine Github repository. For scaling large workloads with Apache Beam and Google Dataflow, see the People, Planet and AI land cover classification demo. For nuts-and-bolts, there are client library reference docs (computePixels, getPixels) and REST reference docs (computePixels, getPixels) with details on request parameters.
We’re looking forward to hearing about all the impactful workflows people build using these new methods. Pixels to the people!
