Batch uploading assets to the GEE

More convenient than uploading your geospatial files one by one.

This is a small tutorial that explains how to upload your own geospatial data to the Google Earth Engine. I assume you have installed the python API as explained here. The upload tool comes with the GEE asset manager which you can download from github.

Step 1: Make sure Git is install on your machine.

sudo apt-get install git

Step 2: Clone the GEE asset manager to your machine

git clone https://github.com/tracek/gee_asset_manager.git

Step 3: Go into the gee_asset_manager folder and install the application

cd gee_asset_manager

sudo python setup.py install

The requirements.txt shows the follow dependencies:

earthengine_api >= 0.1.87
requests >= 2.10.0
retrying >= 1.3.3
bs4 >= 4.5.1
requests_toolbelt >= 0.7.0

Step 4: Install the dependencies.

For example

sudo pip install bs4

Step 5: Test the application by running

geebam -h

I got this error (Linux mint):

File "build/bdist.linux-x86_64/egg/retrying.py", line 47, in wrap
AttributeError: 'module' object has no attribute 'wraps'

So downgraded retrying to make the tool work

sudo -H pip install retrying==1.2.3

Step 6: The line below gives you all options to upload all files from a directory:

geebam upload -h
usage: geebam upload [-h] -u USER -d DIRECTORY [-m METADATA] [-c COLLECTION]
                     [-p PATH] [--large] [--nodata NODATA]

optional arguments:
  -h, --help            show this help message and exit

Required named arguments.:
  -u USER, --user USER  Google account name (gmail address).
  -d DIRECTORY, --directory DIRECTORY
                        Path to the directory with images.

Optional named arguments:
  -m METADATA, --metadata METADATA
                        Path to CSV with metadata.
  -c COLLECTION, --collection COLLECTION
                        Name with path of the collection to create. If not
                        provided, directory name will be used. It assumes the
                        upload goes to the user folder. Need upload to a
                        shared directory? Use --path instead. They are mutuall
                        exclusive
  -p PATH, --path PATH  Absolute upload path. It does not take any assumptions
                        about user folder, so it can be used to upload to a
                        shared folder. Mutually exclusive with --collection.
  --large               (Advanced) Use multipart upload. Might help if upload
                        of large files is failing on some systems. Might cause
                        other issues.
  --nodata NODATA       The value to burn into the raster as NoData (missing
                        data)

Example:

geebam upload -u myemailadress@gmail.com -d /path/to/my/files/ -c mycollectionName --nodata -9999

2016-11-23 09:23:08,175 :: oauth2client.client :: INFO :: Attempting refresh to obtain initial access_token
2016-11-23 09:23:08,175 :: oauth2client.client :: INFO :: Refreshing access_token
Password:
2016-11-23 09:23:16,430 :: urllib3.connectionpool :: INFO :: Starting new HTTPS connection (1): accounts.google.com
2016-11-23 09:23:17,488 :: urllib3.connectionpool :: INFO :: Starting new HTTPS connection (1): accounts.google.com.vn
2016-11-23 09:23:17,912 :: urllib3.connectionpool :: INFO :: Starting new HTTPS connection (1): myaccount.google.com
2016-11-23 09:23:23,871 :: root :: INFO :: New collection users/myname/mycollectionName created
2016-11-23 09:23:27,535 :: root :: INFO :: Processing image 1 out of 132: /path/to/my/files/file1.tif
2016-11-23 09:23:27,536 :: urllib3.connectionpool :: INFO :: Starting new HTTPS connection (1): ee-api.appspot.com
2016-11-23 09:23:28,063 :: urllib3.connectionpool :: INFO :: Starting new HTTPS connection (1): www.google.com
2016-11-23 09:23:28,419 :: urllib3.connectionpool :: INFO :: Starting new HTTPS connection (1): appengine.google.com
2016-11-23 09:23:49,524 :: root :: INFO :: Processing image 2 out of 132: /path/to/my/files/file2.tif
...

Note on parsing metadata:

### Parsing metadata
By metadata we understand here the properties associated with each image.

Thanks to these, GEE user can easily filter collection based on specified criteria. The file with metadata should be organised as follows:

| filename (without extension) | property1 header | property2 header |
|——————————|——————|——————|
| file1                        | value1           | value2           |
| file2                        | value3           | value4           |
 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s