## 1. Project Overview & Quickstart (samapriya/submit) # submit Open-source repository samapriya/submit ### Repository Details - **Repository:** [samapriya/submit](https://github.com/samapriya/submit) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (samapriya/samapriya.github.io) ## File: docs/projects/arcmap_addon.md # ArcMap Addons [](https://doi.org/10.5281/zenodo.1009210) While working on different projects over sometime I had to create and use some tools which acted as valuable addons to arcmap toolbox. I am sharing a few and I will keep on updating these as I keep working on new models.This toolbox was created on ArcMap 10.4 and is not backward compatible. The toolbox can be downloaded and added to existing toolbox to create additional functionalities. ## Table of contents * [Installation](#installation) * [Usage examples](#usage-examples) * [Batch Raster to Point](#batch-raster-to-point) * [Batch Table to CSV](#batch-table-to-csv) * [Email Notification](#email-notification) * [Iterative Clip](#iterative-clip) * [MultiBand to Single Images](#multiband-to-single-images) * [Raster Properties as CSV](#raster-properties-to-csv) * [Raster Copy Iterative](#raster-copy-iterative) * [Feature Select and Copy](#feature-select-and-copy) * [Select and Calculate Field](#select-and-calculate-field) * [Credits](#credits) ## Installation We assume that the user already has a copy of ArcMap >=10.4. The toolbox can be downloaded along with adjoining script and added onto existing toolbox. You can keep this toolbox as part of the default toolboxes by clicking Save Settings>Save as Default ## Usage examples Usage examples will vary and only continue to grow as new tools are added to the toolbox. ### Batch Raster to Point This tool allows you to convert all Raster datasets in a folder into a point with the value field converted to GRID code. The tool supports all raster formats supported by ArcMap and renames the files automatically to the source file name. The raster pixel is converted to a centroid value. ### Batch Table to CSV This tool allows you to batch convert all table files(in this case it looks for '.dbf' files) and converts the fields into csv columns. This is an effective way when you want to handle a large number of dbf files to be imported into other softwares and or processing chains. ### Email Notification I found this to be one of the most useful tools when running a long process and not having to wait for your results. The tool makes use of the smtp library and allows the user to set up an email service for an email to be sent out after a process has been completed. ### Iterative Clip This tool does exactl what the name suggests it uses an iterator tool to clip a raster to different boundaries using shapefiles. Requires a single raster file and a folder with multiple polygons for clips. If using on a private machine the Key is saved as a csv file for all future runs of the tool. ### MultiBand to Single Images The aoijson tab within the toolset allows you to create filters and structure your existing input file to that which can be used with Planet's API. The tool requires inputs with start and end date, along with cloud cover. You can choose from multiple input files types such as KML, Zipped Shapefile, GeoJSON, WKT or even Landsat Tiles based on PathRow numbers. The geo option asks you to select existing files which will be converted into formatted JSON file called aoi.json. If using WRS as an option just type in the 6 digit PathRow combination and it will create a json file for you. ### Raster Properties as CSV The activatepl tab allows the users to either check or activate planet assets, in this case only PSOrthoTile and REOrthoTile are supported because I was only interested in these two asset types for my work but can be easily extended to other asset types. This tool makes use of an existing json file sturctured for use within Planet API or the aoi.json file created earlier ### Raster Copy Iterative Having metadata helps in organising your asstets, but is not mandatory - you can skip it. The downloadpl tab allows the users to download assets. The platform can download Asset or Asset_XML which is the metadata file to desired folders.One again I was only interested in these two asset types(PSOrthoTile and REOrthoTile) for my work but can be easily extended to other asset types. ### Feature Select and Copy These tools allow the users to iteratively select features using a given SQL expression and then extract that as a new feature class. This can be done iteratively for all feature classes in the folder. ### Select and Calculate Field This tool allows you to select a field and then recalculate or calculate it based on expressions. The user has the option of simply filling empty fields based on selection of a different field. Meaning I can calculate a field B based on a selection of a seperate field A if I want. # Credits I would like to thank all those who bring me interesting problems to solve, there is little that is needed to keep one inspired. ### Changelog #### v0.2 - Batch Table to CSV - Batch Raster to Points --- ## File: docs/projects/arcticdem_download.md # ArcticDEM Batch Download & Processing Tools [](https://doi.org/10.5281/zenodo.841299) [](https://www.pgc.umn.edu/) ArcticDEM project was a joint project supported by both the National Geospatial-Intelligence Agency(NGA) and the National Science Foundation(NSF) with the idea of creating a high resolution and high quality digital surface model(DSM). The product is distributed free of cost as time-dependent DEM strips and is hosted as https links that a user can use to download each strip. As per their policy *The seamless terrain mosaic can be distributed without restriction.* The created product is a 2-by-2 meter elevation cells over an over of over 20 million square kilometers and uses digital globe stereo imagery to create these high resolution DSM. The method used for the 2m derivate is Surface Extraction with TIN-based Search-space Minimization(SETSM). Based on their acknowledgements requests you can use *Acknowledging PGC services(including data access)* * Geospatial support for this work provided by the Polar Geospatial Center under NSF OPP awards 1043681 & 1559691. *Acknowledging DEMS created from the ArcticDEM project* * DEMs provided by the Polar Geospatial Center under NSF OPP awards 1043681, 1559691 and 1542736. You can find details on the background, scope and methods among other details [here](https://www.pgc.umn.edu/guides/arcticdem/introduction-to-arcticdem/?print=pdf) A detailed acknowledgement link can be found [here](https://www.pgc.umn.edu/guides/user-services/acknowledgement-policy/) With this in mind and with the potential applications of using these toolsets there was a need to batch download the DEM files for your area of interest and to be able to extract, clean and process metadata. In all fairness this tool has a motive of extending this as an input to Google Earth Engine and hence the last tool which is the metadata parser is designed to create a metadata manifest in a csv file which GEE can understand and associate during asset upload. ## Table of contents * [Installation](#installation) * [Getting started](#getting-started) * [Usage examples](#usage-examples) * [Subset to AOI](#subset-to-aoi) * [Estimate Download Size](#estimate-download-size) * [Download DEM](#download-dem) * [Extract DEM](#extract-dem) * [Metadata Parsing for GEE](#metadata-parsing-for-gee) ## Installation We assume that you have installed the requirements files to install all the necessary packages and libraries required to use this tool. To install packages from the requirements.txt file you can simply use ```pip install -r requirements.txt```. Remember that installation is an optional step and you can run this program by simply browsing to the pgcdem-cli file and typing ```python arcticdem.py```. One of the only other requirement for this tool is the Master Shapefile for all DEM footprints(make sure to use the most updated version which can be found [here](https://www.pgc.umn.edu/data/arcticdem/)) **This toolbox also uses some functionality from GDAL** For installing GDAL in Ubuntu ``` sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update sudo apt-get install gdal-bin ``` For Windows I found this [guide](https://sandbox.idre.ucla.edu/sandbox/tutorials/installing-gdal-for-windows) from UCLA To install **ArcticDEM Batch Download & Processing Tools:** ``` git clone https://github.com/samapriya/ArcticDEM-Batch-Pipeline.git cd ArcticDEM-Batch-Pipeline && pip install . ``` This release also contains a windows installer which bypasses the need for you to have admin permission, it does however require you to have python in the system path meaning when you open up command prompt you should be able to type python and start it within the command prompt window. Post installation using the installer you can just call ppipe using the command prompt similar to calling python. Give it a go post installation type ``` arcticdem -h ``` The advantage of having it installed is being able to execute ppipe as any command line tool. I recommend installation within virtual environment. As a extra addon feature I have wrapped the cli into a Graphical User Interface(GUI) so some people will find it easier to use. To install run ``` python setup.py develop or python setup.py install In a linux distribution sudo python setup.py develop or sudo python setup.py install ``` ## Getting started To obtain help for a specific functionality, simply call it with _help_ switch, e.g.: `arcticdem demextract -h`. If you didn't install arcticdem, then you can run it just by going to _arcticdem-cli_ directory and running `python arcticdem.py [arguments go here]` As usual, to print help `arcticdem -h`: ``` ArcticDEM Batch Download & Processing Tools positional arguments: { ,demaoi,demsize,demdownload,demextract,demmeta} --------------------------------------- -----Choose from ArcticDEM-Download Tools Below----- --------------------------------------- demaoi Allows user to subset Master ArcticDEM to their AOI demsize Allows users to estimate total download size and space left in your destination folder demdownload Allows users to batch download ArcticDEM Strips using aoi shapefile demextract Allows users to extract both image and metadata files from the zipped tar files demmeta Tool to process metadata files into CSV for all strips[For use with Google Earth Engine] optional arguments: -h, --help show this help message and exit ``` ## Subset to AOI The script clips the master ArcticDEM strip file to a smaller subset usgin an area of interest shapefile. This allows to get the strip DEM(s) for only the area of interest and to use that to download these files. The subset allows the user to limit the total amount of strips to be downloaded and processed. The script will create a new shapefile with the clipped subset of the master ArcticDEM strip file. **Make sure you reproject your aoi shapefile to the same projection as the ArcticDEM strip file** ``` usage: arcticdem.py demaoi [-h] [--source SOURCE] [--target TARGET] [--output OUTPUT] optional arguments: -h, --help show this help message and exit --source SOURCE Choose location of your AOI shapefile --target TARGET Choose the location of the master ArcticDEM strip file --output OUTPUT Choose the location of the output shapefile based on your AOI ``` An example setup would be ``` arcticdem demaoi --source "C:\users\aoi.shp" --target "C:\users\masterdem.shp" --output "C:\users\master_aoi.shp" ``` ### Estimate Download Size One of the most common things you want to do is to know if the destination where you want to save these files has enough space before you begin the download process. This script allows you to query the total download size for your area of interest and the destination drive where you want to save the compressed files. It also recursively updates overall download size on the screen and print total size needed along with total download size in GB. ``` usage: arcticdem.py demsize [-h] [--infile INFILE] [--path PATH] optional arguments: -h, --help show this help message and exit --infile INFILE Choose the clipped aoi file you clipped from demaoi tool[This is the subset of the master ArcticDEM Strip] --path PATH Choose the destination folder where you want your dem files to be saved[This checks available disk space] ``` An example setup would be ``` arcticdem demsize --infile "C:\users\master_aoi.shp" --path "C:\users\ArcticDEM" ``` The program might misbehave if the area of interest is extremely large or be sluggish in nature. ## Download DEM What we were mainly interested after we know that we have enough space to download is to download the files. The script used a multi part download library to download the files quicker and in a more managed style to the destination given by the user. ``` usage: arcticdem.py demdownload [-h] [--subset SUBSET] [--desination DESINATION] optional arguments: -h, --help show this help message and exit --subset SUBSET Choose the location of the output shapefile based on your AOI[You got this from demaoi tool] --desination DESINATION Choose the destination where you want to download your files ``` An example setup would be ``` arcticdem demdownload --subset "C:\users\master_aoi.shp" --destination "C:\users\ArcticDEM" ``` ## Extract DEM This downloaded DEM files are tar or tar gz files and need to be extracted. The important thing to note is that the script retains the dem file, the matchtag file and the metadata text files in separate directories within the destination directory. ``` usage: arcticdem.py demextract [-h] [--folder FOLDER] [--destination DESTINATION] [--action ACTION] optional arguments: -h, --help show this help message and exit --folder FOLDER Choose the download file where you downloaded your tar zipped files --destination DESTINATION Choose the destination folder where you want your images and metadata files to be extracted --action ACTION Choose if you want your zipped files to be deleted post extraction "yes"|"no" ``` An example setup would be ``` arcticdem demdextract --folder "C:\users\ArcticDEM" --destination "C:\users\ArcticDEM\Extract" --action "yes" ``` ## Metadata Parsing for GEE One of my key interest in working on the ArcticDEM parsing was to be able to upload this to Google Earth Engine(GEE). The metadata parser script allows you to parse all metadata into a combined csv file to be used to upload images to GEE. The manifest and upload tools are separate from this package tool and included in my gee_asset_manager_addon. ``` usage: arcticdem.py demmeta [-h] [--folder FOLDER] [--metadata METADATA] [--error ERROR] optional arguments: -h, --help show this help message and exit --folder FOLDER Choose where you unzipped and extracted your DEM and metadata files --metadata METADATA Choose a path to the metadata file "example: users/desktop/metadata.csv" --error ERROR Choose a path to the errorlog file "example: users/desktop/errorlog.csv" ``` An example setup would be ``` arcticdem demmeta --folder "C:\users\ArcticDEM\Extract\pgcmeta" --metadata "C:\users\arcticdem_metadata.csv" --error "C:\users\arcticdem_errorlog.csv" ``` # Changelog ## [0.1.1] - 2017-08-12 ### Added - Can now handle ogr input and includes instruction to project aoi in same projection as DEM strip. - Added the capability of skipping over already downloaded files and continues with left over downloads. - Completed recompiling executable to include changes. --- ## File: docs/projects/clip_ship_planet_cli.md # Clip Ship Planet CLI addon [](https://doi.org/10.5281/zenodo.1290193) [](https://www.planet.com/products/education-and-research/) [](https://saythanks.io/to/samapriya) _**Note: The [Clips API has been deprecated](https://developers.planet.com/changelog) - and will no longer be supporting the standalone Clip and Ship service. Deprecation means that there is intention to remove the service at some point; it DOES NOT mean end-of-life for the service, yet. (Deprecate and end-of-life are two distinct terms.) There should be a plan to replace Clips API with new pre-processing functionality, but no new announcement or timetable for the launch of the new service and the removal (end-of-life) for Clips API has been made.**_ Planet's Clip API was a compute API designed to allowed users to clip the images to their area of interest. This would save them time in preprocessing and also allow the user to save on their area quota which might have restrictions. Based on Planet's Education and Research Program this quota is set at 10,000 square kilometers a month, which means saving up on quota is very useful. The discussion also led to an important clarification that users are in fact charged only for the area downloaded post clip if using the clip operation and hence this tool. This tool takes a sequential approach from activation to generating a clip request for multiple images activated and then processing the download tokens to actually download the clipped image files. The tool also consists of a sort function which allows the user to extract the files and sort them by type and deleting the original files to save on space. ## Installation To install the Clip-Ship-Planet-CLI you can simply perform the following action with Linux(Tested on Ubuntu 16): ``` git clone https://github.com/samapriya/Clip-Ship-Planet-CLI.git cd Clip-Ship-Planet-CLI && pip install -r requirements.txt ``` On a windows as well as a linux machine, installation is an optional step; the application can also be run directly by executing pclip.py script. The advantage of having it installed is being able to execute ppipe as any command line tool. I recommend installation within virtual environment but you can also install it to system python and should not create any conflicts. To install on windows download the setup files as a zip package, unpack and run ``` python setup.py develop or python setup.py install ``` In a linux distribution ``` sudo python setup.py develop or sudo python setup.py install pclip -h ``` ## Table of contents * [Getting started](#getting-started) * [Usage examples](#usage-examples) * [Planet Key](#planet-key) * [Planet Quota](#planet-quota) * [AOI JSON](#aoi-json) * [Activate or Check Asset](#activate-or-check-asset) * [List IDs](#list-ids) * [Clipping with GeoJSON](#clipping-with-geojson) * [Clipping with JSON](#clipping-with-json) * [Downloading Clipped Imagery](#downloading-clipped-imagery) * [Sorting](#sorting) ## Getting started As usual, to print help: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Usage examples The tools have been designed to follow a sequential setup from activation, clip, download and even sort and includes steps that help resolve additional issues a user might face trying to download clipped area of interests instead of entire scenes. The system will ask you to enter your API key before the CLI starts(this will prompt you only once to change API key use the Planet Key tool). ### Planet Key This tool basically asks you to input your Planet API Key using a password prompt this is then used for all subsequent tools. Ites makes use of the Planet client and esentially executes ```planet init``` ``` usage: pclip planetkey [-h] optional arguments: -h, --help show this help message and exit ``` ### Planet Quota This tool prints details on your existing quota and your area remaining ``` usage: pclip quota optional arguments: -h, --help show this help message and exit ``` ### AOI JSON The aoijson tab within the toolset allows you to create filters and structure your existing input file to that which can be used with Planet's API. The tool requires inputs with start and end date, along with cloud cover. You can choose from multiple input files types such as KML, Zipped Shapefile, GeoJSON, WKT or even Landsat Tiles based on PathRow numbers. The geo option asks you to select existing files which will be converted into formatted JSON file called aoi.json. If using WRS as an option just type in the 6 digit PathRow combination and it will create a json file for you. ``` usage: pclip aoijson [-h] [--start START] [--end END] [--cloud CLOUD] [--inputfile INPUTFILE] [--geo GEO] [--loc LOC] optional arguments: -h, --help show this help message and exit --start START Start date in YYYY-MM-DD? --end END End date in YYYY-MM-DD? --cloud CLOUD Maximum Cloud Cover(0-1) representing 0-100 --inputfile INPUTFILE Choose a kml/shapefile/geojson or WKT file for AOI(KML/SHP/GJSON/WKT) or WRS (6 digit RowPath Example: 023042) --geo GEO map.geojson/aoi.kml/aoi.shp/aoi.wkt file --loc LOC Location where aoi.json file is to be stored ``` As with the [Planet-GEE-Pipeline-CLI](https://github.com/samapriya/Planet-GEE-Pipeline-CLI) the aoijson tool allows the user to bring any filetype of interest, which includes GEOJSON, WKT, KML or SHP file including but not limited to WRS rowpath setup and structures it to enable filtered query using Planet's data API. A simple setup would be ```pclip aoijson --start "2017-06-01" --end "2017-12-31" --cloud "0.15" --inputfile "GJSON" --geo "C:\planet\myarea.geojson" --loc "C:\planet"``` the output is always named as aoi.json. ### Activate or Check Asset The activate tool allows the users to either check or activate planet assets. This tool makes use of an existing json file sturctured for use within Planet API or the aoi.json file created earlier. This is a necessary step since the clip API can only work with those ID(s) which have been activated. In the future the list ID tool will check for number of activated id and wait for all of them to be activated before generating an ID list. ``` usage: pclip activate [-h] [--aoi AOI] [--action ACTION] [--asst ASST] optional arguments: -h, --help show this help message and exit --aoi AOI Choose aoi.json file created earlier --action ACTION choose between check/activate --asset ASST Choose between planet asset types (PSOrthoTile analytic/PSOrthoTile analytic_dn/PSOrthoTile visual/PSScene4Band analytic/PSScene4Band analytic_dn/PSScene3Band analytic/PSScene3Band analytic_dn/PSScene3Band visual/REOrthoTile analytic/REOrthoTile visual ``` An example setup for asset activation is the following ```pclip activate --aoi "C:\planet\aoi.json" --action "activate" --asset "PSOrthoTile analytic"``` ### List IDs The next step is to list ID(s) that you have activated, this creates a temporary file containing the list of ID(s) which can be used to iteratively call the clips API. This is a modification of the activation function to use only the item id instead of item type and asset id and write to file for future use. ``` usage: pclip idlist [-h] [--aoi AOI] [--asset ASSET] optional arguments: -h, --help show this help message and exit --aoi AOI Input path to the structured json file from which we will generate the clips --asset ASSET Choose from asset type for example:"PSOrthoTile analytic"|"REOrthoTile analytic" ``` The example setup for this command is the following ```pclip idlist --aoi “C:\planet\aoi.json” --asset “PSOrthoTile analytic”``` ### Clipping with GeoJSON A geejson file can be used directly to clip and query the area of interest and then submit clip process. I added this is a functionality but want to make clear that this does not take into consideration any other filters such as cloud cover or start and end date, and hence should be used only when you do not need to apply any filter. ``` usage: pclip geojsonc [-h] [--path PATH] [--item ITEM] [--asset ASSET] optional arguments: -h, --help show this help message and exit --path PATH Path to the geojson file including filename (Example: C:\users ile.geojson) --item ITEM Choose from item type for example:"PSOrthoTile","REOrthoTile" --asset ASSET Choose from asset type for example: "visual","analytic" ``` A simple setup for the JSON tool is the following ```pclip geojsonc --path “C:\planet\aoi.geojson” --item “PSOrthoTile” --asset “analytic"``` ### Clipping with JSON This is the preferred style of submitting the clip requests using the IDlist we generated earlier. This is already structured before even activating assets and includes the additional filters you might have used for selecting the images. ``` usage: pclip jsonc [-h] [--path PATH] [--item ITEM] [--asset ASSET] optional arguments: -h, --help show this help message and exit --path PATH Path to the json file including filename (Example: C:\users ile.json) --item ITEM Choose from item type for example:"PSOrthoTile","REOrthoTile" --asset ASSET Choose from asset type for example: "visual","analytic" ``` A simple setup for the JSON tool is the following ```pclip jsonc --path “C:\planet\aoi.json” --item “PSOrthoTile” --asset “analytic"``` ### Downloading Clipped Imagery The last step includes providing a location where the clipped imagery can be downloaded. This includes the zip files that are generated from the earlier step and include a download token that expires over time. This batch downloads the clipped zip files to destination directory ``` usage: pclip downloadclips [-h] [--dir DIR] optional arguments: -h, --help show this help message and exit --dir DIR Output directory to save the assets. All files are zipped and include metadata ``` A simple setup includes just the location to the download directory for the zipped & clipped files to be downloaded ``` pclip downloadclips --dir “C:\planet\zipped" ``` ### Sorting As an additional measure and because it makes arranging and handling datasets easily, this setup comes completed with a sort tool. If a output directory is provided for the unzipped files, the tool unzips all files, moves the images and metadata to seperate directories and then deletes the original zipped files to save space. ``` usage: pclip sort [-h] [--zipped ZIPPED] [--unzipped UNZIPPED] optional arguments: -h, --help show this help message and exit --zipped ZIPPED Folder containing downloaded clipped files which are zipped --unzipped UNZIPPED Folder where you want your files to be unzipped and sorted ``` A simple would be the following (Images and metadata are sorted into an image and metadata folder inside the unzipped files folder) ``` pclip sort --zipped “C:\planet\zipped” --unzipped “C:\planet\unzipped” ``` ## Changelog ### v0.2.2 - Improved Planet Key Handler - Added new tool to insepect planet account quota ### v0.2.1 - Thanks to commit suggested by [Rabscuttler](https://github.com/Rabscuttler) - Fixed issues with help text and installer ### v0.2.0 - Fixed issues with config files ### v0.1.9 - Now handles running and succeeded status better - Now enumerates during clip and download to allow user estimates on number of assets clipped and/or downloaded ### v0.1.8 - Includes required packages list within installer - Robust GEOJSON Parsing ### v0.1.7 - Fixed issues with processing visual asset types - The Clip function now handles error codes if the post response code is not 202(accepted for processing) then the error code and item and asset type is printed. ### v0.1.6 - Handles single time input API Key, this is needed only once to start the program - Fixed issue with base metadata folder during sort - Updated asset argument for asset activation to match styles ### v0.1.5 - Updated Requirements.txt to include pyshp - Fixed subprocess shell error, for now shell=True ### v0.1.4 - General Improvements ### v0.1.3 - General Improvements ### v0.1.2 - Tested on Ubuntu 16.04 and now handles permissions problem - Temporary files now written to config folders to avoid admin permission ### v0.1.1 - General Improvements --- ## File: docs/projects/ee_data.md # [Earth-Engine-Datasets-List](https://github.com/samapriya/Earth-Engine-Datasets-List) The idea is to make this list machine readable so you can programmatically call assets based on filters and to have a list for users who are not yet registered but want to look at the dataset list within earth engine. Register for a free [Google Earth Engine account](https://earthengine.google.com/signup/). The table is in the form a csv file with the following setup | id | provider | title | start_date | end_date | startyear | endyear | type | tags | |-------------------------------------------------|------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|------------|----------|-----------|---------|-----------------|-----------------------------------------------------| | AHN/AHN2_05M_INT | AHN | AHN Netherlands 0 | 12/31/11 | 12/31/11 | 2011 | 2011 | Image |ahn, lidar, elevation, netherlands, dem, geophysical | The list allows for datasets to be built into a parser and we can pull these as though a dictionary for making selections before applying any other tool. --- ## File: docs/projects/gee_asset_manager_addon.md # [Google Earth Engine Batch Asset Manager with Addons](https://github.com/samapriya/gee_asset_manager_addon) [](https://badge.fury.io/py/geeadd) [](https://doi.org/10.5281/zenodo.3474295) [](https://caniusepython3.com/project/geeadd) [](https://saythanks.io/to/samapriya) Google Earth Engine Batch Asset Manager with Addons is an extension of the one developed by Lukasz [here](https://github.com/tracek/gee_asset_manager) and additional tools were added to include functionality for moving assets, conversion of objects to fusion table, cleaning folders, querying tasks. The ambition is apart from helping user with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as need arises. If you use this tool to download data for your research, and find this tool useful, star and cite it as below ``` Samapriya Roy. (2019, October 6). samapriya/gee_asset_manager_addon: GEE Asset Manager with Addons (Version 0.3.3). Zenodo. http://doi.org/10.5281/zenodo.3474295 ``` ## Table of contents * [Installation](#installation) * [Getting started](#getting-started) * [Uploading](#uploading) * [Usage examples](#usage-examples) * [EE User](#ee-user) * [Asset List](#asset-list) * [Asset Size](#asset-size) * [Earth Engine Asset Report](#earth-engine-asset-report) * [Task Query](#task-query) * [App to Script](#app-to-script) * [Task Report](#task-report) * [Delete a collection with content:](#delete-a-collection-with-content) * [Assets Move](#assets-move) * [Assets Copy](#assets-copy) * [Assets Access](#assets-access) * [Set Collection Property](#set-collection-property) * [Cancel all tasks](#cancel-all-tasks) ## Installation We assume Earth Engine Python API is installed and EE authorised as desribed [here](https://developers.google.com/earth-engine/python_install). Quick installation **```pip install geeadd```** To install using github: ``` git clone https://github.com/samapriya/gee_asset_manager_addon cd gee_asset_manager_addon && pip install -r requirements.txt python setup.py install ``` Installation is an optional step; the application can be also run directly by executing geeadd.py script. The advantage of having it installed is being able to execute geeadd as any command line tool. I recommend installation within virtual environment. To install run ``` python setup.py develop or python setup.py install In a linux distribution sudo python setup.py develop or sudo python setup.py install ``` ## Getting started As usual, to print help: To obtain help for a specific functionality, simply call it with _help_ switch, e.g.: `geeadd upload -h`. If you didn't install geeadd, then you can run it just by going to _geeadd_ directory and running `python geeadd.py [arguments go here]` ## Uploading You can upload tables and rasters using [geeup](https://github.com/samapriya/geeup). This uses selenium to handle uploading and hence cannot be used in a headless environment. ## Usage examples ### EE User This tool is designed to allow different users to change earth engine authentication credentials. The tool invokes the authentication call and copies the authentication key verification website to the clipboard which can then be pasted onto a browser and the generated key can be pasted back ### Create This tool allows you to create a collection or folder in your earth engine root directory. The tool uses the system cli to achieve this and this has been included so as to reduce the need to switch between multiple tools and CLI. ``` usage: geeadd.py create [-h] --typ TYP --path PATH optional arguments: -h, --help show this help message and exit --typ TYP Specify type: collection or folder --path PATH This is the path for the earth engine asset to be created full path is needsed eg: users/johndoe/collection ``` ### Asset List This tool is designed to either print or output asset lists within folders or collections using earthengine ls tool functions. ``` usage: geeadd.py lst [-h] --location LOCATION --typ TYP [--items ITEMS] [--output OUTPUT] optional arguments: -h, --help show this help message and exit Required named arguments.: --location LOCATION This it the location of your folder/collection --typ TYP Whether you want the list to be printed or output as text[print/report] Optional named arguments: --items ITEMS Number of items to list --output OUTPUT Folder location for report to be exported ``` ### Asset Size This tool allows you to query the size of any Earth Engine asset[Images, Image Collections, Tables and Folders] and prints out the number of assets and total asset size in non-byte encoding meaning KB, MB, GB, TB depending on size. ``` usage: geeadd assetsize [-h] --asset ASSET optional arguments: -h, --help show this help message and exit --asset ASSET Earth Engine Asset for which to get size properties ``` ### App to Script This tool writes out or prints the underlying earthengine code for any public earthengine app. The tool has an option to export the code into a javascript file that you can then paste into Google Earth Engine code editor. ``` geeadd app2script -h usage: geeadd app2script [-h] --url URL [--outfile OUTFILE] optional arguments: -h, --help show this help message and exit --url URL Earthengine app url Optional named arguments: --outfile OUTFILE Write the script out to a .js file: Open in any text editor ``` Simple setup can be ``` geeadd app2script --url "https://gena.users.earthengine.app/view/urban-lights" ``` or write to a javascript file which you can then open with any text editor and paste in earthengine code editor ``` geeadd app2script --url "https://gena.users.earthengine.app/view/urban-lights" --outfile "Full path to javascript.js" ``` ### Earth Engine Asset Report This tool recursively goes through all your assets(Includes Images, ImageCollection,Table,) and generates a report containing the following fields [Type,Asset Type, Path,Number of Assets,size(MB),unit,owner,readers,writers]. ``` usage: geeadd.py ee_report [-h] --outfile OUTFILE optional arguments: -h, --help show this help message and exit --outfile OUTFILE This it the location of your report csv file ``` A simple setup is the following ``` geeadd --outfile "C:\johndoe\report.csv"``` ### Task Query This script counts all currently running and ready tasks along with failed tasks. ``` usage: geeadd.py tasks [-h] optional arguments: -h, --help show this help message and exit geeadd.py tasks ``` ### Task Report Sometimes it is important to generate a report based on all tasks that is running or has finished. Generated report includes taskId, data time, task status and type ``` usage: geeadd taskreport [-h] [--r R] optional arguments: -h, --help show this help message and exit --r R Folder Path where the reports will be saved ``` ### Delete a collection with content: The delete is recursive, meaning it will delete also all children assets: images, collections and folders. Use with caution! ``` geeadd delete users/johndoe/test ``` Console output: ``` 2016-07-17 16:14:09,212 :: oauth2client.client :: INFO :: Attempting refresh to obtain initial access_token 2016-07-17 16:14:09,213 :: oauth2client.client :: INFO :: Refreshing access_token 2016-07-17 16:14:10,842 :: root :: INFO :: Attempting to delete collection test 2016-07-17 16:14:16,898 :: root :: INFO :: Collection users/johndoe/test removed ``` ### Delete all directories / collections based on a Unix-like pattern ``` geeadd delete users/johndoe/*weird[0-9]?name* ``` ### Assets Move This script allows us to recursively move assets from one collection to the other. ``` usage: geeadd.py mover [-h] [--assetpath ASSETPATH] [--finalpath FINALPATH] optional arguments: -h, --help show this help message and exit --assetpath ASSETPATH Existing path of assets --finalpath FINALPATH New path for assets geeadd.py mover --assetpath "users/johndoe/myfolder/myponycollection" --destination "users/johndoe/myfolder/myotherponycollection" ``` ### Assets Copy This script allows us to recursively copy assets from one collection to the other. If you have read acess to assets from another user this will also allow you to copy assets from their collections. ``` usage: geeadd.py copy [-h] [--initial INITIAL] [--final FINAL] optional arguments: -h, --help show this help message and exit --initial INITIAL Existing path of assets --final FINAL New path for assets geeadd.py mover --initial "users/johndoe/myfolder/myponycollection" --final "users/johndoe/myfolder/myotherponycollection" ``` ### Assets Access This tool allows you to set asset acess for either folder , collection or image recursively meaning you can add collection access properties for multiple assets at the same time. ``` usage: geeadd.py access [-h] --asset ASSET --user USER --role ROLE optional arguments: -h, --help show this help message and exit --asset ASSET This is the path to the earth engine asset whose permission you are changing folder/collection/image --user USER Full email address of the user, try using "AllUsers" to make it public --role ROLE Choose between reader, writer or delete ``` ### Set Collection Property This script is derived from the ee tool to set collection properties and will set overall properties for collection. ``` usage: geeadd.py collprop [-h] [--coll COLL] [--p P] optional arguments: -h, --help show this help message and exit --coll COLL Path of Image Collection --p P "system:description=Description"/"system:provider_url=url"/"sys tem:tags=tags"/"system:title=title ``` ### Cancel all tasks This is a simpler tool, can be called directly from the earthengine cli as well ``` earthengine cli command earthengine task cancel all usage: geeadd.py cancel [-h] optional arguments: -h, --help show this help message and exit ``` ### Changelog #### v0.3.3 - General improvements - Added tool to get underlying code from earthengine app #### v0.3.1 - Updated list and asset size functions - Updated function to generate earthengine asset report - General optimization and improvements to distribution - Better error handling #### v0.3.0 - Removed upload function - Upload handles by [geeup](https://github.com/samapriya/geeup) - General optimization and improvements to distribution - Better error handling #### v0.2.8 - Uses poster for streaming upload more stable with memory issues and large files - Poster dependency limits use to Py 2.7 will fix in the new version #### v0.2.6 - Major improvement to move, batch copy, and task reporting - Major improvements to access tool to allow users read/write permission to entire Folder/collection. #### v0.2.5 - Handles bandnames during upload thanks to Lukasz for original upload code - Removed manifest option, that can be handled by seperate tool (ppipe) #### v0.2.3 - Removing the initialization loop error #### v0.2.2 - Added improvement to earthengine authorization #### v0.2.1 - Added capability to handle PlanetScope 4Band Surface Reflectance Metadata Type - General Improvements #### v0.2.0 - Tool improvements and enhancements #### v0.1.9 - New tool EE_Report was added #### v0.1.8 - Fixed issues with install - Dependencies now part of setup.py - Updated Parser and general improvements --- ## File: docs/projects/gee_takeout.md # Google Takeout and Transfer: Tools and Guide for Code and Asset Transfer [](https://doi.org/10.5281/zenodo.1185158) [](https://saythanks.io/to/samapriya) Command Line Interface Allows you to copy all codes and assets from one Google account to another **Note: This is something that I have tested and have designed only for a windows machine with python 2.7.14 but can be easily ported into an different operating system. Use these tool and steps at your own risk and backup your scripts always just in case.** If you still want to proceed which I assume you do in case you are still reading, I am including descriptions links to the tool I made and the steps I used to achieve the same. The tool is a single command line interface with three sections. **Before you do this make sure of a few things** * _Both your google accounts have an external password, since it requires that to download and perform a lot of the operations. Also enable [__Let Less Secure App use your account_][1]_ on both these accounts._ * _Your system has native python available in terminal or command prompt depending on what kind of system you are using. You can check this by typing_ `python --version` * _Git is installed on your system. For windows you can find [__installation here_][2] * _Earth Engine Command Line(earthengine cli) interface is installed, instructions are in the [__developer page_][3] * _You have authenticated earthengine cli using _`earthengine authenticate` * _Make sure you visit the [__git source for your account_][4]_ within earth engine and allow access._ * _Check git is accessible via your system path type_ `git help` _and check if the system can reach installed git command line tools._ Now we setup and install the tool by running the following set of steps. The Github repository containing this tool and codes [can be found here][5] For windows [download the zip here][6] and after extraction go to the folder containing "setup.py" and open command prompt at that location and type python setup.py install pip install -r requirements.txt Now call the tool for the first time, by typing in `geetakeout -h`. This will only work if you have python in the system path which you can test for opening up terminal or command prompt and typing `**python**.` If you want, you don't have to install the tool to use it you can browse to the folder geetakeout inside the main zipped folder. You can then call the tool by simple `python geetakeout.py -h` The housekeeping and credential setup is optional since most of you have probably installed the earthengine cli and authenticated it using `**earthengine authenticate**.` > **Anatomy of the Process: How to transfer step by step** Getting first things out of the way is to understand the three sections of this tool. To make life and this process simpler I designed the tool to have a flow so you can run these tools one after the other. The _EE Setup and Housekeeping_ sections are optional , since I will generally updated the selenium driver for mozilla and it assumes you have authenticated your earthengine CLI . The tool might show an error if you have not authenticated using earthengine authenticate If you have installed the tool run geetakeout -h If you have migrated into the folder python geetakeout.py -h ![][7] The GEE Takeout Tool CLI printout **Setting up the case study** For this blog I decided to make the transfer simple I have a university account but since my university if shifting umail services to a google app service it means my domain would change from @umail.iu.edu to @iu.edu which are separate accounts. I created the iu.edu GEE account recently. ![][8] Code Editor comparison Left(my @umail account and right my @iu account) This also means that the root path for my home folder and repository are different. The idea is simple to be able to replicate the codes and assets from one account to the other. This includes every type if assets including collections but also making sure that the structure of the folders are same. That being said you will still have to change the home path in the codes but if the structure is same then only a single root-path changes. ![][9] Comparing the root path and assets folder for both accounts So now that we have a setup, I am going to approach it step by step and have a walk through to explain the process better. **Step 1: Getting your Repository Lists(gee_repo)** This assumes that you have visited the Git Source for your codes in Google Earth Engine and authorized it. If not [allow it here][4] and then you are set to download your repo contents and perform git operations. The tool is setup for accessing all repositories that are shared with you. This downloads the list into an html file which can then be parsed for your repositories. ![][10] Create GEE Repo List **Step 2: Setting up your Git with Earth Engine Credentials(git_auth)** You can do this using two methods * The first simple includes you visiting your [gitsource account page ][4]that we accessed earlier and click `**Generate Password **`and follow instructions. I am going to talk more about the second method because this eliminates the need to get the password again and again since it is save as passkey. This will authenticate your git client with your git password using a browser less login and also store your gitkey ![][11] GIT Auth (saves git key) We will use this again to setup our second account post authorization. This will print our gitkey location and make sure you copy that so you can swap in out as needed. Note the name of the key is in the format `**git-"username"**`** **in this case it is `**git-roysam**` **Step 3: Authorize your Git Client with Git Key(git_swap)** The next step is to use the saved gitkey to authorize the git client. We are setting everything up so that we can clone the repositories to which we have access. ![][12] Authorize using the git key stored earlier **Step 4: Clone your repositories(git_clone)** This tool makes use of your earlier created GIT list, now that your git client has been authorized in step 2, you should be able to download your repos. This tool uses the account already linked to your terminal account. If you are not sure try `**earthengine ls**` to see your username. The export path is noted for the collection of repositories. ![][13] Cloning your Git Repositories **Step 5: Working with Assets: Generating Asset Report(ee_report)** This includes all your assets , including tables, images, image collections and folders. We need to make sure we have this list to work on copying over your assets to the secondary account. Running this is simple and just requires a location for the csv file (the full path). ![][14] Running Earth Engine Reports The output is a csv file consiting of the type of asset and the asset path to be replicated in the new account. And now that we have the list time to get permissions to copy these assets. **Step 6: Setting Permissions to Assets(ee_permissions)** We now use the report file generated to grant read access to all assets in your account. Once this is completed you will be able to copy your assets apart from being able to copy your codes. ![][15] Getting permissions to assets **Let us Begin to Copy : We change gears and switch over to the destination account** **Step 7: Setting up the Destination Account(ee_user and git_swap)** Now we have to do two steps one after the other, do a quick earthengine authenticate and authenticate to your new account and perform Step 2 and Step 3 this time using your new account. The tool `**ee_user**` will also allow you to change your accounts. I already created Step 2 for my secondary account and now I will use that to authorize my git client with the new account. ![][16] Change your earthengine authentication and also validate your git key ![][17] Now we authorized the git client with the second key **Step 8: Replicate Repositories (git_create)** To setup our new account we need to build the outline of the earlier account, the repolists and folders inside these repos and then similary the folders and empty collections in the secondary account. Note: Git cannot push an empty repository so if you have an empty repository delete it before downloading and pushing to new account ![][18] Git Create your folder based on your earlier account The repo lists now look similar ![][19] Repo created on secondary account **Step 9: Push to New Account(git_replicate)** Now we push all codes from our earlier account to our new account, this way our repositories will now be populated with the most recent codes. "Do not push to any repository that already has code because this will overwrite it" ![][20] git_replicate to new account **Step 10: Replicate Asset Structure(asset_create) and Assets (asset_replicate)** This is similar to git_create here we replicate the collection and folder structure so we can push our assets to them. You pass it the original report you created from your primary account and it sets up as needed. ![][21] Creates asset structure (folders and collections) This has replicated the collection and Image folder structures. ![][22] Asset Collections and Folders have been created(Left: asset home before asset_create Right: asset home after asset_create) However this is still empty and the last step makes sure that your assets are actually copied over to your new asset home. I have included a counter to measure transfers left incase it is a large collection. ![][23] Asset Replication: Copying assets to your home folder The final results is your assets and codes all copied, you will still have to edit codes to change your path as needed but for now we have replicated an Earth Engine account into a new location. ![][24] Replicated Assets on Both Accounts Copied from Left to Right There you go, over the last 10 steps we have managed to replicate and move an earth engine account from one place to another. Though I found this useful to move accounts within a university setting, I see some value in moving accounts and replicating when a project member leaves a project or for simply migrating at large. For now if an owner of an account deletes his/her account or looses access to his/her account and even if you are a writer to the repository and the collection, you will loose access to these codes and assets. So this can aid in maintaining continuity by moving codes to more persistent account. Though I have not tested this tools in a linux setting, these setup tools can be adapted and used easily in that framework, since I have tested the individual components in such setups. [1]: https://support.google.com/accounts/answer/6010255?hl=en [2]: https://git-scm.com/downloads [3]: https://developers.google.com/earth-engine/python_install_manual [4]: https://earthengine.googlesource.com/ [5]: https://github.com/samapriya/gee-takeout [6]: https://github.com/samapriya/gee-takeout/archive/master.zip [7]: https://cdn-images-1.medium.com/max/2000/1*CVgzI0ro26lAHD3R37JKDg.jpeg [8]: https://cdn-images-1.medium.com/max/1600/1*ZvsokVlQBbqiUv4UA-ow9Q.jpeg [9]: https://cdn-images-1.medium.com/max/1600/1*uH3e6ORkgm5ikJjez844bw.png [10]: https://cdn-images-1.medium.com/max/2000/1*tP5FzkRiWOTguTPwzXij8Q.gif [11]: https://cdn-images-1.medium.com/max/2000/1*PRAn1xGQ4EDMRGJQWlEN1A.gif [12]: https://cdn-images-1.medium.com/max/2000/1*yZJD-PNWUR9jk4Tv4yDTdA.gif [13]: https://cdn-images-1.medium.com/max/2000/1*EFslzisYtbDEGFCzApaNyw.gif [14]: https://cdn-images-1.medium.com/max/2000/1*4LIeCpEUMozxYmiTL2-STg.gif [15]: https://cdn-images-1.medium.com/max/2000/1*M9lweMgxLlENWyy8SMqXZg.gif [16]: https://cdn-images-1.medium.com/max/2000/1*PH6kz_kieM1nBvxbJ9Zd6Q.gif [17]: https://cdn-images-1.medium.com/max/2000/1*3GvklH9xmcnlOLM17y526A.gif [18]: https://cdn-images-1.medium.com/max/2000/1*aL2aHP1b9HlPY6-dbGgCnQ.jpeg [19]: https://cdn-images-1.medium.com/max/1600/1*X6xtrR-rBfpqGkpmH1TsWQ.jpeg [20]: https://cdn-images-1.medium.com/max/2000/1*4Im_nOru-j8fI0GYQe_duQ.gif [21]: https://cdn-images-1.medium.com/max/2000/1*5OCk2sHYDZ2Pop6dxCWJrA.gif [22]: https://cdn-images-1.medium.com/max/1600/1*bxqnJff5hpFYmGhXb7JU8Q.png [23]: https://cdn-images-1.medium.com/max/2000/1*ulmG1mEu_D3eRsadvBFwiA.gif [24]: https://cdn-images-1.medium.com/max/1600/1*jzQ9cLPIushKqFoZFfn9Sg.png --- ## File: docs/projects/gee2drive.md # gee2drive: Download Earth Engine Public and Private assets to Google Drive [](https://doi.org/10.5281/zenodo.1324456) [](https://badge.fury.io/py/gee2drive) Google Earth Engine currently allows you to export images and assets as either GeoTiff files or TFrecords. The system splits the files if the estimated size is greater than 2GB which is the upper limit and needs the geometry to be parsed in the form of either a fusion table, a user drawn geometry or a table imported into the user's assets. While the javascript frontend is great owing to the queryable catalog whereby you can search and and export your personal and private assets, the limitation lies in batch exports. To resolve this the python API access allows you to call batch export functions but now it is limited to checking for itersects first and running without having a queryable catalog. With the same idea I created this tool which allows you to run a terminal environment where your personal and general catalog images are part of a autosuggest feature. This tool allows you to look for images based on names for example " you can search for Sentinel and it will show you full path of images which have the word sentinel in the title". It also creates a report for your image collections and images so apart from the public datasets this can also find your own datasets as well. You can then generate bandlist to make sure all bands you are exporting are of the same type and then export all images that intersect you aoi. The assumption here is * Every image in the give image have the same band structure, choose the bandlist that you know to common to all images * If the geomery is too complex use the operator feature to use a bounding box instead. * For now all it filters is geometry and date, and it is does not filter based on metadata (however in the examples folder I have shown how to import and use additional filter before exporting an image collection) In the future I will try to integrate some other functionalities to this environment and you can indeed run the tool without the use of the autosuggest terminal as a simple CLI. Hence the terminal feature is optional. ## Table of contents * [Installation](#installation) * [Getting started](#getting-started) * [Google Earth Engine to Drive Manager](#google-earth-engine-to-drive-manager) * [GEE to Google Drive CLI](#gee-to-google-drive-cli) * [gee2drive Terminal](#gee2drive-terminal) * [gee2drive refresh](#gee2drive-refresh) * [gee2drive idsearch](#gee2drive-idsearch) * [gee2drive bandtype](#gee2drive-bandtype) * [gee2drive export](#gee2drive-export) ## Installation This assumes that you have native python & pip installed in your system, you can test this by going to the terminal (or windows command prompt) and trying. This assumes that you are also well aware of Google Earth Engine Python setup and have it installed and authetenticated on your system. If not you can [read about it here](https://developers.google.com/earth-engine/python_install_manual) ```python``` and then ```pip list``` If you get no errors and you have python 2.7.14 or higher you should be good to go. Please note that I have tested this only on python 2.7.15 but can be easily modified for python 3. To install **Python CLI for Digital Ocean** you can install using two methods ```pip install gee2drive``` or you can also try ``` git clone https://github.com/samapriya/gee2drive.git cd gee2drive python setup.py install ``` Use might have to use sudo privileges Installation is an optional step; the application can be also run directly by executing gee2drive.py script. The advantage of having it installed is being able to execute ppipe as any command line tool. I recommend installation within virtual environment. If you don't want to install, browse into the gee2drive folder and try ```python gee2drive.py``` to get to the same result. ## Getting started As usual, to print help: ``` usage: gee2drive [-h] {terminal,refresh,idsearch,bandtype,export} ... Google Earth Engine to Drive Exporter positional arguments: {terminal,refresh,idsearch,bandtype,export} terminal Starts the interactive terminal with autosuggest refresh Refreshes your personal asset list and GEE Asset list idsearch Does possible matches using asset name to give you asseth id/full path bandtype Prints bandtype and generates list to be used for export export Export Collections based on filter optional arguments: -h, --help show this help message and exit ``` To obtain help for a specific functionality, simply call it with _help_ switch, e.g.: `gee2drive idsearch -h`. If you didn't install gee2drive, then you can run it just by going to *gee2drive* directory and running `python gee2drive.py [arguments go here]` ### GEE to Google Drive CLI This tool is designed to augment to the existing facilty of image export using a CLI, whereby you can pass it arguments to filter based on an area of interest geojson file, a start and end date for collection ### gee2drive Terminal This is an autosuggestive terminal which uses the gee2add package to perform all of the functions but has autosuggest for Earth Engine catalog and your own personal catalog. This way you can get access to image id without needing the catalog id in the javascript codeeditor. ``` usage: gee2drive terminal [-h] optional arguments: -h, --help show this help message and exit ``` Once you type ```gee2drive terminal``` you get a shell inside your current terminal where you get autosuggest for image and have full functionality of the terminal. ### gee2drive refresh For the past couple of months I have [maintained a catalog of the most current Google Earth Engine assets](https://github.com/samapriya/Earth-Engine-Datasets-List), within their raster data catalog. I update this list every week. This tool downloads the most current version of this list, and also looks into your personal assets to generate your very own asset report which then serve as a master dataset to feed into autosuggestions. ``` gee2drive refresh -h usage: gee2drive refresh [-h] optional arguments: -h, --help show this help message and exit ``` ### gee2drive idsearch There is a possibility that you don't really remember the full path to your asset or the public asset. Fortunately when I parse and collect the image list and path for you they have names that are searchable so use a keyword. for example search using "MODIS" or "sentinel". Also it is not case sensitive, so you should be able to type "SENTINEl" or "Sentinel" or "sentinel" and it should still work ``` gee2drive idsearch -h usage: gee2drive idsearch [-h] [--name NAME] optional arguments: -h, --help show this help message and exit --name NAME Name or part of name to search for ``` ### gee2drive bandtype Export requires all the bandtypes to be of the same kind. To do this, I simply generate the band types for you and you can select the band list you want , remember to paste it as a list. ``` usage: gee2drive bandtype [-h] [--id ID] optional arguments: -h, --help show this help message and exit --id ID full path for collection or image ``` ### gee2drive export Finally the export tool, that lets you export an image or a collection clipped to your AOI. This makes use of the bandlist you exported. Incase you are exporting an image and not a collection you don't need a start and end date. The tool uses the bounds() function to use a bounding box incase the geometry has a complex geometry or too many vertices simply use the operator ```bb```. If the geojson/json/kml keeps giving parsing error go to [geojson.io](geojson.io) ``` usage: gee2drive export [-h] [--id ID] [--type TYPE] [--folder FOLDER] [--aoi AOI] [--start START] [--end END] [--bandlist BANDLIST] [--operator OPERATOR] optional arguments: -h, --help show this help message and exit --id ID Full path for collection or image --type TYPE Type whether image or collection --folder FOLDER Drive folder path --aoi AOI Full path to geojson/json/kml to be used for bounds Optional named arguments for image collection only: --start START Start date to filter image --end END End date to filter image --bandlist BANDLIST Bandlist we generated from bandtype export must be same bandtype --operator OPERATOR Use bb for Bounding box incase the geometry is complex or has too many vertices ``` A typical setup would be ```gee2drive export --id "COPERNICUS/S2" --folder "sentinel-export" --aoi "C:\Users\sam\boulder.geojson" --start "2018-02-01" --end "2018-03-01" --bandlist ['B2','B3','B4'] --operator "bb" --type "collection" ``` Also as promised earlier , there is a way to add additional filters and then pass it through the export function here is how and I have included this in the Examples folder. This for example uses the Landsat collection but applies the Cloud cover filter before passing it for export ```python import ee import os import sys import gee2drive [head,tail]=os.path.split(gee2drive.__file__) os.chdir(head) sys.path.append(head) from export import exp ee.Initialize() exp(collection=ee.ImageCollection('LANDSAT/LC08/C01/T1_SR').filterMetadata('CLOUD_COVER','less_than',20), folderpath="l8-out",start="2018-02-01",end="2018-06-01", geojson=r"C:\Users\sam\boulder.geojson",bandnames="['B1','B2']", operator="bb",typ="ImageCollection") ``` #### Changelog ##### v0.0.4 * Can now parse gejson, json,kml * Minor fixes and general improvements ##### v0.0.3 * Minor Fixes --- ## File: docs/projects/geeup.md # [geeup: Simple CLI for Earth Engine Uploads with Selenium Support](https://github.com/samapriya/geeup)   [](https://twitter.com/intent/tweet?text=Use%20porder%20CLI%20with%20@planetlabs%20new%20ordersv2%20API&url=https://github.com/samapriya/geeup) [](https://doi.org/10.5281/zenodo.3369484) [](https://badge.fury.io/py/geeup) [](https://opensource.org/licenses/Apache-2.0) This tool came of the simple need to handle batch uploads of both image assets to collections but also thanks to the new table feature the possibility of batch uploading shapefiles into a folder. Though a lot of these tools including batch image uploader is part of my other project [geeadd](https://github.com/samapriya/gee_asset_manager_addon) which also includes additional features to add to the python CLI, this tool was designed to be minimal so as to allow the user to simply query their quota, upload images or tables and also to query ongoing tasks and delete assets. I am hoping this tool with a simple objective proves useful to a few users of Google Earth Engine. -If you find this tool useful, star and cite it as below ``` Samapriya Roy. (2019, August 16). samapriya/geeup: geeup: Simple CLI for Earth Engine Uploads (Version 0.2.5). Zenodo. http://doi.org/10.5281/zenodo.3369484 ``` ## Table of contents * [Installation](#installation) * [Windows Setup](#windows-setup) * [Getting started](#getting-started) * [geeup Simple CLI for Earth Engine Uploads](#geeup-simple-cli-for-earth-engine-uploads) * [geeup init](#geeup-init) * [gee Quota](#gee-quota) * [gee getmeta](#gee-getmeta) * [gee Zipshape](#gee-zipshape) * [gee upload](#gee-upload) * [gee seltabup](#gee-seltabup) * [gee selsetup](#gee-selsetup) * [gee tasks](#gee-tasks) * [gee delete](#gee-delete) ## Installation This assumes that you have native python & pip installed in your system, you can test this by going to the terminal (or windows command prompt) and trying ```python``` and then ```pip list``` If you get no errors and you have python 2.7.14 or higher you should be good to go. Please note that I have tested this only on python 2.7.15, but it should run on Python 3. **This command line tool is dependent on functionality from GDAL** For installing GDAL in Ubuntu ``` sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update sudo apt-get install gdal-bin sudo apt-get install python-gdal ``` ## Windows Setup Shapely and a few other libraries are notoriously difficult to install on windows machines so follow the steps mentioned here **before installing porder**. You can download and install shapely and other libraries from the [Unofficial Wheel files from here](https://www.lfd.uci.edu/~gohlke/pythonlibs) download depending on the python version you have. **Do this only once you have install GDAL**. I would recommend the steps mentioned above to get the GDAL properly installed. However I am including instructions to using a precompiled version of GDAL similar to the other libraries on windows. You can test to see if you have gdal by simply running ```gdalinfo``` in your command prompt. If you get a read out and not an error message you are good to go. If you don't have gdal try Option 1,2 or 3 in that order and that will install gdal along with the other libraries #### Option 1: Starting from porder v0.4.5 onwards: Simply run ```geeup -h``` after installation. This should go fetch the extra libraries you need and install them. Once installation is complete, the porder help page will show up. This should save you from the few steps below. #### Option 2: If this does not work or you get an unexpected error try the following commands. You can also use these commands if you simply want to update these libraries. ``` pipwin refresh pipwin install gdal ``` #### Option 3 For Windows I also found this [guide](https://webcache.googleusercontent.com/search?q=cache:UZWc-pnCgwsJ:https://sandbox.idre.ucla.edu/sandbox/tutorials/installing-gdal-for-windows+&cd=4&hl=en&ct=clnk&gl=us) from UCLA Also for Ubuntu Linux I saw that this is necessary before the install ```sudo apt install libcurl4-openssl-dev libssl-dev``` **This also needs earthengine cli to be [installed and authenticated on your system](https://developers.google.com/earth-engine/python_install_manual) and earthengine to be callable in your command line or terminal** To install **geeup: Simple CLI for Earth Engine Uploads** you can install using two methods. ```pip install geeup``` or you can also try ``` git clone https://github.com/samapriya/geeup.git cd geeup python setup.py install ``` For Linux use sudo or try ```pip install geeup --user```. Installation is an optional step; the application can also be run directly by executing geeup.py script. The advantage of having it installed is that geeup can be executed as any command line tool. I recommend installation within a virtual environment. If you don't want to install, browse into the geeup folder and try ```python geeup.py``` to get to the same result. ## Getting started As usual, to print help: ``` usage: geeup.py [-h] {update,quota,zipshape,upload,selupload,seltabup,tasks,delete} ... Simple Client for Earth Engine Uploads with Selenium Support positional arguments: {update,quota,zipshape,upload,selupload,seltabup,tasks,delete} update Updates Selenium drivers for firefox quota Print Earth Engine total quota and used quota zipshape Zips all shapefiles and subsidary files into individual zip files getmeta Generates generalized metadata for all rasters in folder upload Batch Asset Uploader using Selenium seltabup Batch Table Uploader using Selenium. selsetup Non headless setup for new google account, use if upload throws errors tasks Queries current task status [completed,running,ready,failed,cancelled] delete Deletes collection and all items inside. Supports Unix-like wildcards. optional arguments: -h, --help show this help message and exit ``` To obtain help for specific functionality, simply call it with _help_ switch, e.g.: `geeup zipshape -h`. If you didn't install geeup, then you can run it just by going to *geeup* directory and running `python geeup.py [arguments go here]` ## geeup Simple CLI for Earth Engine Uploads The tool is designed to handle batch uploading of images and tables(shapefiles). While there are image collection where you can batch upload imagery, for vector or shapefiles you have to batch upload them to a folder. ### geeup init **This is a key step since all upload function depends on this step, so make sure you run this**. This downloads selenium driver and places to your local directory for windows and Linux subsystems. This is the first step to use selenium supported upload. ``` geeup init``` ### gee Quota Just a simple tool to print your earth engine quota quickly. ``` usage: geeup quota [-h] optional arguments: -h, --help show this help message and exit ``` ### gee Zipshape So here's how table upload in Google Earth Engine works, you can either upload the component files shp, shx, prj and dbf or you can zip these files together and upload it as a single file. The pros for this is that it reduces the overall size of the shapefile after zipping them along, this tool looks for the shp file and finds the subsidiary files and zips them ready for upload. It also helps when you have limited upload bandwidth. Cons you have to create a replicate structure of the file system, but it saves on bandwidth and auto-arranges your files so you don't have to look for each additional file. ``` usage: geeup zipshape [-h] --input INPUT --output OUTPUT optional arguments: -h, --help show this help message and exit Required named arguments.: --input INPUT Path to the input directory with all shape files --output OUTPUT Destination folder Full path where shp, shx, prj and dbf files if present in input will be zipped and stored ``` ### gee getmeta This script generates a generalized metadata using information parsed from gdalinfo and metadata properties. For now it generates metadata with image name, x and y dimension of images, the pixel resolution and the number of bands. ``` usage: geeup getmeta [-h] --input INPUT --metadata METADATA optional arguments: -h, --help show this help message and exit Required named arguments.: --input INPUT Path to the input directory with all raster files --metadata METADATA Full path to export metadata.csv file ``` ### gee upload The script creates an Image Collection from GeoTIFFs in your local directory. By default, the image name in the collection is the same as the local directory name; with the optional parameter you can provide a different name. ``` usage: geeup upload [-h] --source SOURCE --dest DEST -m METADATA [--nodata NODATA] [-u USER] optional arguments: -h, --help show this help message and exit Required named arguments.: --source SOURCE Path to the directory with images for upload. --dest DEST Destination. Full path for upload to Google Earth Engine, e.g. users/pinkiepie/myponycollection -m METADATA, --metadata METADATA Path to CSV with metadata. -u USER, --user USER Google account name (gmail address). Optional named arguments: --nodata NODATA The value to burn into the raster as NoData (missing data) ``` ### gee seltabup This tool allows you to batch download tables/shapefiles to a folder. It uses a modified version of the image upload and a wrapper around the earthengine upload cli to achieve this while creating folders if they don't exist and reporting on assets and checking on uploads. This only requires a source, destination and your ee authenticated email address. This tool also uses selenium to upload the tables. ``` usage: geeup seltabup [-h] --source SOURCE --dest DEST [-u USER] optional arguments: -h, --help show this help message and exit Required named arguments.: --source SOURCE Path to the directory with zipped folder for upload. --dest DEST Destination. Full path for upload to Google Earth Engine, e.g. users/pinkiepie/myponycollection -u USER, --user USER Google account name (gmail address). ``` ### gee selsetup Once in a while the geckodriver requires manual input before signing into the google earth engine, this tool will allow you to interact with the initialization of Google Earth Engine code editor window. It allows the user to specify the account they want to use, and should only be needed once. ```geeup selsetup``` ### gee tasks This script counts all currently running, ready, completed, failed and canceled tasks along with failed tasks. This tool is linked to your google earth engine account with which you initialized the earth engine client. This takes no argument. ``` usage: geeup tasks [-h] optional arguments: -h, --help show this help message and exit ``` ### gee delete The delete is recursive, meaning it will also delete all children assets: images, collections, and folders. Use with caution! ``` usage: geeup delete [-h] id positional arguments: id Full path to asset for deletion. Recursively removes all folders, collections and images. optional arguments: -h, --help show this help message and exit ``` # Changelog ### v0.3.2 - Fixed issue with selsetup. ### v0.3.1 - Fixed issue with raw_input and input for selsetup. - Fixed selenium path for windows and other platforms. - General improvements to ReadMe ### v0.3.0 - Fixed (issue 13)[https://github.com/samapriya/geeup/issues/13] non relative import. - Fixed issues with package import. ### v0.2.9 - Fixed issues caused by --no-use_cloud_api in earthengine-api package ### v0.2.7 - Fix to handle case senstive platform type for all os Fix to [Issue 11](https://github.com/samapriya/geeup/issues/11) ### v0.2.6 - Fixed geckodriver path to handle macos Fix to [Issue 10](https://github.com/samapriya/geeup/issues/10) ### v0.2.5 - Now allows for downloading geckodriver for macos Fix to [Issue 10](https://github.com/samapriya/geeup/issues/10) - Now includes a metadata tool to generate a generalized metadata for any raster to allow upload. Fix to [Issue 7](https://github.com/samapriya/geeup/issues/7) - Changed from geeup update to init to signify initialization - Added selsetup this tool allows for setting up the gecko driver with your account incase there are issues uploading - Better error handling for selenium driver download ### v0.2.4 - Made general improvements - Better error handling for selenium driver download ### v0.2.2 - Can now handle generalized metadata (metadata is now required field) - Fixed issues with table upload - Overall code optimization and handle streaming upload ### v0.1.9 - Changes to handle PyDL installation for Py2 and Py3 - Removed Planet uploader to make tool more generalized ### v0.1.8 - Multipart encoder using requests toolbelt for streaming upload - Changed manifest upload methodology to match changes in earthengine-api ### v0.1.6 - Fixed issue with [module locations](https://github.com/samapriya/geeup/issues/2) ### v0.1.5 - Fixed issue with gecko driver paths - Fixed issue with null uploads using task, switched to ee CLI upload ### v0.1.4 - OS based geckdriver path fix - General improvements ### v0.1.3 - fixed issues with extra arguments - Upload issue resolved - General dependency ### v0.1.1 - fixed dependency issues - Upload post issues resolved - Removed dependency on poster for now ### v0.0.9 - fixed attribution and dependecy issues - Included poster to improve streaming uploads - All uploads now use selenium ### v0.0.8 - fixed issues with unused imports ### v0.0.7 - fixed issues with manifest lib ### v0.0.6 - Detailed quota readout - Uses selenium based uploader to upload images - Avoids issues with python auth for upload ### v0.0.5 - Removed unnecessary library imports - Minor improvements and updated readme ### v0.0.4 - Improved valid table name check before upload - Improvements to earth engine quota tool for more accurate quota and human readable --- ## File: docs/projects/ghome.md # [ghome: Simple CLI for Google Home & Mini](https://github.com/samapriya/ghome) This is an application of the google home API Blueprint reported [here](https://github.com/rithvikvibhu/GHLocalApi). While a lot of the end points maybe of interest to quite a few people, I used the ones I use most and I thought would be most useful and created a command line tool for you to search for a Google home mini device and then use this tool to interact with it and perform actions as needed. ## Table of contents * [Installation](#installation) * [Getting started](#getting-started) * [ghome Simple CLI for Earth Engine Uploads](#ghome-simple-cli-for-earth-engine-uploads) * [list](#list) * [reboot](#reboot) * [alarm](#alarm) * [do not disturb](#do-not-disturb) * [bluetooth status](#bluetooth-status) * [bluetooth scan](#bluetooth-scan) * [bluetooth paired](#bluetooth-paired) * [bluetooth discovery](#bluetooth-discovery) * [wifi scan](#wifi-scan) ## Installation This assumes that you have native python & pip installed in your system, you can test this by going to the terminal (or windows command prompt) and trying ```python``` and then ```pip list``` If you get no errors and you have python 2.7.14 or higher you should be good to go. Please note that I have tested this only on python 2.7.15, but it should run on Python 3. To install **ghome: Simple CLI for Google Home & Mini** you can install using two methods. ```pip install ghome``` or you can also try ``` git clone https://github.com/samapriya/ghome.git cd ghome python setup.py install ``` For Linux use sudo or try ```pip install ghome --user```. Installation is an optional step; the application can also be run directly by executing ghome.py script. The advantage of having it installed is that ghome can be executed as any command line tool. I recommend installation within a virtual environment. If you don't want to install, browse into the ghome folder and try ```python ghome.py``` to get to the same result. ## Getting started As usual, to print help: ``` usage: ghome [-h] {list,reboot,alarm,dnd,bstat,bscan,bpair,bdisc,wscan} ... Simple Google Home Mini Client positional arguments: {list,reboot,alarm,dnd,bstat,bscan,bpair,bdisc,wscan} list Lists all google home mini devices & IP address reboot Reboot a google home mini using IP address alarm Print out the current alarms setup on your google home mini dnd Enable or disable for a google home mini using IP address bstat Print current bluetooth status for a google home mini using IP address bscan Scan for Bluetooth devices near a google home mini using IP address bpair Print current paired bluetooth devices for a google home mini using IP address bdisc Enable or disable bluetooth discovery for a google home mini using IP address wscan Scan for Wifi networks near a google home mini using IP address optional arguments: -h, --help show this help message and exit ``` To obtain help for specific functionality, simply call it with _help_ switch, e.g.: `ghome wscan -h`. If you didn't install ghome, then you can run it just by going to *ghome* directory and running `python ghome.py [arguments go here]` ## ghome Simple CLI for Earth Engine Uploads The tool is based on curret unofficial API blueprint published for the device and is subject to change in the future. ### list **This is a key step since it lists all google home mini devices in your wifi, it used nmap to identify said devices. You can avoid this tool by using a third party tool like [Fing] to identify your google home devices.Usage is simply ``` ghome list``` ### reboot Just a simple tool to reboot your google home device quickly. ``` usage: ghome reboot [-h] [--ip IP] optional arguments: -h, --help show this help message and exit Required named arguments.: --ip IP Use "ip" for Google Home Mini device ``` ### alarm This tool will simply list all alarms currently on your device including date , time and time zone. ``` usage: ghome alarm [-h] [--ip IP] optional arguments: -h, --help show this help message and exit Required named arguments.: --ip IP Use "ip" for Google Home Mini device ``` ### do not disturb Enable or disable do not disturb mode on the Google home mini. ``` usage: ghome dnd [-h] [--ip IP] [--action ACTION] optional arguments: -h, --help show this help message and exit Required named arguments.: --ip IP Use "ip" for Google Home Mini device --action ACTION enable|disable do not disturb mode ``` ### bluetooth status The bluetooth status prints whether the device discovery is enabled,whether scanning is enabled and whether it is connected to a device. ``` usage: ghome bstat [-h] [--ip IP] optional arguments: -h, --help show this help message and exit Required named arguments.: --ip IP Use "ip" for Google Home Mini device ``` ### bluetooth scan This prints bluetooth status, including is possible bluetooth device name and mac name ``` usage: ghome bstat [-h] [--ip IP] optional arguments: -h, --help show this help message and exit Required named arguments.: --ip IP Use "ip" for Google Home Mini device ``` ### bluetooth paired Check if device is paried via bluetooth to any current device as well as history of all deviced connected, last connected and whether or not they are currently connected. ``` usage: ghome bpair [-h] [--ip IP] optional arguments: -h, --help show this help message and exit Required named arguments.: --ip IP Use "ip" for Google Home Mini device ``` ### bluetooth discovery This is to enable or disable bluetooth discovery to allow for pairing as needed. The action can be enable or disable coupled with the ip. ``` usage: ghome bdisc [-h] [--ip IP] [--action ACTION] optional arguments: -h, --help show this help message and exit Required named arguments.: --ip IP Use "ip" for Google Home Mini device --action ACTION enable|disable bluetooth discovery ``` ### wifi scan Prints the wifi scan results of all available wifi connections and their ssid based on the proximity of the device to other connections. You might want to run this twice, since it may not build the device cache directly. ``` usage: ghome wscan [-h] [--ip IP] optional arguments: -h, --help show this help message and exit Required named arguments.: --ip IP Use "ip" for Google Home Mini device ```