Python API Improvements

[arjxn-py]

Considering this issue as a Parent issue to open sub-issues and keep track for JupyterGIS Python API Improvements.

[brookisme]

WORKFLOW IDEAS

---

VECTOR

The idea is to make a JGIS "layer", "collection" "feature" python objects. The main
interaction is covered in the next section, but here is how I'm imagining it working
under-the-hood

class JGISLayer:
	"""
	- translates jgis-style-json to a python object with various methods
	- some methods
		- `.collection()`: returns a `JGISCollection` object as described below
		- `.to_json()`: which translates it back to jgis-style-json
	- note: maybe `JGISCollection` and `JGISLayer` could be one class. I'm thinking
			* the "layer": is being the thing you grab and references what is on the map.
			* the "collection": being what you sort/filter/maipulate to create a new collection, from which (if you want) you can create a new layer, and add to the map
	"""
	pass

class JGISCollection:
	"""
	this is something like a geodataframe that would allow you to select feature(s), filter/remove/add features...

	- some methods:
		- '[index_value]': select a object by index (list like)  (returns JGISFeature)
		- '.get_feature(name)': select an feature by feature-name (returns JGISFeature)
		- '.add_feature(feat)': adds feature to collection
		- '.update_feature(feat_name, feat)': updates existing feature
		- '.get_features(kwargs)': select an features by properties  (returns JGISCollection) (ex `collection.get_features(height=10, height_op='>', state='CA')`)
		- '.get_selected(kwargs)': features selected on map (returns JGISCollection)
		- '.to_gdf()': create geodataframe from collection
		- 'JGISCollection.from_gdf(gdf)': create collection from geodataframe
		- `.to_layer(...)`: creates (in memory) JGISLayer that can be added to the `doc` instance

		Note: Some common manipuations could be added like `union()`, or `intersection(). However it could be managed through `to_gdf/JGISCollection.from_gdf(gdf)`

		# example on JGISCollection instance
		feats = feats.union()

		# example using gdf
		gdf = feats.to_gdf()
		gdf = buffer_gdf_method(gdf)
		feats = JGISCollection.from_gdf(gdf)
	"""
	pass

class JGISFeature:
	"""
	This is where you manipulate individual features

	- some methods:
		- '.get_shape()': returns SHP object for the underlying geometry
		- '.update_shape(shp)': updates SHP object for the underlying geometry
		- '.properties': returns feature properties as dictionary
		- '.set(**kwargs)': updates properties
		- `.to_json()`: which translates it back to jgis-style-json

	Note: Some common manipuations could be added like `buffer()`, `fillholes()`. However it could be managed through `get_shape`
	(at least) initially.

	# example on JGISFeature instance
	feat = feat.buffer(10)

	# example using shp
	shp = feat.get_shape()
	shp = shp.buffer(10)
	feat = feat.update_shape(shp)
	"""
	pass

Layer, Collection, and Feature objects

Get JGIS

# get JGIS "layer" object
layer = doc.get_layer(layer_name)

# get JGIS "feature" object
feat = layer.get_feature(feat_name)

# - OR - get JGIS "collection" (list-like) of JGIS "feature" objects
feats = layer.get_features(height=10, height_op='>', state='CA')

# - OR - get JGIS "collection" (list-like) of JGIS "feature" objects from selected-features
feats = layer.get_selected()

Example

# 1. get layer
# 2. perform manipuations on collection
# 3. create layer from resulting collection
# 4. add/remove layers to map/doc
layer = doc.get_layer(layer_name)
feats = layer.get_features(height=10, height_op='>', state='CA')
feats = feats.buffer(10)
feats = feats.union()
feats = feats.fillholes()
new_layer = feats.to_layer(name='NewLayer', z=1, color='#3FA')
doc.add_layer(new_layer)
doc.remove_layer(layer.name)

---

RASTER

Rasters will be fundamentally different. Its going to be harder to manipulate and create in memory rasters, that could then be added back to the map as layers. Note it should be possible - creating numpy arrays etc, but for instance globaly rasters would bog everything down, if not completely crash the system (probably the later).

Three things I think should be possible (in order assumed of difficulty):

1. Manipulate properties like layer-name/opacity - I'm assuming this is already possible.
2. Create a method that transforms a tile layer to displayed altered tiles. Something like:

def normalize_difference(arr, b1, b2):
	return (arr[b1] - arr[b2])/(arr[b1] + arr[b2])

def tile_transformer(x, y, z):
	arr, meta = ... get the tile data (arr is a np.array, meta maybe isnt there but is some dictionary maybe...)
	new_arr = normalize_difference(arr, 0, 1)
	return new_arr  # or maybe new_arr, meta

raster_layer = doc.get_layer(layer_name)
transformed_layer = raster_layer.transform(tile_transformer)
doc.add_layer(transformed_layer)

3. Compute with Features:

I am imagining a worflow like (somewhat inspired by GEE)

# `vector_layer` is a bunch of points
# `raster_layer` is a raster with at least 2 bands
vector_layer = doc.get_layer(layer_name)
feats = vector_layer.get_features(height=10, height_op='>', state='CA')
feats = feats.buffer(10)

raster_layer = doc.get_layer(raster_layer_name)

def add_mean_ndvi(feat):
	arr = raster_layer.get_pixels(feat.geometry(), z=ZLEVEL)
	feat = feat.set('mean_ndvi', arr.mean())
	return feat

feats = feats.map(add_mean_ndvi)
ndvi_layer = feats.to_layer(name='NewLayer', z=1, color='#3FA')
doc.add_layer(ndvi_layer)

[mfisher87]

Wow, amazing detail, Brookie! I think we can break off a couple new issues from this.

[brookisme]

A bit of a side note but...

I'm wondering how collaboration will work. I feel like maybe real-time-sharing of manipulations of in-memory vector data is already possible in some sense. I'm thinking of the JupyterCad stuff where someone draws a shape and someone else cuts out the middle. It seem like once you call doc.add_layer() there should be something that someone else can play with.

Persistence is a related question. And maybe this is already being done, but if I'm collaborating with someone else how do we save it so that both me an my collaborators have access to the same file (rather than each our own copy). Can we have local config files with keys/details to save to various cloud solutions. So for instance if me and my collaborators setup (and have permissions) to use a gcs bucket. Could my collaborator click a "save (remote)" button. And I in a different session just open from the remote (in this case GCP) file?

[mfisher87]

Thinking more about reproducibility.

layer = doc.get_layer(layer_name)
feats = layer.get_features(height=10, height_op='>', state='CA')

This will only be reproducible if the doc stays the same over time. Perhaps instead we should try for a one-click experience to both export data from the UI and generate code to open that exported data.

@GondekNP suggested that we could record a manifest of vector editing actions taken in the UI and generate code to reproduce those actions. Say for example in a future version of JupyterGIS, I open a vector dataset, simplify the geometry in the GUI, remove a feature I'm not interested in. Then I do layer = doc.get_layer(...); feats = layer.get_features(...). This code depends on the point-and-click actions I took in the GUI. If instead we recorded the point-and-click actions and generated the equivalent code, you could make a reproducible notebook.

Example of generating code from a GUI operation in ArcGIS: https://www.youtube.com/watch?v=sCkVI4VHdXo