Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix 567: task graph processing #604

Merged
merged 44 commits into from
Dec 5, 2023
Merged

fix 567: task graph processing #604

merged 44 commits into from
Dec 5, 2023

Conversation

ungarj
Copy link
Owner

@ungarj ungarj commented Nov 16, 2023 •
edited
Loading

fixes #567

  • complete refactor of mapchete.commands
  • remove mapchete.Job and use observer pattern to track execution state and progress
  • refactor mapchete.processing
  • replace mapchete.processing.compute with mapchete.processing.execute

@coveralls
Copy link

coveralls commented Nov 16, 2023 •
edited
Loading

Coverage Status

coverage: 100.0%. remained the same
when pulling f282d61 on 567_task_graph_processing
into e56b027 on main.

ungarj and others added 23 commits November 16, 2023 11:51
@ungarj
clean up
d161743
@ungarj
add pretty_bytes function
4ef22d5
@ungarj
add TaskBatches class to rule them all
a4774ee
@ungarj
dev commit
17f4d70
@ungarj
Release 2023.11.0
a882ef1
@ungarj
remove mapchete.Job class and replace with observer pattern to get ex…
4da4c1d 
…ecution state and progress
@ungarj
remove mapchete.processing.compute
95275bd
@ungarj
remove batch_process and batch_processor methods
1e450ae
@ungarj
remove Mapchete.compute() and all calls from test suite
b22cb8d
@ungarj
handle single file outputs
1966264
@ungarj
change deprecated kwarg
ef83fa9
@ungarj
add DaskSettings configuration model to combine multiple dask specifi…
618d3c1 
…c settings
@ungarj
pass on preprocessing task results to following task batches
3ec8a32
@ungarj
share wrapper functions between task graph processing and other metho…
2022ce5 
…ds; combine test functions
@ungarj
unify task wrapper functions and improve tests
df45a78
@ungarj
remember preprocessing tasks optionally
46928b9
@ungarj
handle task dependencies; Executor initialized with 1 worker will not…
8453b61 
… automatically return a SequentialExecutor
@ungarj
remove task skip
8ff334e
@ungarj
fix batches generator for test
97baedf
@ungarj
fix DaskSettings in commands and cli
0885b87
@ungarj
update documentation
1448d44
@ungarj
doc build: add dask & fsspec to mock imports
2488378
@ungarj
Merge branch 'main' into 567_task_graph_processing
6ef6f76
ungarj added 15 commits November 29, 2023 14:51
@ungarj
avoid aiobotocore 2.8.0
6753ac8
@ungarj
update docs
67da5e2
@ungarj
improve tests
8d19d52
@ungarj
increase test coverage
589e75a
@ungarj
speed up slowest tests
b64b3d2
@ungarj
adapt to recent changes to fsspec.AbstractFileSystem.protocol
4e0fbff
@ungarj
required fsspec 2023.12.0 but release dependency restriction of aiobo…
c046096 
…tocore
@ungarj
deactivate GDAL VSI cache for test
9e39718
@ungarj
refactor task batch generation
d78a532
@ungarj
analyze tasks and only use preprocessing tasks intersecting with actu…
743fffe 
…al process_aoi; add MapcheteConfig.overview_levels and MapcheteConfig.processing_levels; clean up dead code
@ungarj
use ProcessingConfig enum where possible; increase test coverage
6f560e0
@ungarj
simplify Tasks class
16cd59d
@ungarj
make sure no tasks are returned when out of bounds; add more typing
6663ddc
@ungarj
allow kwargs
3f91dc3
@ungarj
fix index generation
f282d61
@ungarj ungarj merged commit 45d42c3 into main Dec 5, 2023
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

decide intelligently whether to use task graph processing
2 participants
@ungarj @coveralls