Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GTiff: MultiThreadedRead(): make it take into account AdviseRead() limit to reduce the number of I/O requests #9694

Merged
merged 3 commits into from
Apr 19, 2024

Conversation

rouault
Copy link
Member

@rouault rouault commented Apr 18, 2024

Fixes #9682

With those changes, the number of GET requests for #9682 use case is down to 6:

$ time GDAL_NUM_THREADS=4 CPL_DEBUG=ON python -c "from osgeo import gdal; gdal.UseExceptions(); src=gdal.Open('/vsicurl/http://localhost:8080/ortho.tiff'); src.ReadAsArray()"
HTTP: libcurl/7.68.0 GnuTLS/3.6.13 zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
VSICURL: GetFileSize(http://localhost:8080/ortho.tiff)=279686928  response_code=200
VSICURL: Downloading 0-16383 (http://localhost:8080/ortho.tiff)...
VSICURL: Got response_code=206
VSICURL: Downloading 32768-49151 (http://localhost:8080/ortho.tiff)...
VSICURL: Got response_code=206
GTiff: Using up to 4 threads for compression/decompression
GDAL: GDALOpen(/vsicurl/http://localhost:8080/ortho.tiff, this=0x2ec3e50) succeeds as GTiff.
VSICURL: Downloading 16384-32767 (http://localhost:8080/ortho.tiff)...
VSICURL: Got response_code=206
GTiff: Splitting request (0,0,7255x6234) into (0,0,7255x3072) and (0,3072,7255x3162)
VSICURL: AdviseRead(): fetching 1 ranges
VSICURL: Downloading 70597872-156443454 (http://localhost:8080/ortho.tiff)...
GDAL: GDAL_CACHEMAX = 1587 MB
GTiff: Splitting request (0,3072,7255x3162) into (0,3072,7255x1536) and (0,4608,7255x1626)
VSICURL: AdviseRead(): fetching 1 ranges
VSICURL: Downloading 156443463-234346925 (http://localhost:8080/ortho.tiff)...
VSICURL: AdviseRead(): fetching 1 ranges
VSICURL: Downloading 234346934-279686923 (http://localhost:8080/ortho.tiff)...
GDAL: GDALClose(/vsicurl/http://localhost:8080/ortho.tiff, this=0x2ec3e50)

real	0m2,144s
user	0m3,233s
sys	0m3,385s

@rouault rouault added this to the 3.9.0 milestone Apr 18, 2024
@coveralls
Copy link
Collaborator

Coverage Status

coverage: 69.026% (+0.001%) from 69.025%
when pulling 5b948d1 on rouault:fix_9682
into eff42b8 on OSGeo:master.

@rouault rouault merged commit ba413cc into OSGeo:master Apr 19, 2024
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Suspected regression in /vsicurl
2 participants