<div class="csl-bib-body">
<div class="csl-entry">Reimer, C., Weidenholzer, L., Hoyal, S., Raml, B., Schobben, M., Schramm, M., Wagner, W., & Briese, C. (2023). Multi-cloud processing with Dask: Demonstrating the capabilities of DestinE Data Lake (DEDL). In P. Soille, S. Lumnitz, & S. Albani (Eds.), <i>Proceedings of the 2023 conference on Big Data from Space (BiDS’23) : From foresight to impact</i> (pp. 161–164). Publications of the European Union. https://doi.org/10.34726/5310</div>
</div>
-
dc.identifier.uri
http://hdl.handle.net/20.500.12708/190731
-
dc.identifier.uri
https://doi.org/10.34726/5310
-
dc.description.abstract
The DestinE Data Lake (DEDL) is one of the three key system components of the Destination Earth (DestinE) initiative of the European Commission. It is in the responsibility of the DEDL to provide seamless data access as well as support for big data processing services to DestinE users. One of the big data processing services offered via DEDL is based on the Dask ecosystem. The DEDL Dask service allows for multi-cloud processing, following the paradigm of data proximate computation. Digital Twins of the Earth will run on HPC system across Europe, producing petabytes of data per day over the next few years. Accordingly, enabling users to do processing close to the data is key to cope with these volumes of data in an efficient way. Those capabilities of the DEDL are presented by two use case demonstrations based on Pangeo core packages such as xarray and Dask.
en
dc.language.iso
en
-
dc.rights.uri
http://creativecommons.org/licenses/by/4.0/
-
dc.subject
Destination Earth
en
dc.subject
DestinE Data Lake
en
dc.subject
Big Data Processing
en
dc.subject
Pangeo
en
dc.subject
Dask
en
dc.subject
STAC
en
dc.subject
Flood
en
dc.subject
Drought
en
dc.title
Multi-cloud processing with Dask: Demonstrating the capabilities of DestinE Data Lake (DEDL)
en
dc.type
Inproceedings
en
dc.type
Konferenzbeitrag
de
dc.rights.license
Creative Commons Namensnennung 4.0 International
de
dc.rights.license
Creative Commons Attribution 4.0 International
en
dc.identifier.doi
10.34726/5310
-
dc.contributor.affiliation
EODC Earth Observation Data Centre for Water Resources Monitoring GmbH, Austria
-
dc.contributor.affiliation
EODC Earth Observation Data Centre for Water Resources Monitoring GmbH, Austria
-
dc.contributor.affiliation
EODC Earth Observation Data Centre for Water Resources Monitoring GmbH, Austria
-
dc.contributor.affiliation
EODC Earth Observation Data Centre for Water Resources Monitoring GmbH, Austria
