e-Research Summer Hackfest: The Ophidia platform

Alessandro D'Anca

05 July 2016

Abstract: In thispresentation we introduce and present the Ophidia project, which is a research effort on big data analytics facing scientific data analysis challenges in the climate change domain. Ophidia provides declarative, server-side, and parallel data analysis, jointly with an internal storage model able to efficiently deal with multidimensional data and a hierarchical data organization to manage large data volumes (“datacubes”). The project relies on a strong background on high performance database management and OLAP systems to manage large scientific datasets. The Ophidia analytics platform provides several data operators to manipulate datacubes, and array-based primitives to perform data analysis on large scientific data arrays. Metadata management support is also provided. The server front-end exposes several interfaces to address interoperability requirements: WS-I+, GSI/VOMS and OGC-WPS (through PyWPS). From a programmatic point of view a Python module (PyOphidia) makes straightforward the integration of Ophidia into Python-based environments and applications (e.g. IPython). The system offers a CLI (e.g. bash-like) with a complete set of commands. A key point of the video is the description of workflow capabilities offered by Ophidia. In this regard, the framework stack includes an internal workflow management system, which coordinates, orchestrates, and optimises the execution of multiple scientific data analytics & visualization tasks. Specific macros are also available to implement loops, or to parallelize them in case of data independence. Real-time workflow monitoring execution is also supported through a graphical user interface. Some real workflows implemented at CMCC and related to different EU projects are also presented. e-Research Summer Hackfest (http://www.sci-gaia.eu/summer-hackfest).

Keyword(s): Sci-GaIA ; INDIGO-DataCloud ; e-Research Summer Hackfest ; Ophidia ; Big Data Analytics
Identifier(s): 10.15169/sci-gaia:1469699844.55

Licence: cc-by-4.0

The record appears in these collections:
Lessons > Lessons Sci-GaIA

 Record created 2016-07-28, last modified 2016-07-28

Download Resource

Rate this document:

Rate this document:
(Not yet reviewed)