: Scalable Data Discovery Using Profiles

NextiaJD is a system that supports data discovery over data lakes (i.e., large scale heterogeneous data repositories). This website is a companion of the research and demonstration papers submitted to EDBT 2021, where we present the method underlying our approach. NextiaJD's novelty lies on a learning-based approach to data discovery relying on dataset profiles. These are succinct representations that capture the underlying characteristics of the schemata and data values of datasets, which can be efficiently extracted in a parallel and distributed fashion. Profiles are then compared, to predict the quality of a join operation among a pair of attributes from different datasets. NextiaJD is implemented as an extension of Apache Spark.






Software repository

The source code of the system can be found in the following Github repository.

The easy way to use NextiaJD is with Maven. For SBT just add the following dependency in your build.sbt

For more ways to add NextiaJD using Maven, please go here

You can check how to use NextiaJD here or see the zeppelin notebook with an explanation step by step, see demonstration section

Ground truth datasets

Datasets used in this work have been obtained from open data repositories with no copyright such as Kaggle and OpenML. The datasets used to generate both our ground truth and to evaluate our method are available in the following links:


We believe in transparent and shareable research [1], [2]. Hence, we provide you with detailed instructions on how to reproduce the experiments presented in our work:


Live demo

Live demos of NextiaJD are available as Zeppelin notebooks. Bear in mind that, in order to access them you must first login with the following credentials (user: user1, password: nextiajd).

Interactive GUI

A live demo of the user interface is available here.

Library usage

We also provide with a code-oriented demo showcasing how proficient data analysts can take full benefit of our tool here.


Last update: 2021/03/14 by Javier Flores