Description

HORIZON-CL4-2022-DATA-01-01

Extreme data characteristics (volume, speed, heterogeneity, diverse quality, etc.) challenge the state-of-the-art datadriven analytics and decision-making approaches and technologies. At the same time, data-driven outcomes need to be extremely accurate, precise, fit-for-purpose, and trustworthy, so that they can be useful. ExtremeXP aims to handle the complexity of matching such extreme needs with extreme characteristics by relying on user intents and running experiments (i.e., trial and error) to prune the vast solution space of possible analytics. As a result, ExtremeXP puts the end user at the centre of complex analytics processes (i.e., processes that involve and combine ML, data analysis, and simulation components) and corresponding visualizations. An end user is a domain expert (e.g., first respondent in disasters situations, transportation planner, maintainer of industrial plant) that does not necessarily possess advanced technical skills (e.g., on database management, machine learning, or database querying languages such as SQL), but has a vested interest in the outcome of complex analytics for supporting decision making in their domain.

ExtremeXP’s main goal is to create a next generation decision support system that integrates novel research results from the domains of data integration, machine learning, visual analytics, explainable AI, decentralised trust, knowledge engineering, and model-driven engineering into a common framework. The main, overarching idea of the ExtremeXP framework is to optimise the properties of a complex analytics process that the end user cares about (e.g., accuracy, time-to-answer, specificity, recall, precision, resource consumption) by associating user profiles to computation variants.

A main novelty is that such association is not known a priori but learned by consecutive interactions between (different) end users and the ExtremeXP framework. An interaction starts by the end user providing an intent – a knowledge requirement, a required outcome – together with potential preferences and constraints. For instance, an end user may need to decide on upgrading a specific equipment, in which case it is of utmost importance to “predict the mean time to repair” for a certain machine in a supply chain and indicate that accuracy should be maximised (preference), while computation time should not be more than 10 minutes (constraint). Extra information about the user profile and context is also provided. The framework then examines the alternative ways to satisfy this intent, e.g., alternative datasets, ML algorithms and models, deployment options. When presenting the results, the framework also examines whether there are alternative visualization options (different graphs, different slices on the results). In both cases, the framework decides on one of them based on the knowledge it has so far on the alternatives and the profile and context of the end user having the intent. For instance, the framework may use the rule that “always (i.e., for every user profile and context) use decision trees when predicting mean time to repair”, if it knows such a rule. In general, however, such rules are very difficult to be known beforehand, especially when dealing with dynamic settings, when the available datasets, algorithms, deployment options, and user intents change.

To address this challenge, ExtremeXP proposes to learn such rules by performing experiments that evaluate different complex analytics variants, like comparing different versions of a product (e.g., a website) via exposing both versions to real users and performing A/B testing on key retention metrics. Following this novel idea, in the presence of several variants without an association rule to select among them, the ExtremeXP framework allocates a variant to a particular user interaction, schedules the corresponding analytics process (or uses the corresponding visualisation), and measures both system (e.g., outcome accuracy, resources used, runtime) and user feedback metrics.

In summary, a user interaction starts by providing an intent, together with possible preferences and constraints, and ends by providing feedback on the usefulness, clarity, and trust (which we aim to increase using appropriate explainable AI techniques) on the outcomes. The ExtremeXP framework gradually builds up knowledge (as association rules with precise semantics) via multiple user interactions (belonging to the several intents specific to each use case). Crucially, it keeps this knowledge continuously updated, since it can always create and run new experiments when e.g., the users and the complex analytics capabilities change. Such automation is crucial for the vision of a self-improving decision support system that ExtremeXP aims to provide.