AI-driven DataOps pipeline for real-time skills extraction in Europe

Descrizione

Context & overview

In a rapidly changing labour market, conventional forecasting methods often fail to capture emerging needs in Europe. To address this, institutions in some European countries launched a collaborative initiative connecting employment agencies, universities and data science centres. The goal was to design an AI-driven pipeline able to detect, extract and classify skills in real time useful for training providers and policymakers.

 

Challenge addressed

Persistent mismatch between supply and demand of skills.

Vast volumes of unstructured data difficult to understand with traditional tools.

Slow feedback cycles, as policy reports often describe yesterday’s needs instead of the ones of tomorrow’s.

Fragmentation of classifications (ESCO, national systems) which limits cross-border comparability.

 

Solution implemented

A DataOps pipeline was implementing. It integrates data engineering and machine learning. The major steps included an integration of heterogeneous data such as vacancy portals and CV databases, a machine learning application to identify both technical and transversal skills and the mapping of competences extracted from European tools like ESCO.

 

Impact & results

Accuracy

Skill extraction achieved over 80% accuracy and above 94% recall.

Efficiency

Automated processing reduced analysis time from days of manual coding to minutes through the use of AI.

AI’s support

AI provide powerful signals, but expert interpretation remains essential.

Link di riferimento

https://arxiv.org/abs/2104.01966?utm_

Parole chiave

Skills extraction, labour market intelligence, DataOps pipeline, machine learning, ESCO integration

×

Partner

Il sostegno della Commissione europea alla realizzazione di questa pubblicazione non costituisce un'approvazione dei contenuti, che riflettono esclusivamente il punto di vista degli autori, e la Commissione non può essere ritenuta responsabile per qualsiasi uso che possa essere fatto delle informazioni in essa contenute.
I materiali pubblicati sul sito web del progetto AIVET sono classificati come “risorse educative aperte” (OER) e possono essere liberamente (senza l'autorizzazione dei loro creatori) scaricati, utilizzati, riutilizzati, copiati, adattati e condivisi dagli utenti, previo citazione della loro origine.
Reading assistant
Listen to Olena