AI-driven DataOps pipeline for real-time skills extraction in Europe

Descripción

Context & overview

In a rapidly changing labour market, conventional forecasting methods often fail to capture emerging needs in Europe. To address this, institutions in some European countries launched a collaborative initiative connecting employment agencies, universities and data science centres. The goal was to design an AI-driven pipeline able to detect, extract and classify skills in real time useful for training providers and policymakers.

 

Challenge addressed

Persistent mismatch between supply and demand of skills.

Vast volumes of unstructured data difficult to understand with traditional tools.

Slow feedback cycles, as policy reports often describe yesterday’s needs instead of the ones of tomorrow’s.

Fragmentation of classifications (ESCO, national systems) which limits cross-border comparability.

 

Solution implemented

A DataOps pipeline was implementing. It integrates data engineering and machine learning. The major steps included an integration of heterogeneous data such as vacancy portals and CV databases, a machine learning application to identify both technical and transversal skills and the mapping of competences extracted from European tools like ESCO.

 

Impact & results

Accuracy

Skill extraction achieved over 80% accuracy and above 94% recall.

Efficiency

Automated processing reduced analysis time from days of manual coding to minutes through the use of AI.

AI’s support

AI provide powerful signals, but expert interpretation remains essential.

Enlace de referencia

https://arxiv.org/abs/2104.01966?utm_

Palabras clave

Skills extraction, labour market intelligence, DataOps pipeline, machine learning, ESCO integration

×

Socios

El apoyo de la Comisión Europea a la producción de esta publicación no implica la aprobación de su contenido, que refleja únicamente la opinión de sus autores, y la Comisión no se responsabiliza del uso que pueda hacerse de la información contenida en ella.
Los materiales publicados en el sitio web del proyecto AIVET están clasificados como «Recursos Educativos Abiertos» (REA) y los usuarios pueden descargarlos, utilizarlos, reutilizarlos, copiarlos, adaptarlos y compartirlos libremente (sin autorización de sus creadores), con información sobre su origen.
Reading assistant
Listen to Olena