Personalized feedback service for national qualification test result – Digital Mentor Service of National qualification (DMS)

"Why did I fail? How should I develop my career?" To address such concerns, we developed the Digital Mentor Service of National Qualification. It leverages massive data on national qualification tests to identify each individual's weaknesses and why they fail. It also provides personalized information by matching examination history with jobs and training offered by government agencies. Supporting test takers from skills assessment and development to job placement.

Innovation Summary

Innovation Overview

The National Technical Qualification is a system through which the Korean government recognizes an individual's vocational ability. Candidates can improve their skills while preparing for the qualification, which can be used for employment, starting a business, and further education. In Korea, more than 3.5 million people participate in national technical qualifications every year, and about 700,000 have been certified. However, after taking an exam, examinees were only told whether they passed or not and had no clue how much they scored in each subject and where they should work on more to obtain the qualification. In addition, even after getting qualified, the rest was up to the individual, everything from finding a job and getting more information on additional training and education to improving their skills. Not much information was given, such as what kind of company would hire them, how much salary they could get, and what kind of additional training was available. Although 3.5 million people participate in the exams every year, and a total of 30 million have been accredited, the amount and quality of the information provided to test takers fell way below expectations.

Recently, the Korean government has been taking an active approach towards accelerating digital transformation and opening and sharing public data to stay ahead of the 4th industrial revolution. In response to such government policies, the HRDKorea launched the "Digital Mentor Service of National Qualification" to meet the needs for quality information and ultimately improve the country's vocational skills. The Digital Mentor Service of National Qualification provides the following services to participants of national technical qualification tests.

Analysis and diagnosis of test results (pass/fail, scores by subject and the average score of passers, weaknesses);
Relevant information on qualifications (qualifications for which test takers may be exempted from the written test if they acquired a certificate in the same career field, other exams frequently taken by those who were qualified for the same skill, etc.);
Timelines (application and exam schedule);
Information on vocations, jobs, education and training (training institutions, offerings, jobs related to the qualifications the candidates applied for, future job prospects, job postings, current status of workers in relevant fields, wages).

Through these services, test takers can:

Assess their level of achievement and
Identify the skills they lack and areas they need to work more on.
Those who have passed will be able to prepare for further skills development by obtaining information on relevant qualifications. Even those who failed would get a stronger motivation to acquire qualifications.
Individuals can benefit from the information on jobs, training, and employment related to qualifications as it helps them establish career development plans and saves the time and effort required to search for information.
Unsuccessful candidates will again be motivated to retry as the service helps them better prepare for the next test by diagnosing their weaknesses.

The Digital Mentor Service of National Qualification was launched in 2021 to meet the needs of test takers eager to know why they failed and learn more about career development and job opportunities that the qualification would provide. Also, we adopted the Agile development methodology. The service was introduced as a pilot program incorporating the most essential features and will go through a three-step plan of gradual improvement backed by user feedback. In 2021, the DMS served 14,094 people who tested for five qualification subjects, and as of 2022, it is already supporting job seekers, with 2,784 of the 14,904 users having landed new jobs (20% got a new job within 12 months of service use). The results of the first user survey showed a high satisfaction score of 4.3 out of 5, and we are continuously collecting user feedback and suggestions and applying them to the service model. The Digital Mentor Service of National Qualification (DMS) is a one-stop, comprehensive HRD service tool for test takers, which saved KRW 350 million by reducing the time it took them to search for information by offering information on qualifications, occupation, and jobs in one place (20,464 annual expected users X 2 hours - time saved - X KRW 9,160 - minimum hourly wage). It is also expected to improve the vocational abilities of the test takers, who in turn will contribute to national competitiveness.

The Digital Mentor Service of National Qualification has been recognized by the Korean government for its innovativeness and has secured a budget (KRW 3.9 billion) for future expansion. Hopefully, following the three-steps roadmap, it will establish itself as a service that supports the improvement of people's vocational competency by providing useful information.

Innovation Description

What Makes Your Project Innovative?

The Digital Mentor Service of National Qualification (DMS) is innovative for the following reasons:

A Paradigm Shift in National Qualification Services: Going beyond simply reporting test results as pass or fail, the DMS sees qualifications as the beginning of the journey. It’s designed to user in a paradigm shift by integrating skills qualification, education and training, and job placement assistance, driving the scalability and effectiveness of public services.
Comprehensive HRD Service: The DMS is a comprehensive service encompassing skills assessment, skills development, and employment support. In addition to encouraging further skills development, it provides analysis and diagnosis results for those who failed the exam to help them do better next time.
Convenience: The DMS integrates useful information scattered across different employment and labor services and big data, and provides personalized feedback so that users can easily access quality services.

What is the current status of your innovation?

Background: A 2021 study on civil complaints showed that the examinees of national qualification exams wanted to know why they failed. A solution to drive innovation by actively opening public data was sought after.
Approach: A pilot for the Digital Mentor Service of National Qualification was introduced in 2021. A cross-functional collaborative system was formed between industry, academia, and research institutes. A pilot service was developed with essential functionalities through an agile development strategy.
Project: The project is about processing data on employment and labor scattered across different services and integrating them into our service platform Q-Net. The data is analyzed to provide test takers with personalized feedback, including their improvement areas and information on training and jobs.
Current State: The pilot for five test qualification ended in June 2022. The scope of service will gradually expand per the Information Strategy Planning roadmap.

Innovation Development

Collaborations & Partnerships

The Smart Employment Service Forum, joined by businesses, scholars, and researchers, assisted the project in planning, advice, and review. An MOU was signed with the Korea Employment Information Service for big data services. HRDKorea received help with big data and improved service quality through service process and model validation and user surveys. DMS is not just a service designed and led by the state, but an innovative service enabled through the participation of businesses and academia.

Users, Stakeholders & Beneficiaries

User: DMS encourages further development and supports those who pass qualification exams. Those who fail will get training guidance and personalized feedback on weaknesses.
Stakeholder: Educational institutions minimize the mismatch between the supply and demand of training by offering courses on weaknesses. Companies will benefit as their job postings will be shown to qualified job-seekers.
Beneficiary: Improvement of vocational abilities, industrial and national competitiveness.

Innovation Reflections

Results, Outcomes & Impacts

Service innovation: The DMS is the first big data-based HRD service that analyzes the reason why examinees fail national qualification exams (internal development saved development costs by $78,500). The pilot provided high-quality services to 14,094 examinees of five test qualifications (User satisfaction: 4.3/5 points)
Job search: Out of 14,094 pilot users, 2,784 found a new job after using the DMS for qualification, training, and job search (20% found a new job within 12 months of using the service).
Improved convenience: The one-stop comprehensive HRD service reduces annual social costs wasted due to fragmented services by KRW 350m (20,464 people X 2 hours (time saved) X KRW 9,160 (minimum wage).
Expectations: It is expected that the service will serve 3.5m people a year once the scope includes all subjects. Its effectiveness will continue to grow thanks to service improvement through industry-university-research collaboration.

Challenges and Failures

System expansion: The lack of IT infrastructure created difficulties in service expansion. In response, we established an information strategy plan to smoothly provide services amid the accelerating digital transformation. We secured a budget of $2.7 million from the government, which allowed us to build IT infrastructure and stably expand the service.
Data management process: The data needed for the service had to be managed around the National Competency Standards(NCS) because qualification and training programs are designed and operated based on the NCS. So, we had to change our business processes into NCS-based data management process, which improved the quality of the information provided through the DMS. The 4th industrial revolution and digital transformation posed obstacles to infrastructure expansion and business transformation. However, we could overcome such difficulties and innovate thanks to the collaborative system, complementary partnerships, and an agile strategy.

Conditions for Success

While we did not have the resources and infrastructure necessary for the smooth expansion of services, the HRDKorea carried out tasks efficiently thanks to the industry-university-research collaboration with experts specialized in employment and labor and accurate identification of user needs.

The first success factor was "effective project management." We were able to develop a service model on time because we adopted the "Agile" method, which shortened the time and reduced the costs required to introduce a new service compared to the traditional, time-consuming, and costly methods.
The second success factor was "active user engagement." One good example is that 2,784 out of 14,094 users found a new job through our integrated service of qualification, training, and job search support. We were able to continuously expand and improve our services thanks to the active use of our customers.

Replication

All countries have their own qualification system and services. However, what makes our Digital Mentor Service of National Qualification special is that it seeks a paradigm shift by providing personalized feedback and information on various employment and labor services in one spot previously scattered across different functions and services. In order to respond to the changes facing the future labor market, such as the 4th industrial revolution and digital transformation, it has become more important than ever to develop the vocational skills of the national workforce. Against this background, we believe sharing our know-how with the idea that obtaining a qualification is “just the beginning, not the end of competency development” will help different national qualification systems and HRD services to build on what they have now and achieve even more.

Lessons Learned

Concerns often raised by test takers include: “I want to know why I failed,” “How can I develop my career after passing the exam?” “Where can I get a job if I pass the exam?” HRDKorea, the 40-year-old organization, has gained expertise from providing national qualification services and understands the difficulties experienced by test takers and the requirements for national qualification services better than anyone else. Such expertise in qualification services and an accurate understanding of customer needs were behind our success.

We sought to improve the national qualification service through the DMS by taking advantage of changes that were perceived as uncertainties in the labor market, such as the 4th industrial revolution and digital transformation. HRDKorea will continue to build the vocational skills of the Korean people and continuously enhance the competitiveness of businesses and the Republic of Korea through continuous efforts to innovate public services.

Anything Else?

The plan to expand the 'National Qualified Digital Mentor Service' is as follows.

Step 1: Trial service phase (2021~2022): Establish service model and start service for 5 national technical qualifications. Investigate the satisfaction of beneficiaries with the service and collect opinions for improvement. Securing human and material resources necessary for service expansion.
Step 2: Service expansion (2023~2024): We improve the service by collecting feedback from beneficiaries of the pilot service, while expanding the number of service recipients to test takers in 20 qualification items. Expand data management systems such as big data collection and analysis for more accurate service implementation.
Step 3: Full service (from 2025~): We provide services for test takers in all of the qualifications we run.

🇰🇷