The Management of Data Quality Assessment in Big Data Presents a Complex Challenge, Accompanied by Various Issues Related to Data Quality

Abstract

The precondition for effective utilization and analysis of big data is high-quality data. However, comprehensive research and examination of effective estimation and appraisal techniques for big data are still lacking. This paper aims to provide an overview of data quality studies and address the challenges faced in the context of big data. The proposed data quality framework includes measurements, attributes, and files, with a focus on meeting the needs of data users. Additionally, a dynamic appraisal technique for data quality is developed based on this framework, which is scalable and flexible to address the challenges associated with big data quality appraisal.

Maintaining the quality of data is crucial for effective decision-making in support management. However, the sources of big data are vast and the data systems are complex, often leading to issues such as errors, inconsistencies, and noise. Data cleansing is a process that involves identifying and removing such issues to enhance data quality. There are four approaches to data cleansing: manual execution, use of specialized software, generic data cleaning, and domain-specific data cleaning. Among these approaches, the third one holds great practical value and can be effectively applied.