Architecture and software tools for Big Data management in computational environments

Abstract

In the digital age, the processing of large volumes of data (Big Data) has become a fundamental challenge for organizations across various sectors. It is projected that by 2026, the global volume of data will reach approximately 200 to 210 zettabytes, driven primarily by the growth of IoT devices, social networks, and corporate information systems. In this context, software infrastructure plays a crucial role in the efficient collection, analysis, and storage of massive datasets.

This article presents a critical review of the leading technologies and architectures used in the Big Data ecosystem, with particular emphasis on widely adopted tools such as Apache Hadoop, Apache Spark, and NoSQL databases (e.g., MongoDB, Cassandra, and HBase). The capabilities of these technologies are examined in relation to the fundamental characteristics of Big Data—volume, velocity, and variety—while comparing their approaches to distributed processing, in-memory computation, and response times across different environments.

Furthermore, reference architectures such as Lambda and Kappa are analyzed, highlighting their contributions to real-time and batch processing. Finally, current industry challenges are addressed, including scalability, the integration of heterogeneous data sources, and security and privacy concerns. The article concludes with a discussion of emerging trends such as Artificial Intelligence, Machine Learning, Edge Computing, and Cloud infrastructures, which are redefining the possibilities of large-scale data analysis.