Data crunching is an information science approach that makes it possible to prepare vast volumes of data and information (Big Data) for automated processing. Data crunching consists of planning and modelling a device or application that is used: data is stored, sorted and organized to run algorithms and programme sequences on it. Therefore, the word crunched data refers to data that has already been imported and processed into a system. Related terms include data munging and data wrangling — more of manual or semi-automatic data processing, which is why they are substantially different from data crunching.
Most data-crunching tasks can be simplified into three stages. First, raw data is read to convert it to a selected format as a next step. Then, the data is output in the correct format so that it can be further interpreted or analyzed. This trichotomy has the advantage that individual data (input, output) can also be applied for other scenarios.
The ultimate aim of data processing is to have a more in-depth insight into the matter that needs to be transmitted via data, such as in the field of business intelligence so that informed decisions can be made. Areas, where data cracking applies, are medicine, physics, chemistry, biology, economics, criminology or web analytics. Depending on the context, various programming languages and tools are used: while Excel, Batch and Shell programming has been historically used earlier, languages such as Java, Python, R, SAS, SPSS, or Ruby are preferred today.
Data crunching, however, does not apply to exploratory research or data visualization – it is performed by special programs adapted to their field of use. Data crunching is all about the right retrieval so that the machine can do something about the information and the data format. Data crunching is, therefore, an upstream data analysis process. This method, like the data processing itself, can be iterative if the output of the crunching process contains new data or errors. It ensures that program sequences can be replicated until the desired result is achieved: an accurate, valid data set that can be further processed directly or imported and does not contain errors or bugs or skewed values.
Five data-crunching applications are:
• More processing of the inherited data in the programme code
• Conversion to uniform formats across columns
• Conversion from one format to another, e.g. plain text to XML data records
• Correcting errors in data sets, including spelling errors or software errors
• Extraction of raw data to prepare for subsequent assessment.
As a rule, a lot of time can be saved with data crunching because the procedures do not have to be done manually. Data crunching, particularly with large data sets and relational databases, can therefore be a significant advantage. However, there is a need for sufficient infrastructure to provide computing capacity for such operations. For example, a system like Hadoop distributes computer loads across multiple resources and performs arithmetic processes on computer clusters. It uses the principle of division of labour.