Data Engineering Terms Explained

Combine data from multiple sources into a single dataset.

Aligning data can mean one of three things: aligning datasets, meeting business rules, or arranging data elements in memory.

Identify data points or events that deviate significantly from expected patterns or behaviors.

Remove personal or identifying information from data.

Adding or attaching new records or data items to the end of an existing dataset, database table, file, or list.

Move rarely accessed data to a low-cost, long-term storage solution to reduce costs. Store data for long-term retention and compliance.

Speed up execution with asynchronous I/O.

Add new data or information to an existing dataset to enhance its value.

The automatic execution of computations and the persistence of their results.

A mechanism to handle situations where data is produced faster than it can be consumed.

Create a copy of data to protect against loss or corruption.

Process large volumes of data all at once in a single operation or batch.

Process large volumes of data in parallel and distributed computing environments to improve performance.

Store expensive computation results so they can be reused, not recomputed.

Organizing and classifying data into different categories, groups, or segments.

Saving the state of a process at certain points so that it can be restarted from that point in case of failure.

Remove invalid or inconsistent data values, such as empty fields or outliers.

Group data points based on similarities or patterns to facilitate analysis and modeling.

Reducing the size of data while preserving its essential information.

Reduce the size of data to save storage space and improve processing performance.

Combine multiple datasets into one to create a more comprehensive view of the data.

A measure of similarity between two entities used in text analysis, natural language processing, etc.

Select, organize, and annotate data to make it more useful for analysis and modeling.

Remove personally identifiable information (PII) from data to protect privacy and comply with regulations.

Identify and remove duplicate records or entries to improve data quality.

Remove noise or artifacts from data to improve its accuracy and quality.

Optimize data for faster read access by reducing the number of joins needed to retrieve related data.

Extracting, transforming, and generating new data from existing datasets.

Deserialization is essentially the reverse process of serialization. See: 'Serialize'.

Analyzing the number of features or attributes in the data to improve performance.

Transform continuous data into discrete categories or bins to simplify analysis.

Reduce the amount of data for analysis, storage, or processing.

Extract, transform, and load data between different systems.

The bundling of data with the methods that operate on that data.

Convert categorical variables into numerical representations for ML algorithms.

Enhance data with additional information from external sources.

Understand the data, identify patterns, and gain insights.

Extract data from a system for use in another system or application.

Predict values outside a known range, based on the trends or patterns identified within the available data.

A pipeline design in which one operation is broken into - or results in - many parallel downstream tasks.

Identify and extract relevant features from raw data for use in analysis or modeling.

Identify and select the most relevant and informative features for analysis or modeling.

Extract a subset of data based on specific criteria or conditions.

Break data down into smaller chunks for storage and management purposes.

Analyze data that has geographic or spatial components to identify patterns and relationships.

A powerful tool to model and understand intricate relationships within our data systems.

Convert data into a fixed-length code to improve data security and integrity.

Make data uniform, consistent, and comparable.

An operation that produces the same result each time it is performed.

Fill in missing data values with estimated or imputed values to facilitate analysis.

Create an optimized data structure for fast search and retrieval.

The initial collection and import of data from various sources into your processing environment.

Combine data from different sources to create a unified view for analysis or reporting.

Use known data values to estimate unknown data values.

Understand how data moves through a pipeline, including its origin, transformations, dependencies, and ultimate consumption.

Ensure that each individual operation on a distributed system appear to occur instantaneously.

Transforming the relationship between variables to make datasets approximately linear.

Insert data into a database or data warehouse, or your pipeline for processing.

Obfuscate sensitive data to protect its privacy and security.

Executing a computation and persisting the results into storage.