Earlier this week, we introduced the Databricks implementation of Alloy and Ember in DataForge 10.0. Today, we are expanding that same architecture to Snowflake. DataForge now brings structured refinement, metadata-driven execution, and predictable processing to a second major cloud data platform.

Snowflake users can now run the Alloy Architecture natively inside Snowflake with table-backed refinement, metadata-driven change detection, enrichment, and merging. All stages are powered by Ember’s structured metadata model. This provides Snowflake customers with a consistent, governed refinement flow without relying on hidden logic or custom layers.

Alloy, Now Running Natively on Snowflake

Alloy’s five-layer processing flow of ORE, MINERAL, ALLOY, INGOT, and PRODUCT is now implemented as Snowflake tables and views. This allows refinement to occur directly in the warehouse using Snowflake’s native capabilities. Each stage is visible, queryable, and aligned with Snowflake’s execution patterns.

Snowflake teams benefit from a consistent refinement model that is easy to inspect, audit, and govern. This implementation preserves Alloy’s structured behavior while integrating smoothly with Snowflake’s SQL-first environment and performance model.

Powered by Ember: Structured Metadata for Snowflake

Ember acts as the definition layer behind Alloy on Snowflake. Ember stores explicit metadata that describes:

• how sources are interpreted

• how incremental change detection operates

• how attributes relate across domains

• how enrichment logic is applied

• how merge behavior functions during refresh

• how outputs are shaped and delivered

These definitions map directly into Snowflake’s table structures and allow Alloy to execute deterministically across domains.

Snowflake Local Agent Support

DataForge 10.0 introduces Snowflake Local Agent support. This enables ingestion, metadata synchronization, and connectivity to run inside a customer-controlled network while refinement runs inside Snowflake. It provides a hybrid option for organizations with secure networking or on-premises requirements.

Snowpark SDK Availability

DataForge 10.0 also includes a Snowpark-based SDK for Snowflake developers. This allows custom ingestion logic, connectors, and extensions to be built using Snowflake-native programming patterns while still participating in Alloy’s structured refinement model.

Incremental Processing Built Into the Architecture

Incremental behavior is implemented the same way on Snowflake as on Databricks. MINERAL isolates new or changed records. ALLOY enriches this reduced dataset before it scales. INGOT merges updates back into the complete dataset.

Because Ember defines attribute behavior at each stage, Alloy can push logic earlier in the process, reduce warehouse compute consumption, and eliminate the need for custom orchestration. Incremental workloads become predictable, efficient, and consistent across domains.

A Familiar Experience for Existing DataForge Customers

Teams already using DataForge will find that the Snowflake experience matches their existing workflow. The development model is identical:

• the same declarative configurations

• the same attribute-level refinement model

• the same Alloy flow across five stages

• the same operational experience in Talos and the DataForge UI

Snowflake becomes simply another execution environment, rather than a separate implementation.

Performance and Operational Optimization for Snowflake

DataForge 10.0 includes performance optimizations specific to Snowflake environments:

• improved ingestion paths optimized for Snowflake’s loading patterns

• efficient merge logic in INGOT using Snowflake’s micro-partitioning

• improved Data Profile extraction aligned with Snowflake’s INFORMATION_SCHEMA

• reduced warehouse consumption through Alloy’s incremental design

• improved agent-level performance that minimizes round-trips to Snowflake

These enhancements provide faster and more cost-effective refinement cycles.

A Unified Multi-Platform Architecture

With Snowflake support, Alloy and Ember now operate consistently across Databricks and Snowflake. This allows organizations to adopt either platform without learning a new data engineering model or maintaining multiple pipeline designs.

Alloy provides the structure.
Ember provides the definitions.
Snowflake provides the warehouse execution environment.

Together, they offer a modern, predictable refinement experience without hidden layers or platform-specific rewrites.

What’s Next

The release of DataForge 10.0 completes our largest platform launch to date. Over the past week, we introduced the Alloy Architecture, a new structured refinement model, and Ember, the metadata catalog that defines and governs Alloy’s behavior. We then brought both technologies to Databricks with a fully table-native implementation and Unity Catalog alignment. Today’s announcement extends that same architecture to Snowflake, enabling Alloy and Ember to operate consistently across multiple cloud data platforms.

With the core architecture now fully deployed, our upcoming content will focus on deep-dive technical walkthroughs, transformation framework details, Ember metadata patterns, and best practices for Alloy across platforms. We will also be sharing new customer stories that highlight how organizations are adopting Alloy and Ember to simplify complex pipelines and accelerate time-to-value.

More updates will follow throughout the next few weeks and months.

Last week, we introduced the Alloy Architecture: a structured refinement model designed to eliminate hidden layers and create predictable, repeatable workflows. We also unveiled Ember, the metadata catalog that defines Alloy’s behavior using clear, declarative configuration.

Today, we’re announcing how Alloy and Ember come to life inside DataForge 10.0 for Databricks. This release reflects a major reinvestment in our Databricks architecture, delivering Unity Catalog integration, table-native refinement, federated metadata access, and a modernized compute layer.

Table-Native Refinement Inside Databricks

Older versions of DataForge relied on file-based refinement stages. While functional, this approach made it harder for teams to understand how data moved through the system or to align with Databricks-native governance tools.

In DataForge 10.0, each stage of the Alloy Architecture (ORE, MINERAL, ALLOY, INGOT, and PRODUCT) is now represented as a Delta table or view. This change makes refinement far more transparent and intuitive. Teams can inspect intermediate results directly, query refinement stages using Databricks SQL, and leverage Unity Catalog controls without needing to interpret abstract internal flows.

The end result is an Alloy execution model that feels native on Databricks, with refinement exposed through standard lakehouse primitives rather than internal file structures.

Ember: The Structured Definition Layer Behind Alloy

Alloy’s consistency is enabled by Ember - the relational metadata repository that defines how refinement should behave across domains. Ember stores explicit definitions for:

• Source interpretation and normalization

• Change detection logic

• Attribute relationships and enrichment rules

• Merge behavior in INGOT

• Output shaping and delivery patterns

This metadata is stored in a Postgres-backed relational schema designed specifically for Alloy’s five-layer model. Rather than inferring logic from code, Ember provides an authoritative, declarative description of how data should be transformed.

Ember Available Directly in Unity Catalog via Lakehouse Federation

One of the most meaningful enhancements in DataForge 10.0 is that Ember’s metadata can now be queried directly in Databricks. Through Lakehouse Federation, Ember’s catalog appears as Unity Catalog tables without requiring extra connectors or APIs. Teams can join metadata to operational data, build governance dashboards, or analyze domain relationships using standard SQL, all inside Databricks.

Incremental Processing Built Into the Architecture

Incremental refinement is traditionally one of the most complex aspects of building pipelines. Alloy makes it a native part of the design. MINERAL isolates new and changed records; ALLOY enriches this reduced dataset before it scales; and INGOT merges updates back into the full table. Because Ember defines attribute behavior across these layers, Alloy can push heavy transformations earlier in the flow and reduce the amount of data processed at each stage.

This built-in incremental pattern reduces compute costs, speeds up refresh cycles, and eliminates the custom branching logic that often accumulates in traditional pipelines.

Performance and Developer Experience Improvements

DataForge 10.0 includes a series of enhancements designed to improve reliability and performance on Databricks:

• Faster hub table checks using metadata-driven pruning

• Smarter Delta merge strategies with reduced conflicts

• More responsive connection tests to cut down on debugging time

• Quicker Data Profile introspection across new sources

• Improved job orchestration with clearer stage boundaries

These improvements shorten iteration loops and deliver a smoother developer experience.

Deep Alignment with Unity Catalog

Unity Catalog has become the governance backbone of Databricks deployments, and DataForge 10.0 aligns tightly with that model. New workspaces build hub tables natively under UC, enabling consistent lineage, access control, and oversight. Existing Hive deployments can be upgraded smoothly, bringing old environments into the new governance model without major disruptions.

DataForge now fits naturally into UC-driven lakehouse architectures, offering:

• Unified governance across both data (Delta) and metadata (Ember)

• Easier auditing and exploration

• Consistent behavior across environments

Modernized Compute Layer

We’ve also refreshed the compute experience:

• Support for DBR 16.4 LTS

• Updated instance type recommendations

• Ability to restart Talos directly from the UI

• Terminology aligned with Databricks (“Compute” replacing “Clusters”)

These updates simplify administration and ensure a modern operational foundation.

A Unified Databricks Experience

Alloy provides the structure. Ember provides the metadata. Databricks provides the runtime and governance environment. Together, they create a version of DataForge that is more transparent, more predictable, and easier to operate than ever before.

Every stage is inspectable.
Every definition is queryable.
Every refinement behaves consistently across domains.

And this is only the first half of the 10.0 rollout.

What’s Coming Next

Later this week, we’ll extend Alloy and Ember to a new ecosystem as part of the second major platform announcement for DataForge 10.0.

Thursday’s announcement completes the release. Stay tuned!

Today, we’re excited to introduce the system that makes Alloy possible:

Ember is DataForge’s structured data catalog and declarative metadata engine.

Ember is more than a catalog of tables and fields. It is the definition layer that tells Alloy exactly how each step of the refinement process should behave.

Where Alloy provides the architectural flow, Ember provides the detailed instructions that drive it - clearly, consistently, and without ambiguity.

Why Ember Is Different From Traditional Data Catalogs

Most data catalogs work the same way:
They scan SQL, code, notebooks, or dbt models to infer what pipelines are doing and attempt to document them after the fact.

This creates challenges most data teams know well:

• Hidden logic scattered across CTEs, temp tables, scripts, and models

• Catalogs drifting out of sync as code evolves

• Lineage that describes what happened, not what should happen

• Difficulty enforcing consistency across domains

• Metadata that reflects intentions rather than guarantees

Ember is designed for the opposite model.

Instead of scanning pipelines, Ember stores the rules that create them.

It doesn’t observe developer logic.
It defines it — up front, in a structured, relational model.

This shift from descriptive to prescriptive metadata is what makes Ember fundamentally different from observability tools and traditional catalogs.

Ember as a New Kind of Data Catalog

Ember replaces the traditional “after-the-fact” catalog with a catalog that defines behavior, not just documentation.

Ember is:

• A structured data catalog

• A metadata engine

• A definition layer for transformation rules

• A control plane for the Alloy Architecture

• A single source of truth for how data should be refined

It bridges the familiarity of a catalog with the precision of a declarative execution model.

This makes Ember both recognizable and entirely new.

What Ember Stores and Manages

Ember contains a set of metadata objects that describe:

• What each data source looks like

• How new and changed records should be detected

• How attributes relate to one another

• What validation or quality rules apply

• Which enrichment steps should occur

• How data should be merged and refined

• What outputs should be produced for downstream systems

Each object has a well-defined purpose.
Each relationship is encoded explicitly.
Each rule is unambiguous.

Ember becomes the source of truth for how data is processed end-to-end — without relying on handwritten pipeline code or scattered configuration files.

How Ember Enables Alloy’s “No Hidden Layers” Approach

A major benefit of the Alloy Architecture is that it eliminates the hidden intermediate layers typically found in notebooks, models, and procedural pipelines.

This is only possible because Ember defines transformation behavior in a way that:

• Keeps related logic together rather than spreading it across tables

• Stores rules at the attribute level instead of burying them inside CTE chains

• Separates incremental processing rules from full-dataset refinements

• Provides clear instructions for how to enrich and merge data

• Removes the need for ad-hoc staging tables and temporary constructs

In other words:

Ember centralizes transformation rules so that Alloy can execute them cleanly — without creating additional layers behind the scenes.

This is the foundation of Alloy’s predictability, and the core reason Ember needed to be invented.

Ember + Alloy: Definition + Execution

Alloy establishes a structured model for data refinement.
Ember provides the definitions that make that model consistent across every domain.

Together, they deliver:

• Clear, predictable refinement behavior

• Built-in incremental processing

• Uniform patterns across all sources

• Zero hidden layers or ad-hoc transformations

• Faster onboarding of new data domains

• Stronger governance and lineage

• A foundation for true declarative data engineering

This pairing represents a major step forward in how data pipelines can be modeled and executed.

What’s Coming Next

If you’d like to explore the foundations behind Ember and the Alloy Architecture, we’ve published two in-depth introductions:

• Introduction to the DataForge Declarative Transformation Framework
  https://www.dataforgelabs.com/blog/introduction-dataforge-framework

• Introduction to the DataForge Object Model
  https://www.dataforgelabs.com/blog/dataforge-intro-2

These articles provide background on the principles that guided the design of Ember’s metadata structures and Alloy’s refinement flow.

Next week, we’ll begin detailing the physical implementation of Alloy and Ember inside DataForge version 10.0 — starting with our Databricks experience early in the week, and concluding with a second major announcement on Thursday.

More to come soon.

For years, data teams have been asked to deliver reliable, governed, high-quality data products using architectural patterns that look clean in diagrams but break apart in practice. Medallion-style models are often framed as three simple layers: Bronze, Silver, Gold - but real pipelines contain dozens or even hundreds of hidden transformation stages buried inside:

• Common Table Expressions

• Temp tables

• dbt models and sub-models

• Spark/Dataset transformations

• User-defined staging tables

• Ad-hoc orchestration logic

These implicit layers introduce operational risk, slow down teams, and make it nearly impossible to achieve end-to-end lineage or enforce consistent transformation logic across domains.

The Alloy Architecture solves this problem by introducing a structured, statically defined processing flow that eliminates hidden layers entirely.

A Five-Layer Refinement Model - Designed for Clarity and Consistency

At the heart of Alloy is a deterministic refinement sequence:

Each stage has one clear purpose, and every pipeline follows the same structured flow.

ORE

Unrefined data captured exactly as delivered from the source.

MINERAL

Purpose-built change detection isolates only the new or updated records.

ALLOY

Business logic, joins, and enrichment are applied only to the incremental batch - improving both performance and predictability.

INGOT

The enriched batch is merged into the full dataset through a consistent refinement process that ensures clean, canonical results.

PRODUCT

Final data outputs are materialized for analytics, operational systems, and downstream consumers.

This architecture makes the refinement process explicit, stable, and uniformly repeatable across every domain and every data source.

Incremental Processing - Built In, Not Bolted On

Traditional data engineering frameworks treat incremental loads as an optimization problem left to developers:

• Custom merge logic

• Timestamp parsing

• Soft/hard delete handling

• State tracking tables

• Conditional pipeline branches

• Fragile assumptions buried in code

This approach creates unnecessary variability and technical risk across domains.

Alloy takes a fundamentally different approach.

Incremental behavior is architecturally enforced, not optional.
MINERAL and ALLOY are designed to process only new or updated data, while INGOT performs the structured, full-dataset refinement.

This design provides:

• Predictable scaling

• No per-table incremental logic

• Consistent change handling

• Simplified failure recovery

• Less custom code and fewer edge-case rules

Incrementalism isn’t an optimization. It’s a first-class architectural principle of Alloy.

Modernizing the Foundation - A Major Upgrade from DataForge’s Legacy Architecture

Prior versions of DataForge used a file-based refinement model, generating RAW, CDC, ENR, and HUB datasets as files in cloud storage.
This approach enabled rapid early adoption but introduced challenges around:

• Navigability

• Governance and metadata consistency

• Managing partition evolution

• Cross-layer relationships

• Efficient incremental processing patterns

• Integration with native lakehouse governance tools

Alloy modernizes this foundation by transitioning the entire refinement flow to a fully table-native architecture.

This upgrade improves:

• Layer-to-layer transparency

• Debugging and troubleshooting

• Partitioning strategy consistency

• Observability across the refinement process

• Incremental execution reliability

• Cross-domain structural alignment

This modernization is a major upgrade for DataForge customers and represents a generational shift in how the platform models and processes data.

How Alloy Relates to Medallion

The Medallion Architecture is intentionally broad and flexible, which is why it has been widely adopted across the industry. Its three-tier model (Bronze, Silver, Gold) can describe a wide range of pipeline patterns depending on how teams choose to implement it.

The Alloy Architecture is something different.

While Alloy can conceptually map to Medallion, it introduces:

• Explicit, enforceable refinement stages

• A deterministic sequence without hidden layers

• Incremental processing built directly into the architecture

• A uniform pattern that applies consistently across domains

Alloy is not a simple interpretation of Medallion, it is an evolution.

It is a new architectural model designed to solve a different set of problems: reducing hidden complexity, improving consistency, and enabling predictable, declarative data engineering.

We will publish a dedicated follow-on article that provides a deeper comparison between Alloy and Medallion, including where they align and where they meaningfully diverge.

Why Alloy Matters

1. No More Hidden Layers

A static, five-step process replaces scattered logical operations inside notebooks, CTEs, or ephemeral tables.

2. Predictable Performance from Structural Incrementalism

Incremental behavior is part of the architecture itself - not a pattern left to individual developers.

3. Consistency Across Every Pipeline

Each source, domain, and transformation path follows the same refinement model.

4. Better Governance and Debugging

Structured refinement produces cleaner, more discoverable lineage and easier troubleshooting.

5. A Foundation for Declarative Transformation

Less code. More structure. More repeatability. More resilience.

The Beginning of a New Chapter

The Alloy Architecture brings structure to what was previously ambiguous, simplicity to what was previously fragmented, and determinism to what was previously unpredictable.

This release marks a major milestone for DataForge - and the foundation for everything that comes next.

More updates later this week.

DataForge, the AI-powered declarative data platform for modern data teams, today announced the launch of DataForge Cloud 9.0, delivering a transformative development experience powered by Talos, the platform’s embedded AI agent. With this release, users can now develop data models, operate pipelines, and manage complex dependencies using natural language prompts—bringing unprecedented speed and accessibility to data engineering.

“Talos is now capable of turning a simple business question into a fully operational data workflow,” said Matt Kosovec, CEO of DataForge. “With DataForge Cloud 9.0, we’re giving teams the ability to develop reliable, scalable data infrastructure without writing a single line of code.”

AI-Powered Development with Talos

At the center of the DataForge Cloud 9.0 release is a new AI development experience powered by Talos. Users describe the outcomes they want, such as reporting logic, pipeline rules, or metrics, and Talos generates the full declarative configuration to implement them. This includes defining entities, setting pipeline conditions, establishing dependencies, and generating reusable components across the platform.

Natural Language to Production-Ready Models
Talos understands high-level prompts like “build a customer lifetime value model” or “track revenue by product and region.” It then generates the necessary tables, metrics, joins, and pipeline logic required to support the request.

End-to-End Configuration in minutes
From data models and filters to joins and execution conditions, Talos handles every layer of configuration, drastically reducing the time it takes to go from idea to deployed workflow.

Intelligent Suggestions in Context
As users iterate, Talos becomes context-aware, recommending additional configurations based on what’s already in place. Whether building a new model or modifying an existing pipeline, Talos ensures consistency and reliability.

Visual Feedback and Guided Edits
Each configuration generated by Talos is accompanied by visual feedback and edit suggestions, allowing users to fine-tune logic or explore variations, all without navigating complex codebases.

DataForge Cloud 9.0 also introduces a new dark mode user experience, enhanced observability, visual dependency tracing, and SDK upgrades, but the spotlight remains on Talos and its ability to bring AI-native development to every data team, regardless of technical background.

Availability

DataForge Cloud 9.0 is available today for all customers. New users can explore the Talos experience and request access at www.dataforgelabs.com.

About DataForge

DataForge is a declarative data platform that enables lean teams to build, manage, and scale automated data workflows with speed and confidence. Talos, the platform’s embedded AI agent, accelerates development across the data lifecycle—from modeling to orchestration—without requiring traditional coding. DataForge is trusted by fast-growing companies and private equity-backed organizations looking to modernize their data infrastructure while keeping teams agile.

For more information, visit www.dataforgelabs.com.

Refresh is the process in a data pipeline that consolidates individual batches (or event streams) of data into the most current representation of the dataset (such as the gold layer in a medallion architecture). The Refresh process operates on each non-static dataset within a data platform.

The main purpose of a refresh process is to consolidate multiple batches of data, remove duplicate records, and provide the latest version of the data to downstream consumers.

A few secondary features are:

• Capture the history of changes in dataset over time

• Perform atomic and transactional-style updates to ensure consistent dataset state

• Update data incrementally if possible

• Enable easy reprocessing of data after configuration change or error

• Allow for unload and reload processes to remove, correct, and re-add bad batches of data

• Capture and expose observability metrics such as new, changed, and updated record counts, snapshot timestamps, etc.

The most common approach to refreshing data is a basic truncate-reload pattern, where all data is fully replaced. The second most popular and more performant approach is an incremental data refresh of datasets by leveraging a primary key. An extension of primary key incremental refresh is SCD (Slowly Changing Dimension) Type 1 and 2, where the history of data for each key is tracked in addition to maintaining the most up to date records. This concept was covered in a previous blog where Databricks’ Delta Live Tables and DafaForge Cloud approaches were compared. Before jumping in to this material further, it is likely helpful to review this previous blog first.

Unfortunately, not every data set has a natural primary key. In many tools and architectures, the only way to support incremental refresh with these non-keyed tables is via a custom-developed pipeline and dataset-specific refresh logic.

Refresh Types in DataForge

At DataForge, we are strong believers in column-pure, functional, and declarative transformation logic. Applying these ideals to the universal batch data processing problem of Data Refresh, we developed six different and declaratively-definable refresh patterns:

Key Refresh:

Starting with the most commonly used Refresh Type, Keyed Refresh has the following parameters:

Key Refresh Processing steps

1. Calculate a hash value of all non-key attributes in the latest batch, excluding columns specified by the “Exclude From CDC” parameter. Store the value in a new system-managed column: s_hash

2. For each record, store the value from the Date Column in a new system-managed column: s_update_timestamp

3. Compare s_hash and s_update_timestamp values in the current batch with the last received record for each key in the Hub table, and calculate s_record_status_code with one of the following values: 
   N - new
   C - updated
   U - unchanged
   S - unchanged with more recent s_update_timestamp 
   O - old record: updated, but with older s_update_timestamp

4. Apply filter of s_record_status_code IN (‘N’,’C’)  for the next processing step

5. Merge the batch into main dataset (Hub table) by updating changed (‘C’) records, and appending new (‘N’) records

While the key and hash approach is pretty standard and has been widely used in ETL pipelines, the configurable Date Column is fairly unique to DataForge. It enables to accurately identify and filter out “old” records, which have been updated earlier, before the current update timestamp. This ensures accuracy of the most recent record and optimizes processing. It helps in scenarios where input data batches (or files) are received out of order, such as historic data loads.

Full Refresh:

A Simple truncate/reload pattern. Used on small datasets where it’s often simpler and more efficient to reprocess the entire dataset vs. identifying what changed and processing incrementally. It keeps a history of previous batches and allows restoring prior versions of data.

Full Refresh Processing steps

1. Create new partition in the hub table

2. Copy batch data into new partition

3. Update hub view definition to mark the partition as current

None Refresh:

Simple append pattern. May be used when source data has already been deduplicated or it is desirable to have full history of changes. Each subsequent batch of data is appended to the dataset.

None Refresh Processing steps

1. Create new partition in the hub table

2. Copy batch data into new partition

Timestamp Refresh:

Frequently used for time series and transactional data that follows these pattern:

• Does not have natural primary key

• Has non-null date or datetime column representing action or transaction timestamp

• Each batch includes a continuous range of records by transaction timestamp. Each batch may have different ranges, from several minutes to several days, month or more. When ranges overlap in batches, the latest received batch replaces data within its range

Timestamp Refresh Processing steps

1. Store configured timestamp column value in new system s_timestamp attribute

2. Calculate and store MIN and MAX s_timestamp values for the batch

3. Add batch data to the main dataset by replacing overlapping timestamp range, and appending non-overlapping range record, as illustrated below:

Sequence Refresh:

Sequence is identical to Timestamp refresh, except that timestamp column is replaced with incrementing numeric sequence attribute.

Custom Refresh:

This pattern allows us to handle more complex data refresh scenarios that don’t fit into 5 categories described above.

Custom Refresh Processing steps

1. Evaluate Delete Query and delete hub table records satisfying the criteria

2. Append latest batch of data to hub table, optionally pre-filtering it using Append Filter

Secondary Subsystems of Refresh in DataForge

Capture Historic Data

These DataForge refresh types provide means for exposing historic changes to a dataset over time: 

Atomic Processing

This ensures that in case of any processing errors, the main dataset will not get corrupted. It is achieved by 3 complementary mechanisms:

• Separation of refresh processing into 2 distinct phases:

• Capture Data Changes (CDC) phase reads the main data set and current batch, but updates the current batch only. This is typically where most errors occur, caused by bad configuration parameters or bad data in the batch. When this phase fails, the failure has no impact on the main dataset.

• “Refresh” phase updates the main dataset, leveraging system attribute values pre-calculated during the CDC phase. This simplifies the update process as much as possible, minimizing the chance for errors.

• Integrated versioning of the main dataset, which enables rolling back to the last stable version of the data when the Refresh phase fails.

• Reset All CDC function allows to fully re-process the main dataset from “scratch”, using individual raw batch files stored in the data lake.

Incremental Processing and Source System Querying

When the source dataset contains an accurate timestamp or sequence column that represents the last insert/update datetime for each record, a watermark column parameter can be configured to enable incremental ingestion and processing. Specifying this optional parameter enables use of special reserved tokens in source query that filter data using the last timestamp or sequence value observed in the watermark column. An example of source query using the watermark token:

SELECT * FROM Orders WHERE modified_date >= <latest_timestamp>

During ingestion, the value of <latest_timestamp> token is evaluated as MAX(<configured timestamp column>) in the current Hub table.

Reprocessing

Developers often start with the Full refresh pattern - it is the most simple to configure and best suited for initial development while still exploring and discovering dataset characteristics. Later in the development cycle, a developer can update the refresh pattern and associated parameters. After the change, the system will display a confirmation prompt and initialize a reprocessing of the dataset using the new configuration.

Unload and Reload

Mistakes happen, and wrong data batches get loaded into the dataset. It is possible to delete any batch from any DataForge source with a simple click of a button. This is enabled by DataForge’s metadata repository that stores relevant information about every received batch of data for every dataset. The “Delete” process removes one or more selected inputs (batches), updating the dataset to the same state as if deleted batches were never loaded.

Once invalid batches are deleted, the corrected data can be reloaded, even if more recent datasets were loaded into the table. This feature saves a lot of time and stress to developers and support engineers responsible for monitoring platform data feeds.

Observability Metrics

The DataForge Meta Database captures extensive observability metrics for every dataset, batch, and processing step. Here are a few of the many metrics captured and exposed via the DataForge UI that are specific to the Refresh process. Many of these metrics apply to all refresh types, but some are type-specific.

Conclusion

The Data Refresh process is critical in managing and updating datasets within a data pipeline, consolidating individual batches, or event streams to provide a unified, up-to-date representation of data. Its primary purpose is to eliminate duplicates, capture the latest changes, and maintain data consistency for users. Key features include capturing change history, performing atomic updates, supporting incremental processing, and enabling reprocessing or corrections after configuration changes or errors.

DataForge supports various refresh types tailored for different dataset structures and update needs, from Key-based refresh (leveraging primary keys and timestamps) to simple full reloads, time-series (timestamp or sequence-based) refresh, and even custom criteria-based refreshes. These patterns allow DataForge users to manage data across diverse scenarios with simple configurations, whether the source has natural keys or not.

Lastly, the DataForge Refresh process type includes mechanisms for capturing historical changes, ensuring atomic processing, and offering observability metrics. These features not only safeguard data integrity during updates but also enable easy troubleshooting and performance optimization by exposing key metrics like record counts and effective data ranges.

Stop building your refresh processes by hand and start declaratively defining your pipelines today with a DataForge 30 day free trial.

1. Specifying Data Types from Untyped Files

One of the initial design choices in building a BI data model is whether to define data types at the earliest stages. Joe advocates for establishing "type stages" for raw data tables, where data types are validated and set at the outset. This early assignment of types reduces downstream dependencies on casting, minimizes potential error handling, and simplifies subsequent transformations. Without this initial type-setting step, downstream processes face an increased risk of casting errors, adding complexity to data validation. While introducing type stages creates an additional processing layer, it enhances pipeline durability and ensures data integrity for later stages.

2. Grouping Data to Achieve Consistent Granularity

Engineering teams frequently encounter source tables with different levels of granularity, requiring careful consideration of data aggregation methods. For example, customer data might be available at a delivery-detail level instead of the invoice-detail level, necessitating aggregation for consistent granularity. Joe demonstrated the use of a "group stage" to align customer and delivery data, which facilitates accurate joins and preserves data fidelity. Here, engineers face a trade-off: creating a separate group stage enhances modularity and reuse but can add processing steps. Consolidating aggregation logic into fewer steps can reduce complexity but at the cost of flexibility.

3. Managing Multi-Item Orders Through Unpivoting

In many data models, handling complex record structures, such as rows containing multiple items, can complicate aggregation and reporting. Joe showcased an unpivoting technique, where each item in a multi-item order is broken into distinct rows, creating a consistent granularity that simplifies calculations. This choice allows BI tools to retrieve accurate insights without complex transformations. However, unpivoting introduces additional stages, impacting processing resources. Engineers must evaluate the balance between increased granularity and processing efficiency, as unpivoted structures may better support analysis but add to the pipeline’s complexity.

4. Choosing When to Materialize Stages for Data Stability

Materializing intermediate stages—creating physical tables to store transformations—adds stability to the data pipeline, allowing validated data to serve as a secure source for downstream processes. Joe highlighted the choice to materialize type and aggregation stages to provide consistent access to cleaned data, enhancing the robustness of future data transformations. However, fully materializing each stage increases storage costs and can add maintenance overhead. Virtual stages, on the other hand, keep data transient, passing it through without permanent storage. While this approach saves on storage, it demands recalculations in high-use scenarios. Engineers must carefully consider access patterns and computational needs to make the most effective choice.

5. Balancing Modularity and Performance

One overarching engineering design choice in ETL pipeline development is balancing modularity against performance. Joe’s example emphasized how separating tasks into discrete stages (e.g., typing, grouping, unpivoting) improves modularity and debugging. However, each additional stage introduces processing demands and can increase the pipeline’s maintenance requirements. Consolidating logic by using common table expressions (CTEs) can reduce the number of stages, streamlining operations but potentially complicating troubleshooting and limiting flexibility. Engineering teams must evaluate these trade-offs in light of data complexity, operational load, and resource constraints.

Summary

These engineering design choices are fundamental in crafting BI data pipelines that balance simplicity, performance, and adaptability. By critically assessing each choice based on organizational needs and technical requirements, engineering teams can build scalable, resilient data models that consistently deliver accurate insights and support evolving analytics demands.

Part 2: Eliminating Stage Design with DataForge

In part 2 of this blog series, Joe will explore how these many decision points and considerations are eliminated or automated by the DataForge framework, enabling developers to spend more time building the core logic and less time worrying about code re-use, performance, or architecture considerations.

Avoid these design decisions and common pitfalls by signing up for a 30-day trial of DataForge to explore the future of data engineering.

Star and follow our open-source DataForge library on Github to stay updated on new features and enhancements to DataForge Core.

As we outlined in Introduction to the DataForge Declarative Transformation Framework blog, using declarative, column-pure transformations in data pipelines allows engineers to take advantage of inheritance and object composition in data engineering. These foundational computer science concepts have been used in software engineering for decades, but have yet to gain wide adoption within the data engineering discipline. They are the key ingredients for successfully building, maintaining and extending large and complex software products with millions lines of code.

The historical inability to apply these concepts to data engineering pipelines often leads to an explosion of spaghetti code that’s been copy-pasted and tweaked multiple times, which geometrically increases the complexity of the data platform. This leads to rampant increase in costs to support and extend your data platform, requiring more effort, resources, and talent.  

In this blog and linked video, I will showcase multi-layered mechanisms for centralized transformation code management and re-use available in DataForge Cloud

Rule Template

A Rule Template is an abstract, column-pure function that has properties identical to a Rule. It is created as an abstract object, and initially not linked to any Source. When the rule template is applied to the source, the system validates that rule expression is valid in the context of the source and then creates a new rule in that source. The rule is linked to the template and cannot be modified directly. 

The rule template is analogous to the single-method trait or mixin in software engineering. It allows centralized definition and management of transformation logic that is common for multiple data sources.

When the Rule Template changes, all rules linked to the template are automatically updated.

Relation Template

A Relation Template is an abstract, column-pure, boolean-typed function that has properties similar to a Relation. Similar to Rule Templates, they are created as an abstract object, and initially not linked to any pair of sources.

When the relation template is applied to the source, the system validates that the expression is valid in the context of the both sources, and then creates a new relation between them. The created relation is then linked to the template and cannot be modified directly.

Similar to Rule Templates, Relation Templates are analogous to a mixin and enable centralized management of relations between sources with identical logic.

Relation Templates are used to support Rule Templates that include Source Traversals to leverage Rules or Raw Attributes from other Sources.

Token

A Token represents a variable that can be used in any expression (Rule, Relation, and Template counterparts) in 2 ways: 

• as constant value

• as complete or partial attribute alias in the expression

Token values are defined for each source. When the rule or relation template is applied to the source, the token value defined for the target source is substituted in the template expression.

Examples:

Groups and Cloning

Similar to the module concept in programming, a Group organizes multiple sources, outputs, and connections with preconfigured transformations together. When a Group is Cloned, the system creates copies of all objects contained within the Group, replacing their names according to the new group name and source, output or connection template for each object. Group object includes a special type of Token (variable) reserved for use in a source, output, or connection names.  Diagram below illustrates cloning of GroupA into GroupB: 

Here is the breakdown of the steps that happen during a Clone operation:

• User selects Group A as the base and instructs DataForge to Clone the Group

• ConnectionB is Cloned from ConnectionA

• Two new sources Source1B and Source2B are Cloned from GroupA sources Source1A and Source2A. This includes all source settings and rules (both templated and regular)

• Relation1 between Source1 and Source2 is Cloned and linked to Relation Template1

• Output1B is Cloned from Output1A, including all channels and mappings

Groups and cloning are commonly used when you have multiple data feeds from similarly structured source systems (e.g. ERP or CRM). Each additional feed can be easily Cloned with a couple clicks, while allowing you to tweak, add or override specific rules and transformations that are unique for each source system. This is similar to importing a module from the library and extending it to accommodate your specific requirements. This mechanism enables centralized management of common transformation logic via rule and relation templates, while providing flexibility to adjust each group via tokens and regular, non-templated rules.

Source and Output Template

Source and Output Templates provide a naming mechanism that ensures uniqueness of source/output name in the project when cloning a Group. These templates are especially important when you want to control if a Cloned Output writes to the same target table(s) as the base Group, or create a new set of target tables specific to the generated Group within the same target database/connection.

Connection Template

Connection Templates allow for cloning of the connection referenced by Sources and Outputs within the Group. When a group is cloned, all templated connections are also cloned. A user just needs to update the cloned connection to point it to the new source/output system. Connection Templates help when the new cloned Group should ingest or write data from/to a different server or database.

Conclusion

Templates, Tokens and Groups provide a powerful, multi-layered mechanism for centralized management of data transformation code. It is based on the established concepts that have been successfully used in software engineering for multiple decades: inheritance, object composition, traits or mixins and  modules.

The ultimate benefit it provides is drastically reduced size of the code base in your data platform. This is achieved by enabling DRY principle and reusable code architecture, which eliminates traditionally used copy-pasting that often leads to bloat and spaghettification of code in data pipeline transformations.

Start scaling out your data codebase today with a 30 day free trial of DataForge cloud.

According to the recent poll posted on Reddit data engineering community, the most common reason for production data pipelines failures is source data schema changes. That includes adding or removing attributes and their data type changes.

SQL is the leading language for data transformation in modern data engineering and is widely supported by most contemporary data tools. However, as discussed in a previous blog, despite its power, SQL has some significant limitations. Today, we’ll take a closer look at two often-overlooked but crucial issues: Type Safety and Schema Evolution. This blog will explore these challenges, and in the linked video, I’ll demonstrate how DataForge addresses and solves them.

Type Safety

Type Safety is an important part of Compile-time validation. Together they minimize the risk of runtime crashes and exceptions.

Almost all modern programming languages support type safety and strong typing. While SQL is a strongly-typed language, it typically does not enforce type safety due to several reasons:

• SQL scripts often don’t have a distinct compilation phase. Compilation occurs dynamically immediately prior to execution of the script. Any syntax or type errors detected will fail at execution time.

• Variable and column data types used by SQL scripts are highly dependent on the schema of the tables, and it’s often impossible to perform type checking without having a schema of each table referenced by the script.

• Example: it’s impossible to validate SELECT * query without having a schema of the table.

• The SQL expression type is also dependent on the data contained in the table. Consider this example in Spark SQL:

price * quantity,  where price is DECIMAL(4,2) and quantity is INT

Depending on the value contained in the quantity column, the expression result type may vary between 

DECIMAL(6,2) and DECIMAL(15,2) when executing on different batches of data

This leads to the data type of the result of the SQL query and transformation changing unpredictably. The typical way to handle this is to apply a manual CAST function to the result of the transformation. Unfortunately, this is not only cumbersome, but also does not fully address the problem because the CAST data type may not be compatible with the latest batch of data’s result type. Even worse, in Spark SQL specifically, a failed CAST silently creates a NULL value for that row rather than failing the query by default.

These unsafe type changes lead to cascading failures when the changed attribute is referenced in downstream transformations. These issues are notoriously difficult to debug and fix because changes may flow through long chains of transform and store operations. 

Schema Evolution

Modern data platforms live in a permanent state of change: new datasets being constantly added, existing dataset schemas and business logic change frequently. These changes require a data transformation pipeline to manage Schema Evolution of each dataset it stores, otherwise the pipeline will fail when writing to intermediate or target tables. 

Here are common column or data type changes that need to be addressed by a schema evolution process:

• New column added to the dataset

• Column removed from the dataset

• Data type of the existing column has changed

These changes may be caused by schema changes in the most recent batch of data, or business logic changes requiring data type change of one of the existing rules.

Note that the data type change can also include changes to the schema of a nested, complex data type column., e.g. STRUCT.  It is for this reason that Type Safety and Schema Evolution concepts are closely linked together and need to be considered as a whole. In my experience, these 2 linked issues represent one of the most tedious aspects of maintaining the data pipelines. The problem is also exacerbated by multiple downstream schemas that need to be changed, that include multiple layers of data lake medallion architecture, intermediate persisted stages, and all impacted downstream datasets. 

After decades of frustration over this common challenge, we asked ourselves at DataForge: how could we simplify and automate the schema evolution process? Are there a set of common patterns we can pre-build and re-use depending on the type of evolution?

The DataForge Approach

First, we need to implement compile/design-time Type Safety for each columnar transformation (Rule). This requires, at design time:

1. Validating that all references used by expressions are valid: Hub (data lake table) names, Raw Attribute (source column) names, and Rule (calculated column) names.

2. Checking the DataForge columnar expression syntax by evaluating the expression on the target SQL platform (Databricks SQL in our case)

3. Determine the transformation result data type, and save it to a metadata repository separate from the lake/database schema.

4. Recursively querying all downstream attributes and transformations referencing the current attribute in this metadata repository and performing the same checks

Only when the above validations have succeeded, we allow the developer to save the Rule and update the logic.

This process ensures that each Rule data type will never change inadvertently, and any allowed changes are persisted downstream.

Next, we looked at the different types of changes that can occur and categorized them, thus creating dataset-specific options to handle the schema evolution triggered by schema changes in the ingested data batch.

Each configuration category can be either:

• Enabled: Performs specific action

• Disabled: Stops ingestion and generates error. This setting is typically used in highly controlled environments. It will require human approval before ingesting any new attribute.

Upcast and Extend for Complex Types

For complex/nested data types (STRUCT, ARRAY) the Upcast/Extend option applies the schema evolution parameters recursively to each nested attribute.

To illustrate this, let’s use column product. In the latest batch of data, its type is different from the type defined in the dataset: it lacks the color attribute. After ingestion, the system applies Upcast, which adds color with NULL value and stores it in the existing table.

With DataForge, we have simplified schema evolution strategies into easy to understand and configurable settings. This saves you from having to custom build this logic for every data set, and protects you from expensive failures caused by changes data structures. You can select exactly what types of changes are allowed on a per-dataset basis while also ensuring that downstream logic will continue to operate correctly.

Conclusion

DataForge provides customizable schema evolution functionality combined with robust, compile-time type safety.

This results in the following advantages:

• Massively improves pipeline reliability, reducing time developers spend debugging code

• Speeds up development by automating what would otherwise require tedious and time consuming custom code

• Enforces standard architectural patterns for the entire data lakehouse

Take advantage of this innovative approach and accelerate your development by starting a 30 day trial of DataForge Cloud today!

With these new capabilities, DataForge strengthens its alignment with the principles of Lambda Architecture, a robust framework for building data processing systems that can handle both real-time (speed layer) and historical (batch layer) data in parallel. By integrating streaming data with batch datasets, DataForge provides the foundation for comprehensive real-time analytics, significantly enhancing your ability to manage, analyze, and react to fast-moving data.

Stream Processing: Key Features

DataForge’s stream processing capabilities include:

• Kafka and Streaming Delta Table Integration: Seamlessly ingest data from Kafka topics and streaming Delta tables into the DataForge platform.

• Batch Data Enrichment: Enrich real-time data streams with historical batch data already residing in the DataForge Managed Lakehouse, allowing comprehensive, up-to-the-moment insights.

• Downstream Data Processing: Write enriched data back to Kafka topics and Delta tables for real-time consumption by downstream systems.

Building a Lambda Architecture with DataForge

Lambda Architecture is based on the idea that real-time systems benefit from both a speed layer (handling real-time data) and a batch layer (processing large volumes of historical data). With DataForge’s new stream processing capabilities, you can easily implement both layers, allowing for fast responses to new data while simultaneously maintaining the accuracy and depth of batch processing.

In the video above, Co-founder and Lead Software Engineer Joe Swanson showcases how to build a pipeline that exemplifies this architecture. We pulled streaming order data from Confluent’s Apache Kafka Cloud, enriched it with batch data already stored in DataForge, and output the results back into Kafka, ensuring real-time availability for downstream consumers.

Key Steps in the Demo:

1. Real-Time Data Ingestion: DataForge ingested streaming order data from a Kafka topic, with each message containing fields such as order ID, time, and item ID.

2. Enriching with Batch Data: Using DataForge’s rules engine, the real-time stream was enriched with item details (item cost, type) from a preloaded batch dataset. The transformation logic was implemented as a simple, automated rule.

3. Output to Kafka: The enriched order data, now containing additional information from the batch layer, was written back to Kafka for real-time processing by downstream systems.

Technical Advantages of a Lambda Architecture

By leveraging a Lambda Architecture, you unlock several benefits:

• Real-Time and Batch Unification: By leveraging a single tool and technology stack for both real-time (speed layer) and historical (batch layer) data, it is easy to blend data from any integration point and ensure delivery meets the target application’s demands.

• Full Lifecycle ML/AI Solution: Use batch data processing for feature engineering, model training, and evaluation. Then leverage stream processing combined with Databricks Model Serving to easily serve real-time AI solutions in production - all in one platform.

• Simplified Operations: With all the technologies under one roof, it becomes easy to track, monitor, and maintain all data sets and pipelines across the enterprise, regardless of their real-time or batch nature.

Why DataForge's Stream Processing Matters

DataForge’s integration of stream processing provides a fully managed Lambda Architecture without the need for custom stage or processing design. By connecting real-time streams to historical batch data, DataForge enables organizations to act on up-to-date insights and expand to advanced, high performance analytics use cases.

Technical Highlights:

• Accelerated Development: Build a streaming data solution integrated with your Lakehouse with easy to use configuration in minutes.

• Zero-Effort Infrastructure: DataForge automates cluster management in Databricks, dynamically provisioning job clusters for stream processing and enrichment, minimizing the need for manual intervention.

• Flexible Rules Engine: DataForge’s intuitive rules engine simplifies the complexity of data transformation, allowing for a common coding structure and patterns across both streaming and batch transformations.

Build Your Lambda Architecture with DataForge

The Stream Processing Integration makes it easier than ever to implement Lambda Architecture in your data pipelines, providing both real-time responsiveness and batch data accuracy. Whether you are working with IoT data, live transaction streams, or operational analytics, DataForge gives you the tools to handle data at any scale.

Get started with stream processing in DataForge today and transform your data infrastructure to meet the demands of modern real-time data processing.

For technical documentation, visit our developer portal, academy, or contact our team for a tailored demo.

In DataForge Cloud 8.1, we’ve taken complex data types functionality to the next level with a new, innovative concept: sub-source.

Problem Statement: Nested Complex Arrays (NCAs) are hard

When a column of data is structured as a nested complex array (NCA) with an ARRAY<STRUCT<..>> data type, it presents a unique set of challenges. 

For example, consider the ORDER dataset with the following single row structure expressed in JSON format:

The “order_detail” key contains an ARRAY of JSON values, with each element representing details about the ordered product. This structure logically represents a nested ORDER_DETAIL table that is physically stored in the ORDER.ORDER_DETAIL column.  ARRAY elements represent rows, and STRUCT keys (qty, price, …) represent columns of the nested table. ORDER and ORDER_DETAIL tables are logically related via 1:M relation. This logical-only relation is implicit by virtue of the order_detail data being contained in the parent ORDER record. Physically this data format is considered denormalized.

This data structure is fairly common in NoSQL databases, Event Hubs, and RESTful API integrations. Although the NCA structure represents a table, standard SQL DML cannot be directly applied to it, as it lacks the syntax to deal with the implicit 1:M relation. Querying and transforming NCA data is a common challenge data engineers face.

There are a couple common solutions to working around this lack of direct syntax support: 

Option 1: Normalization

The most common approach is to normalize the data structure into two related tables as illustrated below

 The Normalization process involves the following steps:

1. Unnest order_detail array into rows

2. Explode json value into individual table columns

3. Copy the primary key (order_id) of the parent ORDER record to each row 

4. Store results in new ORDER_DETAIL table, related to ORDER via M:1 relation on order_id

This approach produces a normalized data structure and enables engineers to use standard SQL operations on the ORDER_DETAIL table.

Normalization is a fairly tedious process with several complications that require extra manual effort:

• When a primary key does not exist on the parent table, it must be generated, stored, and managed on both tables

• When there are multiple layers of nesting without keys, or a mix of keyed/non-keyed NCA, the complexity of key building and management increases exponentially.

• The potential lack of keys to replace the implicit relation of the data structure is why normalization cannot be fully automated and requires a data engineer to design a unique solution for every dataset. 

• When the dataset changes, e.g. order_detail gets updated with new rows, changes need to be propagated to both tables.

• When data structure of the incoming dataset changes, e.g. product_category attribute is added to order_detail, the order_detail schema needs to be updated accordingly

To summarize, normalization solves the problem of accessing and transforming the nested data but it can be a complex and tedious process that is very difficult or often impossible to fully automate.

Option 2: Platform-specific SQL Extensions

For simple transformations limited to single value selections and aggregations, the normalization step can be avoided by using non-standard SQL such as the TRANSFORM lambda function combined with one or more ARRAY functions available in Spark SQL and Snowflake standards.

The major downside of these functions is their unintuitive, non-SQL-style syntax that becomes extremely difficult to read and debug when used for anything but the most basic operations.

The DataForge Solution: Sub-Source Approach

What if you could get all the syntax benefits of the normalization approach without processing the data into physically separate tables? With DataForge, we added a capability to create logically separate tables without physically altering the underlying data structures, allowing engineers to write standard SQL expressions on datasets containing NCAs.

DataForge compiles the logical meta-structure definitions and standard SQL expressions to generate the syntax (such as TRANSFORM functions) to process the NCA in its native format.

More simply: Work with complex datatypes as if they were flat tables with standard SQL syntax without the need for normalization.

Here is how it works in DataForge:

1. A Sub-Source is created by defining a DataForge Rule pointing to an existing NCA column in the parent Source.

• A Sub-Source is a “virtual” DataForge Source that reads the data from the parent Sources’ NCA column and writes data to a new target column in the parent Source.

• This process maintains the column-pure nature of Rules while allowing for operations on NCA data elements

2. The Sub-Source grain is defined by the NCA array, i.e. the number or elements in the array equals the number of rows in the Sub-Source.

3. A Sub-Source is implicitly related to the parent source via M:1 DataForge Relation automatically created by the system.

• Unlike standard SQL JOIN syntax, a DataForge Relation is conceptually flexible enough to natively support this implicit connection between two tables.

4. Once created, the new Sub-Source behaves exactly like a normal source for all transformation code such as Rules, Relations, and Mappings within the DataForge framework, allowing for standard SQL-expression syntax to be used.

Here is a diagram showing the new setup in DataForge:

Sub-Sources differ from regular sources in a few ways:

• Data stays physically stored in the parent source table

  • The Raw Attribute NCA column is not modified

  • Results are stored in the enriched Rule column that defines the Sub-Source

• All data processing is performed in the context of the parent source

  • Sub-sources share the inputs (batches), process, and log records with the parent source

• Refresh type is inherited from the parent source, and operates at the grain of the parent source

• Unique rules are not supported within the sub-source

How to use Sub-sources in DataForge:

• Configure relations to other DataForge sources, enabling to pull/push data from/to Sub-Source

• Configure rules within Sub-Source in the same manner as with regular DataForge source.

  • Sub-Source rules can reference attributes from any related source, including Parent.

  • Rules created in the sub-source are calculated during parent source processing, and the results are saved into the parent source hub table, within the Sub-Source rule column. Physically, each Sub-source rule adds new STRUCT key to the original STRUCT schema of the NCA column

• Define rule in another source that calculates value in the Sub-Source, using available implicit and user-defined relation paths.

  • Common use case is aggregating the value contained in the Sub-Source, e.g. SUM([ORDER_DETAIL].qty * [ORDER_DETAIL].price) at the higher level ([Order])

• Sub-Sources can be nested at multiple levels: when your data structure contains nested ARRAY<STRUCT<... <ARRAY<STRUCT>..>> schemas

• Sub-Sources can also be mapped to Output Channels

  • When mapped, it will explode/flatten the NCA, allowing for easy normalization and de-nestings in the target system when needed

Conclusion

DataForge Sub-Sources enable a fast and effective way to transform nested semi-structured data, abstracting its complexity. A Sub-Source is virtually identical to regular DataForge Source for all intents and purposes: it supports rules, relations and outputs just like regular Source. Sub-Source data is physically stored in the parent source column, saving data engineers from having to design, build and maintain complex normalization pipelines with additional storage layers and transformations.

Take advantage of this innovative new approach and accelerate your development by starting a 30 day trial of DataForge Cloud today!

We are excited to announce the release of our latest video, where Vadim Orlov, CTO and co-founder of DataForge, walks us through the automation of Change Data Capture (CDC) processes using DataForge Cloud in comparison to Databricks Delta Live Tables.

Key Highlights of the Video:

1. Introduction to DataForge: Vadim explains how DataForge, a platform designed by data engineers, simplifies and automates the complex aspects of data engineering. He begins by addressing the frequent question from customers—why choose DataForge when Databricks already offers a powerful data platform?

2. Exploring Change Data Capture (CDC): The video focuses on a critical data engineering pattern—CDC—where Vadim compares and contrasts its implementation using Databricks’ Delta Live Table (DLT) library and DataForge Cloud. He highlights the nuances of configuring CDC pipelines in Databricks while showing how DataForge automates this traditionally tedious process.

3. Manual CDC in Databricks Delta Live Tables: Vadim reviews how Databricks handles CDC, including setting up pipeline snapshots and applying changes to source tables using DLT. He explains the steps involved in accumulating snapshots and the extensive boilerplate code needed to make it work, emphasizing the use of Python for this process.

4. Automating with DataForge Cloud: Vadim then shifts focus to DataForge Cloud and demonstrates how the platform automates the entire CDC process—from ingesting source data and configuring connections to storing and processing data in bronze, silver, and gold layers of a Delta Lake. With just a few clicks, DataForge Cloud handles tasks that typically take hours with DLT.

5. Real-time Demonstration: The video features a live demo where Vadim shows how to set up and manage CDC pipelines in both Databricks and DataForge Cloud. He performs key operations such as pulling data from a SQL server, tracking changes using DataForge’s built-in observability tools, and handling Slowly Changing Dimensions (SCD) types 1 and 2.

6. Advanced Features of DataForge Cloud: Vadim also covers advanced features such as exporting configuration settings into YAML files for easy management, making the platform even more flexible for complex projects. This feature helps users manage, track, and share configurations in source control systems.

Why You Should Watch This Video:

• Time-Saving Automation: See firsthand how DataForge Cloud automates repetitive and complex data tasks, cutting down hours of manual configuration into just minutes.

• Detailed Comparison: Gain insights into the strengths and limitations of using Databricks Delta Live Tables versus DataForge Cloud for CDC.

• Live Demo: Follow along with Vadim as he walks through a real-world example, showcasing the simplicity and efficiency of DataForge Cloud.

Whether you're a data engineer looking to streamline your workflows or a business interested in automating your data pipelines, this video is packed with valuable insights.

The video is also available on YouTube with chapters to quickly navigate between sections.

Make sure to check out our website for additional resources, demos, and a 30-day free trial of DataForge Cloud!

What’s New in Plus?

Our Plus plan delivers a perfect blend of functionality and value, providing everything you need to accelerate your data projects while keeping costs predictable. Here’s what’s included:

• Flat-Rate Pricing: At just $400 per month, you get comprehensive access to our robust suite of tools without any hidden costs or surprises.

• 30-Day Free Trial: We want you to experience the benefits of Plus risk-free. Enjoy a full 30-day free trial to explore and implement our features before committing.

Key Features of the Plus Plan:

• Everything in Community: Access to all features in our Community plan, including unlimited users, functional transformation code, CLI, Spark SQL compiler, and local development.

• One DataForge Workspace: Manage your data engineering tasks with a dedicated DataForge workspace.

• Up to 50 DataForge Sources: Integrate up to 50 distinct data sources for your projects.

• Up to 500 Processes/Month: Run up to 500 processes each month to handle your data workflows efficiently.

• Browser-Based IDE: Enjoy the convenience of a powerful, browser-based integrated development environment (IDE) for seamless coding and project management.

• Automated Orchestration: Simplify complex workflows with automated orchestration capabilities.

• Change Data Capture & Schema Evolution: Keep your data up-to-date with change data capture and adapt to evolving data schemas with ease.

• 25+ Input/Output Connectors: Leverage over 25 connectors to integrate with various data sources and destinations.

• Observability Repository: Gain insights into your data operations with an observability repository.

• Managed Infra Configurations: Benefit from pre-configured infrastructure to streamline your setup and reduce management overhead.

• Monitoring and Logging: Keep track of your data processes with comprehensive monitoring and logging tools.

• Community Support: Access support from our vibrant community to help troubleshoot and optimize your use of our platform.

Why Choose Plus?

The Plus plan is crafted for startups and small enterprises seeking a scalable solution to manage and transform their data efficiently. With its flat-rate pricing, you can easily budget for your data engineering needs while accessing powerful features that support growth and innovation.

Ready to take your data engineering to the next level? Start your 30-day free trial of the Plus plan today and experience firsthand how our solutions can transform your data workflows.

For more information on how to get started or to discuss your specific needs, feel free to contact our sales team. We’re excited to help you elevate your data engineering capabilities with the Plus plan!

Part 1 of this blog series introduced a new generalized framework for working with SQL transformations by using column-pure and row-pure functions and the associated benefits for regression testing, impact analysis, and extensibility if implemented carefully.

This blog will detail how the team at DataForge implemented these concepts and created a new way for engineers to quickly build and extend data platforms.

At the heart of everything is DataForge Core - the open-source implementation of the column and row-pure transformation framework. DataForge Core focuses on taking raw data already landed into the Warehouse/Lakehouse and transforming it into the appropriate target structure(s) by applying the pure transformations in two distinct steps - one for column-pure functions and one for row-pure functions.

Because the DataForge framework works at a more granular level than tables and SQL statements, new logical objects and code constructs are required to help developers work declaratively and provide compile-time checks to ensure purity of functions within each stage, referential integrity, etc.

This blog will introduce these new logical objects, their analogous structures in computer science, and how using them further promotes software engineering best practices previously unavailable or challenging for data engineers to adhere to.

The DataForge Core Object Model

While the concepts and the associated advantages of the DataForge framework are technology agnostic, it quickly becomes cumbersome to develop and enforce this separation of logic and concerns with vanilla SQL statements, Spark, or dbt - as they require large amounts of boilerplate, duplicative code, and have no built-in structures to protect against accidental mixing of column and row functions.

To solve these issues and improve the developer experience, the new logical objects must balance a few key tenants:

1. Limit or eliminate code duplication

2. Each type of transform logic should have only one associated object type

Create as few new Class Types and instantiated objects as possible (less is more!)

Rules: Column-Pure SQL Expressions

Starting at the most atomic and fundamental level of the DataForge framework, a Rule is a Class Type representing column-pure functions that define data transformations using SQL expression syntax. They take a set of columns as inputs and produce a single column as an output. These Rules can be chained together to produce a wide variety of transformations and calculations.

Here is an example of a Rule using the DataForge yaml syntax:

rules:
- name: net_price_int
  expression: "([This].l_extendedprice - [This].l_tax - [This].l_discount)*100"

This function subtracts l_tax and l_discount from l_extendedprice and multiples the result by 100. But what is [This] and where do l_extendedprice, l_tax, and l_discount come from?

Raw Attributes: External Columns

To differentiate data generated by external systems from transformation results, incoming data columns are given a separate class type: Raw Attributes. From a computer science perspective, these can be thought of as immutable input variables and they act as the starting point for all transformation logic.

In DataForge Core, Raw Attributes must be specified in the codebase to ensure compile time type safety and referential integrity - something only available at runtime in other SQL-based frameworks.

Here is an example of a few Raw attributes in the DataForge yaml syntax:

raw_attributes:
- l_comment string
- l_commitdate date
- l_discount decimal
- l_extendedprice decimal
- l_tax decimal
- l_returnflag boolean
- l_orderkey long
- l_partkey long
- l_suppkey long

Each Raw Attribute is independent and has both a name and type, allowing for developers to think of them more as variables to be used in downstream functions, rather than in the context of a table for set-based logic.

Rules can use both Raw Attributes and other Rules as inputs for their expressions.

Source: The Column-Pure Parent Object/Class

Despite the desire to think functionally and operate at only the column level, columns ultimately live within tables/schema. In addition, there are numerous other bits of metadata and structures that may operate at the table level besides just the schema and physical table definition.

Because the new class type in question will hold Raw Attributes which act as the starting point for all other logic, it is given the name of “Source”.

A DataForge Source is a class of objects that hold Raw Attributes, Rules, and other attributes that define the context of the dataset being operated on.

Returning to the earlier example of a Rule, the DataForge syntax includes a lightweight SQL aliasing to help developers quickly specify which Source contains the column (Rule or Raw Attribute) they are referencing in the formula.

rules:
- name: net_price_int
expression: "([This].l_extendedprice - [This].l_tax - [This].l_discount)*100"

Following established syntax set by other languages, [This] allows developers to reference Raw Attributes or Rules contained within the same source as the Rule being defined.

In DataForge Core, a Source has three top level properties:

• source_name: A globally unique name for the instantiated class

• source_table: What table the Raw Attributes come from

• target_table: Where to save the resulting table that contains all Raw Attributes and all Rules

In the DataForge Framework, every Source gets its own file to help with code organization.

Here is a simple example of a Source file:

tpch_lineitem.yaml
---
source_name: "tpch_lineitem"
source_table: "samples.tpch.lineitem"
target_table: "enriched_lineitem"
raw_attributes:
- l_comment string
- l_commitdate date
- l_discount decimal
- l_extendedprice decimal
- l_tax decimal
- l_returnflag boolean
- l_orderkey long
- l_partkey long
- l_suppkey long
- l_linenumber int
rules:
- name: "rule_net_price_int"
  expression: "([This].l_extendedprice - [This].l_tax - [This].l_discount)*100"
- name: "rule_net_price_no_returns"
  expression: "CASE WHEN [This].l_returnflag IS TRUE
               THEN [This].rule_net_price_int
               ELSE 0
               END"
- name: "rule_partsupp_pkey"
  expression: "CONCAT([This].l_partkey,'|',[This].l_suppkey)"

With these three classes developers can write chains of Rules to create new calculated columns to cleanse and enrich the Raw Attributes. While working with Raw Attributes functionally with Rules is powerful, most data transformations include columns from multiple Source Tables. To support this, a new class type is needed to represent the connections between tables.

Relation: JOINs with cardinality

One critical weakness of SQL statements and other frameworks is the embedding of JOIN conditions into the table-level syntax. This almost guarantees code duplication as developers must copy-paste JOIN conditions between tables that are referenced multiple times across different transformations and complex calculations. Even worse, there is no guidance or best practices on how to manage and update changes or tweaks to the source data model. A change to a foreign key constraint in an incoming set of tables can potentially result in a refactoring and reprocessing the majority of the transformation logic.

To promote DRY code, DataForge Relations are defined and managed separately from both columnar and row-pure transformation code. By decoupling the connections between Sources from the underlying transformation code, developers can now build, manage, update, and reuse JOINs between tables.

Even better - chains of Relations between multiple Sources create a meta-model at the transformation layer that is separate from both the source system model and the target dataset model(s). This creates an explicit and clear separation of concerns between external systems, transformation, and target data models.

In other frameworks, this transformation data model is scattered among scripts, embedded in the code, and often not supported by the physical tables supporting the transformation code - making it impossible to understand and manage.

In DataForge Core, relations are stored in a dedicated file and contain three components:

• Name: Globally unique name with Source Names included on either side

• Expression: SQL JOIN ON condition detailing the logical connection

• Cardinality: Many to Many, Many to One, or One to One specified in the order of the two Sources in the Name

Here is an example of 3 different relations in DataForge Core syntax:

relations.yaml
---
- name: "[tpch_lineitem]- orderkey -[tpch_orders]"
  expression: "[This].l_orderkey = [Related].o_orderkey"
  cardinality: "M-1"
- name: "[tpch_orders]- custkey -[tpch_customer]"
  expression: "[This].o_custkey = [Related].c_custkey"
  cardinality: "M-1"
- name: "[tpch_lineitem]- partsupp_pkey -[tpch_partsupp]"
  expression: "[This].rule_partsupp_pkey = [Related].rule_ps_pkey"
  cardinality: "M-1"

The relations.yaml references the previous tpch_lineitem source example above in addition to three additional Sources: tpch_orders, tpch_customer, and tpch_partsupp.

Here are example files for these three new Sources:

tpch_orders.yaml
---
source_name: "tpch_orders"
source_table: "samples.tpch.orders"
target_table: "enriched_orders"
raw_attributes:
- o_custkey long
- o_orderdate date
- o_orderkey long
- o_orderstatus string
- o_totalprice decimal

tpch_customer.yaml
---
source_name: "tpch_customer"
source_table: "samples.tpch.customer"
target_table: "enriched_customer"
raw_attributes:
- c_acctbal decimal
- c_address string
- c_custkey long
- c_mktsegment string
- c_name string

tpch_partsupp.yaml
---
source_name: "tpch_partsupp"
source_table: "samples.tpch.partsupp"
target_table: "enriched_partsupp"
raw_attributes:
- ps_availqty int
- ps_partkey long
- ps_suppkey long
- ps_supplycost decimal
rules:
- name: "rule_ps_pkey"
expression: "CONCAT([This].ps_partkey,'|',[This].ps_suppkey)"

These files together with the tpch_lineitem.yaml example above create the combined transformation model:

Some important notes:

• The transformation model includes both Raw Attributes and Rules

• Relations can be defined with one or more Raw Attributes and/or Rules

• The Relations often reflect the source system model and augment or extend it with enhanced calculations or combinations of quality checks and cross-table calculations

• An example of a Rule being used in a relation is [tpch_lineitem]- partsupp_pkey -[tpch_partsupp] where the JOIN ON statement uses two Rule expressions to define the Relation rather than Raw Attributes.

• The transformation model is global across all target datasets and can easily be incrementally extended with additional column-pure functions and relations - further promoting DRY principles.

• In DataForge, this transformation model and the associated persisted tables are also known as the Hub.

Traversal: Using Relations to retrieve data across Sources

In the examples above, the Rules only used Rules and Raw Attributes available on the Source where the Rule is defined. To retrieve data from other Sources, Relations must be used to traverse between Sources and reference Raw Attributes or Rules defined in remote Sources.

Because all Rules have a parent Source object, they have a starting point for navigation of the Hub. A Traversal is when moving logically from one Source to another by using a Relation. Traversals do not have their own object or class, but are distinct from Relations as they have a procedural direction of movement. Relations are defined declaratively and can be traversed in either direction.

Cardinality is critical to include as part of Relation definitions to enforce column-purity of rules - as the number of rows in the resulting dataset from a JOIN between two tables is determined by the cardinality and the direction of the traversal.

When traversing a cardinality of many-to-one or one-to-one, the number of rows is unchanged - meaning any Rule that uses the Relation in that direction maintains column-purity. When traversing across a one-to-many or a Many-to-Many cardinality, the Rule must include a pre-aggregation statement on the many side to avoid data duplication or the rule will fail at compile time as Rules must be column-pure.

Without any pre-aggregation, traversals must be many-to-one or one-to-one to maintain column-purity. They can be chained together indefinitely so long as they continue to move from many-to-one or one-to-one.

When moving down grain (from many-to-one) in order to maintain column-purity, a pre-transformation must be defined to change the many-to-one relation into a one-to-one. The most common way to do this is by simply grouping by the ID(s) used in the one-to-many JOIN expression. In other words, performing a “rollup” calculation from the lower grain table to the higher. There are other ways to create this new one-to-one relation, but due to the common nature of rollups, this group-by-id pattern is baked into the DataForge Rules syntax and compiler.

Here is an example of a rule using a lookup from tpch_lineitem to orders:

tpch_lineitem.yaml
...
rules:
...
- name: "rule_discount_ratio"
expression: "[tpch_orders].o_totalprice / [This].l_discount"

To reference a Raw Attribute or Rule from another Source that is traversable via many-to-one relations from the current Source, the simple aliasing syntax of [<Source_Name>] can be used rather than [This]. When there are multiple traversal paths between the current Source and the Source that contains the Raw Attribute or Rule, an extended syntax is required. For an example, see this Rule in the sample project included with DataForge Core on Github.

Here is an example of a rule performing the rollup pattern discussed above:

tpch_orders.yaml
...
rules:
...
- name: "rule_total_price_less_tax"
expression: "[This].o_totalprice - SUM([tpch_lineitem].l_tax)"

When referencing a Source field that includes a one-to-many traversal, DataForge requires the reference to be wrapped in an aggregate function to perform the “rollup” operation. If the reference is not wrapped in an aggregate function, the rule will fail to compile.

A few final notes about Relations, Traversals, and Rules:

• Multiple Relations can be defined between two Sources with different cardinalities

• A Source can have a self-referencing Relation (parent/child hierarchy for example)

• Rules and Relations are infinitely composable with each other

• Smaller (less complex logic) Rules and Relations are preferable and best practice when possible

• Small rules promote code reuse across multiple different use cases

• Multi-Source circular references will cause compile-time failure in DataForge Core

• Solutions for these patterns are available in DataForge Cloud, but are outside the scope of this blog

Visual representation of a data flow using the Class Types discussed so far:

For contrast, here is a visual example of using dbt models or other frameworks to perform similar transformations:

Some key differences between the two approaches:

• In DataForge, there is exactly one target table for every incoming table

• In dbt or Spark, an infinite number of models/scripts can be created with associated target tables

• When to create a new model vs. augment an existing one?

• In DataForge, target tables contain all columns from original table plus Rules, allowing for easy use and re-use downstream without the need to update the transformation code

• In dbt or Spark, it is at the developer’s discretion to pass through columns and sometimes impossible due to lack of column-purity, resulting in potential refactoring or regression issues with new changes or additions of columns

• Each model is a “black-box” of logic, with JOINs and other logic embedded into the model and difficult or impossible to share across models

• Some code snippets can be shared with Jinja templates and macros, but quickly become clunky to manage at scale

• Due to black-box nature, column-level lineage becomes a complex calculation and requires separate tooling to visualize and understand logical dependencies

Output Columns: Target table definition(s)

Now that all column-pure transformations are completed, the next step is to perform any required row-pure transformations and save the final results to a target table.

Output Columns are aliases and datatypes of a target table. These act as the Data Definition Language (DDL) for target tables, allowing developers to control target table schemas within the DataForge framework.

In DataForge syntax, Output Columns follow a very SQL-like pattern similar to Raw Attributes:

...
columns:
- "Order Name" string
- "Order Date" date
- "Revenue Less Tax" decimal(18,2)
...

Column Map (Mapping): Final aliasing and new-grain aggregates

With the final target columns defined by Output Columns, the next Class Type to explore are Mappings.

In their basic form, Mappings connect Raw Attributes and Rules to Output Columns. They define where the data for a target Output Column comes from within the Hub. Said differently, they are a column-pure aliasing operation with no other calculations. Whereas Rules allow for any Column-pure transformation, Mappings only allow for aliasing.

This extra layer of column-pure aliasing is provided in the DataForge framework to allow developers to define a lightweight semantic layer with different column names from the ones used in Raw Attributes, Rules, and Relations. Column names convenient to use in the transformation layer(s) are often different from the final metrics and dimensional attributes used by business analysts and data scientists. Additionally, the same data can often have different required target names depending on the department, target end users, or API endpoint.

Here are a few basic mappings in DataForge syntax, with the Rule or Raw Attribute listed first and the Output Column second:

...
mappings:
- o_name "Order Name"
- o_orderdate "Order Date"
- rule_total_price_less_tax "Revenue Less Tax"
...

Aggregate Mappings behave similarly to a SQL Group By statement, but have the aggregate flag set as the operation_type and include at least one mapping which has an aggregate function applied. All columns that are not wrapped in an aggregate session are implicitly included in the “group by” statement.

...
operation_type: "Aggregate"
mappings:
- o_name "Order Name"
- o_orderdate "Order Date"
- sum(rule_total_price_less_tax) "Revenue Less Tax"
...

Important note:

Aggregates in Mappings should only be used when the grain of data does not exist naturally in another Source upstream. If the same operation can be performed within the column-pure Source layer via Rules and Relations, it should be. Rules/Relations are preferred for extensibility and code reuse when compared to Mappings.

Channel: What Source to get the data from

In the previous examples, we provided target columns and mappings, but failed to provide any syntax or instructions to let the framework know what Source in the Hub to look for the Raw Attributes and Rules as the input for the mappings.

A Channel points the Mappings to a Source that holds the Raw Attributes and Rules.

...
channels:
- source_name: tpch_orders
  operation_type: "Aggregate"
  mappings:
  - o_name "Order Name"
  - o_orderdate "Order Date"
  - sum(rule_total_price_less_tax) "Revenue Less Tax"
...

With the Source_name specified, any Raw Attributes or Rules can be used in the Mappings to populate the target Output Columns.

Filter: Row-pure, pre-aggregate WHERE clause

Arguably the most used row-pure transformation, the Filter class type follows the same patterns as a SQL WHERE expression. Any boolean expression using any column from the Source specified in the Channel (including those not in any Mappings!) can be used to reduce the number of rows feeding into the aggregation mappings or target table.

The syntax in DataForge is simple and easy to understand if familiar with SQL WHERE expressions:

...
channels:
- source_name: tpch.orders
  filter: [This].o_orderstatus == ‘P’
  operation_type: "Aggregate"
  mappings:
  - o_name "Order Name"
  - o_orderdate "Order Date"
  - sum(rule_total_price_less_tax) "Revenue Less Tax"
...

Despite their importance, Filters are relatively simple, and thus can be handled as a property on the Channel class rather than their own dedicated Class/Object.

Output: The (mostly) Row-pure parent Object/Class

An Output acts similarly to a Source as a container for all of the Row-pure functions as well as the semantic layer provided by the mapping aliasing. Combining Output Columns, Channels, Filters, and Mappings, Outputs define the target table DataForge produces.

Bringing the full syntax together and adding a name for the target table:

revenue_by_order_date.yaml
output_name: revenue_by_order_date
columns:
- "Order Name" string
- "Order Date" date
- "Revenue Less Tax" decimal(18,2)
channels:
- source_name: tpch_orders
  filter: [This].o_orderstatus == ‘P’
  operation_type: "Aggregate"
  mappings:
  - o_name "Order Name"
  - o_orderdate "Order Date"
  - sum(rule_total_price_less_tax) "Revenue Less Tax"

Similarly to the Source, an Output holds all of the individual, broken-down components that make up row-pure functions, and their associated references to upstream columns from a Source.

The only missing row-pure operation is the UNION. This fits cleanly into the Output syntax by simply adding an additional Channel with associated mappings.

Here is an example of unioning two sources together:

reporting.yaml
output_name: reporting
columns:
- "Part Supplier Key" string
- "Part Key" string
- "Supplier Key" string
- "Revenue No Returns" decimal(18,2)
- "Supplier Cost" decimal(18,2)
channels:
- source_name: tpch_lineitem
  mappings:
  - rule_partsupp_pkey "Part Supplier Key"
  - l_partkey "Part Key"
  - l_suppkey "Supplier Key"
  - rule_net_price_no_returns "Revenue No Returns"
- source_name: tpch_partsupp
  mappings:
  - rule_ps_pkey "Part Supplier Key"
  - ps_partkey "Part Key"
  - ps_suppkey "Supplier Key"
  - ps_supplycost "Supplier Cost"

By specifying two different channels in the Output, DataForge stacks the two channels together via UNION. For convenience, not all columns must be specified in the mappings. Any missing mappings will be NULL for all rows produced in the Output Columns by that Channel.

With this final piece of the puzzle, DataForge has provided exactly one place for every type of relational algebra or metadata - with the only exception being column-pure alias operations available in both Rules and Mappings.

The Full Framework:

Although there are eight new Class Types introduced in the DataForge framework, the intent is to organize structures that already exist within data engineering and SQL relational algebra - not create brand new syntax or logic for transforming data. By breaking down the Spark Dataframe, dbt model, or SQL stored procedure to this more granular, organized, and structured framework, the foundation has been set to develop following computer science principles and best practices, as discussed in Part 1.

The DataForge framework is also meant to be easy to understand and work with, no matter if your background is in data engineering or software development. The reason both the ERD and UML visuals are provided is because DataForge is simultaneously both a database schema and a class structure.

When developing in DataForge Core using the YAML syntax, developers can engage with the framework in the Class structure to make things such as inheritance and dependencies easy to work with and manage.

If you dig into the codebase for DataForge Core, however, it becomes clear that the framework is a database-first architecture built primarily in PostgreSQL functions that has matching physical tables for every Class type discussed in this blog.

In the process of compiling the codebase (dataforge build command), all Class objects are stored in tables that are easy to query, analyze, and build custom reports on. This means that with a large codebase of 1,000s of Sources and millions of Rules can be easily searched with the most powerful and flexible tool for any data professional: SQL.

Here is a combined view covering both Sources and Outputs of the dataflow for all tables processed by DataForge:

All Sources create a dedicated Hub table, even if no Rules are defined, allowing for easy extensions in the future. Outputs can use as many Sources as needed, and can also branch off a Source multiple times. This means that the same Rules can be reused across multiple target use cases and datasets, encouraging developers to simply branch off the existing Combined Transformation Model and Rules rather than build a parallel and largely duplicative pipeline for their own needs.

A few final notes about DataForge not mentioned in the sections above:

• Sources and Outputs have a many-to-many relationship via channels

• Any source can have an unlimited number of Output definitions and an Output can have an unlimited number of Channels (UNIONs) powering it.

• If adding new row-pure logic and target table, it is often as easy as creating a new Output and mapping to existing Raw Attributes and Rules.

• DataForge is typesafe at compile time

• This is why all Raw Attributes and Output Columns must be defined with their datatypes in the code.

• DataForge Cloud provides code automation tools to automatically hydrate Raw Attributes from existing tables, external files, etc. as well as handle complex schema evolution scenarios, and more!

• Orchestration is automated at compile time

• The DataForge compiler is able to understand how different Raw Attributes, Rules, and Relations are used by each other and can build a DAG to process everything appropriately for both the Hub and Outputs.

• This SQL is saved in the Targets folder in DataForge Core for analysis and tweaking if needed.

• DataForge Core cannot handle a few corner cases such as complex multi-traversal circular logic.

• DataForge Cloud approaches Orchestration automation in a different way than Core. Cloud builds on the Core scheduler with an event-based microservices-style workflow engine which covers infrastructure, storage, incremental loads, and many other considerations - allowing for any type of orchestration to be automated.

At DataForge, our mission is to automate the tedious and simplify the complex. While at first glance, DataForge Core may seem more complex than writing full SQL statements, when working with enterprise-scale codebases it suddenly becomes much more important to have a system and structure to help developers work together and grow the codebase over time than scraping together a quick script.

Combined with code automations and the web-based development environment provided by DataForge Cloud, it has never been easier to build and evolve a world-class data transformation codebase that will allow your organization to focus on the business logic and avoid the traditional pitfalls of spaghetti code that proliferates with imperative scripts and table-based models.

In the next blog in this series, we will cover the standard processing stages and the out-of-the-box, fully automated Medallion Architecture provided by DataForge Cloud.

At DataForge, we have spent the past 6+ years researching the fastest, easiest, and most extensible ways to build and maintain transformation code. Our mission is to help teams focus on the logic and skip as many tedious or complex steps as possible - without sacrificing a developer’s ability to build custom logic.

The Problem

The majority of enterprise data engineering code deployed today is structured as a monolith, combining transformation logic with persisted database tables. Monolithic code lacks flexibility and often results in an unmanageable mess of spaghetti code as new source and target datasets are added over time. Software engineering has solved this problem by separating transformation logic and state. This separation combined with concepts such as functional programming, microservices, and other principals make code easier to build, extend, and manage.

There are two primary reasons data engineering has lagged behind software best practices and principals:

1. Data transformation code by nature changes the state of external objects (database tables).

• This side-effecting forces code to follow imperative computer science patterns rather than declarative.

2. Many data transformation tools are scoped at the table/dataset level and can’t be split into modular and composable functional code blocks.

To illustrate these points, let’s use SQL: the most popular data transformation language today. SQL DML is an immensely powerful scripting language that can handle very complex transformations. SQL SELECT statements are declarative in isolation: every SQL platform includes a query engine that translates a SQL statement into an optimal physical data execution plan, allowing developers to focus on logic.  

However SQL has some key traits that make it challenging to use for complex transformation development:

• SQL was created as an interactive command tool. It does not support key modern programming language features such as object oriented or functional programming structures, libraries, or packages.

• The minimal unit of transformation is a dataset (table), limiting the way transformation code can be structured, managed, and re-used.

• Multi-statement queries with INSERT, DELETE, UPDATE, and other built-in statements are imperative and require external procedural scripting harnesses to ensure proper order of operations.

In addition to these key traits, SQL alone lacks structures or frameworks to prevent multiple statements or scripts from side-effecting the same object or table. This opens the door to imperative patterns and complex webs of state-dependent logic.

As illustrated above, every black-box of transformation logic not only operates at the table level, but multiple different transformation scripts can side-effect or modify a target table. Target tables can also be used as inputs to the same script, creating stateful “can only run once” or “must be run exactly twice” patterns in the processing chain.

Apache Spark took the first step towards declarative programming schemes by introducing the concept of immutable DataFrames (and later Datasets). Spark also introduced Scala and Python APIs that with careful use can be implemented with declarative patterns.

Tools like dbt then furthered this trend by replacing DataFrames with dbt models and simplifying the code from Python or Scala to something more SQL-like. The primary tradeoff is transformations can only be strictly defined at the table-level due to the SQL-like nature.

By forcing the target data of each spark function/dbt model to be immutable and protected from side-effects by other functions/models, Spark and dbt force developers to think more declaratively when developing data pipelines and transformation code, resulting in improved pipeline architectures and easier to manage data platforms.

So if Spark and dbt have made it possible to be more declarative, why are so many data platforms still built as monoliths and devolve into spaghetti code?

Two main reasons:

1. The table scope of immutability is too broad, resulting in code duplication.

2. Developers mix and match different transformation types freely and custom design the pipeline framework and procedure independent of logic contained in other data flows.

TL;DR Modern data transformation frameworks like Spark and dbt do a great job helping teams build transformation code quickly, but fall short when large codebases need to be extended, managed, or augmented due to their wide scope of immutability and lack of structure defining when in the processing chain to perform specific types of transformation.

Breaking up the monolith

In order to introduce additional declarative patterns and functional coding best practices, the scope of immutability must be broken down to a lower level than tables.

The reason the minimum scope of immutability for both DataFrames and dbt models is a table is because data engineers most commonly think about transformations using set-based operations using SQL. Written as a function, SQL statements look like the following:

Where T represents input/output tables of the function.

With Spark and dbt, multiple immutable tables are passed in to a function that produces a single immutable output table. While straightforward and useful for simple calculations and transformations, a single function’s definition can potentially contain thousands of lines of code (or more!), making it unwieldy to develop, manage, debug, and modify. This means seemingly simple tasks such as adding a single column to SQL pipeline can require months of development.

Spark DataFrames and dbt models give developers the option to break down these functions into smaller and more manageable chunks, however even when broken down to their most atomic levels, they still operate at the table-level for input and output arguments:

While a step in the right direction, by only allowing developers to write table-level functions as the minimum scope, it limits their ability to encapsulate and compose certain types of logical transforms. In addition to this minimum scoping problem, developers must design the structure and organization of these functions based on a difficult balance between ease of development, performance, and consistency with patterns platform-wide.

While skilled developers and architects can manage this balance during a green-field build of a few hundred pipelines, deciding when or how to restructure a solution due to new requirements that has thousands of pipelines becomes an impossible equation to solve effectively without a different approach.

Grouping transforms by dimension

In order to build up a lower-level framework to address these problems with table-level functions, the table logic must be decomposed and reorganized into smaller parts using a smaller scope of operation (immutability).

Taking inspiration from linear algebra, table-level functions can be broken down by what dimension the component functional logic operates on: cell, column, or row. To maximize composability and downstream benefits, it is also important to ensure that the logic is broken down in a way that results in pure functions. This means that no other function or external process can affect the same output data (cell, column, or row, respectively).

Cell-pure Functions:

Starting with the lowest-level and most versatile group, cell-pure functions take a set of cells as arguments and generate a single value.

These can be most easily understood as “excel-like-functions” and while they are the most basic and atomic version of a pure function, they are not traditionally useful in the context of data engineering, as they do not follow set-based logic or relational algebra algorithms.

Column-pure and Row-pure Functions:

In order to make cell-pure functions more useful and applicable to data transformations, both the inputs and output must be contextualized within the structure of a table.

To do so, and continuing on our application of linear algebra to data engineering, one of the two dimensions (row or column) of the table must be held constant while the other is allowed to change. Applying cell-pure functions with this constraint creates a new type of pure function in the dimension of change.

Thus:

• Column-pure functions are transformations where the number of rows do not change between input and output

• Row-pure functions are transformations where the number of columns do not change between input and output

Column-pure Functions:

1. Are always in the context of a single immutable starting table (T₁)
• T₁ is the table that is constrained to one dimension via no row changes
2. Are made up of a cell-pure function(s) evaluated for every row in T₁
3. The output is stored as new immutable column(s) (X_1..n) appended to T₁, creating a new immutable table R

The usefulness of these column-pure functions starts to become apparent when the independence of the inputs and output are analyzed:

• Columns A, B, and C are all independent of each other
• X₁ is dependent on A and B, but not C

With column-pure functions, transformation dependencies can be broken down to the columnar level rather than the table-level. Additionally, the dependencies are implicit and transparent from the functional definition of the transformation, rather “black-box” at the table-level.

Another powerful feature of column-pure functions is their ability to be combined together without the need for intermediary storage or orchestration.

Extending our original example, above, the result table R can now be treated as the starting Table T₁ for use by a new column-pure function:

In this second example:

• A, B, and C are independent columns
• X₁ is dependent on A and B, but not C
• X₂ is dependent on B and X₁
• • Transitive: X₂ is dependent on A and B, but not C

These combined examples highlight how developers can now think in columns rather than tables, and by maintaining pure functional transforms, key benefits emerge:

• Logic can be written and managed at the column-level, while processing can be managed at the column, multi-column, or full table level for performance.

• New logic can be branched from any intermediary function or column, preventing code duplication and promoting DRY principals.

• New column-pure logic never impacts existing target data, eliminating the need for impact analysis or regression testing.

• If an existing function must be modified, impact analysis is simple to calculate due to implicit functional chains.

Here is another alternative visual aid to help illustrate how new logic can be easily added to the existing codebase without needing to modify existing functions, thus eliminating regression testing:

Complex Column-pure Functions (JOINs)

Arguably the most powerful and important operator in SQL, JOIN statements are complex and over-loaded with many different types of functional transformations. In order to support JOIN operations within the context of column-pure functions the different JOIN types must be segregated (LEFT, INNER, etc.) and the cardinality of the JOINs must be considered. This is because a JOIN statement can potentially change the number of rows - and thus immediately disqualify it from being column-pure.

With the definition of column-pure including “Number of output rows is determined by number of input rows (no grain change)” the starting table of the JOIN and direction is critically important in adhering to these constraints. The easiest way to approach this problem is to assume all JOINs are LEFT JOINs and the number of rows of the function output is defined by the first table argument (the table furthest on the “left”).

Now that multiple tables are involved, a sample data model is helpful for illustration:

Complex column-pure functions that include a join must:

• Start with a set grain of data (the “left”-most table)

• Define JOIN condition(s) as a boolean-typed column-pure function

• Optionally apply a pre-calculation to the “right-most” table in the JOIN chain to ensure no new rows are created (e.g. aggregation)

Examples:

These two examples show how even complex calculations and SQL statements can adhere to the restrictions of column-pure functions.

In the Rollup example, a WITH statement (Common Table Expression) is written first to calculate a rollup of the invoice table and generate a set at the same grain as the customer table. Now that the grains match, a JOIN can be executed without changing the number of rows in sales.customer (the left-most table) as part of the operation.

Row-pure Functions:

While column-pure functions can cover a large variety of different transformations, they only handle one dimension of the table.

Row-pure functions output a table with different number of rows:

• Output is a table with more or less rows (different grain)

• Output columns (schema) are identical to input columns

• No new columns are added or removed

• Can be first-order (Union) or higher-order function (Filter)

Examples:

Unlike column-pure functions, where the examples discussed above were just a small sample of different possible transformations, row-pure functions are much more limited in diversity, but are equally powerful constructs.

Because they require zero change in the column set (schema), row-pure functions are by definition limited to simply adding or removing rows.

Any SQL transformation that changes the number of rows in a table or result can be decomposed into one of the two examples of row-pure functions in the table above. For example

• INSERT is different syntax for UNION

• DELETE is different syntax for FILTER

• DISTINCT is a column-pure ranking window function followed by a FILTER

• UPDATE is a DELETE followed by an INSERT

• CROSS JOIN is UNION repeated in a loop

• GROUP BY is DISTINCT combined with column-pure aggregate window functions

• HAVING is a FILTER

Row-Pure Function Summary:

• Row-pure functions operate at the table-level, but specifically avoid schema changes to improve composability with other row-pure functions.

• Operations that change the number of rows are often blended with column-creating functions (such as Group By), and must be decomposed into UNION and FILTER statements to identify the row-pure component(s) of the higher-order function.

Bring it all together:

With these two strictly defined functional types, all SQL transformation logic can be broken down into either Column-Pure and Row-Pure functions:

With the pure functions now defined and table-level transformations decomposed, the original problem statements can now be addressed:

• Data engineering code is monolithic and hard to extend or modify once built due to its freeform procedural nature.

• SQL is the gold standard for transformation code, but has some major limitations that make it difficult to follow software engineering best practices and principles.

Key findings from breaking down the logic:

• Column-pure functions are easy to combine with each other and new logic can be added without the need for downstream regression testing.

• Row-pure functions are easily composed with each other and should be separated from column logic to avoid table-level dependencies.

• Unlike column-pure functions, if new row-pure functions are added, then it creates an entirely new logical table vs just adding a column to an existing table and could affect downstream processes.

• Due to their minimized dependencies, both Column and Row-pure functions can be easily reused and integrated into various data pipelines, akin to the mixin pattern.

Thus, to make data engineering code as declarative as possible and avoid the monolithic patterns and architectures, the process is as follows:

1. Analyze transformation logic and decompose the relational algebra into column-pure and row-pure functions.

2. Group as many column-pure functions and row-pure functions together into as few processing stages (each stage only has one type) as possible.

• The large majority of column-pure functions should be grouped together to maximize advantages and extensibility.

3. As new requirements or feature additions are made, take advantage of the functional nature of the pure processing stages and add more functions to an existing stage if columnar, or an additional branch stage at the very end for row-pure logic.

Take all logic and decompose it into column-pure and row-pure functions, eliminating any higher order combination row and column combined functions such as GROUP BY.

Determine the fewest number of stages possible to process the datasets while grouping as many column-pure functions together as possible to maximize framework efficiency.

New column-pure logic can be added as a new function in the column-pure stage, with no downstream regression impact as none of the existing target columns are affected.

For new row-pure functions, it naturally creates a new table definition, so a dedicated parallel processing stage is often the most efficient design and avoids any regression impact to existing processes.

This simplified example shows how developers can use these concepts and frameworks to develop a better pipeline architecture that is easier to manage and change over time.

Rather than a web of spaghetti code and monolithic table-level custom pipeline flows, the majority of logic can be broken down into column-level expressions and standardized stages.

This allows teams to scale-up their solution to thousands of pipelines without compromise - meaning the 1,000’th new column in a target dataset is just as quick to build and deploy as the 1st.

At DataForge, we developed an enterprise data management platform around this framework for pipeline development.

To learn more about the extended framework built around column-pure and row-pure functions, check our Part 2 of the blog series.

Our open source product DataForge Core is a free way for engineers to explore pure functional development and start to think about data transformations declaratively rather than procedurally.

DataForge Cloud extends DataForge Core with additional framework components, a web-based development environment, automated infrastructure, and many more powerful features that allow developers to build, run, and manage functional data transformation code faster and easier than any alternative.

In the next blog in this technical deep-dive series, we will detail how DataForge applied these base concepts and extended the framework to handle complex incremental loads, circular functional dependencies, and other advanced patterns.

Integrate Event Data using Kafka into your workspace

DataForge Cloud 8.0 now enables event integration to do batch writes and reads from any Kafka topic. Data is read directly from the topic in batches and then decoded to String or Struct fields using Avro or JSON schemas to help decode Event stream messages. Schema Registry is supported for platforms that implement it, such as Confluent. For output, data is encoded using Avro or JSON schemas, with Schema Registry also being supported for encoding messages.

In this update, we’ve simplified the process of connecting to a Kafka server and pulling from a Kafka topic or writing to a Kafka topic with our Connection features for extending sensitive and standard JDBC connection parameters. This means users can now connect more efficiently and securely, reducing the complexity and potential errors in the setup process. Users can also reuse the connection to a Kafka server to pull various topics hosted within, rather than defining or storing connection information for every custom notebook written to pull from Kafka in Databricks. This significantly streamlines the workflow, saving time and reducing redundancy, which enhances productivity and minimizes the risk of configuration mistakes.

Offset information is stored on each ingestion, enabling incremental pulls from a topic without the need to manage offset starting and ending points, which can be a pain point for creating a reusable notebook in Databricks that doesn’t pull the entire stream every time. This feature ensures more efficient data processing by only pulling new data, preventing unnecessary data reprocessing, and reducing computational load, which is particularly beneficial for maintaining optimal performance and resource management in Databricks.

Currently, only batch processing is available; however, the new connector works natively with the existing scheduling services, allowing incremental processing every 15 minutes. Ingestion supports defining starting and ending offsets, with the ability to set the starting offsets to "deltas" to ensure each ingestion always pulls the latest batch from the Kafka topic. Additional parameters exist within the Source ingestion parameters for further refining the data flow.

Key/value columns are supported for output, with the mapping of String and Struct fields being allowed in the key and value. To create a Struct field in DataForge Cloud, a rule must be created on the Source to build the struct using the rule expression “struct([col] as col1, [col] as col2…)”. This rule-based approach ensures consistent and accurate struct creation, reducing manual errors and streamlining the data preparation process.

Using the Output schema encoding for JSON or Avro and structuring Output mapping to write key-value pairs directly from DataForge greatly simplifies the notebook code needed to format data for Kafka writes. This simplification of notebook code not only reduces development time and complexity but also enhances the maintainability and scalability of data pipelines, allowing for quicker iterations and easier debugging.

Future Enhancements

• Full streaming integration for ingestion and output

• Support for schema definition in JSON Schema format

These enhancements will enable more use cases and integrations for data sources in DataForge Cloud.

Meet Talos: DataForge's Virtual AI Assistant, powered by advanced large language models (LLMs). Talos brings a new level of efficiency and intuitiveness to data management. Here’s what Talos can do for you:

• Identify existing connections by name

• List available external source database schemas within a remote connection

• Locate tables within a connection - with optional filtering by schema and table name

• Create new sources from tables - with optional data extraction and provide a link to open the source and view the data

• Find existing DataForge Cloud sources by name or table name - and provide a link to open them

• Search for attributes (raw, rule, or output columns) by name, type, and container name, and provide a link to view the full lineage screen for the attribute

Many of these tasks can also be accomplished using the standard DataForge Cloud UI, but Talos offers a more intuitive alternative. Notably, finding attributes and displaying lineage, previously only possible through metastore database queries, is now easily handled by Talos - showcasing the unique advantage of an LLM-powered tool.

Future Enhancements

We have exciting plans to expand Talos's capabilities. Here are a few areas we're considering for future releases:

• DataForge Online Help Integration: "Explain the concept of relation and relation types"

• Operational Status Queries: "Summarize failed processes by source in project X since last Friday"

• AI-Augmented Configuration Assistance: "Create a new connection from PostgreSQL" or "Add a rule to source X to calculate Y"

DataForge focuses on features that fully utilize LLMs' natural language processing capabilities, aiming to enhance the user experience for both new and experienced DataForge users.

As always, we look forward to your feedback. Let’s shape the future of data engineering together! 🚀

DataForge Cloud 8.0 enabled full support of struct and array complex types. For those unfamiliar with Apache Spark's complex types: a struct is akin to a table row containing one or more fields (similar to table columns) with specific data types. Structs can be nested at multiple levels, with fields containing other structs. Arrays represent collections of elements of a specific type, such as integers or strings. Arrays of structs are particularly noteworthy because they essentially encapsulate an entire table within a single cell of a parent table.

Common Uses

These types are widely used in scenarios involving messaging, events, APIs, and other systems generating semi-structured (JSON) data. Despite their power, working with nested struct types has historically been complex, especially when managing evolving schemas that include multi-level nested arrays of structs commonly found in many messaging protocols and API responses.

Our Approach

At DataForge, our philosophy when managing schemas has always been KISS+YAGNI. DataForge Cloud automates most data type decisions - from detecting the data type during ingestion, capturing it for each columnar transformation, and mapping it to the corresponding output type. We’ve applied the same principles to complex types by:

• Automatically capturing schemas of raw attributes during ingestion

• Determining transformed column types during rule validation

• Mapping output column types to appropriate complex types, like Snowflake’s VARIANT

This approach helps users navigate complex attribute hierarchies when defining transformations, as illustrated below:

Schema Evolution

Handling schema evolution for complex types adds another layer of difficulty. Spark complex types are rigid and not easily castable like simple types. Recognizing this, we extended our schema evolution model to accommodate complex types. Here are five new configuration options that control schema evolution for each DataForge Cloud source:

The Upcast option is particularly powerful for handling JSON data with sparse attributes, enabling seamless backfilling of incoming data into existing column types when schemas are compatible.

These options give DataForge Cloud users fine-grained control over schema changes. The default settings of "Add, Remove, Upcast" work for most scenarios, with additional options providing further control.

Future Enhancements

We plan to introduce the following features to enhance complex type management and schema evolution:

• A sub-source feature for working with nested array<struct> attributes similar to regular DataForge sources.

• A manual "Merge" option to consolidate multiple column clones with compatible schemas.

These enhancements will complete our vision for managing complex types and schema evolution in DataForge Cloud.

DataForge continues its mission to make data management, integration, and analysis faster and easier than ever

DataForge, the Declarative Data Management platform, has released its latest version, enhancing ease of use and adding new connectivity options for Databricks customers. With these updates, DataForge continues its mission to empower enterprises to build, grow, and expand data products.

Introducing DataForge Cloud 8.0! Packed with powerful new features and enhancements, this version is designed to elevate your data management capabilities. Here’s a glimpse of what’s new:

• Handle Complex Data Types with Ease: DataForge Cloud 8.0 now supports array and struct data types commonly found in semi-structured datasets like JSON, API results, and streaming data. Leverage these complex types seamlessly in rules, relations, templates, output mappings, and filters. The intuitive expression editor helps you navigate nested types effortlessly, ensuring a smoother data processing experience.

• Integrate Event Data with Kafka: DataForge Cloud 8.0 introduces Kafka integration for Sparky Ingestion and Output, enabling batch reads and writes from any Kafka topic. Support for Avro and JSON schemas, along with Schema Registry connections, ensures seamless event data integration into your workspace.

• Introducing Talos AI Assistant: Meet Talos, DataForge's new virtual assistant. Talos can help you find connections, list and search schemas and tables, create sources, and much more. Available in preview for DataForge Cloud Professional and Enterprise customers on AWS, Talos is here to enhance your data management experience.

For more details, please see the release notes below or visit www.dataforgelabs.com.

Table of Contents

1. Handle Complex Data Types with ease

2. Extended Schema Evolution

3. Integrate Event Data using Kafka into your workspace

4. Streamlined Import/Export format 2.1

5. Project management puts more control in your hands

6. Easily duplicate Workspace level configurations

7. Enhanced Processing details user experience

8. Benefit from upgraded Databricks Runtime

9. Google Cloud Platform (GCP) support

10. Talos AI Assistant

Handle Complex Data Types with ease

DataForge added support for array and struct data types. These types are common in semi-structured datasets (JSON), API results, and streaming.

You can now use these complex types along with scalar types in rules, relations, templates, output mappings, and filters. The expression editor intelli-sense will help you navigate nested complex types by listing child keys for each nested level. Structs and Arrays display a metadata icon with further schema definitions. You can also use all available spark built-in functions in rules.

In output mappings, you have the option to expand mapped struct attributes. This will create new columns named as <original struct column name>_<key> and map corresponding struct keys into new columns.

You can also do this for all structs in the channel by clicking Expand Structs in the channel options menu. 

Clicking this option recursively will expand nested structs to the next level.

DataForge recommends using Databricks' higher-order functions for working with complex nested arrays. In a future release, DataForge plans to introduce a new sub-source feature to fully integrate nested arrays of struct operations into the platform at the columnar level, further streamlining the experience of working with complex types.

Extended Schema Evolution

A new “Schema Evolution” parameter extends DataForge capabilities with the following settings: 

The Schema Evolution parameter is in the source parameters Parsing section. It contains a drop-down with 7 preselected combinations of the above settings. The default setting is Add, Remove, Upcast.

Integrate Event Data using Kafka into your workspace

Event integration using Kafka is now supported for Sparky Ingestion (Databricks cluster-based) and Output, to do batch writes and reads from any Kafka topic. Avro and JSON schemas and Schema Registry connection are supported using parameters defined in the Connection.

Currently, only batch reads and writes are supported, with full streaming integration coming in a future release. Starting and ending offsets are supported on Ingestion, with the ability to set the starting offsets to "deltas" to ensure each ingestion is always pulling the latest batch from the Kafka topic. Additional parameters exist within the Source ingestion parameters for further refining the data flow.

For output, key/value columns are supported, and users set up the value column in a rule if it's necessary to make a complex type in the value.

Streamlined Import/Export format 2.1

New export format 2.1 removes all attributes containing default values from YAML files. This results in a significant 2-3x reduction in the size of export YAML files and less code to manage and source control.

Additionally, relations have been moved to a single standalone relations.yaml file in the root folder of the import zip. The relations format has been simplified as in this example:

Centralized relation files make it easier to maintain and avoid duplication.

Format version is defined in the meta.yaml file in the root folder of the import zip.

Project Management puts more control in your hands 

Ensure Projects that are live in production are never accidentally modified by using the new Lock Project option.  Easily enable or disable the project lock by updating the flag on the project settings and saving the change.

When a project is locked, only changes made via project import are allowed.  Trying to edit any configurations or settings in the project manually while it is locked is prevented. 

Locked projects are indicated on any page of the Workspace by the lock icon presented to the right of the project name drop-down.

Easily duplicate Workspace level configurations 

Easily duplicate workspace-level configurations within the Workspace, including Process Configurations, Cluster Configurations, and Connections. 

Use the Duplicate button on any configuration and DataForge Cloud will launch a new tab with the same settings and a name of "<configuration name> COPY". Update the name of the config and any settings you need to change and save. 

Enhanced Processing details user experience

DataForge Cloud now supports copying and sending direct hyperlinks to specific processes.  Share process-specific links for direct context with others while troubleshooting.

Click the Process ID hyperlink to highlight the process ID and copy the new URL from the browser window. Alternatively, right-click the Process ID hyperlink and copy the link address. Available on all Processing screens (main, source, output).

Easily filter many processes at a time to narrow down your search results by filtering on start and end hours. The start and end times are based on your browser's local time setting.  Change the hours and hit the Enter key, or click the Refresh button to adjust your view easily. Available on all Processing screens (main, source, output).

Search for a specific process by entering the Process ID in the Process ID filter to narrow it down to a specific process.  This feature is available on all Processing pages.

Benefit from upgraded Databricks Runtime

The default Databricks version for cluster configurations has been updated to DBR 14.3 LTS.  For more information, visit Databricks 14.3 LTS documentation.

Google Cloud Platform (GCP) Support

DataForge Cloud now offers support for Google Cloud. In addition to AWS and Azure, DataForge Cloud seamlessly integrates with Google's powerful cloud services and resources.

Talos AI Assistant

DataForge is excited to introduce DataForge's Talos virtual assistant. Here is a list of things Talos can help with:

• Find Connections by name

• List & search schemas and tables within metadata for the connection

• Create source and pull data from table(s)

• Find a source and open it

• Find an attribute (raw, rule, or output column) and open lineage for it

Talos is available in preview for DataForge Cloud Professional and Enterprise (AWS only) customers.

DataForge Cloud 8.0 is here to transform your data operations with advanced features and user-friendly enhancements. Upgrade now and experience the future of data management with DataForge!

Have you ever contemplated the distinction between Software Engineering and Data Engineering? Although both deal with data in some capacity, their methodologies are fundamentally different.

Software engineers harness a combination of computer science principles and libraries to facilitate and uphold best practices such as DRY, SRP, and OCP. These tools empower developers to craft, sustain, and scale dependable and extensible codebases.

Conversely, in modern Data Engineering, SQL scripting reigns supreme as the go-to method for data transformations. Originally conceived in the 1970s alongside relational databases, SQL streamlines both schema (DDL) and data management (DML) within databases, making it an indispensable tool for those purposes.

However, employing SQL within data engineering pipelines can yield negative repercussions:

• Recurring copy-paste patterns (e.g., change data capture, refresh, audit, data quality)

• As SQL scripts are table-scoped, they can expand into lengthy documents, obfuscating lineage and necessitating substantial effort to debug, maintain, and evolve

• SQL joins may inadvertently result in data loss or duplication, posing formidable challenges for debugging

• Any alteration to the output schema of any single SQL script mandates schema changes and the "backfilling" of all intermediate tables, as well as updates to all downstream SQL scripts

These issues precipitate an exponential trend in the complexity of data engineering pipelines with growth, often resulting in what is colloquially known as "spaghetti code." This proliferation of complexity directly translates into out-of-control costs associated with building, maintaining, supporting, and evolving an enterprise data platform.

I am thrilled to announce that DataForge has open-sourced its data transformation framework: DataForge Core. This framework enforces modern software engineering best practices and allows data engineers to delineate transformations at the most granular, columnar level. Each transformation is articulated as a pure, single-column-valued function expressed in SQL columnar syntax, with implicit arguments.

The DataForge framework operates on the principle of inversion of control, compiling columnar transformations specified by developers, validating syntax, determining data types, and generating executable SQL scripts that facilitate transformations in the correct order based on dependencies. Additionally, the framework generates DDL SQL CREATE and ALTER statements for both the final output and all intermediate tables.

Discover more about DataForge Core on Github, and please share your feedback with us!