KCM guide artwork showing one Iceberg table with Spark and Snowflake workloads

SNOWFLAKE / ICEBERG GUIDE

One Iceberg Table, Two Engines

Choose the catalog and write path for Snowflake and Spark.

Spark may prepare inspection events while Snowflake serves quality reporting. Sharing one Iceberg table can remove a separate reporting export, but the team still needs to decide where the files live, which catalog is authoritative and who may write. Start with the business workloads, then test one table before widening access.

Read the full article ↓

A data engineering team may need Spark to prepare inspection results while an analytics team uses Snowflake to report on production quality. Keeping a separate reporting export can mean another pipeline to operate, another delay to explain and another dataset to reconcile.

Apache Iceberg offers a way for both engines to use the same table. The business reason is to support two useful workloads without maintaining a second table solely to cross the platform boundary. That does not mean every intermediate result, cache or report disappears. It means the team has a deliberate shared-table design instead of an unmanaged collection of copies.

Before choosing that design, separate three things. Storage holds the table’s files. The catalog identifies the table and its current metadata. The engines, such as Spark and Snowflake, perform the work. Iceberg is the table format that describes the data and its snapshots, not a replacement for the engines or the catalog. Snowflake’s Iceberg overview explains these components.

The first decision is therefore not simply whether Snowflake supports Iceberg. It is which catalog will remain authoritative, which engine may change the table and who will keep the arrangement working.

Why this decision deserves attention now

Snowflake made external-engine writes to Snowflake-managed Iceberg tables through Horizon Catalog generally available on May 26, 2026. On October 16, it announced general availability of Iceberg v2-to-v3 upgrades and expression defaults. These changes make older comparisons that assume a simple read-only relationship less useful. External-engine write release, October 16 release.

More capability also creates more decisions. Permission to write from two engines is not a requirement to do so. A shared-table project can start with one writer and a second reader, then expand only when a specific business need justifies the additional responsibility.

Choose the catalog before adding another writer

If Snowflake owns the main data pipeline and table administration, a Snowflake-managed Iceberg table may be the sensible starting point. External engines can access supported tables through Horizon’s Iceberg REST interface. Check the actual client, operations and restrictions rather than treating protocol support as blanket compatibility. External-engine access documentation.

If an established Spark pipeline already publishes an Iceberg table into an external REST catalog, retaining that catalog may avoid an unnecessary ownership change. Snowflake can link to the table. Current documentation also supports Snowflake writes to qualifying externally managed tables, subject to restrictions, including autocommit-only transactions. Read-only access can be a chosen permission boundary, not a product limitation. Externally managed writes.

If there is no concrete need for another engine to access the table in Iceberg form, keep a standard Snowflake table in the comparison. Adding another operating model needs a reason beyond a feature announcement.

Catalog ownership does not settle storage ownership. Snowflake-managed Iceberg tables can use Snowflake storage or customer-managed external storage. External catalogs require external storage. Protection differs too: permanent Iceberg tables in Snowflake storage have Fail-safe, while tables on customer-managed external volumes do not. Storage options.

Follow one inspection batch through the design

Consider a North American manufacturer whose Spark pipeline already writes inspection events to an Iceberg table in its own S3 storage. Its supported external REST catalog is established, and Snowflake quality reporting needs to read that table. The pilot keeps both workloads in the same AWS region and leaves Spark as the only authorized writer.

This is an append-only inspection-event pilot. Each event has a unique inspection ID and a recorded result of Pass or Hold. Corrections and later business actions are outside this first test.

Through business batch B41, the table contains 1,200 inspections: 1,180 Pass and 20 Hold. Batch B42 adds 200 inspections: 192 Pass and eight Hold. After B42 is committed and available to both readers, the expected result is 1,400 inspections, with 1,372 Pass and 28 Hold.

The important evidence is more than a matching total. Both engines should find the same inspection IDs and results at the same accepted cutoff. B41 and B42 are business batch labels, not Iceberg snapshot IDs. Record the actual catalog snapshot or commit reference separately.

Suppose Spark sees the new batch while Snowflake still shows 1,200 inspections. That observation alone does not establish missing data. First check which committed version each engine is reading. Snowflake’s automated refresh for externally managed tables polls the external catalog to synchronize metadata; sharing storage does not make every reader advance at the same instant. Automated metadata refresh.

The team should define how long Snowflake may take to expose a committed batch and what the report does when that limit is missed. A visible last-accepted batch is more useful than a report that silently presents an older result as current.

Prove the access path, not just the connection

A successful administrator query is weak evidence for production access. Run the test with the intended reporting identity and confirm that its permissions do not allow table changes. Also test a user who should not see the inspection data.

For Snowflake-managed tables accessed through Horizon, supported data-protection paths can enforce row and masking policies, but the configuration matters. A connector path can route work through Snowflake; a Scan Plan API path can supply a temporary policy-filtered dataset. Neither is a reason to assume that arbitrary direct file access inherits every Snowflake policy. Policy enforcement options.

That distinction can change the architecture. If the required access controls depend on a supported query path, include that path in the design and in the operating estimate.

Keep upgrades and maintenance under named ownership

The October release is a reason to review compatibility, not to upgrade every shared table. A v3 upgrade is permanent, and every participating engine must support the required features. Snowflake documents a Time Travel restoration route for Snowflake-managed tables, not a universal downgrade button. Version-upgrade guidance.

Name the person or team who can approve a format change, adjust retention and coordinate maintenance. For externally managed tables, Snowflake does not delete external metadata or snapshots simply because its own retention period expires. The external owner still needs a maintenance plan. Metadata and retention.

For the manufacturer, the first decision is narrow: keep the existing external catalog, retain Spark as the writer and prove Snowflake reading at an agreed freshness target. A second writer, a format upgrade or a broader catalog rollout would each require a separate reason and test.

That is a useful outcome for a first Iceberg project: a clear owner, a working access path and evidence that both teams can use the same inspection results.

The companion eGuide includes the catalog decision worksheet, inspection-batch reconciliation and a practical acceptance record. To discuss where this design fits your environment, talk with KCM.