Delta Lake gives Apache Spark data sets new powers

Cisco has announced a range of new data center offerings (Image CasarsaGuru / iStockPhoto)
Databricks has released an open source storage layer for Apache Spark (Image CasarsaGuru / iStockPhoto)

Databricks, the company founded by the original developers of Apache Spark, has released Delta Lake, an open source storage layer for Spark that provides ACID transactions and other data-management functions for machine learning and other big data work.

Many kinds of data work need features like ACID transactions or schema enforcement for consistency, metadata management for security, and the ability to work with discrete versions of data. Features like those don’t come standard with every data source out there, so Delta Lake provides those features for any Spark DataFrame data source.

Delta Lake can be used as a drop-in replacement to access storage systems like HDFS. Data ingested into Spark through Delta Lake is stored in Parquet format in a cloud storage service of your choice. Devlopers can use their choice of Java, Python, or Scala to access Delta Lake’s API set.

Delta Lake supports most of the existing Spark SQL DataFrame functions for reading and writing data. It also supports Spark Structured Streaming as a source or destination, although not the DStream API. Every read and write through Delta Lake has an ACID transaction guarantee, so that multiple writers will have their writes serialized and multiple readers will see consistent snapshots.

Reading a specific version of a data set—what the Delta Lake documentation calls “time travel”—works by simply reading a DataFrame with an associated time stamp or version ID. Delta Lake also ensures the schema of the DataFrame being written matches the table it’s being written to; if there’s a mismatch, it throws an exception rather than change the schema. (Spark’s file APIs will replace the table in such a case.)

Future releases of Delta Lake may support more of Spark’s public API set, although DataFrameReader/Writer are the main focus for now.

InfoWorld

Read more on

Suggested Articles

Verizon Media will expand its Yahoo Studio production studio in Hong Kong as part of its growth strategy for the market

Cloud security solutions provider Bitglass' regional channel boss Brendon Thwaites has left the company

An agreement links Cisco Meraki MX Security/SD-WAN appliances and its Auto VPN technology to Teridion’s cloud-based WAN service