Given the following PySpark code snippet in a Databricks notebook:
filtered_df = spark.read.format("delta").load("/mnt/data/large_table") \
.filter("event_date > '2024-01-01'")
filtered_df.count()
The data engineer notices from the Query Profiler that the scan operator for filtered_df is reading almost all files, despite the filter being applied.
What is the probable reason for poor data skipping?

• A.The Delta table lacks optimization that enables dynamic file pruning.
• B.The filter is executed only after the full data scan, preventing data skipping.
• C.The event_date column is outside the table's partitioning and Z-ordering scheme.
• D.The filter condition involves a data type excluded from data skipping support.

Comment: *

Name: *

Email: *

Verification: *

What is the probability that the total of two dice will be greater than 8, given that the first die is a 6?

• A.2/6
• B.1/3
• C.1/6
• D.2/3

Comment: *

Name: *

Email: *

Verification: *

All records from an Apache Kafka producer are being ingested into a single Delta Lake table with the following schema:
key BINARY, value BINARY, topic STRING, partition LONG, offset LONG, timestamp LONG There are 5 unique topics being ingested. Only the "registration" topic contains Personal Identifiable Information (PII). The company wishes to restrict access to PII. The company also wishes to only retain records containing PII in this table for 14 days after initial ingestion. However, for non-PII information, it would like to retain these records indefinitely.
Which of the following solutions meets the requirements?

• A.All data should be deleted biweekly; Delta Lake's time travel functionality should be leveraged to maintain a history of non-PII information.
• B.Data should be partitioned by the registration field, allowing ACLs and delete statements to be set for the PII directory.
• C.Because the value field is stored as binary data, this information is not considered PII and no special precautions should be taken.
• D.Separate object storage containers should be specified based on the partition field, allowing isolation at the storage level.
• E.Data should be partitioned by the topic field, allowing ACLs and delete statements to leverage partition boundaries.

Comment: *

Name: *

Email: *

Verification: *

In order to prevent accidental commits to production data, a senior data engineer has instituted a policy that all development work will reference clones of Delta Lake tables. After testing both deep and shallow clone, development tables are created using shallow clone.
A few weeks after initial table creation, the cloned versions of several tables implemented as Type 1 Slowly Changing Dimension (SCD) stop working. The transaction logs for the source tables show that vacuum was run the day before.
Why are the cloned tables no longer working?

• A.The data files compacted by vacuum are not tracked by the cloned metadata; running refresh on the cloned table will pull in recent changes.
• B.Because Type 1 changes overwrite existing records, Delta Lake cannot guarantee data consistency for cloned tables.
• C.The metadata created by the clone operation is referencing data files that were purged as invalid by the vacuum command
• D.Running vacuum automatically invalidates any shallow clones of a table; deep clone should always be used when a cloned table will be repeatedly queried.

Comment: *

Name: *

Email: *

Verification: *

A junior data engineer on your team has implemented the following code block.

The view new_events contains a batch of records with the same schema as the events Delta table. The event_id field serves as a unique key for this table.
When this query is executed, what will happen with new records that have the same event_id as an existing record?

• A.They are merged.
• B.They are ignored.
• C.They are updated.
• D.They are inserted.
• E.They are deleted.

Comment: *

Name: *

Email: *

Verification: *