Which approach demonstrates a modular and testable way to use DataFrame.transform for ETL code in PySpark?

• A.class Pipeline:
  def transform(self, df):
  return df.withColumn("value_upper", upper(col("value")))
  pipeline = Pipeline()
  assertDataFrameEqual(pipeline.transform(test_input), expected)
• B.def upper_value(df):
  return df.withColumn("value_upper", upper(col("value")))
  def filter_positive(df):
  return df.filter(df["id"] > 0)
  pipeline_df = df.transform(upper_value).transform(filter_positive)
• C.def upper_transform(df):
  return df.withColumn("value_upper", upper(col("value")))
  actual = test_input.transform(upper_transform)
  assertDataFrameEqual(actual, expected)
• D.def transform_data(input_df):
  # transformation logic here
  return output_df
  test_input = spark.createDataFrame([(1, "a")], ["id", "value"])
  assertDataFrameEqual(transform_data(test_input), expected)

Comment: *

Name: *

Email: *

Verification: *

A Delta Lake table was created with the below query:

Realizing that the original query had a typographical error, the below code was executed:
ALTER TABLE prod.sales_by_stor RENAME TO prod.sales_by_store
Which result will occur after running the second command?

• A.The table reference in the metastore is updated and no data is changed.
• B.The table name change is recorded in the Delta transaction log.
• C.All related files and metadata are dropped and recreated in a single ACID transaction.
• D.The table reference in the metastore is updated and all data files are moved.
• E.A new Delta transaction log Is created for the renamed table.

Comment: *

Name: *

Email: *

Verification: *

A Delta table of weather records is partitioned by date and has the below schema:
date DATE, device_id INT, temp FLOAT, latitude FLOAT, longitude FLOAT
To find all the records from within the Arctic Circle, you execute a query with the below filter:
latitude > 66.3
Which statement describes how the Delta engine identifies which files to load?

• A.All records are cached to an operational database and then the filter is applied
• B.The Parquet file footers are scanned for min and max statistics for the latitude column
• C.All records are cached to attached storage and then the filter is applied
• D.The Delta log is scanned for min and max statistics for the latitude column
• E.The Hive metastore is scanned for min and max statistics for the latitude column

Comment: *

Name: *

Email: *

Verification: *

A member of the data engineering team has submitted a short notebook that they wish to schedule as part of a larger data pipeline. Assume that the commands provided below produce the logically correct results when run as presented.

Which command should be removed from the notebook before scheduling it as a job?

• A.Cmd 2
• B.Cmd 3
• C.Cmd 4
• D.Cmd 5
• E.Cmd 6

Comment: *

Name: *

Email: *

Verification: *

A junior data engineer seeks to leverage Delta Lake's Change Data Feed functionality to create a Type 1 table representing all of the values that have ever been valid for all rows in abronzetable created with the propertydelta.enableChangeDataFeed = true. They plan to execute the following code as a daily job:

Which statement describes the execution and results of running the above query multiple times?

• A.Each time the job is executed, newly updated records will be merged into the target table, overwriting previous values with the same primary keys.
• B.Each time the job is executed, the entire available history of inserted or updated records will be appended to the target table, resulting in many duplicate entries.
• C.Each time the job is executed, the target table will be overwritten using the entire history of inserted or updated records, giving the desired result.
• D.Each time the job is executed, the differences between the original and current versions are calculated; this may result in duplicate entries for some records.
• E.Each time the job is executed, only those records that have been inserted or updated since the last execution will be appended to the target table giving the desired result.

Comment: *

Name: *

Email: *

Verification: *