A data engineer is creating a data ingestion pipeline to understand where customers are taking their rented bicycles during use. The engineer noticed that, over time, data being transmitted from the bicycle sensors fail to include key details like latitude and longitude. Downstream analysts need both the clean records and the quarantined records available for separate processing.
The data engineer already has this code:
import dlt
from pyspark.sql.functions import expr
rules = {
"valid_lat": "(lat IS NOT NULL)",
"valid_long": "(long IS NOT NULL)"
}
quarantine_rules = "NOT({})".format(" AND ".join(rules.values()))
@dlt.view
def raw_trips_data():
return spark.readStream.table("ride_and_go.telemetry.trips")
How should the data engineer meet the requirements to capture good and bad data?

• A.@dlt.table(name="trips_data_quarantine")
  def trips_data_quarantine():
  return (
  spark.readStream.table("raw_trips_data")
  .filter(expr(quarantine_rules))
  )
• B.@dlt.view
  @dlt.expect_or_drop("lat_long_present", "(lat IS NOT NULL AND long IS NOT NULL)") def trips_data_quarantine():
  return spark.readStream.table("ride_and_go.telemetry.trips")
• C.@dlt.table
  @dlt.expect_all_or_drop(rules)
  def trips_data_quarantine():
  return spark.readStream.table("raw_trips_data")
• D.@dlt.table(partition_cols=["is_quarantined", ])
  @dlt.expect_all(rules)
  def trips_data_quarantine():
  return (
  spark.readStream.table("raw_trips_data")
  .withColumn("is_quarantined", expr(quarantine_rules))
  )

Comment: *

Name: *

Email: *

Verification: *

A data engineer is designing an append-only pipeline that needs to handle both batch and streaming data in Delta Lake. The team wants to ensure that the streaming component can efficiently track which data has already been processed.
Which configuration should be set to enable this?

• A.checkpointLocation
• B.overwriteSchema
• C.partitionBy
• D.mergeSchema

Comment: *

Name: *

Email: *

Verification: *

The data engineer team has been tasked with configured connections to an external database that does not have a supported native connector with Databricks. The external database already has data security configured by group membership. These groups map directly to user group already created in Databricks that represent various teams within the company.
A new login credential has been created for each group in the external database. The Databricks Utilities Secrets module will be used to make these credentials available to Databricks users.
Assuming that all the credentials are configured correctly on the external database and group membership is properly configured on Databricks, which statement describes how teams can be granted the minimum necessary access to using these credentials?

• A.''Read'' permissions should be set on a secret key mapped to those credentials that will be used by a given team.
• B.No additional configuration is necessary as long as all users are configured as administrators in the workspace where secrets have been added.
• C."Read" permissions should be set on a secret scope containing only those credentials that will be used by a given team.
• D."Manage" permission should be set on a secret scope containing only those credentials that will be used by a given team.

Comment: *

Name: *

Email: *

Verification: *

A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records.
In addition to de-duplicating records within the batch, which of the following approaches allows the data engineer to deduplicate data against previously processed records as it is inserted into a Delta table?

• A.Set the configuration delta.deduplicate = true.
• B.VACUUM the Delta table after each batch completes.
• C.Perform an insert-only merge with a matching condition on a unique key.
• D.Perform a full outer join on a unique key and overwrite existing data.
• E.Rely on Delta Lake schema enforcement to prevent duplicate records.

Comment: *

Name: *

Email: *

Verification: *

Below table temp_data has one column called raw contains JSON data that records temperature for every four hours in the day for the city of Chicago, you are asked to calculate the maximum temperature that was ever recorded for 12:00 PM hour across all the days. Parse the JSON data and use the necessary array function to calculate the max temp.
Table: temp_date
Column: raw
Datatype: string

Expected output: 58

• A.1.select max(raw.chicago.temp[3]) from temp_data
• B.1.select array_max(raw.chicago[*].temp[3]) from temp_data
• C.1.select array_max(from_json(raw['chicago'].temp[3],'array<int>')) from temp_data
• D.1.select array_max(from_json(raw:chicago[*].temp[3],'array<int>')) from temp_data
• E.1.select max(from_json(raw:chicago[3].temp[3],'array<int>')) from temp_data

Correct Answer: D

Explanation
Note: This is a difficult question, more likely you may see easier questions similar to this but the more you are prepared for the exam easier it is to pass the exam.
Use this below link to look for more examples, this will definitely help you,
https://docs.databricks.com/optimizations/semi-structured.html
Here is the solution, step by step
Text Description automatically generated

Use this below link to look for more examples, this will definitely help you,
https://docs.databricks.com/optimizations/semi-structured.html
If you want to try this solution use below DDL,
1.create or replace table temp_data
2. as select ' {
3. "chicago":[
4.{"date":"01-01-2021",
5."temp":[25,28,45,56,39,25]
6.},
7.{"date":"01-02-2021",
8."temp":[25,28,49,54,38,25]
9.},
10.{"date":"01-03-2021",
11."temp":[25,28,49,58,38,25]
12. }]
13. }
14. ' as raw
15.
16.select array_max(from_json(raw:chicago[*].temp[3],'array<int>')) from temp_data
17.

Comment: *

Name: *

Email: *

Verification: *