A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure.
The silver_device_recordings table will be used downstream to power several production monitoring dashboards and a production model. At present, 45 of the 100 fields are being used in at least one of these applications.
The data engineer is trying to determine the best approach for dealing with schema declaration given the highly-nested structure of the data and the numerous fields.
Which of the following accurately presents information about Delta Lake and Databricks that may impact their decision-making process?

• A.The Tungsten encoding used by Databricks is optimized for storing string data; newly-added native support for querying JSON strings means that string types are always most efficient.
• B.Because Delta Lake uses Parquet for data storage, data types can be easily evolved by just modifying file footer information in place.
• C.Human labor in writing code is the largest cost associated with data engineering workloads; as such, automating table declaration logic should be a priority in all migration workloads.
• D.Because Databricks will infer schema using types that allow all observed data to be processed, setting types manually provides greater assurance of data quality enforcement.
• E.Schema inference and evolution on .Databricks ensure that inferred types will always accurately match the data types used by downstream systems.

Comment: *

Name: *

Email: *

Verification: *

You are currently asked to work on building a data pipeline, you have noticed that you are currently working on a very large scale ETL many data dependencies, which of the following tools can be used to address this problem?

• A.AUTO LOADER
• B.JOBS and TASKS
• C.SQL Endpoints
• D.DELTA LIVE TABLES
• E.STRUCTURED STREAMING with MULTI HOP

Correct Answer: D

Explanation
The answer is, DELTA LIVE TABLES
DLT simplifies data dependencies by building DAG-based joins between live tables. Here is a view of how the dag looks with data dependencies without additional meta data,
1.create or replace live view customers
2.select * from customers;
3.
4.create or replace live view sales_orders_raw
5.select * from sales_orders;
6.
7.create or replace live view sales_orders_cleaned
8.as
9.select sales.* from
10.live.sales_orders_raw s
11. join live.customers c
12.on c.customer_id = s.customer_id
13.where c.city = 'LA';
14.
15.create or replace live table sales_orders_in_la
16.selects from sales_orders_cleaned;
Above code creates below dag

Documentation on DELTA LIVE TABLES,
https://databricks.com/product/delta-live-tables
https://databricks.com/blog/2022/04/05/announcing-generally-availability-of-databricks-delta-live-tables-dlt.htm DELTA LIVE TABLES, addresses below challenges when building ETL processes
1.Complexities of large scale ETL
a.Hard to build and maintain dependencies
b.Difficult to switch between batch and stream
2.Data quality and governance
a.Difficult to monitor and enforce data quality
b.Impossible to trace data lineage
3.Difficult pipeline operations
a.Poor observability at granular data level
b.Error handling and recovery is laborious

Comment: *

Name: *

Email: *

Verification: *

What is the purpose of a gold layer in Multi-hop architecture?

• A.Optimizes ETL throughput and analytic query performance
• B.Eliminate duplicate records
• C.Preserves grain of original data, without any aggregations
• D.Data quality checks and schema enforcement
• E.Powers ML applications, reporting, dashboards and adhoc reports.

Correct Answer: E

Explanation
The answer is Powers ML applications, reporting, dashboards and adhoc reports.
Review the below link for more info,
Medallion Architecture - Databricks
Gold Layer:
1.Powers Ml applications, reporting, dashboards, ad hoc analytics
2.Refined views of data, typically with aggregations
3.Reduces strain on production systems
4.Optimizes query performance for business-critical data
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.

Comment: *

Name: *

Email: *

Verification: *

You are working on a process to query the table based on batch date, and batch date is an input parameter and expected to change every time the program runs, what is the best way to we can parameterize the query to run without manually changing the batch date?

• A.Create a notebook parameter for batch date and assign the value to a python variable and use a spark data frame to filter the data based on the python variable
• B.Create a dynamic view that can calculate the batch date automatically and use the view to query the data
• C.There is no way we can combine python variable and spark code
• D.Manually edit code every time to change the batch date
• E.Store the batch date in the spark configuration and use a spark data frame to filter the data based on the spark configuration.

Comment: *

Name: *

Email: *

Verification: *

You are asked to set up an alert to notify in an email every time a KPI indicater increases beyond a threshold value, team also asked you to include the actual value in the alert email notification.

• A.Use notebook and python code to run every minute, using python variables to capture send the information in an email
• B.Setup an alert but use the default template to notify the message in email's subject
• C.Setup an alert but use the custom template to notify the message in email's subject
• D.Use the webhook destination instead so alert message can be customized
• E.Use custom email hook to customize the message

Correct Answer: C

Explanation
Alerts support custom template supports using variables to customize the default message, set up the query to compare the KPI current value to the threshold and use the variable QUE-RY_RESULT_VALUE to display the value in the email notification.
here is a simple alert, that uses variables in the custom template to present these values in the email notification message, when the alert is fired these variables get replaced with actual values.
Alert with custom template
Graphical user interface, application Description automatically generated

When you enable preview you can see how the alert looks when you substitute the variables.
Graphical user interface, text, application, email Description automatically generated

Below are additional template variables available to you with the custom template.
Alerts | Databricks on AWS
Graphical user interface, text, application, email Description automatically generated

Comment: *

Name: *

Email: *

Verification: *