What are the potential challenges associated with schema inference in data processing pipelines?

• A.Performance overhead due to schema discovery
• B.Inaccuracies in inferred schemas leading to data processing errors
• C.Increased storage costs for schema metadata
• D.Handling complex nested structures and arrays
• E.The need for manual schema updates

Comment: *

Name: *

Email: *

Verification: *

Which of the following commands is used to install PySpark in your development environment?

• A.pip install pyspark
• B.npm install pyspark
• C.yarn add pyspark
• D.brew install pyspark

Comment: *

Name: *

Email: *

Verification: *

How can "Explain Plan" help in optimizing query performance regarding data partitioning?

• A.By showing the number of partitions created on the fly
• B.By indicating whether the query is able to take advantage of partition pruning
• C.By displaying the total size of all partitions
• D.By revealing the encryption method used for partitioned data

Comment: *

Name: *

Email: *

Verification: *

How can you ensure that a set of tasks in an Airflow DAG are executed in parallel after a specific initial task is completed?

• A.Use the SequentialExecutor
• B.Set depends_on_past=True for all tasks
• C.Use the parallelism parameter in the airflow.cfg file
• D.Use the ]] and [[ operators to set task dependencies

Comment: *

Name: *

Email: *

Verification: *

In Apache Airflow, what is the purpose of setting max_active_runs in a DAG's configuration?

• A.To limit the number of task instances that can run concurrently within the DAG.
• B.To specify the maximum number of DAG runs that can be executed in parallel.
• C.To control the number of retries for a failed task.
• D.To determine the maximum number of DAG files that can be parsed at any given time.

Comment: *

Name: *

Email: *

Verification: *