What mechanism does Airflow provide to retry failed tasks?

• A.Airflow Scheduler's automatic rerun feature
• B.The retry_delay and retries parameters in task definitions
• C.Manual intervention and rerun via the Airflow Webserver
• D.The on failure callback function in DAG definitions

Comment: *

Name: *

Email: *

Verification: *

When writing a DataFrame to a CSV file, what potential issues should you consider and how can you address them?

• A.No specific issues need to be considered, as CSV is a simple format
• B.Ensure proper handling of special characters and delimiters to avoid data corruption
• C.Choose an appropriate compression format like Gzip to reduce file size
• D.All of the above

Comment: *

Name: *

Email: *

Verification: *

You need to join a Spark DataFrame with a Hive table. How can you achieve this efficiently?

• A.Use Spark SQL syntax with the JOIN clause, specifying the join type and condition
• B.Convert the Hive table to a temporary table and then perform the join with the DataFrame
• C.Implement custom logic using Spark's RDD operations to join the data
• D.Load the Hive table data into the DataFrame and then perform an in-memory join

Comment: *

Name: *

Email: *

Verification: *

You need to design a DAG that can be easily monitored and visualized for performance insights. How can you achieve this?

• A.All of the above
• B.Implement custom logic within each task to send detailed performance metrics to external monitoring tools.
• C.Implement alerts and notifications within the DAG to trigger upon specific events or performance thresholds.
• D.Utilize Airflow's built-in metrics and monitoring features like the Airflow web UI to track DAG execution and task performance.

Comment: *

Name: *

Email: *

Verification: *

You're writing a Spark application that processes streaming data in real-time. How can you create DataFrames from streaming data sources?

• A.Use the same methods as for batch processing, reading data from a file
• B.Leverage Spark Streaming's functions like createDataFrame with appropriate parameters
• C.Utilize Spark SQL's streaming capabilities with the FROM clause
• D.All of the above

Comment: *

Name: *

Email: *

Verification: *