You're working with a DataFrame containing customer data, including a "purchase_date" column. How can you calculate the average purchase amount per month for the past year?

• A.Use Spark SQL's MONTH function and AVG function with appropriate windowing
• B.Loop through the DataFrame and group purchases by month, calculating the average manually
• C.Utilize Spark's machine learning library (MLIiB. for time series analysis
• D.Convert the "purchase_date" column to a string and filter based on the year

Comment: *

Name: *

Email: *

Verification: *

You're working with a complex data pipeline involving Spark and Hive, and you need to monitor its performance and identify potential bottlenecks. Which tools and techniques can you employ for effective monitoring?

• A.Manually analyze Spark and Hive logs after job completion
• B.Leverage Spark's web UI and Hive logs for basic information
• C.Utilize YARN resource manager and Spark/Hive metrics for detailed monitoring
• D.Implement custom instrumentation code within your Spark application

Comment: *

Name: *

Email: *

Verification: *

When leveraging Spark's DataFrame API for caching, what implicit optimization does Spark perform to enhance processing efficiency?

• A.Automatic conversion of all operations to map-reduce jobs
• B.Inlining of functions to reduce the overhead of JVM method calls
• C.Automatic selection of the optimal storage level based on the DataFrame's size
• D.Predicate pushdown and other logical optimizations before physical execution

Comment: *

Name: *

Email: *

Verification: *

Consider a PySpark application that calculates the count of rows in a CSV file. The main application file is 'count_rows.py', and it uses a custom library 'data_utils.py'. After packaging, which 'spark-submit' command correctly submits this job including the custom library?

• A.'spark-submit -py-files data utils.py count_rows.py'
• B.'spark-submit --jars data utils.py count_rows.py'
• C.'spark-submit --files data_utils.py count_rows.py'
• D.'spark-submit --archives data_utils.py count_rows.py'

Comment: *

Name: *

Email: *

Verification: *

In a distributed environment like Apache Spark, under what conditions might a sort merge join perform more efficiently than a broadcast join?

• A.When both datasets are small enough to fit into the memory of a single node.
• B.When one of the datasets is significantly larger than the other, and broadcasting the smaller dataset would lead to excessive memory usage.
• C.When both datasets are large, roughly equal in size, and already partitioned on the join key.
• D.Whenever there is sufficient network bandwidth to handle data shuffling.

Comment: *

Name: *

Email: *

Verification: *