Databricks-Certified-Associate-Developer-for-Apache-Spark-3.0 Exam Dumps - Databricks Certified Associate Developer for Apache Spark 3.0 Exam

Explanation

Reroute a query in case of an executor failure.

Correct. Although this feature exists in Spark, it is not a feature of Adaptive Query Execution. The cluster manager keeps track of executors and will work together with the driver to launch an

executor and assign the workload of the failed executor to it (see also link below).

Replace a sort merge join with a broadcast join, where appropriate.

No, this is a feature of Adaptive Query Execution.

Coalesce partitions to accelerate data processing.

Wrong, Adaptive Query Execution does this.

Collect runtime statistics during query execution.

Incorrect, Adaptive Query Execution (AQE) collects these statistics to adjust query plans. This feedback loop is an essential part of accelerating queries via AQE.

Split skewed partitions into smaller partitions to avoid differences in partition processing time.

No, this is indeed a feature of Adaptive Query Execution. Find more information in the Databricks blog post linked below.

More info: Learning Spark, 2nd Edition, Chapter 12, On which way does RDD of spark finish fault-tolerance? - Stack Overflow, How to Speed up SQL Queries with Adaptive Query Execution

Explanation

Correct code block:

transactionsDf.withColumn("cos", round(cos(degrees(transactionsDf.value)),2))

This QUESTION NO: is especially confusing because col, "cos" are so similar. Similar-looking answer options can also appear in the exam and, just like in this question, you need to pay attention to

the

details to identify what the correct answer option is.

The first answer option to throw out is the one that starts with withColumnRenamed: The QUESTION NO: speaks specifically of adding a column. The withColumnRenamed operator only renames

an

existing column, however, so you cannot use it here.

Next, you will have to decide what should be in gap 2, the first argument of transactionsDf.withColumn(). Looking at the documentation (linked below), you can find out that the first argument of

withColumn actually needs to be a string with the name of the column to be added. So, any answer that includes col("cos") as the option for gap 2 can be disregarded.

This leaves you with two possible answers. The real difference between these two answers is where the cos and degree methods are, either in gaps 3 and 4, or vice-versa. From the QUESTION

NO: you

can find out that the new column should have "the values in column value converted to degrees and having the cosine of those converted values taken". This prescribes you a clear order of

operations: First, you convert values from column value to degrees and then you take the cosine of those values. So, the inner parenthesis (gap 4) should contain the degree method and then,

logically, gap 3 holds the cos method. This leaves you with just one possible correct answer.

More info: pyspark.sql.DataFrame.withColumn — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 49 (Databricks import instructions)

Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.

This is almost correct. All of these features, except for dynamically injecting scan filters, are part of Adaptive Query Execution. Dynamically injecting scan filters for join operations to limit the amount

of data to be considered in a query is part of Dynamic Partition Pruning and not of Adaptive Query Execution.

Adaptive Query Execution reoptimizes queries at execution points.

No, Adaptive Query Execution reoptimizes queries at materialization points.

Adaptive Query Execution is enabled in Spark by default.

No, Adaptive Query Execution is disabled in Spark needs to be enabled through the spark.sql.adaptive.enabled property.

Adaptive Query Execution applies to all kinds of queries.

No, Adaptive Query Execution applies only to queries that are not streaming queries and that contain at least one exchange (typically expressed through a join, aggregate, or window operator) or

one subquery.

More info: How to Speed up SQL Queries with Adaptive Query Execution, Learning Spark, 2nd Edition, Chapter 12 (https://bit.ly/3tOh8M1)

Explanation

The trick in this QUESTION NO: is knowing the "modes" of the DataFrameWriter. Mode ignore will ignore if a file already exists and not replace that file, but also not throw an error. Mode

errorifexists will throw an error, and is the default mode of the DataFrameWriter. The QUESTION NO: explicitly calls for the DataFrame to be "silently" written if it does not exist, so you need to

specify mode("ignore") here to avoid having Spark communicate any error to you if the file already exists.

The `overwrite' mode would not be right here, since, although it would be silent, it would overwrite the already-existing file. This is not what the QUESTION NO: asks for.

It is worth noting that the option starting with spark.DataFrameWriter(itemsDf) cannot work, since spark references the SparkSession object, but that object does not provide the DataFrameWriter.

As you can see in the documentation (below), DataFrameWriter is part of PySpark's SQL API, but not of its SparkSession API.

More info:

DataFrameWriter: pyspark.sql.DataFrameWriter.save — PySpark 3.1.1 documentation

SparkSession API: Spark SQL — PySpark 3.1.1 documentation

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 59 (Databricks import instructions)

Explanation

Correct code block:

transactionsDf.withColumnRenamed("storeId", "storeNumber")

More info: pyspark.sql.DataFrame.withColumnRenamed — PySpark 3.1.1 documentation

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 38 (Databricks import instructions)

Explanation

Correct code block:

itemsDf.filter(col('supplier').contains('et'))

A mixup can easily happen here between isin and contains. Since we want to check whether a column "contains" the values et, this is the operator we should use here. Note that both methods are

methods of Spark's Column object. See below for documentation links.

A specific Column object can be accessed through the col() method and not the Column() method or through col[], which is an essential thing to know here. In PySpark, Column references a generic

column object. To use it for queries, you need to link the generic column object to a specific DataFrame. This can be achieved, for example, through the col() method.

More info:

- isin documentation: pyspark.sql.Column.isin — PySpark 3.1.1 documentation

- contains documentation: pyspark.sql.Column.contains — PySpark 3.1.1 documentation

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 51 (Databricks import instructions)

Explanation

itemsDf.printSchema()

Correct! Here is an example of what itemsDf.printSchema() shows, you can see the tree-like structure containing both column names and types:

root

|-- itemId: integer (nullable = true)

|-- attributes: array (nullable = true)

| |-- element: string (containsNull = true)

|-- supplier: string (nullable = true)

itemsDf.rdd.printSchema()

No, the DataFrame's underlying RDD does not have a printSchema() method.

spark.schema(itemsDf)

Incorrect, there is no spark.schema command.

print(itemsDf.columns)

print(itemsDf.dtypes)

Wrong. While the output of this code blocks contains both column names and column types, the information is not arranges in a tree-like way.

itemsDf.print.schema()

No, DataFrame does not have a print method.

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 36 (Databricks import instructions)

Explanation

spark.udf.register("LIMIT_FCN", to_limit)

spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf")

Correct! First, you have to register to_limit as UDF to use it in a sql statement. Then, you can use it under the LIMIT_FCN name, correctly naming the resulting column result.

spark.udf.register(to_limit, "LIMIT_FCN")

spark.sql("SELECT transactionId, LIMIT_FCN(predError) AS result FROM transactionsDf")

No. In this answer, the arguments to spark.udf.register are flipped.

spark.udf.register("LIMIT_FCN", to_limit)

spark.sql("SELECT transactionId, to_limit(predError) AS result FROM transactionsDf")

Wrong, this answer does not use the registered LIMIT_FCN in the sql statement, but tries to access the to_limit method directly. This will fail, since Spark cannot access it.

spark.sql("SELECT transactionId, udf(to_limit(predError)) AS result FROM transactionsDf")

Incorrect, there is no udf method in Spark's SQL.

spark.udf.register("LIMIT_FCN", to_limit)

spark.sql("SELECT transactionId, LIMIT_FCN(predError) FROM transactionsDf AS result")

False. In this answer, the column that results from applying the UDF is not correctly renamed to result.

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 52 (Databricks import instructions)