Associate-Developer-Apache-Spark Practice Exam and Study Guides - Verified By Exam4PDF Updated 179 Questions
2022 Updated Verified Pass Associate-Developer-Apache-Spark Study Guides & Best Courses
NEW QUESTION 100
Which of the following describes tasks?
- A. Tasks transform jobs into DAGs.
- B. A task is a command sent from the driver to the executors in response to a transformation.
- C. A task is a collection of slots.
- D. Tasks get assigned to the executors by the driver.
- E. A task is a collection of rows.
Answer: D
Explanation:
Explanation
Tasks get assigned to the executors by the driver.
Correct! Or, in other words: Executors take the tasks that they were assigned to by the driver, run them over partitions, and report the their outcomes back to the driver.
Tasks transform jobs into DAGs.
No, this statement disrespects the order of elements in the Spark hierarchy. The Spark driver transforms jobs into DAGs. Each job consists of one or more stages. Each stage contains one or more tasks.
A task is a collection of rows.
Wrong. A partition is a collection of rows. Tasks have little to do with a collection of rows. If anything, a task processes a specific partition.
A task is a command sent from the driver to the executors in response to a transformation.
Incorrect. The Spark driver does not send anything to the executors in response to a transformation, since transformations are evaluated lazily. So, the Spark driver would send tasks to executors only in response to actions.
A task is a collection of slots.
No. Executors have one or more slots to process tasks and each slot can be assigned a task.
NEW QUESTION 101
Which of the following is a problem with using accumulators?
- A. Accumulator values can only be read by the driver, but not by executors.
- B. Accumulators do not obey lazy evaluation.
- C. Only unnamed accumulators can be inspected in the Spark UI.
- D. Only numeric values can be used in accumulators.
- E. Accumulators are difficult to use for debugging because they will only be updated once, independent if a task has to be re-run due to hardware failure.
Answer: A
Explanation:
Explanation
Accumulator values can only be read by the driver, but not by executors.
Correct. So, for example, you cannot use an accumulator variable for coordinating workloads between executors. The typical, canonical, use case of an accumulator value is to report data, for example for debugging purposes, back to the driver. For example, if you wanted to count values that match a specific condition in a UDF for debugging purposes, an accumulator provides a good way to do that.
Only numeric values can be used in accumulators.
No. While pySpark's Accumulator only supports numeric values (think int and float), you can define accumulators for custom types via the AccumulatorParam interface (documentation linked below).
Accumulators do not obey lazy evaluation.
Incorrect - accumulators do obey lazy evaluation. This has implications in practice: When an accumulator is encapsulated in a transformation, that accumulator will not be modified until a subsequent action is run.
Accumulators are difficult to use for debugging because they will only be updated once, independent if a task has to be re-run due to hardware failure.
Wrong. A concern with accumulators is in fact that under certain conditions they can run for each task more than once. For example, if a hardware failure occurs during a task after an accumulator variable has been increased but before a task has finished and Spark launches the task on a different worker in response to the failure, already executed accumulator variable increases will be repeated.
Only unnamed accumulators can be inspected in the Spark UI.
No. Currently, in PySpark, no accumulators can be inspected in the Spark UI. In the Scala interface of Spark, only named accumulators can be inspected in the Spark UI.
More info: Aggregating Results with Spark Accumulators | Sparkour, RDD Programming Guide - Spark 3.1.2 Documentation, pyspark.Accumulator - PySpark 3.1.2 documentation, and pyspark.AccumulatorParam - PySpark 3.1.2 documentation
NEW QUESTION 102
The code block shown below should return a two-column DataFrame with columns transactionId and supplier, with combined information from DataFrames itemsDf and transactionsDf. The code block should merge rows in which column productId of DataFrame transactionsDf matches the value of column itemId in DataFrame itemsDf, but only where column storeId of DataFrame transactionsDf does not match column itemId of DataFrame itemsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Code block:
transactionsDf.__1__(itemsDf, __2__).__3__(__4__)
- A. 1. join
2. transactionsDf.productId==itemsDf.itemId, how="inner"
3. select
4. "transactionId", "supplier" - B. 1. join
2. [transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId]
3. select
4. "transactionId", "supplier" - C. 1. filter
2. "transactionId", "supplier"
3. join
4. "transactionsDf.storeId!=itemsDf.itemId, transactionsDf.productId==itemsDf.itemId" - D. 1. select
2. "transactionId", "supplier"
3. join
4. [transactionsDf.storeId!=itemsDf.itemId, transactionsDf.productId==itemsDf.itemId] - E. 1. join
2. transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId
3. filter
4. "transactionId", "supplier"
Answer: B
Explanation:
Explanation
This question is pretty complex and, in its complexity, is probably above what you would encounter in the exam. However, reading the question carefully, you can use your logic skills to weed out the wrong answers here.
First, you should examine the join statement which is common to all answers. The first argument of the join() operator (documentation linked below) is the DataFrame to be joined with. Where join is in gap 3, the first argument of gap 4 should therefore be another DataFrame. For none of the questions where join is in the third gap, this is the case. So you can immediately discard two answers.
For all other answers, join is in gap 1, followed by .(itemsDf, according to the code block. Given how the join() operator is called, there are now three remaining candidates.
Looking further at the join() statement, the second argument (on=) expects "a string for the join column name, a list of column names, a join expression (Column), or a list of Columns", according to the documentation. As one answer option includes a list of join expressions (transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId) which is unsupported according to the documentation, we can discard that answer, leaving us with two remaining candidates.
Both candidates have valid syntax, but only one of them fulfills the condition in the question "only where column storeId of DataFrame transactionsDf does not match column itemId of DataFrame itemsDf". So, this one remaining answer option has to be the correct one!
As you can see, although sometimes overwhelming at first, even more complex questions can be figured out by rigorously applying the knowledge you can gain from the documentation during the exam.
More info: pyspark.sql.DataFrame.join - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
NEW QUESTION 103
Which of the following code blocks performs a join in which the small DataFrame transactionsDf is sent to all executors where it is joined with DataFrame itemsDf on columns storeId and itemId, respectively?
- A. itemsDf.join(broadcast(transactionsDf), itemsDf.itemId == transactionsDf.storeId)
- B. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, "broadcast")
- C. itemsDf.merge(transactionsDf, "itemsDf.itemId == transactionsDf.storeId", "broadcast")
- D. itemsDf.join(transactionsDf, broadcast(itemsDf.itemId == transactionsDf.storeId))
- E. itemsDf.join(transactionsDf, itemsDf.itemId == transactionsDf.storeId, "right_outer")
Answer: A
Explanation:
Explanation
The issue with all answers that have "broadcast" as very last argument is that "broadcast" is not a valid join type. While the entry with "right_outer" is a valid statement, it is not a broadcast join. The item where broadcast() is wrapped around the equality condition is not valid code in Spark. broadcast() needs to be wrapped around the name of the small DataFrame that should be broadcast.
More info: Learning Spark, 2nd Edition, Chapter 7
Static notebook | Dynamic notebook: See test 1
tion and explanation?
NEW QUESTION 104
The code block shown below should read all files with the file ending .png in directory path into Spark.
Choose the answer that correctly fills the blanks in the code block to accomplish this.
spark.__1__.__2__(__3__).option(__4__, "*.png").__5__(path)
- A. 1. read
2. format
3. "binaryFile"
4. "pathGlobFilter"
5. load - B. 1. open
2. format
3. "image"
4. "fileType"
5. open - C. 1. open
2. as
3. "binaryFile"
4. "pathGlobFilter"
5. load - D. 1. read()
2. format
3. "binaryFile"
4. "recursiveFileLookup"
5. load - E. 1. read
2. format
3. binaryFile
4. pathGlobFilter
5. load
Answer: A
Explanation:
Explanation
Correct code block:
spark.read.format("binaryFile").option("recursiveFileLookup", "*.png").load(path) Spark can deal with binary files, like images. Using the binaryFile format specification in the SparkSession's read API is the way to read in those files. Remember that, to access the read API, you need to start the command with spark.read. The pathGlobFilter option is a great way to filter files by name (and ending). Finally, the path can be specified using the load operator - the open operator shown in one of the answers does not exist.
NEW QUESTION 105
Which of the following describes Spark's Adaptive Query Execution?
- A. Adaptive Query Execution features are dynamically switching join strategies and dynamically optimizing skew joins.
- B. Adaptive Query Execution is enabled in Spark by default.
- C. Adaptive Query Execution applies to all kinds of queries.
- D. Adaptive Query Execution reoptimizes queries at execution points.
- E. Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.
Answer: A
Explanation:
Explanation
Adaptive Query Execution features include dynamically coalescing shuffle partitions, dynamically injecting scan filters, and dynamically optimizing skew joins.
This is almost correct. All of these features, except for dynamically injecting scan filters, are part of Adaptive Query Execution. Dynamically injecting scan filters for join operations to limit the amount of data to be considered in a query is part of Dynamic Partition Pruning and not of Adaptive Query Execution.
Adaptive Query Execution reoptimizes queries at execution points.
No, Adaptive Query Execution reoptimizes queries at materialization points.
Adaptive Query Execution is enabled in Spark by default.
No, Adaptive Query Execution is disabled in Spark needs to be enabled through the spark.sql.adaptive.enabled property.
Adaptive Query Execution applies to all kinds of queries.
No, Adaptive Query Execution applies only to queries that are not streaming queries and that contain at least one exchange (typically expressed through a join, aggregate, or window operator) or one subquery.
More info: How to Speed up SQL Queries with Adaptive Query Execution, Learning Spark, 2nd Edition, Chapter 12 (https://bit.ly/3tOh8M1)
NEW QUESTION 106
The code block shown below should return a DataFrame with columns transactionsId, predError, value, and f from DataFrame transactionsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__)
- A. 1. select
2. col(["transactionId", "predError", "value", "f"]) - B. 1. where
2. col("transactionId"), col("predError"), col("value"), col("f") - C. 1. select
2. ["transactionId", "predError", "value", "f"] - D. 1. filter
2. "transactionId", "predError", "value", "f" - E. 1. select
2. "transactionId, predError, value, f"
Answer: C
Explanation:
Explanation
Correct code block:
transactionsDf.select(["transactionId", "predError", "value", "f"])
The DataFrame.select returns specific columns from the DataFrame and accepts a list as its only argument.
Thus, this is the correct choice here. The option using col(["transactionId", "predError",
"value", "f"]) is invalid, since inside col(), one can only pass a single column name, not a list. Likewise, all columns being specified in a single string like "transactionId, predError, value, f" is not valid syntax.
filter and where filter rows based on conditions, they do not control which columns to return.
Static notebook | Dynamic notebook: See test 2
NEW QUESTION 107
Which of the following code blocks returns a copy of DataFrame transactionsDf in which column productId has been renamed to productNumber?
- A. transactionsDf.withColumnRenamed(col(productId), col(productNumber))
- B. transactionsDf.withColumnRenamed("productId", "productNumber")
- C. transactionsDf.withColumnRenamed("productNumber", "productId")
- D. transactionsDf.withColumnRenamed(productId, productNumber)
- E. transactionsDf.withColumn("productId", "productNumber")
Answer: B
Explanation:
Explanation
More info: pyspark.sql.DataFrame.withColumnRenamed - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2
NEW QUESTION 108
In which order should the code blocks shown below be run in order to return the number of records that are not empty in column value in the DataFrame resulting from an inner join of DataFrame transactionsDf and itemsDf on columns productId and itemId, respectively?
1. .filter(~isnull(col('value')))
2. .count()
3. transactionsDf.join(itemsDf, col("transactionsDf.productId")==col("itemsDf.itemId"))
4. transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId, how='inner')
5. .filter(col('value').isnotnull())
6. .sum(col('value'))
- A. 4, 1, 2
- B. 3, 5, 2
- C. 4, 6
- D. 3, 1, 6
- E. 3, 1, 2
Answer: A
Explanation:
Explanation
Correct code block:
transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId,
how='inner').filter(~isnull(col('value'))).count()
Expressions col("transactionsDf.productId") and col("itemsDf.itemId") are invalid. col() does not accept the name of a DataFrame, only column names.
Static notebook | Dynamic notebook: See test 2
NEW QUESTION 109
The code block displayed below contains an error. The code block is intended to join DataFrame itemsDf with the larger DataFrame transactionsDf on column itemId. Find the error.
Code block:
transactionsDf.join(itemsDf, "itemId", how="broadcast")
- A. Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.
- B. The join method should be replaced by the broadcast method.
- C. broadcast is not a valid join type.
- D. The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.
- E. The syntax is wrong, how= should be removed from the code block.
Answer: C
Explanation:
Explanation
broadcast is not a valid join type.
Correct! The code block should read transactionsDf.join(broadcast(itemsDf), "itemId"). This would imply an inner join (this is the default in DataFrame.join()), but since the join type is not given in the question, this would be a valid choice.
The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.
This option does not apply here, since the syntax around broadcasting is incorrect.
Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.
No, it is enabled by default, since the spark.sql.autoBroadcastJoinThreshold property is set to 10 MB by default. If that property would be set to -1, then broadcast joining would be disabled.
More info: Performance Tuning - Spark 3.1.1 Documentation (https://bit.ly/3gCz34r) The join method should be replaced by the broadcast method.
No, DataFrame has no broadcast() method.
The syntax is wrong, how= should be removed from the code block.
No, having the keyword argument how= is totally acceptable.
NEW QUESTION 110
Which of the following is the deepest level in Spark's execution hierarchy?
- A. Executor
- B. Stage
- C. Slot
- D. Job
- E. Task
Answer: E
Explanation:
Explanation
The hierarchy is, from top to bottom: Job, Stage, Task.
Executors and slots facilitate the execution of tasks, but they are not directly part of the hierarchy. Executors are launched by the driver on worker nodes for the purpose of running a specific Spark application. Slots help Spark parallelize work. An executor can have multiple slots which enable it to process multiple tasks in parallel.
NEW QUESTION 111
Which of the following code blocks applies the boolean-returning Python function evaluateTestSuccess to column storeId of DataFrame transactionsDf as a user-defined function?
- A. 1.from pyspark.sql import types as T
2.evaluateTestSuccessUDF = udf(evaluateTestSuccess, T.BooleanType())
3.transactionsDf.withColumn("result", evaluateTestSuccess(col("storeId"))) - B. 1.evaluateTestSuccessUDF = udf(evaluateTestSuccess)
2.transactionsDf.withColumn("result", evaluateTestSuccessUDF(storeId)) - C. 1.from pyspark.sql import types as T
2.evaluateTestSuccessUDF = udf(evaluateTestSuccess, T.IntegerType())
3.transactionsDf.withColumn("result", evaluateTestSuccess(col("storeId"))) - D. 1.evaluateTestSuccessUDF = udf(evaluateTestSuccess)
2.transactionsDf.withColumn("result", evaluateTestSuccessUDF(col("storeId"))) - E. 1.from pyspark.sql import types as T
2.evaluateTestSuccessUDF = udf(evaluateTestSuccess, T.BooleanType())
3.transactionsDf.withColumn("result", evaluateTestSuccessUDF(col("storeId")))
Answer: E
Explanation:
Explanation
Recognizing that the UDF specification requires a return type (unless it is a string, which is the default) is important for solving this question. In addition, you should make sure that the generated UDF (evaluateTestSuccessUDF) and not the Python function (evaluateTestSuccess) is applied to column storeId.
More info: pyspark.sql.functions.udf - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
NEW QUESTION 112
Which of the following statements about broadcast variables is correct?
- A. Broadcast variables are immutable.
- B. Broadcast variables are occasionally dynamically updated on a per-task basis.
- C. Broadcast variables are serialized with every single task.
- D. Broadcast variables are local to the worker node and not shared across the cluster.
- E. Broadcast variables are commonly used for tables that do not fit into memory.
Answer: A
Explanation:
Explanation
Broadcast variables are local to the worker node and not shared across the cluster.
This is wrong because broadcast variables are meant to be shared across the cluster. As such, they are never just local to the worker node, but available to all worker nodes.
Broadcast variables are commonly used for tables that do not fit into memory.
This is wrong because broadcast variables can only be broadcast because they are small and do fit into memory.
Broadcast variables are serialized with every single task.
This is wrong because they are cached on every machine in the cluster, precisely avoiding to have to be serialized with every single task.
Broadcast variables are occasionally dynamically updated on a per-task basis.
This is wrong because broadcast variables are immutable - they are never updated.
More info: Spark - The Definitive Guide, Chapter 14
NEW QUESTION 113
Which of the following code blocks returns approximately 1000 rows, some of them potentially being duplicates, from the 2000-row DataFrame transactionsDf that only has unique rows?
- A. transactionsDf.take(1000)
- B. transactionsDf.take(1000).distinct()
- C. transactionsDf.sample(True, 0.5, force=True)
- D. transactionsDf.sample(True, 0.5)
- E. transactionsDf.sample(False, 0.5)
Answer: D
Explanation:
Explanation
To solve this question, you need to know that DataFrame.sample() is not guaranteed to return the exact fraction of the number of rows specified as an argument. Furthermore, since duplicates may be returned, you should understand that the operator's withReplacement argument should be set to True. A force= argument for the operator does not exist.
While the take argument returns an exact number of rows, it will just take the first specified number of rows (1000 in this question) from the DataFrame. Since the DataFrame does not include duplicate rows, there is no potential of any of those returned rows being duplicates when using take(), so the correct answer cannot involve take().
More info: pyspark.sql.DataFrame.sample - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
NEW QUESTION 114
Which of the following code blocks returns a single-row DataFrame that only has a column corr which shows the Pearson correlation coefficient between columns predError and value in DataFrame transactionsDf?
- A. transactionsDf.select(corr(col("predError"), col("value")).alias("corr")) (Correct)
- B. transactionsDf.select(corr(["predError", "value"]).alias("corr")).first()
- C. transactionsDf.select(corr(predError, value).alias("corr"))
- D. transactionsDf.select(corr(col("predError"), col("value")).alias("corr")).first()
- E. transactionsDf.select(corr("predError", "value"))
Answer: A
Explanation:
Explanation
In difficulty, this question is above what you can expect from the exam. What this question NO:
wants to teach you, however, is to pay attention to the useful details included in the documentation.
pyspark.sql.corr is not a very common method, but it deals with Spark's data structure in an interesting way.
The command takes two columns over multiple rows and returns a single row - similar to an aggregation function. When examining the documentation (linked below), you will find this code example:
a = range(20)
b = [2 * x for x in range(20)]
df = spark.createDataFrame(zip(a, b), ["a", "b"])
df.agg(corr("a", "b").alias('c')).collect()
[Row(c=1.0)]
See how corr just returns a single row? Once you understand this, you should be suspicious about answers that include first(), since there is no need to just select a single row. A reason to eliminate those answers is that DataFrame.first() returns an object of type Row, but not DataFrame, as requested in the question.
transactionsDf.select(corr(col("predError"), col("value")).alias("corr")) Correct! After calculating the Pearson correlation coefficient, the resulting column is correctly renamed to corr.
transactionsDf.select(corr(predError, value).alias("corr"))
No. In this answer, Python will interpret column names predError and value as variable names.
transactionsDf.select(corr(col("predError"), col("value")).alias("corr")).first() Incorrect. first() returns a row, not a DataFrame (see above and linked documentation below).
transactionsDf.select(corr("predError", "value"))
Wrong. Whie this statement returns a DataFrame in the desired shape, the column will have the name corr(predError, value) and not corr.
transactionsDf.select(corr(["predError", "value"]).alias("corr")).first() False. In addition to first() returning a row, this code block also uses the wrong call structure for command corr which takes two arguments (the two columns to correlate).
More info:
- pyspark.sql.functions.corr - PySpark 3.1.2 documentation
- pyspark.sql.DataFrame.first - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
NEW QUESTION 115
The code block displayed below contains an error. The code block should return a DataFrame where all entries in column supplier contain the letter combination et in this order. Find the error.
Code block:
itemsDf.filter(Column('supplier').isin('et'))
- A. The expression only returns a single column and filter should be replaced by select.
- B. Instead of isin, it should be checked whether column supplier contains the letters et, so isin should be replaced with contains. In addition, the column should be accessed using col['supplier'].
- C. The Column operator should be replaced by the col operator and instead of isin, contains should be used.
- D. The expression inside the filter parenthesis is malformed and should be replaced by isin('et', 'supplier').
Answer: D
Explanation:
Explanation
Correct code block:
itemsDf.filter(col('supplier').contains('et'))
A mixup can easily happen here between isin and contains. Since we want to check whether a column
"contains" the values et, this is the operator we should use here. Note that both methods are methods of Spark's Column object. See below for documentation links.
A specific Column object can be accessed through the col() method and not the Column() method or through col[], which is an essential thing to know here. In PySpark, Column references a generic column object. To use it for queries, you need to link the generic column object to a specific DataFrame. This can be achieved, for example, through the col() method.
More info:
- isin documentation: pyspark.sql.Column.isin - PySpark 3.1.1 documentation
- contains documentation: pyspark.sql.Column.contains - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
NEW QUESTION 116
The code block displayed below contains an error. The code block should trigger Spark to cache DataFrame transactionsDf in executor memory where available, writing to disk where insufficient executor memory is available, in a fault-tolerant way. Find the error.
Code block:
transactionsDf.persist(StorageLevel.MEMORY_AND_DISK)
- A. The DataFrameWriter needs to be invoked.
- B. The storage level is inappropriate for fault-tolerant storage.
- C. Data caching capabilities can be accessed through the spark object, but not through the DataFrame API.
- D. The code block uses the wrong operator for caching.
- E. Caching is not supported in Spark, data are always recomputed.
Answer: B
Explanation:
Explanation
The storage level is inappropriate for fault-tolerant storage.
Correct. Typically, when thinking about fault tolerance and storage levels, you would want to store redundant copies of the dataset. This can be achieved by using a storage level such as StorageLevel.MEMORY_AND_DISK_2.
The code block uses the wrong command for caching.
Wrong. In this case, DataFrame.persist() needs to be used, since this operator supports passing a storage level.
DataFrame.cache() does not support passing a storage level.
Caching is not supported in Spark, data are always recomputed.
Incorrect. Caching is an important component of Spark, since it can help to accelerate Spark programs to great extent. Caching is often a good idea for datasets that need to be accessed repeatedly.
Data caching capabilities can be accessed through the spark object, but not through the DataFrame API.
No. Caching is either accessed through DataFrame.cache() or DataFrame.persist().
The DataFrameWriter needs to be invoked.
Wrong. The DataFrameWriter can be accessed via DataFrame.write and is used to write data to external data stores, mostly on disk. Here, we find keywords such as "cache" and "executor memory" that point us away from using external data stores. We aim to save data to memory to accelerate the reading process, since reading from disk is comparatively slower. The DataFrameWriter does not write to memory, so we cannot use it here.
More info: Best practices for caching in Spark SQL | by David Vrba | Towards Data Science
NEW QUESTION 117
Which of the following describes the difference between client and cluster execution modes?
- A. In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.
- B. In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.
- C. In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.
- D. In cluster mode, the driver runs on the worker nodes, while the client mode runs the driver on the client machine.
- E. In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.
Answer: D
Explanation:
Explanation
In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.
This is wrong, since execution modes do not specify whether workloads are run in the cloud or on-premise.
In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.
Wrong, since in both cases executors run on worker nodes.
In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.
Wrong - in cluster mode, the driver runs on a worker node. In client mode, the driver runs on the client machine.
In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.
No. In both modes, the cluster manager is typically on a separate node - not on the same host as the driver. It only runs on the same host as the driver in local execution mode.
More info: Learning Spark, 2nd Edition, Chapter 1, and Spark: The Definitive Guide, Chapter 15. ()
NEW QUESTION 118
The code block displayed below contains multiple errors. The code block should remove column transactionDate from DataFrame transactionsDf and add a column transactionTimestamp in which dates that are expressed as strings in column transactionDate of DataFrame transactionsDf are converted into unix timestamps. Find the errors.
Sample of DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+----------------+
2.|transactionId|predError|value|storeId|productId| f| transactionDate|
3.+-------------+---------+-----+-------+---------+----+----------------+
4.| 1| 3| 4| 25| 1|null|2020-04-26 15:35|
5.| 2| 6| 7| 2| 2|null|2020-04-13 22:01|
6.| 3| 3| null| 25| 3|null|2020-04-02 10:53|
7.+-------------+---------+-----+-------+---------+----+----------------+ Code block:
1.transactionsDf = transactionsDf.drop("transactionDate")
2.transactionsDf["transactionTimestamp"] = unix_timestamp("transactionDate", "yyyy-MM-dd")
- A. Column transactionDate should be dropped after transactionTimestamp has been written. The string indicating the date format should be adjusted. The withColumn operator should be used instead of the existing column assignment. Operator to_unixtime() should be used instead of unix_timestamp().
- B. Column transactionDate should be wrapped in a col() operator.
- C. Column transactionDate should be dropped after transactionTimestamp has been written. The string indicating the date format should be adjusted. The withColumn operator should be used instead of the existing column assignment.
- D. The string indicating the date format should be adjusted. The withColumnReplaced operator should be used instead of the drop and assign pattern in the code block to replace column transactionDate with the new column transactionTimestamp.
- E. Column transactionDate should be dropped after transactionTimestamp has been written. The withColumn operator should be used instead of the existing column assignment. Column transactionDate should be wrapped in a col() operator.
Answer: C
Explanation:
Explanation
This question requires a lot of thinking to get right. For solving it, you may take advantage of the digital notepad that is provided to you during the test. You have probably seen that the code block includes multiple errors. In the test, you are usually confronted with a code block that only contains a single error. However, since you are practicing here, this challenging multi-error question will make it easier for you to deal with single-error questions in the real exam.
You can clearly see that column transactionDate should be dropped only after transactionTimestamp has been written. This is because to generate column transactionTimestamp, Spark needs to read the values from column transactionDate.
Values in column transactionDate in the original transactionsDf DataFrame look like 2020-04-26 15:35. So, to convert those correctly, you would have to pass yyyy-MM-dd HH:mm. In other words:
The string indicating the date format should be adjusted.
While you might be tempted to change unix_timestamp() to to_unixtime() (in line with the from_unixtime() operator), this function does not exist in Spark. unix_timestamp() is the correct operator to use here.
Also, there is no DataFrame.withColumnReplaced() operator. A similar operator that exists is DataFrame.withColumnRenamed().
Whether you use col() or not is irrelevant with unix_timestamp() - the command is fine with both.
Finally, you cannot assign a column like transactionsDf["columnName"] = ... in Spark. This is Pandas syntax (Pandas is a popular Python package for data analysis), but it is not supported in Spark.
So, you need to use Spark's DataFrame.withColumn() syntax instead.
More info: pyspark.sql.functions.unix_timestamp - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3
NEW QUESTION 119
Which of the following code blocks returns only rows from DataFrame transactionsDf in which values in column productId are unique?
- A. transactionsDf.distinct("productId")
- B. transactionsDf.dropDuplicates(subset="productId")
- C. transactionsDf.unique("productId")
- D. transactionsDf.dropDuplicates(subset=["productId"])
- E. transactionsDf.drop_duplicates(subset="productId")
Answer: D
Explanation:
Explanation
Although the question suggests using a method called unique() here, that method does not actually exist in PySpark. In PySpark, it is called distinct(). But then, this method is not the right one to use here, since with distinct() we could filter out unique values in a specific column.
However, we want to return the entire rows here. So the trick is to use dropDuplicates with the subset keyword parameter. In the documentation for dropDuplicates, the examples show that subset should be used with a list. And this is exactly the key to solving this question: The productId column needs to be fed into the subset argument in a list, even though it is just a single column.
More info: pyspark.sql.DataFrame.dropDuplicates - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
NEW QUESTION 120
Which of the following code blocks returns a copy of DataFrame transactionsDf that only includes columns transactionId, storeId, productId and f?
Sample of DataFrame transactionsDf:
1.+-------------+---------+-----+-------+---------+----+
2.|transactionId|predError|value|storeId|productId| f|
3.+-------------+---------+-----+-------+---------+----+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.+-------------+---------+-----+-------+---------+----+
- A. transactionsDf.drop(col("value"), col("predError"))
- B. transactionsDf.drop("predError", "value")
- C. transactionsDf.drop(["predError", "value"])
- D. transactionsDf.drop([col("predError"), col("value")])
- E. transactionsDf.drop(value, predError)
Answer: B
Explanation:
Explanation
Output of correct code block:
+-------------+-------+---------+----+
|transactionId|storeId|productId| f|
+-------------+-------+---------+----+
| 1| 25| 1|null|
| 2| 2| 2|null|
| 3| 25| 3|null|
+-------------+-------+---------+----+
To solve this question, you should be fmailiar with the drop() API. The order of column names does not matter
- in this question the order differs in some answers just to confuse you. Also, drop() does not take a list. The *cols operator in the documentation means that all arguments passed to drop() are interpreted as column names.
More info: pyspark.sql.DataFrame.drop - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 2
NEW QUESTION 121
......
The Exam cost of Databricks Associate Developer Apache Spark Exam?
The cost of the Databricks Associate Developer Apache Spark Exam is 200 USD per attempt.
Importance of Databricks Associate Developer Apache Spark Exam to Secure your future
Data science is a rapidly growing field that's being used in a wide range of industries today. It's an essential skill for any developer or business person looking to make the most out of their data. Whether you want to be a part of the rapidly expanding world of data science or are just starting out with your career, the Data Science Associate Developer exam is the best way to test your knowledge and skills in this field. Databricks Associate Developer Apache Spark exam dumps are the best way to prepare for this exam.
In the current market scenario, the demand for Data Scientist is increasing day by day. As a Data Scientist, you are required to analyze the data and predict the future trends. The only thing that you need to do is to find out the right data and use the right tools. There are various tools that are available for you to perform data analysis and you can choose the one that suits your needs. Some of the tools that are used in data analysis are R, Python, Tableau, SAS, etc. Data Scientists are required to work on many different platforms and tools and they should be able to switch between them easily.
How to Register for the Databricks Associate-Developer-Apache-Spark Exam
The on-screen steps will show you how to arrange an exam with our partner.
You can see all the available certificate exams by Clicking on the Certifications tab.
You can register for the exam by clicking the Register button.
Go to create an account.
Ultimate Guide to the Associate-Developer-Apache-Spark - Latest Edition Available Now: https://www.exam4pdf.com/Associate-Developer-Apache-Spark-dumps-torrent.html
2022 Updated Verified Pass Associate-Developer-Apache-Spark Exam - Real Questions and Answers: https://drive.google.com/open?id=1hSYKkpAC1HmOIQ58OCR61miFc0RPOzBM
