Databricks-Machine-Learning-Professional Exam Dumps - Databricks Certified Machine Learning Professional

MLflow provides a utility function mlflow.models.signature.infer_signature that can infer the input schema and the output schema of a model from a sample dataset. The input schema and the output schema are represented as mlflow.types.schema.Schema objects, which define the names, data types, and shapes of the model inputs and outputs. The infer_signature function can be used for any model that implements a predict method that takes a pandas DataFrame or a numpy array as input and returns a pandas DataFrame, a pandas Series, a numpy array, or a list as output. The infer_signature function can also be used for models that implement a transform method that takes a pandas DataFrame as input and returns a pandas DataFrame as output. The infer_signature function can be used before logging the model with MLflow, as it does not require the model to be in MLflow format or registered with MLflow Model Registry. References:

• MLflow Models Documentation, Model Signature and Input Example, p. 6-7
• mlflow.types.schema Documentation, p. 1-2
• Machine Learning Engineering with MLflow, Managing Model Signatures and Schemas, p. 1

To obtain run-level results for an experiment ID in a Spark DataFrame, you can use the mlflow.search_runs(exp_id) function. This function takes a filter string, which acts as a filter to the query, and returns a pandas.DataFrame of runs, where each metric, parameter, and tag are expanded into their own columns named metrics., params., and tags.* respectively1. You can then convert the pandas.DataFrame to a Spark DataFrame using the spark.createDataFrame() method2. The other options are incorrect because:

• Option A: client.list_run_infos(exp_id) will return a list of RunInfo objects, not a DataFrame3.
• Option B: spark.read.format(“delta”).load(exp_id) will not work for MLflow experiments, as it expects a Delta table path, not an experiment ID4.
• Option C: There is a way to programmatically return row-level results from an MLflow experiment using the mlflow.search_runs(exp_id) function1.
• Option E: spark.read.format(“mlflow-experiment”).load(exp_id) is not a valid format for reading MLflow experiments, as there is no such format registered in Spark5. References: Search Runs, Create a DataFrame from an RDD, a list or a pandas.DataFrame, MlflowClient.list_run_infos, Delta Lake Quickstart, spark.read.format

The correct line of code is “MODEL_VERSION_TRANSITIONED_STAGE”, which is the name of the event that occurs when a model version transitions to any stage in the MLflow Model Registry. This event can be used to trigger a webhook that will execute a Databricks Job based on the model version and the stage it transitions to. The other options are either invalid event names or too specific for the task.

References: MLflow Model Registry Webhooks, MLflow Model Registry Events

A pyfunc model is a generic Python model that can be loaded and used to make predictions using any Python ML library. By logging a custom model class that implements the preprocessing logic as a pyfunc model, the same logic will be applied when loading the model and calling the predict method on new data. This ensures consistency and reproducibility of the model predictions. The fit method is not relevant for loading a logged model, as it is only used for training the model. The pyfunc model does not affect the parallelizability or the need for pipeline-like objects of the model deployment. References: mlflow.pyfunc, Creating custom Pyfunc models

Streaming with Spark as a model deployment strategy means applying a machine learning model to data streams that are processed incrementally and continuously by Spark Structured Streaming. Spark Structured Streaming is a scalable and fault-tolerant stream processing engine that enables complex analytics on live data streams using the Dataset/DataFrame API1. Spark Structured Streaming supports various sources and sinks for streaming data, such as Kafka, Kinesis, TCP sockets, Delta tables, etc2. Spark Structured Streaming also supports various types of operations on streaming data, such as aggregations, windowing, joins, and stateful transformations3. To deploy a machine learning model on streaming data, you can use the MLflow model registry to managethe model lifecycle and versioning4. You can also use the MLflow model serving feature to serve the model as a REST API endpoint that can be invoked by Spark Structured Streaming5. Alternatively, you can use the UDF (user-defined function) feature to apply the model to streaming data within Spark Structured Streaming6.

The inference of incrementally processed records as soon as trigger is hit describes the streaming with Spark as a model deployment strategy. A trigger defines when the results of a streaming query should be written to the output sink. A trigger can be based on a processing time interval, a data size limit, or a continuous mode that writes the results as soon as they are available. The trigger ensures that the streaming query is executed incrementally and continuously, and the model inference is applied to the latest available data. The other options are incorrect because:

• Option A: The inference of batch processed records as soon as a trigger is hit does not describe streaming with Spark, but rather batch processing with Spark. Batch processing means applying a machine learning model to a finite set of data that is processed as a single job. Batch processing does not require a trigger, as the results are written to the output sink when the job is completed.
• Option B: The inference of all types of records in real-time does not describe streaming with Spark, but rather a generic definition of real-time processing. Real-time processing means applying a machine learning model to data streams that are processed as soon as they arrive, with minimal latency. Real-time processing does not necessarily use Spark Structured Streaming, as there are other frameworks and tools that can support it, such as Apache Flink, Apache Storm, etc.
• Option C: The inference of batch processed records as soon as a Spark job is run does not describe streaming with Spark, but rather batch processing with Spark. Batch processing means applying a machine learning model to a finite set of data that is processed as a single job. Batch processing does not depend on a Spark job, as the model inference can be done outside of Spark, such as using a REST API endpoint, a command-line tool, etc.
• Option E: The inference of incrementally processed records as soon as a Spark job is run does not describe streaming with Spark, but rather a contradiction. Incrementally processed records imply streaming processing, while a Spark job implies batch processing. Streaming processing and batch processing are different paradigms of data processing, and cannot be mixed in this way. References: Structured Streaming Programming Guide, Input Sources and Output Sinks, Operations on streaming DataFrames/Datasets, MLflow Model Registry, MLflow Model Serving, Apply machine learning models, [Triggers], [Trigger Types], [Batch Processing], [Real-time Processing], [Real-time Data Processing Frameworks], [Deploy machine learning models], [Batch vs Streaming Processing]