Free Access to Databricks.Databricks-Generative-AI-Engineer-Associate.v2025-02-03.q15 with Valid Practice Test (Page 4)

Question 11

A Generative Al Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author's web forum. The fantasy novel texts are chunked and embedded into a vector store with metadata (page number, chapter number, book title), retrieved with the user' s query, and provided to an LLM for response generation. The Generative AI Engineer used their intuition to pick the chunking strategy and associated configurations but now wants to more methodically choose the best values.
Which TWO strategies should the Generative AI Engineer take to optimize their chunking strategy and parameters? (Choose two.)

A.Change embedding models and compare performance.
B.Add a classifier for user queries that predicts which book will best contain the answer. Use this to filter retrieval.
C.Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapters.
Choose the strategy that gives the best performance metric.
D.Pass known questions and best answers to an LLM and instruct the LLM to provide the best token count. Use a summary statistic (mean, median, etc.) of the best token counts to choose chunk size.
E.Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chunk. Optimize the chunking parameters based upon the values of the metric.

Correct Answer: C,E

To optimize a chunking strategy for a Retrieval-Augmented Generation (RAG) application, the Generative AI Engineer needs a structured approach to evaluating the chunking strategy, ensuring that the chosen configuration retrieves the most relevant information and leads to accurate and coherent LLM responses.
Here's whyCandEare the correct strategies:
Strategy C: Evaluation Metrics (Recall, NDCG)
* Define an evaluation metric: Common evaluation metrics such as recall, precision, or NDCG (Normalized Discounted Cumulative Gain) measure how well the retrieved chunks match the user's query and the expected response.
* Recallmeasures the proportion of relevant information retrieved.
* NDCGis often used when you want to account for both the relevance of retrieved chunks and the ranking or order in which they are retrieved.
* Experiment with chunking strategies: Adjusting chunking strategies based on text structure (e.g., splitting by paragraph, chapter, or a fixed number of tokens) allows the engineer to experiment with various ways of slicing the text. Some chunks may better align with the user's query than others.
* Evaluate performance: By using recall or NDCG, the engineer can methodically test various chunking strategies to identify which one yields the highest performance. This ensures that the chunking method provides the most relevant information when embedding and retrieving data from the vector store.
Strategy E: LLM-as-a-Judge Metric
* Use the LLM as an evaluator: After retrieving chunks, the LLM can be used to evaluate the quality of answers based on the chunks provided. This could be framed as a "judge" function, where the LLM compares how well a given chunk answers previous user queries.
* Optimize based on the LLM's judgment: By having the LLM assess previous answers and rate their relevance and accuracy, the engineer can collect feedback on how well different chunking configurations perform in real-world scenarios.
* This metric could be a qualitative judgment on how closely the retrieved information matches the user's intent.
* Tune chunking parameters: Based on the LLM's judgment, the engineer can adjust the chunk size or structure to better align with the LLM's responses, optimizing retrieval for future queries.
By combining these two approaches, the engineer ensures that the chunking strategy is systematically evaluated using both quantitative (recall/NDCG) and qualitative (LLM judgment) methods. This balanced optimization process results in improved retrieval relevance and, consequently, better response generation by the LLM.

Question 12

A Generative Al Engineer is building a RAG application that answers questions about internal documents for the company SnoPen AI.
The source documents may contain a significant amount of irrelevant content, such as advertisements, sports news, or entertainment news, or content about other companies.
Which approach is advisable when building a RAG application to achieve this goal of filtering irrelevant information?

A.Keep all articles because the RAG application needs to understand non-company content to avoid answering questions about them.
B.Include in the system prompt that any information it sees will be about SnoPenAI, even if no data filtering is performed.
C.Include in the system prompt that the application is not supposed to answer any questions unrelated to SnoPen Al.
D.Consolidate all SnoPen AI related documents into a single chunk in the vector database.

Question 13

A Generative AI Engineer is building an LLM to generate article summaries in the form of a type of poem, such as a haiku, given the article content. However, the initial output from the LLM does not match the desired tone or style.
Which approach will NOT improve the LLM's response to achieve the desired response?

A.Provide the LLM with a prompt that explicitly instructs it to generate text in the desired tone and style
B.Use a neutralizer to normalize the tone and style of the underlying documents
C.Include few-shot examples in the prompt to the LLM
D.Fine-tune the LLM on a dataset of desired tone and style

Question 14

A Generative AI Engineer has created a RAG application which can help employees retrieve answers from an internal knowledge base, such as Confluence pages or Google Drive. The prototype application is now working with some positive feedback from internal company testers. Now the Generative Al Engineer wants to formally evaluate the system's performance and understand where to focus their efforts to further improve the system.
How should the Generative AI Engineer evaluate the system?

A.Use cosine similarity score to comprehensively evaluate the quality of the final generated answers.
B.Curate a dataset that can test the retrieval and generation components of the system separately. Use MLflow's built in evaluation metrics to perform the evaluation on the retrieval and generation components.
C.Benchmark multiple LLMs with the same data and pick the best LLM for the job.
D.Use an LLM-as-a-judge to evaluate the quality of the final answers generated.

Question 15

When developing an LLM application, it's crucial to ensure that the data used for training the model complies with licensing requirements to avoid legal risks.
Which action is NOT appropriate to avoid legal risks?

A.Reach out to the data curators directly before you have started using the trained model to let them know.
B.Use any available data you personally created which is completely original and you can decide what license to use.
C.Only use data explicitly labeled with an open license and ensure the license terms are followed.
D.Reach out to the data curators directly after you have started using the trained model to let them know.

Question 11

Question 12

Question 13

Question 14

Question 15

Download PDF File