Row 6721

Row ID: 6721 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 6721 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

I have been tinkering with DSPy and thought I will share my 2 cents here for anyone who is planning to explore it:

The core idea behind DSPy are two things:

1. ⁠Separate programming from prompting 2. ⁠incorporate some of the best practice prompting techniques under the hood and expose it as a “signature”

Imagine working on a RAG. Today, the typical approach is to write some retrieval and pass the results to a language model for natural language generation. But, after the first pass, you realize it’s not perfect and you need to iterate and improve it. Typically, there are 2 levers to pull:

1. ⁠Document Chunking, insertion and Retrieval strategy 2. ⁠Language model settings and prompt engineering

Now, you try a few things, maybe document the performance in a google sheet, iterate and arrive at an ideal set of variables that gives max accuracy.

Now, let’s say after a month, model upgrades, and all of a sudden the accuracy of your RAG regresses. Again you are back to square one, cos you don’t know what to optimize now - retrieval or model? You see what the problem is with this approach? This is a very open ended, monolithic, brittle and unstructured way to optimize and build language model based applications.

This is precisely the problem DSPy is trying to solve. Whatever you can achieve with DSPy can be achieved with native prompt engineering and program composition techniques but it is purely dependent on the programmers skill. But DSPy provides native constructs which anyone can learn and use for trying different techniques in a systematic manner.

DSPy the concept:

Separate prompting from programming and signatures

DSPy does not do any magic with the language model. It just uses a bunch of prompt templates behind the scenes and exposes them as signatures. Ex: when you write a signature like ‘context, question -> answer’, DSPy adds a typical RAG prompt before it makes the call to the LLM. But DSPy also gives you nice features like module settings, assertion based backtracking and automatic prompt optimization.

Basically, you can do something like this with DSPy,

“Given a context and question, answer the following question. Make sure the answer is only “yes” or “no””. If the language model responds with anything else, traditionally we prompt engineer our way to fix it. In DSPy, you can assert the answer for “yes” or “no” and if the assertion fails, DSPy will backtrack automatically, update the prompt to say something like, “this is not a correct answer- {previous_answer} and always only respond with a “yes” or “no”” and makes another language model call which improves the LLMs response because of this newly optimized prompt. In addition, you can also incorporate things like multi hops in your retrieval where you can do something like “retrieve -> generate queries and then retrieve again using the generated queries” for n times and build up a larger context to answer the original question.

Obviously, this can also be done using usual prompt engineering and programming techniques, but the framework exposes native easy to use settings and constructs to do these things more naturally. DSPy as a concept really shines when you are composing a pipeline of language model calls where prompt engineering the entire pipeline or even module wise can lead to a brittle Pipeline.

DSPy the Framework:

Now coming to the framework which is built in python, I think the framework as it stands today is

1. ⁠Not production ready 2. ⁠Lacks clear documentation 3. ⁠Poorly designed with not so clean interfaces and abstractions

To me it felt like a rushed implementation with little thought for design thinking, testing and programming principles. The framework code is very hard to understand with a lot of meta programming and data structure parsing and construction going behind the scenes that are scary to run in production.

This is a huge deterrent for anyone trying to learn and use this framework. But, I am sure the creators are thinking about all this and are working to reengineer the framework. There’s also a typescript implementation of this framework that is fairly less popular but has a much better and cleaner design and codebase:

https://github.com/dosco/llm-client/

My final thought about this framework is, it’s a promising concept, but it does not change anything about what we already know about LLMs. Also, hiding prompts as templates does not mean prompt engineering is going away, someone still needs to “engineer” the prompts the framework uses and imo the framework should expose these templates and give control back to the developers that way, the vision of separate programming and prompting co exists with giving control not only to program but also to prompt.

Finally, I was able to understand all this by running DSPy programs and visualizing the LLM calls and what prompts it’s adding using my open source tool - https://github.com/Scale3-Labs/langtrace . Do check it out and let me know if you have any feedback.

FieldValue
text I have been tinkering with DSPy and thought I will share my 2 cents here for anyone who is planning to explore it: The core idea behind DSPy are two things: 1. ⁠Separate programming from prompting 2. ⁠incorporate some of the best practice prompting techniques under the hood and expose it as a “signature” Imagine working on a RAG. Today, the typical approach is to write some retrieval and pass the results to a language model for natural language generation. But, after the first pass, you reali…
label r/deeplearning
dataType post
communityName r/deeplearning
datetime 2024-05-13
username_encoded Z0FBQUFBQm5LakwzYWk0LTBmYURFMW1yLXBjenBtWmx4UHhBaG4tVUlGcXRWc0VfczRQNFhHczRDUFh1Y1Bpb1AyTmhZNnhuNHVySzZ5dTRYNWRoZlY2MlQ0c0lObkt4RGc9PQ==
url_encoded Z0FBQUFBQm5Lak9HTDExZDBGSGFGY2N4Q2dXT3Z1dGZSWUg4aWFPS3BxdmN3UVNzRGw4eTY2VjJOeFhXazFfV1AzNnlGZWpfM2xoSG8tamZkRldjTWZFcTBzSmJGSmhCUUJSQnZQUFBNcVo0SzRzQktBYnkwWEl5dFVtTEpRLWZCUFE5VW9Xckd5RXYxcVg2MVltODc5eENyY2QwVXM0VXpyV2xfMHkzRWx5WXg0M3BScVpYMWhVPQ==

Raw Record

{
  "text": "\nI have been tinkering with DSPy and thought I will share my 2 cents here for anyone who is planning to explore it:\n\nThe core idea behind DSPy are two things:\n\n1.\t⁠Separate programming from prompting\n2.\t⁠incorporate some of the best practice prompting techniques under the hood and expose it as a “signature”\n\nImagine working on a RAG. Today, the typical approach is to write some retrieval and pass the results to a language model for natural language generation. But, after the first pass, you realize it’s not perfect and you need to iterate and improve it. Typically, there are 2 levers to pull:\n\n1.\t⁠Document Chunking, insertion and Retrieval strategy\n2.\t⁠Language model settings and prompt engineering\n\nNow, you try a few things, maybe document the performance in a google sheet, iterate and arrive at an ideal set of variables that gives max accuracy.\n\nNow, let’s say after a month, model upgrades, and all of a sudden the accuracy of your RAG regresses. Again you are back to square one, cos you don’t know what to optimize now - retrieval or model? You see what the problem is with this approach? This is a very open ended, monolithic, brittle and unstructured way to optimize and build language model based applications.\n\nThis is precisely the problem DSPy is trying to solve. Whatever you can achieve with DSPy can be achieved with native prompt engineering and program composition techniques but it is purely dependent on the programmers skill. But DSPy provides native constructs which anyone can learn and use for trying different techniques in a systematic manner.\n\nDSPy the concept:\n\nSeparate prompting from programming and signatures\n\nDSPy does not do any magic with the language model. It just uses a bunch of prompt templates behind the scenes and exposes them as signatures. Ex: when you write a signature like ‘context, question -> answer’, DSPy adds a typical RAG prompt before it makes the call to the LLM. But DSPy also gives you nice features like module settings, assertion based backtracking and automatic prompt optimization.\n\nBasically, you can do something like this with DSPy,\n\n“Given a context and question, answer the following question. Make sure the answer is only “yes” or “no””. If the language model responds with anything else, traditionally we prompt engineer our way to fix it. In DSPy, you can assert the answer for “yes” or “no” and if the assertion fails, DSPy will backtrack automatically, update the prompt to say something like, “this is not a correct answer- {previous_answer} and always only respond with a “yes” or “no”” and makes another language model call which improves the LLMs response because of this newly optimized prompt. In addition, you can also incorporate things like multi hops in your retrieval where you can do something like “retrieve -> generate queries and then retrieve again using the generated queries” for n times and build up a larger context to answer the original question.\n\nObviously, this can also be done using usual prompt engineering and programming techniques, but the framework exposes native easy to use settings and constructs to do these things more naturally. DSPy as a concept really shines when you are composing a pipeline of language model calls where prompt engineering the entire pipeline or even module wise can lead to a brittle Pipeline.\n\nDSPy the Framework:\n\nNow coming to the framework which is built in python, I think the framework as it stands today is\n\n1.\t⁠Not production ready\n2.\t⁠Lacks clear documentation\n3.\t⁠Poorly designed with not so clean interfaces and abstractions\n\nTo me it felt like a rushed implementation with little thought for design thinking, testing and programming principles. The framework code is very hard to understand with a lot of meta programming and data structure parsing and construction going behind the scenes that are scary to run in production.\n\nThis is a huge deterrent for anyone trying to learn and use this framework. But, I am sure the creators are thinking about all this and are working to reengineer the framework. There’s also a typescript implementation of this framework that is fairly less popular but has a much better and cleaner design and codebase:\n\nhttps://github.com/dosco/llm-client/\n\nMy final thought about this framework is, it’s a promising concept, but it does not change anything about what we already know about LLMs. Also, hiding prompts as templates does not mean prompt engineering is going away, someone still needs to “engineer” the prompts the framework uses and imo the framework should expose these templates and give control back to the developers that way, the vision of separate programming and prompting co exists with giving control not only to program but also to prompt.\n\nFinally, I was able to understand all this by running DSPy programs and visualizing the LLM calls and what prompts it’s adding using my open source tool - https://github.com/Scale3-Labs/langtrace . Do check it out and let me know if you have any feedback.",
  "label": "r/deeplearning",
  "dataType": "post",
  "communityName": "r/deeplearning",
  "datetime": "2024-05-13",
  "username_encoded": "Z0FBQUFBQm5LakwzYWk0LTBmYURFMW1yLXBjenBtWmx4UHhBaG4tVUlGcXRWc0VfczRQNFhHczRDUFh1Y1Bpb1AyTmhZNnhuNHVySzZ5dTRYNWRoZlY2MlQ0c0lObkt4RGc9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9HTDExZDBGSGFGY2N4Q2dXT3Z1dGZSWUg4aWFPS3BxdmN3UVNzRGw4eTY2VjJOeFhXazFfV1AzNnlGZWpfM2xoSG8tamZkRldjTWZFcTBzSmJGSmhCUUJSQnZQUFBNcVo0SzRzQktBYnkwWEl5dFVtTEpRLWZCUFE5VW9Xckd5RXYxcVg2MVltODc5eENyY2QwVXM0VXpyV2xfMHkzRWx5WXg0M3BScVpYMWhVPQ=="
}
Explore Dataset Explore Row

Entry Information