Row 52781

Row ID: 52781 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 52781 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

It's interesting because the discussion here is very similar to the issues I had with my own team. A lot of the controversy seems to be in defining what an LLM agent even is. Of course, this is different from the traditional AI/ML agent definition. In this setting, we're talking more about what I usually call "agentic" LLMs, that is, LLMs that are given agency somehow. Usually, this is done through something like function calling in an RAG setting.

I think what most people are trying to define and understand here is multiple agentic LLMs acting together to accomplish some common goal.

For example, let's say you are creating an LLM to diagnose cancer. It might initially seem as though breaking the diagnostic process into steps is advantageous. Maybe you have one agent looking through a certain set of knowledge bases and one looking through another, then they come together at the end with their findings.

I've found there seems to be more of an advantage in understanding the full context of a problem. The smaller agent is simply more likely to make a less useful or more error-prone prediction because it has less context of the overall problem.

The main argument I see that is pro multi-agent systems is that because of context length, having multiple agents "focus in" on specific steps in the diagnostic process, you can somehow take better advantage of the attention mechanism, and somehow the agent's output will be better attended to, or simply superior in some way.

My experience has shown something different. And as I said before, I think if you can, the IDEAL is always going to be a larger parameter LLM, with a larger, more powerful attention mechanism. (Secretly, I believe continuing to scale up LLMs will actually be advantageous to an extent, and the reason we're seeing diminishing returns has more to do with how we benchmark LLMs, rather than the systems peaking in what they can do. But I have 0 evidence to support that.)

Still, there are definitely some reasons to have a multi-agent LLM system. I think defining what agents should even be in such a system is useful.

Essentially, you can say you need separate agents in a system when your two "agents" have entirely different functionalities.

For instance, you have one LLM diagnosing cancer and another LLM verifying questions being input into the system.

The security LLM does not rely on context from the cancer LLM, and the cancer LLM does not rely on context from the security LLM.

The tasks can be usefully separated into two agents, without impeding the overall goal. If an LLM COULD have more context that helps it solve its problem, then it ALWAYS should get it. If we have more information about cancer or the patient, we should give it to the diagnosis LLM. If that information in no way contributes to its overall goal, then it should not get it. Getting unverified malicious user input to the cancer diagnosis LLM contributes nothing to helping the LLM diagnose cancer. Likewise, getting information about cancer does nothing to help verify the user's input.

We always want the highest amount of high-quality information that directly pertains to the problem that needs to be solved. Simply put, getting less information is never better in any situation. Unless that information contributes nothing or impedes the overall goal, then of course, we want to exclude it, or maybe send it to another agent.

A model with an infinite window size and perfect attention can make the best decisions. This sounds quite obvious, but this is a core reason as to why "monolithic" models like GPT-4 with 128k context size outperform multi-agent systems.

Multi-agent systems will always be less effective than monolithic LLMs on a given problem.

Obviously, in the real world, this is not always practical, and therefore you are sometimes forced to use multi-agent systems, but I think it's important to realize "monolithic" LLMs are the ideal. It's what you should work towards.

FieldValue
text It's interesting because the discussion here is very similar to the issues I had with my own team. A lot of the controversy seems to be in defining what an LLM agent even is. Of course, this is different from the traditional AI/ML agent definition. In this setting, we're talking more about what I usually call "agentic" LLMs, that is, LLMs that are given agency somehow. Usually, this is done through something like function calling in an RAG setting. I think what most people are trying to define …
label r/machinelearning
dataType comment
communityName r/MachineLearning
datetime 2024-05-22
username_encoded Z0FBQUFBQm5Lak1VSnk0YUU3bEM0dE5vM0V2OFdNNHM4emtNQXFUZTR5Z2doa3REQVVDMElmczM4VVh4UmFYRDVNV2lzb3ZYU0FVazZGUUxBTWl2c3Z4MTdEZ0otV2txT3AtSWs5bDN5dTFud0VDTHF1TTUxMzA9
url_encoded Z0FBQUFBQm5Lak9qQWdidVZ0M2JUZ3lET2ZqeV9DOVpWVWdRaDNacDhmdGpnb2JTWHJoY3hrRE9acWN1aGNxRk1sLTNzeFU4TnNrQ2pJY2RJelBubEktWmtPOExVNXdnT19DVmRlWnpxVVd5elJWVHlMNTc0NVlBYldpaHVFWkZ6LTJIQjdfTDVkQmJjUDQtRzdYQkVSVTFwOTJTUXFMZUhlVTNZTkxoTnZ5R0Z5bG9STU1GQlk0OHlheE8wS2U1YWRDVFlkMHJ4dGt3RWgtRldScGZZTURzMVp4RS1USmJpY0ltazNOajNyVElPVWpBOXN3eUlaTT0=

Raw Record

{
  "text": "It's interesting because the discussion here is very similar to the issues I had with my own team. A lot of the controversy seems to be in defining what an LLM agent even is. Of course, this is different from the traditional AI/ML agent definition. In this setting, we're talking more about what I usually call \"agentic\" LLMs, that is, LLMs that are given agency somehow. Usually, this is done through something like function calling in an RAG setting.\n\nI think what most people are trying to define and understand here is multiple agentic LLMs acting together to accomplish some common goal.\n\nFor example, let's say you are creating an LLM to diagnose cancer. It might initially seem as though breaking the diagnostic process into steps is advantageous. Maybe you have one agent looking through a certain set of knowledge bases and one looking through another, then they come together at the end with their findings.\n\nI've found there seems to be more of an advantage in understanding the full context of a problem. The smaller agent is simply more likely to make a less useful or more error-prone prediction because it has less context of the overall problem.\n\nThe main argument I see that is pro multi-agent systems is that because of context length, having multiple agents \"focus in\" on specific steps in the diagnostic process, you can somehow take better advantage of the attention mechanism, and somehow the agent's output will be better attended to, or simply superior in some way. \n\nMy experience has shown something different. And as I said before, I think if you can, the IDEAL is always going to be a larger parameter LLM, with a larger, more powerful attention mechanism. (Secretly, I believe continuing to scale up LLMs will actually be advantageous to an extent, and the reason we're seeing diminishing returns has more to do with how we benchmark LLMs, rather than the systems peaking in what they can do. But I have 0 evidence to support that.)\n\nStill, there are definitely some reasons to have a multi-agent LLM system. I think defining what agents should even be in such a system is useful.\n\nEssentially, you can say you need separate agents in a system when your two \"agents\" have entirely different functionalities.\n\nFor instance, you have one LLM diagnosing cancer and another LLM verifying questions being input into the system. \n\nThe security LLM does not rely on context from the cancer LLM, and the cancer LLM does not rely on context from the security LLM. \n\nThe tasks can be usefully separated into two agents, without impeding the overall goal. If an LLM COULD have more context that helps it solve its problem, then it ALWAYS should get it. If we have more information about cancer or the patient, we should give it to the diagnosis LLM. If that information in no way contributes to its overall goal, then it should not get it. Getting unverified malicious user input to the cancer diagnosis LLM contributes nothing to helping the LLM diagnose cancer. Likewise, getting information about cancer does nothing to help verify the user's input. \n\nWe always want the highest amount of high-quality information that directly pertains to the problem that needs to be solved. Simply put, getting less information is never better in any situation. Unless that information contributes nothing or impedes the overall goal, then of course, we want to exclude it, or maybe send it to another agent.\n\nA model with an infinite window size and perfect attention can make the best decisions. This sounds quite obvious, but this is a core reason as to why \"monolithic\" models like GPT-4 with 128k context size outperform multi-agent systems.\n\nMulti-agent systems will always be less effective than monolithic LLMs on a given problem.\n\nObviously, in the real world, this is not always practical, and therefore you are sometimes forced to use multi-agent systems, but I think it's important to realize \"monolithic\" LLMs are the ideal. It's what you should work towards.",
  "label": "r/machinelearning",
  "dataType": "comment",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-22",
  "username_encoded": "Z0FBQUFBQm5Lak1VSnk0YUU3bEM0dE5vM0V2OFdNNHM4emtNQXFUZTR5Z2doa3REQVVDMElmczM4VVh4UmFYRDVNV2lzb3ZYU0FVazZGUUxBTWl2c3Z4MTdEZ0otV2txT3AtSWs5bDN5dTFud0VDTHF1TTUxMzA9",
  "url_encoded": "Z0FBQUFBQm5Lak9qQWdidVZ0M2JUZ3lET2ZqeV9DOVpWVWdRaDNacDhmdGpnb2JTWHJoY3hrRE9acWN1aGNxRk1sLTNzeFU4TnNrQ2pJY2RJelBubEktWmtPOExVNXdnT19DVmRlWnpxVVd5elJWVHlMNTc0NVlBYldpaHVFWkZ6LTJIQjdfTDVkQmJjUDQtRzdYQkVSVTFwOTJTUXFMZUhlVTNZTkxoTnZ5R0Z5bG9STU1GQlk0OHlheE8wS2U1YWRDVFlkMHJ4dGt3RWgtRldScGZZTURzMVp4RS1USmJpY0ltazNOajNyVElPVWpBOXN3eUlaTT0="
}
Explore Dataset Explore Row

Entry Information