Row 9446

Row ID: 9446 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 9446 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

# History of Mixture of Experts (MoE)?

The idea behind Mixture of Expert is quite old. It’s not even an idea that was invented for Deep Learning architectures. It has been around for quite some time, the entire theory of Random Forest and Boosting trees is basically based on the Mixture of Experts, just a different kind of experts than LLMs.

>

Though much of the modern implementation of the mixture of expert setups was developed over (roughly) the past decade, the core premise behind MoE models originates from the 1991 paper “Adaptive Mixture of Local Experts.” The paper proposed training an AI system composed of separate networks that each specialized in a different subset of training cases. This entailed training both the “expert networks” themselves and a *gating network* that determines which expert should be used for each subtask.

# Full article: [https://medium.com/aiguys/moe-and-llm-merging-recipes-741b11da2b4c](https://medium.com/aiguys/moe-and-llm-merging-recipes-741b11da2b4c)

# Evolutionary Algorithms

Optimization of merging configurations is achieved using evolutionary algorithms like CMA-ES (Covariance Matrix Adaptation Evolution Strategy). These algorithms iteratively improve the merging process based on performance metrics such as accuracy or ROUGE scores.

To determine the best merges, they use the CMA-ES Algorithm (Covariance matrix adaptation evolution strategy) which is a strategy for numerical optimization.

Evolutionary algorithms are ideal for settings where you do not have a ground truth set of actions to take. For example, in this scenario, we do not know the exact parameter merges that would lead to an optimal model.

They discover the optimal merges by experimenting with merges, and getting some sort of signal for how well we are doing.

The steps for CMA-ES are as follows:

1. Create a “population” of models (could be from LLM leaderboard) 2. **Loop:**

* Evaluate each model in the environment, returning the average accuracy over the training data (in this case 1069 examples) * Breed the model weights from the ones with the best scores to create new members of the population. The breeding step could use a variety of merging techniques adding some randomness to the system. * You could also add a randomness to the parameters, like genetic mutation (note they did not state this in the paper, but I think it would be interesting, and seen it done in other CMA-ES algos) * Update the population pool by adding the newly created high performing models and removing poorly performing models.

FieldValue
text # History of Mixture of Experts (MoE)? The idea behind Mixture of Expert is quite old. It’s not even an idea that was invented for Deep Learning architectures. It has been around for quite some time, the entire theory of Random Forest and Boosting trees is basically based on the Mixture of Experts, just a different kind of experts than LLMs. > Though much of the modern implementation of the mixture of expert setups was developed over (roughly) the past decade, the core premise behind MoE mode…
label r/deeplearning
dataType post
communityName r/deeplearning
datetime 2024-05-20
username_encoded Z0FBQUFBQm5Lakw0dkNXcEhrOGJ6LTdBNFRTemZmeHhkaU9UVTNWd1RCaUlMTkc5WDJkNFl3TTR0WmFLWEFSMlI5U09fVFU2aWQzXzVvNzdyQzlQbmlBME1VQWQ5S2FzTDZEWGxUWDFNNXhtSmRWNTZudFpvZDg9
url_encoded Z0FBQUFBQm5Lak9JSDJRckw0S29xLTRQTXlIYzEtME1xdFRGTTJYc3ZQWlBhdXFrSXROQ1JyQWFodnF5WkZMNXBaRktjMWhINnRYUWVBYktnRHd3UU9QWWlxUjhvdHRzU2ktU2tBLVRqZTlEYV91TERIaDF4bDg4WmxNeFhWdDNUU3ZCekFtQWlUaFZNLWw2eDQ1SWxicW5FSXJRZGNibDhBUUk0cjNyM250b1pRZnh2QmFocUd6Tll6MF9OaVFIUzhSeW5vVXdaQkxTRUQtNHlrOWdTamw4eU4zblRJR1dDQT09

Raw Record

{
  "text": "# History of Mixture of Experts (MoE)?\n\nThe idea behind Mixture of Expert is quite old. It’s not even an idea that was invented for Deep Learning architectures. It has been around for quite some time, the entire theory of Random Forest and Boosting trees is basically based on the Mixture of Experts, just a different kind of experts than LLMs.\n\n>\n\nThough much of the modern implementation of the mixture of expert setups was developed over (roughly) the past decade, the core premise behind MoE models originates from the 1991 paper “Adaptive Mixture of Local Experts.” The paper proposed training an AI system composed of separate networks that each specialized in a different subset of training cases. This entailed training both the “expert networks” themselves and a *gating network* that determines which expert should be used for each subtask.\n\n\n\n# Full article: [https://medium.com/aiguys/moe-and-llm-merging-recipes-741b11da2b4c](https://medium.com/aiguys/moe-and-llm-merging-recipes-741b11da2b4c)\n\n# Evolutionary Algorithms\n\nOptimization of merging configurations is achieved using evolutionary algorithms like CMA-ES (Covariance Matrix Adaptation Evolution Strategy). These algorithms iteratively improve the merging process based on performance metrics such as accuracy or ROUGE scores.\n\nTo determine the best merges, they use the CMA-ES Algorithm (Covariance matrix adaptation evolution strategy) which is a strategy for numerical optimization.\n\nEvolutionary algorithms are ideal for settings where you do not have a ground truth set of actions to take. For example, in this scenario, we do not know the exact parameter merges that would lead to an optimal model.\n\nThey discover the optimal merges by experimenting with merges, and getting some sort of signal for how well we are doing.\n\nThe steps for CMA-ES are as follows:\n\n1. Create a “population” of models (could be from LLM leaderboard)\n2. **Loop:**\n\n* Evaluate each model in the environment, returning the average accuracy over the training data (in this case 1069 examples)\n* Breed the model weights from the ones with the best scores to create new members of the population. The breeding step could use a variety of merging techniques adding some randomness to the system.\n* You could also add a randomness to the parameters, like genetic mutation (note they did not state this in the paper, but I think it would be interesting, and seen it done in other CMA-ES algos)\n* Update the population pool by adding the newly created high performing models and removing poorly performing models.\n\n  \n",
  "label": "r/deeplearning",
  "dataType": "post",
  "communityName": "r/deeplearning",
  "datetime": "2024-05-20",
  "username_encoded": "Z0FBQUFBQm5Lakw0dkNXcEhrOGJ6LTdBNFRTemZmeHhkaU9UVTNWd1RCaUlMTkc5WDJkNFl3TTR0WmFLWEFSMlI5U09fVFU2aWQzXzVvNzdyQzlQbmlBME1VQWQ5S2FzTDZEWGxUWDFNNXhtSmRWNTZudFpvZDg9",
  "url_encoded": "Z0FBQUFBQm5Lak9JSDJRckw0S29xLTRQTXlIYzEtME1xdFRGTTJYc3ZQWlBhdXFrSXROQ1JyQWFodnF5WkZMNXBaRktjMWhINnRYUWVBYktnRHd3UU9QWWlxUjhvdHRzU2ktU2tBLVRqZTlEYV91TERIaDF4bDg4WmxNeFhWdDNUU3ZCekFtQWlUaFZNLWw2eDQ1SWxicW5FSXJRZGNibDhBUUk0cjNyM250b1pRZnh2QmFocUd6Tll6MF9OaVFIUzhSeW5vVXdaQkxTRUQtNHlrOWdTamw4eU4zblRJR1dDQT09"
}
Explore Dataset Explore Row

Entry Information