Row 7063

Row ID: 7063 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 7063 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

Last month, I published the paper [On the Efficiency of Convolutional Neural Networks](https://arxiv.org/pdf/2404.03617) on Arxiv. It is the result of five years of research and development, and I hope you find it useful for understanding the performance of deep neural networks. At 52 pages, the paper may appear daunting, but I think you will find it readable. The figures tell the story of the paper; simply scanning the figures and captions is a good approach, then drill down into the sections you find interesting.

Although many have noticed the disconnect between model efficiency (accuracy versus FLOPs) and actual efficiency (accuracy versus latency), previous attempts to improve actual efficiency have been unsatisfying. The convnets we use today were designed to circumvent the limitations of existing inference engines. They are not the models that would perform best with optimized software.

In fact, previous works do not compare model efficiency and actual efficiency directly. The prevailing view is that these two measurements are irreconcilable, and that counting the number of model FLOPs has limited usefulness.

I set out to create a solid foundation for model efficiency research by addressing these shortcomings. The first contribution is the "Efficiency Gap" plot, which shows model and actual efficiency in the same graph. We divide the model FLOPs by the processor's peak arithmetic throughput, yielding the "ideal latency" of the model. Ideal latency is the time that model inference would take if the software ran with 100% computational efficiency (i.e. 100% arithmetic utilization).

We plot ideal and actual latency as two curves on the same accuracy versus latency graph. We use log-scale on the latency axis, and the horizontal gap between the two curves equals the logarithm of the computational efficiency (i.e., the ratio between ideal and actual latency).

Efficiency gap plots of well known baseline models give us a precise analytical view of model performance. For example, the efficiency gap plots for EfficientNet show that its ideal latency is much lower than its actual latency, a result of impressive model efficiency and poor computational efficiency.

The plots also show us where modern convnets fall short. ConvNeXt has worse model efficiency than EfficientNet, yet still has lower actual latency because of better computational efficiency. This suggests that we could do better by finding a model that has low ideal latency and that affords high computational efficiency.

The next chapter of the paper creates a new performance model for neural networks to better understand the sources of computational efficiency. We correct the roofline model, which was never meant to be applied to a sequence of parallel kernels. Previous works overestimate the operational intensity of neural networks by computing a single op: byte ratio for the entire application, despite the fact that this measurement only has meaning for operations that run in parallel. Our "Waterline Model" computes the minimum attainable latency for each kernel separately, ensuring an accurate performance estimate for a sequence of parallel kernels.

Waterline analysis shows that the performance of modern convnets suffer because the traditional conv2d layer has been replaced with a sequence of degenerate conv2d layers, each having less operational intensity than the original conv2d. Thus layer-by-layer execution of modern convnets is memory bound for many of the individual layers.

The intuition behind models like ConvNeXt that increased performance by making the models wider helps somewhat, but does not solve the computational efficiency problem. Also, these adaptations caused the model to use more operations to achieve the same accuracy.

In the next chapter, we address the computational efficiency head-on by writing GPU kernels that fuse all the degenerate layers of a convnet block into a single kernel. We also replace depth-wise convolutions with grouped convolutions with group-width equal to eight to fully utilize the NVIDIA tensor core 16m x 8n x 8k matrix multiply instruction. Block fusion replaces a sequence of low operational intensity, degenrate conv2d layers with a single, high operational intensity kernel. Block fusion algorithms create temporal locality, reduce workspace size, and avoid communication to DRAM.

Our block-fusion kernels run 3x to 14x faster than PyTorch Inductor kernels on an NVIDIA Ampere A5000 GPU with float16 tensor core arithmetic. We implemented MBConv with Squeeze & Excitation (MBConv+SE), the block which created the impressive model efficiency of EfficientNet. We also implemented a new block, ConvFirst, which is very similar to ConvNeXt, but better suited for tensor-core arithmetic.

(An aside: I created the ConvFirst block in 2019, but did not publish it until now due to engineering workload. ConvNeXt was discovered independently).

In the final chapter, we created a new convnet called ConvFirstNet using ConvFirst blocks in the early stages and MBConv+SE block in the late stages. This separation between light-weight blocks for early vision and heavy blocks for late vision was used in EfficientNetV2. Our ConvFirst block is a light-weight alternative to the FusedMBConvBlock used by EfficientNetV2, producing better model efficiency.

The result is that ConvFirstNet has better model efficiency than EfficientNet and better computational efficiency than ConvNeXt. Using our kernels, ConvFirstNet runs 4x as fast as ConvNeXt using PyTorch Inductor. In fact, the actual latency of ConvFirstNet is less than ideal latency of ConvNeXt.

I hope that this work contributes to the progress of model efficiency research. I look forward to all of the wonderful applications of efficient convnets.

FieldValue
text Last month, I published the paper [On the Efficiency of Convolutional Neural Networks](https://arxiv.org/pdf/2404.03617) on Arxiv. It is the result of five years of research and development, and I hope you find it useful for understanding the performance of deep neural networks. At 52 pages, the paper may appear daunting, but I think you will find it readable. The figures tell the story of the paper; simply scanning the figures and captions is a good approach, then drill down into the sections y…
label r/machinelearning
dataType post
communityName r/MachineLearning
datetime 2024-05-14
username_encoded Z0FBQUFBQm5LakwzZGRPNFl6S3hfUWZzRnp2M1JmX0s4ZU1OZzFJQVFtN3dOWWpHNlg3U3pNMEs5MnphLUhORHJRam9ja3FPUVNqbzVnUXExbVZGZm04WndLU0NWcFRtVHc9PQ==
url_encoded Z0FBQUFBQm5Lak9HRFR0cmp3UFhHeWhXSXVYS1lrb055R204X1dvU1puSVlUQk84UEpKbVJXeGFUNHBnYWZuV1EyS3VyS3ZNdGR0eTd5T0QtYVBhRTFIRFo5TVVONktkSWZjVGpSZy14djIteE5NS003R3Nqem1zRG9JNnliUnJIM0hMSzZJdG9sRWw2cFhNdWZNaFJVRDk2NFhvNU5MeUx1elhGM2IwNlRkenNyYzEzZGEzZkd2MzNxc1lwS0pSR3NJOEJ5UHJnRDZWY0EzTHp2bmRpRkthMFFVVlNlbS1mUT09

Raw Record

{
  "text": "Last month, I published the paper [On the Efficiency of Convolutional Neural Networks](https://arxiv.org/pdf/2404.03617) on Arxiv. It is the result of five years of research and development, and I hope you find it useful for understanding the performance of deep neural networks. At 52 pages, the paper may appear daunting, but I think you will find it readable. The figures tell the story of the paper; simply scanning the figures and captions is a good approach, then drill down into the sections you find interesting.\n\nAlthough many have noticed the disconnect between model efficiency (accuracy versus FLOPs) and actual efficiency (accuracy versus latency), previous attempts to improve actual efficiency have been unsatisfying. The convnets we use today were designed to circumvent the limitations of existing inference engines. They are not the models that would perform best with optimized software.\n\nIn fact, previous works do not compare model efficiency and actual efficiency directly. The prevailing view is that these two measurements are irreconcilable, and that counting the number of model FLOPs has limited usefulness.\n\nI set out to create a solid foundation for model efficiency research by addressing these shortcomings. The first contribution is the \"Efficiency Gap\" plot, which shows model and actual efficiency in the same graph. We divide the model FLOPs by the processor's peak arithmetic throughput, yielding the \"ideal latency\" of the model. Ideal latency is the time that model inference would take if the software ran with 100% computational efficiency (i.e. 100% arithmetic utilization).\n\nWe plot ideal and actual latency as two curves on the same accuracy versus latency graph. We use log-scale on the latency axis, and the horizontal gap between the two curves equals the logarithm of the computational efficiency (i.e., the ratio between ideal and actual latency).\n\nEfficiency gap plots of well known baseline models give us a precise analytical view of model performance. For example, the efficiency gap plots for EfficientNet show that its ideal latency is much lower than its actual latency, a result of impressive model efficiency and poor computational efficiency.\n\nThe plots also show us where modern convnets fall short. ConvNeXt has worse model efficiency than EfficientNet, yet still has lower actual latency because of better computational efficiency. This suggests that we could do better by finding a model that has low ideal latency and that affords high computational efficiency.\n\nThe next chapter of the paper creates a new performance model for neural networks to better understand the sources of computational efficiency. We correct the roofline model, which was never meant to be applied to a sequence of parallel kernels. Previous works overestimate the operational intensity of neural networks  by computing a single op: byte ratio for the entire application, despite the fact that this measurement only has meaning for operations that run in parallel. Our \"Waterline Model\" computes the minimum attainable latency for each kernel separately, ensuring an accurate performance estimate for a sequence of parallel kernels.\n\nWaterline analysis shows that the performance of modern convnets suffer because the traditional conv2d layer has been replaced with a sequence of degenerate conv2d layers, each having less operational intensity than the original conv2d. Thus layer-by-layer execution of modern convnets is memory bound for many of the individual layers.\n\nThe intuition behind models like ConvNeXt that increased performance by making the models wider helps somewhat, but does not solve the computational efficiency problem. Also, these adaptations caused the model to use more operations to achieve the same accuracy.\n\nIn the next chapter, we address the computational efficiency head-on by writing GPU kernels that fuse all the degenerate layers of a convnet block into a single kernel. We also replace depth-wise convolutions with grouped convolutions with group-width equal to eight to fully utilize the NVIDIA tensor core 16m x 8n x 8k matrix multiply instruction. Block fusion replaces a sequence of low operational intensity, degenrate conv2d layers with a single, high operational intensity kernel. Block fusion algorithms create temporal locality, reduce workspace size, and avoid communication to DRAM.\n\nOur block-fusion kernels run 3x to 14x faster than PyTorch Inductor kernels on an NVIDIA Ampere A5000 GPU with float16 tensor core arithmetic. We implemented MBConv with Squeeze & Excitation (MBConv+SE), the block which created the impressive model efficiency of EfficientNet. We also implemented a new block, ConvFirst, which is very similar to ConvNeXt, but better suited for tensor-core arithmetic.\n\n(An aside: I created the ConvFirst block in 2019, but did not publish it until now due to engineering workload. ConvNeXt was discovered independently).\n\nIn the final chapter, we created a new convnet called ConvFirstNet using ConvFirst blocks in the early stages and MBConv+SE block in the late stages. This separation between light-weight blocks for early vision and heavy blocks for late vision was used in EfficientNetV2. Our ConvFirst block is a light-weight alternative to the FusedMBConvBlock used by EfficientNetV2, producing better model efficiency.\n\nThe result is that ConvFirstNet has better model efficiency than EfficientNet and better computational efficiency than ConvNeXt. Using our kernels, ConvFirstNet runs 4x as fast as ConvNeXt using PyTorch Inductor. In fact, the actual latency of ConvFirstNet is less than ideal latency of ConvNeXt.\n\nI hope that this work contributes to the progress of model efficiency research. I look forward to all of the wonderful applications of efficient convnets.",
  "label": "r/machinelearning",
  "dataType": "post",
  "communityName": "r/MachineLearning",
  "datetime": "2024-05-14",
  "username_encoded": "Z0FBQUFBQm5LakwzZGRPNFl6S3hfUWZzRnp2M1JmX0s4ZU1OZzFJQVFtN3dOWWpHNlg3U3pNMEs5MnphLUhORHJRam9ja3FPUVNqbzVnUXExbVZGZm04WndLU0NWcFRtVHc9PQ==",
  "url_encoded": "Z0FBQUFBQm5Lak9HRFR0cmp3UFhHeWhXSXVYS1lrb055R204X1dvU1puSVlUQk84UEpKbVJXeGFUNHBnYWZuV1EyS3VyS3ZNdGR0eTd5T0QtYVBhRTFIRFo5TVVONktkSWZjVGpSZy14djIteE5NS003R3Nqem1zRG9JNnliUnJIM0hMSzZJdG9sRWw2cFhNdWZNaFJVRDk2NFhvNU5MeUx1elhGM2IwNlRkenNyYzEzZGEzZkd2MzNxc1lwS0pSR3NJOEJ5UHJnRDZWY0EzTHp2bmRpRkthMFFVVlNlbS1mUT09"
}
Explore Dataset Explore Row

Entry Information