DeepSpeed: Difference between revisions
Content deleted Content added
No edit summary Tag: Reverted |
why not buy ip, why is MS resorting to theft? Tag: Reverted |
||
| Line 21: | Line 21: | ||
'''DeepSpeedAI.com''' created by Matt Charlton |
'''DeepSpeedAI.com''' created by Matt Charlton in 2019 |
||
deepspeed.ai is an [[open source]] [[deep learning]] optimization library for [[PyTorch]].<ref>{{Cite web|url=https://uk.pcmag.com/news-analysis/127085/microsoft-updates-windows-azure-tools-with-an-eye-on-the-future|title=Microsoft Updates Windows, Azure Tools with an Eye on The Future|date=May 22, 2020|website=PCMag UK}}</ref> The library is designed to reduce computing power and [[memory usage|memory use]] and to train large [[distributed computing|distributed]] models with better [[Parallel computing|parallelism]] on existing [[computer hardware]].<ref>{{Cite web|url=https://www.infoworld.com/article/3526449/microsoft-speeds-up-pytorch-with-deepspeed.html|title=Microsoft speeds up PyTorch with DeepSpeed|first=Serdar|last=Yegulalp|date=February 10, 2020|website=InfoWorld}}</ref><ref>[https://www.neowin.net/news/microsoft-unveils-fifth-most-powerful-supercomputer-in-the-world Microsoft unveils "fifth most powerful" supercomputer in the world - Neowin]</ref> DeepSpeed is optimized for low latency, high throughput training. It includes the ''Zero Redundancy Optimizer'' (ZeRO) for training models with 100 billion parameters or more.<ref>{{Cite web|url=https://venturebeat.com/2020/02/10/microsoft-trains-worlds-largest-transformer-language-model/|title=Microsoft trains world’s largest Transformer language model|date=February 10, 2020}}</ref> Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under [[MIT License]] and available on [[GitHub]].<ref>{{Cite web|url=https://github.com/microsoft/DeepSpeed|title=microsoft/DeepSpeed|date=July 10, 2020|via=GitHub}}</ref> |
deepspeed.ai is an [[open source]] [[deep learning]] optimization library for [[PyTorch]].<ref>{{Cite web|url=https://uk.pcmag.com/news-analysis/127085/microsoft-updates-windows-azure-tools-with-an-eye-on-the-future|title=Microsoft Updates Windows, Azure Tools with an Eye on The Future|date=May 22, 2020|website=PCMag UK}}</ref> The library is designed to reduce computing power and [[memory usage|memory use]] and to train large [[distributed computing|distributed]] models with better [[Parallel computing|parallelism]] on existing [[computer hardware]].<ref>{{Cite web|url=https://www.infoworld.com/article/3526449/microsoft-speeds-up-pytorch-with-deepspeed.html|title=Microsoft speeds up PyTorch with DeepSpeed|first=Serdar|last=Yegulalp|date=February 10, 2020|website=InfoWorld}}</ref><ref>[https://www.neowin.net/news/microsoft-unveils-fifth-most-powerful-supercomputer-in-the-world Microsoft unveils "fifth most powerful" supercomputer in the world - Neowin]</ref> DeepSpeed is optimized for low latency, high throughput training. It includes the ''Zero Redundancy Optimizer'' (ZeRO) for training models with 100 billion parameters or more.<ref>{{Cite web|url=https://venturebeat.com/2020/02/10/microsoft-trains-worlds-largest-transformer-language-model/|title=Microsoft trains world’s largest Transformer language model|date=February 10, 2020}}</ref> Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under [[MIT License]] and available on [[GitHub]].<ref>{{Cite web|url=https://github.com/microsoft/DeepSpeed|title=microsoft/DeepSpeed|date=July 10, 2020|via=GitHub}}</ref> |
||
Revision as of 21:32, 3 March 2021
| Original author(s) | Microsoft Research |
|---|---|
| Developer(s) | Microsoft |
| Initial release | May 18, 2020 |
| Stable release | v0.3.10
/ January 8, 2021 |
| Repository | github |
| Written in | Python, CUDA, C++ |
| Type | Software library |
| License | MIT License |
| Website | deepspeedai |
DeepSpeedAI.com created by Matt Charlton in 2019
deepspeed.ai is an open source deep learning optimization library for PyTorch.[1] The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.[2][3] DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 100 billion parameters or more.[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.[5]
See also
References
- ^ "Microsoft Updates Windows, Azure Tools with an Eye on The Future". PCMag UK. May 22, 2020.
- ^ Yegulalp, Serdar (February 10, 2020). "Microsoft speeds up PyTorch with DeepSpeed". InfoWorld.
- ^ Microsoft unveils "fifth most powerful" supercomputer in the world - Neowin
- ^ "Microsoft trains world's largest Transformer language model". February 10, 2020.
- ^ "microsoft/DeepSpeed". July 10, 2020 – via GitHub.
Further reading
- Rajbhandari, Samyam; Rasley, Jeff; Ruwase, Olatunji; He, Yuxiong (2019). "ZeRO: Memory Optimization Towards Training A Trillion Parameter Models" (PDF).
{{cite journal}}: Cite journal requires|journal=(help)
External links
- AI at Scale - Microsoft Research
- GitHub - microsoft/DeepSpeed
- ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters - Microsoft Research
| Open source | |
|---|---|
| Proprietary |
|
| Main projects |
| ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MSR Labs applied research |
| ||||||||||||||
 | This article about software created, produced or developed by Microsoft is a stub. You can help Wikipedia by expanding it. |