Submission information
Submission Number: 98
Submission ID: 141
Submission UUID: f2357cc0-989a-468e-8cfc-be09d5a36a09
Submission URI: /form/project
Created: Tue, 04/27/2021 - 10:19
Completed: Tue, 04/27/2021 - 11:05
Changed: Wed, 07/06/2022 - 15:09
Remote IP address: 47.14.5.69
Submitted by: Chris Hill
Language: English
Is draft: No
Webform: Project
| Project Title | A green, open-source greater than 10B parameter language model. |
|---|---|
| Program | CAREERS |
| Project Image | |
| Tags | |
| Status | Halted |
| Project Leader | Chris Hill |
| Cnh@mit.edu | |
| Mobile Phone | |
| Work Phone | |
| Mentor(s) | Chris Hill |
| Student-facilitator(s) | |
| Mentee(s) | |
| Project Description | We are developing a new language model derived from the EleutherAI GPT-Neo initiative ( https://github.com/EleutherAI/gpt-neo ) for application to two projects. These problems need models the skill close to that of the state-of-the art GPT-3 proprietary model. One project is a demonstration of the model for state-of-the-art image captioning, the other project is the publication of the full model as an open community tools for the research community. For both projects we are interested in collaborating with Cyberteams students to work on model training optimization and testing. The project is looking to run model training and evaluate performance on multi-node configurations of the Aimos 6-GPU/node system. This will allow us to examine scaling and potentially prepare for large experiments with appropriate discussions with IBM teams. The model we will use is efficient and some preliminary work has been undertaken at MGHPCC. Both the RPI and MGHPCC systems have excellent carbon emissions footprints so we also anticipate being able to report energy and emissions statistics that are state-of-the-art for large scale language model training too. |
| Project Deliverables | Start - Core benchmarks on Aimos 4-nodes, 6 GPUs/node Middle - Extended model training with checkpointing and "Turing test" validations against standard benchmarks. Conclusion - Initial paper submitted to NeurIPS and plans for extended experiments explored. Codes and models will be made fully available as open source in a professional form with documentation and available to any research group. |
| Project Deliverables | |
| Student Research Computing Facilitator Profile | A student with interest in ML and some programming experience would be best. |
| Mentee Research Computing Profile | |
| Student Facilitator Programming Skill Level | Can work with any level |
| Mentee Programming Skill Level | |
| Project Institution | MIT |
| Project Address | Cambridge, Massachusetts. 02139 |
| Anchor Institution | NE-MGHPCC |
| Preferred Start Date | 04/29/2021 |
| Start as soon as possible. | Yes |
| Project Urgency | Already behind3Start date is flexible |
| Expected Project Duration (in months) | One month |
| Launch Presentation | |
| Launch Presentation Date | |
| Wrap Presentation | |
| Wrap Presentation Date | |
| Project Milestones |
|
| Github Contributions | |
| Planned Portal Contributions (if any) | Tools, will be openly published via GitHub. |
| Planned Publications (if any) | Two at least 1. NeurIPS 2. TBD |
| What will the student learn? | various ML tools pytorch, NCCL, megatron, deeper speed performance profiling profiling of GPU codes on Aimos and on MGHPCC Satori systems |
| What will the mentee learn? | |
| What will the Cyberteam program learn from this project? | This will be a collaboration involving some of the most energy efficient systems with models that are traditionally seen as very resource hungry. |
| HPC resources needed to complete this project? | 4 Aimos nodes, 6 GPUs per node. |
| Notes | There are two student facilitators (Alex Andonian and David Bau) and one mentor (John Cohn) not currently in the Cyberteams system. |
| What is the impact on the development of the principal discipline(s) of the project? | |
| What is the impact on other disciplines? | |
| Is there an impact physical resources that form infrastructure? | |
| Is there an impact on the development of human resources for research computing? | |
| Is there an impact on institutional resources that form infrastructure? | |
| Is there an impact on information resources that form infrastructure? | |
| Is there an impact on technology transfer? | |
| Is there an impact on society beyond science and technology? | |
| Lessons Learned | |
| Overall results |