TL;DR: The Summer of Reproducibility is a 3-month community-wide sprint organized by Flower Labs with the goal of improving reproducibility of Federated Learning techniques. Our ambition is to support the creation of 50 high quality baselines using the Flower Framework and make these the foundations for the next wave of Federated Learning research. Towards this goal, we are contributing a USD 100K prize to reward contributors. The Summer of Reproducibility starts on 1 July 2023 (noon UTC) and runs until 1 October 2023 (noon UTC) and is open to everyone to contribute. Check all the details on our website: flower.ai/summer and in the Slack channel #summer-of-reproducibility.
How to Participate
The Flower Team has compiled a list of 50 peer-reviewed papers that cover a wide range of sub-fields within Federated Learning. The first step to participate in the Summer of Reproducibility is to choose the baseline you would like to implement with Flower. You can do so from the Summer of Reproducibility GitHub project. From there, you are invited to discuss with members of the Flower Team the experiments in the paper of your choice you would like to reproduce with your contribution. From that point onwards, it's coding time! Each baseline in the GitHub project (which is essentially a GitHub issue) will guide you on what to do next, which involves starting from a template baseline that you can edit to add your code. Once you are happy with your code, create a Pull Request and we will review your contribution. If everything looks good, your work will be merged in to the main Flower branch and you will get your reward!
We are excited to have you onboard the Summer of Reproducibility and we look forward to interacting with you along the process. Don’t forget to check out website for the full details about the Summer of Reproducibility: flower.ai/summer
Rules and FAQ
With the Summer of Reproducibility, the Flower Team aims to build and maintain a catalog of high-quality baselines for Federated Learning. We believe this collective effort will directly benefit the growing Federated Learning community. We want baselines to be easy to run by junior researchers or people curious about Federated Learning while also offering high levels of customization to seasoned researchers. To fulfill this goal, we provide a guided step-by-step process — from how to select a baseline all the way to getting it merged into Flower — that we hope contributors will follow. We set a high bar for contributions to be included in the catalog of baselines. To help with the reviewing process we have defined a series of Rules that will guide how the Flower Team runs the Summer of Reproducibility. You can check these rules and a FAQ we have compiled ahead of time in our website: flower.ai/summer
Alternative ways of Contributing
We understand not everybody has the bandwidth to contribute to this initiative with coding and running experiments. If you believe you have an idea to help contributors or to extend the Summer of Reproducibility to a larger audience, we would love to hear from you. Please reach out to us via Slack.
Flower Summit 2023 Announcement
The Summer of Reproducibility was first announced in the Flower Summit 2023. Go to our YouTube channel to watch this and other announcements as well as the talks and tutorials from the event.