NVIDIA FLARE enables federated multimodal AI workflows with FedUMM
FedUMM, developed by William & Mary and NVIDIA, uses FLARE to federate lightweight adapters over frozen multimodal models. The approach addresses data distribution challenges in vision-language tasks.
NVIDIA FLARE introduces a framework for building federated multimodal AI workflows, with FedUMM as a key example. FedUMM, developed through a collaboration between William & Mary and NVIDIA, demonstrates how federated learning can be applied to vision-language models (VLMs). This approach allows institutions to train models across distributed data without centralizing raw records, addressing privacy and data governance concerns.
Federated learning provides a mechanism to coordinate training across data-local sites, enabling institutions to collaborate on AI development without sharing sensitive data. NVIDIA FLARE supports this by offering tools such as FLARE Tensor and a Python SDK, which facilitate the deployment of federated workflows. This is particularly relevant for tasks like visual question answering, captioning, and image-text reasoning, where data is often siloed across organizations.
FedUMM leverages a frozen backbone from models like BLIP3o, applying lightweight adapters to adapt the model to specific use cases. This method reduces computational overhead while maintaining performance. The framework is supported by the NVIDIA Academic Grant Program and has received recognition, including an Outstanding Student Paper Award at TheWebConf 2026.
The adoption of federated learning frameworks like NVIDIA FLARE may increase operational costs due to the need for distributed infrastructure and specialized tools. Additionally, organizations may face challenges related to vendor lock-in, as reliance on proprietary SDKs and platforms could limit flexibility. Governance and compliance also become critical considerations, as federated workflows must adhere to data protection regulations and ensure transparency across participating entities.
Despite these challenges, the development of FedUMM and NVIDIA FLARE represents a significant step forward in making federated multimodal AI more accessible. As the technology continues to evolve, it may influence how institutions approach collaborative AI projects, balancing the benefits of shared learning with the complexities of distributed governance and implementation.