Workshop
The 2nd Workshop on Foundation Models in the Wild
Xinyu Yang · Huaxiu Yao · Mohit Bansal · Beidi Chen · Junlin Han · Pavel Izmailov · Jinqi Luo · Pang Wei Koh · Weijia Shi · Philip Torr · Songlin Yang · Luke Zettlemoyer · Jiaheng Zhang
In the era of AI-driven transformations, foundation models (FMs) have become pivotal in various applications, from natural language processing to computer vision. These models, with their immense capabilities,reshape the future of scientific research and the broader human society, but also introduce challenges intheir in-the-wild/real-world deployments. The 2nd Workshop on FMs in the Wild delves into the urgent need forthese models to be useful when deployed in our societies. The significance of this topic cannot be overstated,as the real-world implications of these models impact everything from daily information access to criticaldecision-making in fields like medicine and finance. Stakeholders, from developers to end-users, care deeplyabout this because the successful integration of FMs into in-the-wild frameworks necessitates a careful consideration of many properties, including adaptivity, reliability, efficiency, and reasoning ability. Some of thefundamental questions that this workshop aims to address are:1. In-the-wild Adaptation: How can we leverage techniques such as Retrieval-Augmented Generation(RAG), In-context Learning (ICL), or Fine-tuning (FT) to adapt FMs for specific domains, such asdrug discovery, education, or clinical health?2. Reasoning and Planning: How can FMs be enhanced to tackle more complex in-the-wild tasks thatrequire multi-step reasoning or decision-making, such as multi-hop question answering, mathematicalproblem-solving, theorem proving, code generation, or robot planning scenarios?3. Reliability and Responsibility: How can FMs work reliably outside their training distribution?And how can we address issues like hallucination, fairness, ethics, safety and privacy within the society?4. Practical Limitations in Deployment: How can FMs tackle challenges in practical applications,such as system constraints, memory requirements, response time demands, data acquisition barriers,and computational costs for inference-time scaling and long-context input?In summary, our topics of interest include, but are not limited to:* Innovations in techniques for customizing models to individual user preferences, tasks, or domains* Advancements in the reasoning and planning abilities of FMs in complex real-world challenges* Theoretical and empirical investigations into the reliability and responsibility of various FMs* Strategies for overcoming practical limitations (e.g., memory, time, data) of FMs in broad applications* Methods for integrating multiple modalities (e.g., text, images, action) into a unified in-the-wild framework* Discussions on FM agents that perform intricate tasks through interaction with the environment* In-depth discussions exploring the in-the-wild deployments and applications of FMs* Benchmark methodologies for assessing FMs in real-world settings
Schedule
|
Sat 5:30 p.m. - 6:00 p.m.
|
René Vidal
(
Invited Talk
)
>
link
SlidesLive Video |
RenĂ© Vidal 🔗 |
|
Sat 5:30 p.m. - 5:30 p.m.
|
Introduction and Opening Remarks
SlidesLive Video |
🔗 |
|
Sat 6:00 p.m. - 6:30 p.m.
|
Jean Kossaifi
(
Invited Talk 2
)
>
SlidesLive Video |
🔗 |
|
Sat 6:30 p.m. - 7:00 p.m.
|
Xinyun Chen
(
Invited Talk 3
)
>
SlidesLive Video |
🔗 |
|
Sat 7:30 p.m. - 8:00 p.m.
|
Tatsunori Hashimoto
(
Invited Talk 5
)
>
SlidesLive Video |
🔗 |
|
Sat 8:00 p.m. - 8:30 p.m.
|
Reza Shokri
(
Invited Talk 6
)
>
SlidesLive Video |
🔗 |
|
Sat 8:30 p.m. - 8:35 p.m.
|
Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
(
Oral Talk 1
)
>
SlidesLive Video |
🔗 |
|
Sat 8:35 p.m. - 8:40 p.m.
|
CARROT: A Cost Aware Rate Optimal Router
(
Oral Talk 2
)
>
SlidesLive Video |
🔗 |
|
Sat 8:40 p.m. - 8:45 p.m.
|
DeltaProduct: Increasing the Expressivity of DeltaNet Through Products of Householders
(
Oral Talk 3
)
>
SlidesLive Video |
🔗 |
|
Sat 8:45 p.m. - 8:50 p.m.
|
All It Takes Is One Prompt: An Autonomous LLM-MA System
(
Oral Talk 4
)
>
SlidesLive Video |
🔗 |
|
Sat 8:50 p.m. - 8:55 p.m.
|
Reliable and Efficient Amortized Model-based Evaluation
(
Oral Talk 5
)
>
|
🔗 |
|
Sat 8:55 p.m. - 9:00 p.m.
|
PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
(
Oral Talk 6
)
>
SlidesLive Video |
🔗 |
|
Sat 9:00 p.m. - 10:30 p.m.
|
Lunch Break
|
🔗 |
|
Sat 9:30 p.m. - 10:30 p.m.
|
Poster Session 1
(
Poster Session
)
>
|
🔗 |
|
Sat 10:30 p.m. - 11:00 p.m.
|
Le Song
(
Invited Talk 7
)
>
SlidesLive Video |
🔗 |
|
Sat 11:00 p.m. - 11:30 p.m.
|
Yuandong Tian
(
Invited Talk 8
)
>
SlidesLive Video |
🔗 |
|
Sat 11:30 p.m. - 12:00 a.m.
|
Chelsea Finn
(
Invited Talk 9
)
>
SlidesLive Video |
🔗 |
|
Sun 12:00 a.m. - 12:30 a.m.
|
Guohao Li
(
Invited Talk 10
)
>
SlidesLive Video |
🔗 |
|
Sun 12:30 a.m. - 12:35 a.m.
|
Demystifying Long Chain-of-Thought Reasoning in LLMs
(
Oral Talk 7
)
>
SlidesLive Video |
🔗 |
|
Sun 12:35 a.m. - 12:40 a.m.
|
Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation
(
Oral Talk 8
)
>
SlidesLive Video |
🔗 |
|
Sun 12:40 a.m. - 12:45 a.m.
|
Infinite Leagues Under the Sea: Realistic 3D Underwater Terrain Generation Augmented by Visual Foundation Models
(
Oral Talk 9
)
>
SlidesLive Video |
🔗 |
|
Sun 12:45 a.m. - 12:50 a.m.
|
Towards Universal Offline Black-Box Optimization via Learning String Embedding Space
(
Oral Talk 10
)
>
SlidesLive Video |
🔗 |
|
Sun 12:50 a.m. - 12:55 a.m.
|
RoboMorph: Evolving Robot Morphology using Large Language Models
(
Oral Talk 11
)
>
|
🔗 |
|
Sun 12:55 a.m. - 1:00 a.m.
|
Geneshift: Impact of different scenario shift on Jailbreaking LLM
(
Oral Talk 12
)
>
SlidesLive Video |
🔗 |
|
Sun 1:00 a.m. - 1:15 a.m.
|
Best Paper Awards and Closing Remarks
(
closing remarks
)
>
SlidesLive Video |
🔗 |
|
Sun 1:15 a.m. - 2:15 a.m.
|
Poster Session 2
(
Poster Session 2
)
>
|
🔗 |
|
-
|
Junchi Yan
(
Invited Talk 4
)
>
SlidesLive Video |
🔗 |