This talk gives an indepth look at Mozilla Common Voice’s Kiswahili work – an initiative to bring a vital language of East Africa online, and to make voice technology accessible to Kiswahili speakers. The talk will cover an introduction to the Mozilla Common Voice platform, community building, including how we have worked to address gender challenges and the inclusion of speakers of related dialects and variants of Kiswahili, the challenges and successes of building an open-source Kiswahili speech data set, the research questions we have explored along the model development roadmap and our efforts to disseminate and encourage the use of the resources created in this work.

Targeting is a central challenge in the design of anti-poverty programs: given available data, how does one rapidly identify the individuals and families with the greatest need? This talk will discuss recent uses of machine learning, applied to non-traditional data from satellites and mobile phones, in the targeting of anti-poverty programs. It draws on results from several field-based projects -- in Togo, Afghanistan, Nigeria, and Kenya -- that illustrate the promise, as well as some of the potential pitfalls, of this new approach to targeting. Collectively, the results highlight the potential for new data sources to improve humanitarian response in low resource settings, particularly during crises and when traditional data are missing or out of date.

Diversity is important for many areas of machine learning, including generative modeling, reinforcement learning, active learning, and dataset curation. Yet, little effort has gone into formalizing and understanding how to effectively measure or enforce diversity. This talk will describe the Vendi Score, a new metric for measuring diversity that connects and extends ideas from ecology and quantum mechanics. The Vendi Score is defined as the Shannon entropy of the eigenvalues of a user-defined similarity matrix. It is general in that (1) it can be applied to any domain where similarity can be defined and (2) it doesn't require defining a probability distribution over the collection to be evaluated for diversity. The Vendi Score can therefore be used to measure the diversity of datasets, samples from a generative model, outputs from decoding algorithms, or any collection for which we want to assess diversity. We will showcase the Vendi Score as a diversity evaluation metric in several domains and as a means to improve the exploration of molecular conformation spaces.

• Timnit Gebru, DAIR
• Jade Abbott, Retro Rabbit
• Asmelash Teka Hadgu, Lesan
• Paul Azunre, Ghana NLP

Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning: Malte Ostendorff.Ostendorff, Malte*; Rehm, Georg

Adaptive Representations for Semantic Search. Aniket Rege, Aditya Kusupati, Sharan Ranjit S, Sham Kakade, Prateek Jain, Ali Farhadi (University of Washington, Allen Institue for AI, Apple)

Malaria is one of the most significant endemic diseases in Sub-Saharan Africa. In Low developed countries (LDCs), the scourge is further bolstered by the lack of enough skilled lab technologists in health centers to accurately detect the disease using the widely accepted gold standard Microscopy method. Thus, the need for reliable detection interventions. This explains the birth of the Topic Group (TG), Automated malaria detection using Artificial Intelligence (AI). The aim is to harness AI to automate the detection of Malaria in a more fast, accurate, and cost-effective manner. Recently emerging technologies of AI and machine learning that can learn complex image patterns have been successful in different medical image analysis tasks and can improve public health. Therefore, the TG-Malaria under the ITU/WHO Focus Group AI for Health (FGAI4H) aims to develop a standardised benchmarking approach for AI based detection of Malaria. This involves all activities related to the curation of a quality dataset, development of AI models and approaches related to malaria detection, suggestions on scoring metrics, development of a benchmarking framework, and extension of the solution to improve disease surveillance and prediction.

We will present a data-centric framework for making machine learning more trustworthy in developing countries. We will discuss best practices for data in different stages of the ML pipeline: starting with how to design/curate datasets, followed by how to identify informative data for ML, and then how to audit and debug ML models to ensure reliable application in resource-limited scenarios.

MH1

Large machine learning models trained on diverse data have been successful across many applications. Federated learning enables training on private data that may otherwise be inaccessible, such as domain-specific datasets decentralized across many clients. However, federated learning can be difficult to scale to large models when clients have limited resources. This challenge often results in a trade-off between model size and data accessibility. To mitigate this issue and facilitate training of large machine learning models on edge devices, we introduce a simple yet effective strategy, Federated Layer-wise Learning, to simultaneously reduce per-client memory, computation, and communication costs.We train a machine learning model in a layer-wise fashion, allowing each client to train just a single layer, thereby considerably reducing the computational burden with minimal performance degradation. In addition, we introduce Federated Depth Dropout, a technique that randomly drops frozen layers during training, to further reduce resource usage. Coupling these two designs enables us to effectively train large models on edge devices. Specifically, we reduce training memory usage by 5$\times$ or more, and demonstrate that performance in downstream tasks is comparable to conventional federated self-supervised representation learning.

NMT systems trained on Pre-trained Multilingual Sequence-Sequence (PMSS) models flounder when sufficient amounts of parallel data is not available for fine-tuning. This specifically holds for languages missing/under-represented in these models. The problem gets aggravated when the data comes from different domains. In this paper, we show that intermediate-task fine-tuning (ITFT) of PMSS models is extremely beneficial for domain-specific NMT, especially when target domain data is limited/unavailable and the considered languages are missing/under-represented in the PMSS model. We quantify the domain-specific results variations using a domain-divergence test, and show that ITFT can mitigate the impact of domain divergence to some extent.

Learned metrics such as BLEURT have in recent years become widely employed to evaluate the quality of machine translation systems. Training such metrics requires data which can be expensive and difficult to acquire, particularly for lower-resource languages. We show how knowledge can be distilled from Large Language Models (LLMs) to improve upon such learned metrics without requiring human annotators, by creating synthetic datasets which can be mixed into existing datasets, requiring only a corpus of text in the target language. We show that the performance of a BLEURT-like model on lower resource languages can be improved in this way.

Heart attack remain one of the greatest contributors to mortality in the United States and globally. Patients admitted to the intensive care unit (ICU) with diagnosed heart attack (myocardial infarction or MI) are more likely to suffer a secondary episode of MI and are at higher risk of death. The ICU is a setting of low-resource environment with noisy, sparse, and irregular time-series data being produced whose distribution and hence predictive use can vary between hospitals. In this study, we use two retrospective cohorts extracted from the eICU and MIMIC-IV databases, to develop a novel pseudo-dynamic machine learning framework for mortality and recurrent heart attack prediction in the ICU with interpretability. The method provides accurate prediction of both outcomes for ICU patients up to 24 hours before the event and provide time-resolved interpretability results. The performance of the framework relying on extreme gradient boosting was evaluated on a held-out test set from eICU, and externally validated on the MIMIC-IV cohort using the most important features identified by time-resolved Shapley values achieving AUCs of 91.0 (balanced accuracy of 82.3) and 85.6 (balanced accuracy of 74.5) for 6-hour prediction of mortality and recurrent heart attack respectively. We show that our framework successfully leverages time-series physiological measurements by translating them into stacked static prediction problems to be robustly predictive through time in the ICU stay and can offer clinical insight from time-resolved intepretability.

Unsupervised approaches for learning representations invariant to common transformations are used quite often for object recognition. Learning invariances makes models more robust and practical to use in real-world scenarios. Since data transformations that do not change the intrinsic properties of the object cause the majority of the complexity in recognition tasks, models that are invariant to these transformations help reduce the amount of training data required. This further increases the model's efficiency and simplifies training. In this paper, we investigate the generalization of invariant representations on out-of-distribution data and try to answer the question: Do model representations invariant to some transformations in a particular seen domain also remain invariant in previously unseen domains? Through extensive experiments, we demonstrate that the invariant model learns unstructured latent representations that are robust to distribution shifts, thus making invariance a desirable property for training in resource-constrained settings.

Language model based methods are powerful techniques for text classification. However, the models have several shortcomings. (1) It is difficult to integrate human knowledge such as keywords. (2) It needs a lot of resources to train the models. (3) It relied on large text data to pretrain. In this paper, we propose Semi- Supervised vMF Neural Topic Modeling (S2vNTM) to overcome these difficulties. S2vNTM takes a few seed keywords as input for topics. S2vNTM leverages the pattern of keywords to identify potential topics, as well as optimize the quality of topics’ keywords sets. Across a variety of datasets, S2vNTM outperforms existing semi-supervised topic modeling methods in classification accuracy with limited keywords provided. S2vNTM is at least twice as fast as baselines.

Web-scale search systems use a large neural network to embed the query which is then hooked into a separate approximate nearest neighbour search (ANNS) pipeline to retrieve similar data points. Such approaches use a rigid – potentially high-dimensional – representation out of encoder to perform the entire search. This can be far from optimal accuracy-compute trade-off. In this paper, we argue that in different stages of ANNS, we can use representations of different capacities, adaptive representations, to ensure that the accuracy-compute tradeoff can be met nearly optimally. In particular, we introduce AdANNS, a novel ANNS design paradigm that explicitly leverages the flexibility and adaptive capabilities of the recently introduced Matryoshka Representations (Kusupati et al., 2022). We demonstrate that using AdANNS to construct the search data structure (AdANNS-C) provides state-of-the-art accuracy-compute tradeoff; AdANNS powered inverted file index (IVF) is up to 1.5% more accurate or up to 100× faster ImageNet-1K retrieval. We also show that matryoshka representations can power compute-aware adaptive search during inference (AdANNS-D) on a fixed ANNS (IVF) structure and be up to 16× faster for similar accuracy. Finally, we explore the applicability of adaptive representations across ANNS building blocks and further analyze the choice of matryoshka representations for semantic search.

Material characterization using electron micrographs is a crucial but challenging task with applications in various fields, such as semiconductors, quantum materials, batteries, etc. The challenges in categorizing electron micrographs include but are not limited to the complexity of patterns, high level of detail, and imbalanced data distribution(long-tail distribution). Existing methods have difficulty in modeling the complex relational structure in electron micrographs, hindering their ability to effectively capture the complex relationships between different spatial regions of micrographs. We propose a hypergraph neural network(HgNN) backbone architecture, a conceptually alternative approach, to better model the complex relationships in electron micrographs and improve material characterization accuracy. By utilizing cost-effective GPU hardware, our proposed framework outperforms popular baselines. The results of the ablation studies demonstrate that the proposed framework is effective in achieving state-of-the-art performance on benchmark datasets and efficient in terms of computational and memory requirements for handling large-scale electron micrograph-based datasets.

In recent years, large multinational corporations have made claims of creating “general purpose” models that can handle many different tasks within natural language processing. Recent works from Meta for example, give theimpression that they have nearly solved machine translation tasks for more than 200 languages including 55 African languages. In this paper, we outline the harms speakers of non dominant languages have experienced due to these grandiose and inaccurate claims, ranging from diverting resources from local startups serving specific communities, to low quality datasets and models from these corporations. We urge the African NLP and machine learning communities to push back against these claims, and support smaller organizations serving their own communities.

With the current set-up, the success of Deep Neural Network models is highly tiedto their size. Although this property might help them improve their performance, itmakes them difficult to train, deploy them on resource-constrained machines, anditerate on experiments. There is also a growing concern about their environmentaland economic impacts. Model Compression is a set of techniques that are appliedto reduce the size of models without a significant loss in performance. Their use isincreasing as models grow with time. However, these techniques alter the behaviorof the network beyond reducing its size. This paper aims to draw attention tothe matter by highlighting present works with regard to Explaniability, NeuralArchitecture Search, and Fairness before finalizing with a suggestion for futureresearch directions.

The advances of Generative AI with interactive capabilities that have beenachieved in the last few years offer unique opportunities for socio-economic mo-bility. Their potential for scalability, accessibility, affordability, personalizingand convenience consists a first-class opportunity for poverty-stricken countriesto adapt and modernize their educational order. As a result, this position papermakes the case for an educational policy framework that would succeed in thistransformation by prioritizing vocational and technical training over academic ed-ucation in sub-Saharan African countries. We highlight substantial applicationsof Large Language Models, tailor-made to their respective cultural background(s)and needs, that would reinforce their systemic decolonization. Lastly, we pro-vide specific historical examples of diverse states successfully implementing suchpolicies in the elementary steps of their socioeconomic transformation, in order tocorroborate our proposal to sub-Saharan African countries to follow their lead.

Test time domain adaptation has come to the forefront as a challenging scenario in recent times. Although single domain test-time adaptation has been well studied and shown impressive performance, this can be limiting when the model is deployed in a dynamic test environment. We explore this continual domain test time adaptation problem here. Specifically, we question if we can translate the effectiveness of single domain adaptation methods to continuous test-time adaptation scenario. We propose to use the given source domain trained model to continually measure the similarity between the feature representations of the consecutive batches. A domain shift is detected when this measure falls below a certain threshold, which we use as a trigger to reset the model back to source and continue test-time adaptation. We demonstrate the effectiveness of our method by performing experiments across datasets, batch sizes and different single domain test-time adaptation baselines. This can have a significant impact in a variety of applications, from healthcare and agriculture to transportation and finance. As a result, this research has the potential to greatly benefit developing countries by providing new tools and techniques for building more effective and efficient machine learning systems.

Research in deep learning is restrictive in developing countries due to a lack of computational resources, quality training data, and expert knowledge, which negatively impacts the performance of deep networks. Moreover, these models are prone to suffer from distribution shift during testing. To address these challenges, this paper presents a novel approach for fine-tuning deep networks in a Domain Generalization setting. The proposed framework, JumpStyle, comprises two key components: (1) an innovative initialization technique that jumpstarts the adaptation process, and (2) the use of style-aware augmentation with pseudo-labeling, in conjunction with a simple and effective test-time adaptation baseline named Tent. Importantly, JumpStyle only requires access to a pre-trained model and is not limited by the training method. The effectiveness of this approach is extensively evaluated through experiments.

On a given dataset, some models perform better than others. Can we examine this performance w.r.t. different strata of the dataset rather than just focusing on an aggregate metric (such as accuracy)? Given that noise and corruption are natural in real-world settings, can we study model failure under such scenarios? For a particular corruption type, do some classes become more difficult to classify than others? To answer such fine-grained questions, in this paper, we explore the use of Item Response Theory (IRT) in computer vision tasks to gain deeper insights into the behavior of models and datasets, especially under corruption. We show that incorporating IRT can provide instance-level understanding beyond what classical metrics (such as accuracy) can provide. Our findings highlight the ability of IRT to detect changes in the distribution of the dataset when it is perturbed through corruption, using latent parameters derived from IRT models. These latent parameters can effectively identify annotation errors, informative images, and class-level information while highlighting the robustness of different models under consideration.