Pedestrian safety remains a critical challenge in current transportation systems. In the US, fatal pedestrian crashes have increased by nearly 50% over the past decade. Data shows that children, the elderly, men, and people with low income are involved in far greater pedestrian-vehicle crashes compared to the general population. Autonomous Vehicles (AVs) are expected to effectively detect pedestrians and react to potential accidents. However, due to the constrained mobilities of vulnerable road users, data from vulnerable pedestrians, such as the elderly and children, is often limited. For example, children are more likely to exhibit unpredictable behaviors, and elderly pedestrians on average walk slower than the general population, as shown conceptually in Figure 1. The data scarcity and distinct distributions of these pedestrian groups will make their data minor ``mode" or even ``out-of-distribution" compared to the huge amount of training data from other pedestrian groups. This will lead to larger prediction errors for those groups during the testing stage. Error-prone detection and trajectory prediction of vulnerable pedestrians may cause decision-making that compromises their safety.