Patients recovering from COVID-19 commonly exhibit cognitive and brain alterations, yet the specific neuropathological mechanisms and risk factors underlying these alterations remain elusive. Given the significant global incidence of COVID-19, identifying factors that can distinguish individuals at risk of developing brain alterations is crucial for prioritizing follow-up care. Here, we report findings from a sample of patients consisting of 73 adults with a mild to moderate SARS-CoV-2 infection without signs of respiratory failure and 27 with infections attributed to other agents and no history of COVID-19. The participants underwent cognitive screening, a decision-making task, and MRI evaluations. We assessed for the presence of anosmia and the requirement for hospitalization. Groups did not differ in age or cognitive performance. Patients who presented with anosmia exhibited more impulsive alternative changes after a shift in probabilities (r =  − 0.26, p = 0.001), while patients who required hospitalization showed more perseverative choices (r = 0.25, p = 0.003). Anosmia correlated with brain measures, including decreased functional activity during the decision-making task, thinning of cortical thickness in parietal regions, and loss of white matter integrity. Hence, anosmia could be a factor to be considered when identifying at-risk populations for follow-up.