Background: Available evidence on the association of physical activity (PA) or sedentary behavior with cognitive decline is inconclusive. Objective: To assess the association between an active lifestyle score and leisure-time physical activity (LTPA) and changes in cognitive function in the Seguimiento Universidad de Navarra (SUN) prospective cohort. Methods: Cognitive function was evaluated in a subsample of 806 participants of the SUN cohort study using the validated Telephone Interview for Cognitive Status-modified (STICS-m) questionnaire at baseline and after 6 years. LTPA was evaluated with a previously validated 17-item self-administered questionnaire and with information on sedentary lifestyles. We also calculated a multidimensional 8-item PA score. Multivariable linear regression analysis evaluated the association between PA and changes in cognitive function and its interaction by APOE genotype. Results: Mean age of participants was 66 (SD 5.3) years and 69.7% were male. When stratifying by APOE variants, no significant associations between the active lifestyle score or LTPA and changes in cognitive performance over time were found among APOE epsilon 4 carriers. However, we observed that a higher adherence to an active lifestyle (high versus low PA score beta = 0.76 95%CI 0.15,1.36; p trend = 0.011) and a high LTPA (Q4 versus Q1 beta = 0.63; 95%CI -0.01,1.26; p trend = 0.030) were associated with more favorable changes in cognitive function over time among APOE epsilon 4 non-carriers with statistically significant interactions in both cases (p for interaction = 0.042 for PA score, and p = 0.039 for LTPA). Conclusion: The results of the present study suggest that an active lifestyle is associated with a better status of cognitive function over time only among APOE epsilon 4 non-carriers.