Background Physical activity is essential for health and well-being during adolescence, and active behaviour early in life predicts higher physical activity levels in adulthood. Although adolescents with intellectual disability (ID) consistently show lower activity levels than peers without ID, national environments—such as school structures, disability support systems, and access to inclusive leisure activities—may influence these patterns. There is limited evidence from Sweden, a country with distinct educational and support frameworks for youth with ID. The present study aimed to examine physical activity patterns among Swedish adolescents with and without ID using accelerometer data. Methods Physical activity was measured objectively using hip-worn accelerometers (ActiGraph GT3X) over seven consecutive days. This cross-sectional study included 45 adolescents with mild-to-moderate ID (median [IQR], 17.0 [14.0–19.0] years; 45.2% females) and 70 adolescents without ID (16.0 [15.0-16.3] years; 62.2% females). Physical activity was categorised as sedentary behaviour (SB), light physical activity (LPA), and moderate-to-vigorous physical activity (MVPA) and analysed across school days, weekend days, and separately for daytime and evening periods on school days. Results Overall, the relative amount of SB was similar between groups (p > 0.05), but significant differences were observed for LPA and MVPA. Adolescents with ID accumulated more LPA during school-day daytime hours (23% vs. 19%; p = 0.012), while peers without ID accumulated more MVPA during leisure time, such as school-day evenings (p = 0.025) and weekends (p = 0.039). For both groups, MVPA was higher on school days than on weekends (p < 0.001). Among adolescents with ID, SB increased markedly on weekends (72% vs. 77%; p < 0.001). Conclusion Adolescents with ID were generally less physically active than peers without ID, except during schoolday daytime, where the MVPA was similar and LPA was higher. Leisure time, particularly weekends and schoolday evenings, seems to be a critical period in achieving sufficient MVPA among adolescents with ID. Targeted interventions and coordinated support from key stakeholders such as school health services, paediatric health care, social care services and organised sports, with a particular focus on unstructured time, may help promote active lifestyles and reduce health disparities in this population.