Do I always have to walk exactly to the beat?

No. However, the movement experience improves if you walk at roughly the same tempo as the music. The music feedback for the arm swing is also designed to roughly match this tempo. If you walk much slower or faster, the music will therefore sound less full. The rhythm of the music also has a positive effect on your movement if it is roughly linked to the tempo of the music.