It isn’t a single “thing” you are some distance away from. It’s photons remaining from the early universe that can be found everywhere without direction. Pick “one” of them and you can track your speed relative to it. It’s the closest thing we have to a universal reference frame.

Also see the later questions on https://www.astro.ubc.ca/people/scott/faq_basic.html