reference frames - Is this one of the reason we can't travel in time?

I'd like to ask a question that is not in my field but it's bothering me. Based on that, I'm sorry if I make dumb mistake in my assumptions.

The idea of travelling in time means "going" at some place, but not in the same time. If I decide to go back to a few days ago, this does NOT only means I would have to do the complex process of reverting time, but (and this in my point/question), I would also have to be physically moved.

Here's my theory : Today, the earth is at a certain position around the sun. In three days, this position will be different. Suppose we can successfully revert the time. If I go three days ago, I will be in space, because the earth was not at the same place as today.

Now this is even more complex because the earth goes around the sun but that's not our only movement : our galaxy is moving, our universe is moving. From what I know, the planets are not even only rotating, but also falling.

So in order to achieve time travel, wouldn't we also need to know the exact position of where whe want to be, based on the calculation of every elements we rely on, from our earth rotation to our universe expanding, moving and falling?

Isn't that one of the reason we can not achieve travelling in time?

Thank you for your enlightment. And I'm really sorry if I said something stupid.