You can have an expiration for the refresh token itself so that if the user is inactive for that much time, they will get logged out