I think that only brings me to the point where you mentioned Oauth2 where you recommended to use the refresh / access token coming from the oauth call, I am dealing with those manually too, is that bad?
SuperTokens is an open source authentication solution offering features like: Different types of login: Email / password, Passwordless (OTP or Magic link based).