We're seeing an increase in client-side errors whe...
# support-questions-legacyg
goodgravy
09/25/2023, 11:03 AMWe're seeing an increase in client-side errors when hitting the email verification endpoint. The request volume isn't super high (5k/min?) so I don't think we should be hitting any rate-limits.
I can see in APM that we're sometimes timing out when making establishing TCP connections to SuperTokens core. Is there anything you can see about our traffic patterns and usage that could explain what's going on?
Here's a giraffe of errors when hitting the
user/email/verifyr
rp_st
09/25/2023, 11:35 AMhey @goodgravy
rp_st
09/25/2023, 11:35 AMthere has been a random huge spike within the last hour on our end
rp_st
09/25/2023, 11:35 AMg
goodgravy
09/25/2023, 3:08 PMIs this still the case @rp_st ? We made a change ~40m ago
r
rp_st
09/25/2023, 3:32 PMseems better now
rp_st
09/25/2023, 3:32 PMrp_st
09/25/2023, 3:32 PMbut it's not lesser than what it was before.. it's just not higher.
g
goodgravy
09/25/2023, 3:44 PMWe just hit TechCrunch and will be doing some other press, so expecting a moderate uplift in sign-ups over the next couple of days.
r
rp_st
09/25/2023, 3:44 PMAh okay! Congrats on the press! 👏👏
g
goodgravy
09/25/2023, 6:52 PMFYI we got a noticeable spike in 502s ~10 minutes ago hitting various endpoints on core, like
/recipe/user/recipe/jwt/recipe/user/email/verify Copy code
SuperTokens core threw an error for a GET request to path: '/recipe/user/email/verify' with status code: 502 and message: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>r
rp_st
09/25/2023, 7:10 PMRight. We were scaling up. Should be resolved now
g
goodgravy
09/26/2023, 1:19 PM@rp_st we've noticed a big increase in latency on your core endponts in the last 30 mins
goodgravy
09/26/2023, 1:19 PMr
rp_st
09/26/2023, 1:20 PMhmm yea. I see it too. Requests are reaching 80k per min. Which is even more than 1k rps!
rp_st
09/26/2023, 1:20 PMrp_st
09/26/2023, 1:21 PMis this expected amount of traffic?
rp_st
09/26/2023, 1:22 PMwe are further scaling up now.
g
goodgravy
09/26/2023, 1:24 PMWe don't have a notable increase in traffic – 4h scale
r
rp_st
09/26/2023, 1:24 PMhuh.. so this is strange
rp_st
09/26/2023, 1:25 PMmaybe it's just concurrent users?
rp_st
09/26/2023, 1:25 PMbut that should increase your traffic too..
g
goodgravy
09/26/2023, 1:27 PMYeah, that graph is just the raw number of inbound requests to our server, so I think it must be an increase in requests from our server to yours, for a ~constant number of requests from users' browsers
r
rp_st
09/26/2023, 1:27 PMyup. Pretty much
g
goodgravy
09/26/2023, 1:27 PMWe haven't deployed
r
rp_st
09/26/2023, 1:28 PMhmmm
rp_st
09/26/2023, 1:28 PMlet me see the logs. In case there is something
rp_st
09/26/2023, 1:33 PMok, the requests suddenly went back to normal amounts.
rp_st
09/26/2023, 1:34 PMg
goodgravy
09/26/2023, 1:36 PMI see the response times normalising too
r
rp_st
09/26/2023, 1:36 PMbut now the requests are rising again
rp_st
09/26/2023, 1:36 PMrp_st
09/26/2023, 1:37 PMlet me check the IP addresses being used to query. Will get back
rp_st
09/26/2023, 1:40 PM52.204.251.14434.207.186.1752.204.251.144rp_st
09/26/2023, 1:44 PMthese are the request paths that are most common:
- /recipe/user/email/verify GET
- /recipe/user GET
- /recipe/jwt
- /recipe/session/regenerate
Where
/recipe/user GETrp_st
09/26/2023, 1:45 PMMaybe it's an issue witht the caching you had in place? I think you had some code where you would fetch info from your db, and if that returned nothing, then you would query the core for the user info? Just guessing here..
g
goodgravy
09/26/2023, 4:03 PMThere must be something within the library code which is hitting the
/recipe/userr
rp_st
09/26/2023, 4:04 PMYea sure.
rp_st
09/26/2023, 4:06 PMThe thing is, most of the lib code runs when an api is called
rp_st
09/26/2023, 4:07 PMIf there is not much of an increase of the browser calling the backend SDK api, then there shouldn’t be that much of an increase for core calls
rp_st
09/26/2023, 4:57 PMwhen will you be available for a debugging call @goodgravy ?
rp_st
09/26/2023, 5:29 PM@porcellus can help here (you had spoken to him on our last debugging call)
g
goodgravy
09/26/2023, 6:26 PMWe made another infrastructure change 1hr ago and think that that will help with this problem.
Theory is:
- Some of our endpoints / some of the SDK trigger multiple requests to core APIs
- Our DNS resolution infrastructure became overloaded
- That caused some requests to core to timeout
- Something triggered a retry
- We retry all of the multiple requests to core API endpoints
- This results in the increased request volume you were seeing
goodgravy
09/26/2023, 6:26 PMI'm not yet sure where the retry is coming from.
p
porcellus
09/26/2023, 6:48 PMhi. I don't think there's too much I can add based on the above, but the most hit requests paths @rp sent looks like something is updating the email verification claim in sessions (if we are looking for a single thing that calls all those paths on the core). I don't know of anything in our SDK that'd cause that to be retried or called multiple times, so I don't think this helps too much.
r
rp_st
09/27/2023, 4:24 AMthings to seem to be stable for now. The rps has gone back to what it used to be and has been that since the last few hours.
rp_st
09/27/2023, 4:25 AMWe will monitor it for a day and let u know if it rises again.
rp_st
09/27/2023, 1:08 PMit did spike up again in the last few hours.
rp_st
09/27/2023, 1:28 PMand it's gone back to doing 1k rps again now.
2 Views