I am migrating users, and I have a problem with so...
# support-questions-legacyc
CaptainPhoton
06/27/2022, 10:13 AMI am migrating users, and I have a problem with some characters in the metadata. Characters like "ä" are "ä" in the database. I am using the supertokens Metadata recipe to insert the metadata. What charsets and collations does Supertokens expect? Is there anything I can do?
CaptainPhoton
06/27/2022, 10:14 AMthe database is using the utf8mb4_unicode_ci collation and utf8mb4 charset
r
rp_st
06/27/2022, 10:15 AMhey @porcellus can help with this
c
CaptainPhoton
06/27/2022, 10:15 AMhopefully!
p
porcellus
06/27/2022, 10:19 AMHi. I'd expect it uses utf8, but I'm not a 100% sure, I'll check. ä becoming 2 chars feels like a normalization going wrong.
c
CaptainPhoton
06/27/2022, 10:20 AMokay! Is it fixable?
p
porcellus
06/27/2022, 10:28 AMIt should be 🙂 but I need to check what/where it goes wrong
porcellus
06/27/2022, 10:29 AMBtw, are you getting the wrong info back or is it just an issue of what you see in the db?
porcellus
06/27/2022, 10:29 AMI mean: is the metadata recipe returning the malformed string?
c
CaptainPhoton
06/27/2022, 10:31 AMthis is what I see in the DB, and also what it returns
p
porcellus
06/27/2022, 10:31 AMI see, thanks.
porcellus
06/27/2022, 10:32 AMwhat kind of stack (backend/db) are you using?
c
CaptainPhoton
06/27/2022, 10:33 AMmariadb and node/express (with Sails)
CaptainPhoton
06/27/2022, 10:33 AMbut the metadata is going through the Supertokens SDK
p
porcellus
06/27/2022, 10:34 AMYeah, what I want to check is that the node SDK does this encoding correctly 🙂
c
CaptainPhoton
06/27/2022, 10:34 AMah yes!
CaptainPhoton
06/27/2022, 10:39 AMthis is probably the issue: https://stackoverflow.com/questions/40211246/encoding-issue-with-axios
p
porcellus
06/27/2022, 10:39 AMI'm looking at the same thing. we are using axios to post/get data
c
CaptainPhoton
06/27/2022, 10:40 AMyeah I saw that in the code 😄
CaptainPhoton
06/27/2022, 10:42 AMencoding issues are nasty :=
p
porcellus
06/27/2022, 10:43 AMThey are. equality checks through encoding/normalizations are even more nasty. I'm happy we are not doing that 😄
porcellus
06/27/2022, 11:38 AMyep, so axios is not sending a content-encoding header
porcellus
06/27/2022, 11:38 AMso the java implementation doesn't try to parse it as utf8
porcellus
06/27/2022, 11:38 AMI'll come up w/ and provide a fix tonight I think
porcellus
06/27/2022, 11:39 AM(most likely to the node SDK, but I'll discuss it with @rp_st )
c
CaptainPhoton
06/27/2022, 12:27 PMgreat, thanks! 🙏
r
rp_st
06/28/2022, 8:04 AMhey @CaptainPhoton we have fixed this issue (hopefully) - try out version 10.0.1 of the node SDK
c
CaptainPhoton
06/28/2022, 8:04 AMGreat, will try right now! thanks
CaptainPhoton
06/28/2022, 8:12 AM@rp_st @porcellus IT WORKS, amazing! 😄 thanks for the fast fix
r
rp_st
06/28/2022, 8:12 AMall thanks to @porcellus
c
CaptainPhoton
06/28/2022, 8:12 AM👏