Normally when you edit the JSON file in the lang file, you could make your edit and just simply reload the plugin. Is that not the case anymore?

I have tried unloading the plugin first and then making the change, but that didn't make any difference either. It only seems to work when I restart the whole server.

There are edge cases where it requires a server restart, usually something to do with the user's preferences.

Where are my user preferences? Never seen that..

Maybe it helps to know that I used "RustServerInstaller" from codefling.

An installer wouldn't affect it, and user preferences are set by each user based on their client language or on command and stored under oxide/data.