Players getting kicked if a single packet is lost to the server

Hi there

I'm experiencing some very annoying problems, which ends up in all players getting kicked (except me as i connect locally).
The problem not occouring a lot every day, but when it is experienced, it happens a lot.

Running a dedicated Rust server on my local Windows Server, few infos about it:
i7-6700k, 48GB RAM, 500GB SSD OS install, 250GB SSD (where the server files are located).
Internet info: 500Mb down / 500Mb up - stable over the 500, and according to speedtest.net, the ping to that server is usually 3/4. Although the players usually are about 5-10 at a time.

Had long searches online, but with no luck finding an answer to the problem. A guess of mine, would be raising "maximum ping" setting, but can't seem to find out where.
I've also hosted other game servers, Minecraft, Arma3, to name a few, but these have no problems like this.

I am leaning towards an internet problem tho, possibly on the WAN side, outside of my reach.
Been in contact with my ISP, but been told I need info whether this is on their networks or not.

I've from them set up "WinMTR", which pings all IP's in a traceroute to the clients on the server.
This is where i can see, that even if just a single packet is lost on the way, ALL clients gets disconnected (except maybe a lucky 1 or 2 who stays)

Greatly appeciate any help! Thanks.
Hey, what is the reason your players are getting kicked with? Do they lag (unable to open boxes/laggy movement) before getting kicked?
Yes, they can't open boxes, doors, in general interact with anything, but are able to move around for a bit before getting the error message "Disconnected: Timed out".
Does the IN/OUT of your server change in the meanwhile?
Haven't had a look at that in the console. I usually use "RustAdmin" for all administration, as I've experienced players getting kicked, if I fiddle around with it too much. I'll for sure have a look. Although I might get a small delay from me noticing they disconnect, as there are no disconnect output in the console, so I have no idea it happens (if I didn't look at RustAdmin)

I do see with the tool i mentioned at first "WinMTR", an IP that seemed to time out quite a bit just now, which resulted in players getting kicked. Some players shared that IP on the route to them, but another player does not, so I'm also monitoring that.

I'll keep you posted of my findings.

#Edit, I'd imagine the in/out just drops to 0/0, as no traffic is connected and required.