uMod - Server memory leak after transferring. Solutions? - Rust

Posted 7 years ago

Last updated 6 years ago

tl/dr: Moved server to new host. Using a lot more memory than before, but nothing has changed. Host is unsure of what it could be and so am I. Please help.

----------

I transferred my server to a new host a couple days ago and noticed a massive memory spike. On my old host, I used about 4GB at wipe, which grew by about 2GB per week (no decay, minus twig). Never had any resource issues, memory was always solid. I transferred my server over to a new host, and my memory usage went from 8GB to 16GB in one day.

I copied my server folder and Oxide folder over directly. I changed nothing aside from what was necessary, such as the IP. Nothing has changed in the weeks before the transfer and I had no issues before.

The host thinks it's a plugin issue, but I've been running the same plugins for months without problems. I'm very confused as to why this is happening and why it would start out of nowhere. Could this just be an issue with the host, possibly something misconfigured? They have good reviews and I haven't read about this from anyone else. I opened a ticket, but they don't seem to know what it could be.

I copied it over to a dedicated server on my own computer and the memory sat at 8GB for over an hour and didn't budge. I'm at a complete loss. Any possible solutions to this?

Host: Streamline
Map size: 4300
Avg players: 20-40
Plugins: BoxLooters, Building Owners, Country Block, Finder, Inventory Viewer, NoGiveNotices, Prod, Sign Checker, Vanish

Posted 7 years ago

Last updated 6 years ago

The new host may have enabled rust features that the old host did not, i.e. AI. Compare server launch configurations, processor affinity, and finally the types of hardware in both machines (not just maximum capacity). Some hardware may be better suited to operating rust servers.

Posted 7 years ago

Last updated 6 years ago

AI and navmesh are enabled for both. I set the launch configurations to be the same. Seems like some launch options are set by the host and hidden though. For example, the old host has server.updatebatch set to 512 by default, but doesn't show it. The new one has it set to 64, which I changed to default (128). I'm not sure what else to look for though, is there a way to easily see this, aside from going through every possible option?

As for hardware, I can ask the host but I'm not sure how much info they can or will give me aside from just the numbers. Would certain hardware actually cause the memory usage to accumulate like that, where other hardware doesn't? Seems more like it's an issue of it just not freeing anything up where it's maybe supposed to. When I restart my server, it goes back down to it's normal level, but then just starts building up again. What is it doing when I restart and why is it not doing that on its own while the server is still running? Seems like the old server did.

Posted 7 years ago

Last updated 6 years ago

The only plugins in your list that potentially create substantial load are Vanish (when in use) and BoxLooters. If the load remains after disabling those plugins then it's safe to say that it's not a plugin.

Posted 7 years ago

Last updated 6 years ago

I tried disabling plugins (one by one and then all at once), and it made no difference. I thought it might be CountryBlock, because it's been giving me errors recently, but the usage is still going up. It doesn't even seem to be related to the player count or anything that's going on, because it was spiking the hardest early this morning when there were only a few people online. It's gone up by almost 5 GB in a day and not much is even happening this late into the wipe.

The host said they have no way of pinpointing why this might be occurring. Is there anything I can do on my end to see what is actually using up resources like this and why they aren't being freed?

Posted 7 years ago

Last updated 6 years ago

Debug information, aside from logs, is sparse because Rust runs in a production mode. That said, it isn't entirely true that they have no way of pinpointing the issue. Presumably they host other rust servers, and given your moderate player count, they probably have other servers in your range. If those servers share the same configuration as yours, then they should have similar issues. I can safely say that your plugins are not the issue.

Posted 7 years ago

Last updated 6 years ago

According to them, no other servers are having this issue. They offered to move me to different hardware, although this would mean changing IP, which essentially means moving the server again. What are the chances that could clear it up?

Is it possible wiping would fix it? I don't see how the map/save would cause a memory leak, but if that's possible I guess I could tolerate it until next week and see how that goes. I don't know anything about this, unfortunately, so I'm just sort of taking shots in the dark and trying to piss off my players as little as possible.

Edit: Also, what about plants? I've heard they can cause resource issues, though I don't know if they'd have more of an impact than all the buildings. Didn't really experience any problems with them before either, as far as I know.

Posted 7 years ago

Last updated 6 years ago

High entity count and AI are typically the largest performance bottlenecks of a rust server. I understand not wanting to switch to another IP address and I don't think all diagnostic options have been exhausted just yet.

What is your entity count? Are you monitoring the entity count? Is there any correlation between an increase in entity count and the decline in server FPS?

Some plugins that might help you manage this..

https://umod.org/plugins/collider-count
https://umod.org/plugins/entity-reducer
https://umod.org/plugins/entity-limit
https://umod.org/plugins/block-remover

Posted 7 years ago

Last updated 6 years ago

The current entity count is about 245,000. The server FPS is okay, it hovers around 180-200, depending on how many players are on. The performance hasn't really been a problem, it's just that the memory grows so quickly throughout the day that I have to restart every 24 hours now at least, whereas before I could easily run it for up to a week. Around this time in the wipe, the server would be using up about 8-10 GB at launch, and that's where it would stay for days at a time. Typically, the entity count brought the base memory usage up by about 2GB per week as it grew, but it was always steady. It never caused it to start randomly spiking, especially when not much is even going on.

I've been monitoring it closely for a few days, and sometimes the memory usage is solid for almost the whole day but then it will start shooting up by 2-3 GB in only a couple hours. This will happen when the entity count has barely grown, or even been reduced.

Posted 7 years ago

Last updated 6 years ago

#10

I'm having a similar problem, the ram on my server stays around 8 -9 + gb all the time Memory: 8.92 to be exact as of right now). I've been having daily crashes at night since I moved to the new host Im now, I and my Host have been communicating back and forth to see if we could find the issue, so far nothing solid . My plugins are way more extensive than yours 74-79 plugins at times. AND I have also started pulling plugins to see if it is plugin related. I dont know if your issue is the same as mine but just wanted to share.

 01 "AdminMenu" (0.1.32) by k1lly0u (0.74s) - AdminMenu.cs
  02 "AdvanceGather" (1.0.0) by Hougan (0.10s) - AdvanceGather.cs
  03 "AlphaChristmas" (1.0.16) by redBDGR (0.56s) - AlphaChristmas.cs
  04 "AlphaLoot" (2.4.17) by FuJiKuRa (13.52s) - AlphaLoot.cs
  05 "AutoCodeLock" (2.2.5) by FuJiCuRa (0.09s) - AutoCodeLock.cs
  06 "AutoDoors" (3.1.2) by Wulf/lukespragg (0.10s) - AutoDoors.cs
  07 "Auto Fuel" (1.0.9) by redBDGR (95.78s) - AutoFuel.cs
  08 "AutoPurge" (1.6.1) by Fujikura/Norn (1.24s) - AutoPurge.cs
  09 "Backpacks" (2.1.8) by LaserHydra (3.32s) - Backpacks.cs
  10 "Bank" (1.0.1) by Calytic (0.74s) - Bank.cs
  11 "Better Chat" (5.0.19) by LaserHydra (0.17s) - BetterChat.cs
  12 "Better Warp" (1.2.2) by Ryan (0.16s) - BetterWarp.cs
  13 "BoatCommander" (0.1.18) by k1lly0u (4.36s) - BoatCommander.cs
  14 "BradleyControl" (0.2.2) by Mattparks (6.19s) - BradleyControl.cs
  15 "Bridge" (2.0.12) by Colon Blow (1.64s) - Bridge.cs
  16 "BuildingBlocker" (2.4.0) by Vlad-00003 (0.55s) - BuildingBlocker.cs
  17 "BuildingWrapper" (0.1.4) by ignignokt84 (0.00s) - BuildingWrapper.cs
  18 "BuriedTreasure" (1.0.5) by Colon Blow (88.14s) - BuriedTreasure.cs
  19 "CarCommander" (0.2.56) by k1lly0u (17.78s) - CarCommander.cs
  20 "Copy Paste" (3.6.8) by Reneb (0.01s) - CopyPaste.cs
  21 "DeathMarker" (1.0.8) by redBDGR (14.39s) - DeathMarker.cs
  22 "Death Notes" (6.1.0) by LaserHydra (6.92s) - DeathNotes.cs
  23 "Economics" (3.1.6) by Wulf/lukespragg (0.22s) - Economics.cs
  24 "Enhanced Hammer" (1.3.4) by Fuji/Visa (0.24s) - EnhancedHammer.cs
  25 "Entity Owner" (3.1.8) by Calytic (0.04s) - EntityOwner.cs
  26 "FancyDrop" (2.7.8) by FuJiCuRa (2.16s) - FancyDrop.cs
  27 "FauxAdmin" (1.0.15) by Colon Blow (0.04s) - FauxAdmin.cs
  28 "Friends" (2.2.5) by Nogrod (0.02s) - Friends.cs
  29 "GrTeleport" (1.0.12) by carny666 (2.63s) - GrTeleport.cs
  30 "GUIAnnouncements" (1.23.83) by JoeSheep (3.54s) - GUIAnnouncements.cs
  31 "GUI Shop" (1.4.6) by Nogrod / Reneb (0.35s) - GUIShop.cs
  32 "HeliControl" (1.3.1) by Shady (4.46s) - HeliControl.cs
  33 "HomeRecycler" (1.3.5) by wazzzup (1.40s) - HomeRecycler.cs
  34 "Image Library" (2.0.45) by Absolut & K1lly0u (0.05s) - ImageLibrary.cs
  35 "InfoPanel" (0.9.8) by Ghosst / Nogrod (13.65s) - InfoPanel.cs
  36 "Kits" (3.2.92) by Reneb (0.53s) - Kits.cs
  37 "LockOnRockets" (0.2.12) by k1lly0u (50.76s) - LockOnRockets.cs
  38 "Loot Bouncer" (0.3.0) by Sorrow (0.57s) - LootBouncer.cs
  39 "NameFix" (1.0.1) by Visagalis (0.04s) - NameFix.cs
  40 "NeonSkins" (1.0.10) by Colon Blow (1.14s) - NeonSkins.cs
  41 "NeverWear" (0.1.4) by k1lly0u (0.12s) - NeverWear.cs
  42 "NightVision" (1.3.1) by Jake_Rich (0.45s) - NightVision.cs
  43 "No Give Notices" (0.2.0) by Wulf/lukespragg (0.00s) - NoGiveNotices.cs
  44 "NTeleportation" (1.0.32) by RFC1920 (2.78s) - NTeleportation.cs
  45 "PathFinding" (1.1.3) by Reneb / Nogrod (0.36s) - PathFinding.cs
  46 "HumanNPC" (0.3.18) by Reneb/Nogrod/Calytic (20.19s) - HumanNPC.cs
  47 "PlaneCrash" (0.1.99) by k1lly0u (2.22s) - PlaneCrash.cs
  48 "PlayerAdministration" (1.3.18) by ThibmoRozier (0.00s) - PlayerAdministration.cs
  49 "PlayerRanks" (1.3.9) by Steenamaroo (4.42s) - PlayerRanks.cs
  50 "PooBot" (1.0.11) by Colon Blow (15.03s) - PooBot.cs
  51 "PowerSmelt" (1.0.10) by redBDGR (4.33s) - PowerSmelt.cs
  52 "PreventLooting" (1.7.0) by CaseMan (0.22s) - PreventLooting.cs
  53 "Quick Sort" (1.3.3) by Frenk92 (39.68s) - QuickSort.cs
  54 "RandomSpawns" (0.2.0) by k1lly0u (0.09s) - RandomSpawns.cs
  55 "RemoverTool" (4.2.10) by Reneb/Fuji (2.44s) - RemoverTool.cs
  56 "Rewards" (1.3.14) by Tarek (0.31s) - Rewards.cs
  57 "ServerInfo" (0.5.3) by Fujikura/baton (0.22s) - ServerInfo.cs
  58 "Signal Cooldown" (1.0.3) by Vliek (0.01s) - SignalCooldown.cs
  59 "Sign Artist" (1.1.2) by Mughisi (0.01s) - SignArtist.cs
  60 "SkinBox" (1.13.5) by FuJiCuRa (7.39s) - SkinBox.cs
  61 "Slots" (1.0.10) by Colon Blow (19.57s) - Slots.cs
  62 "StacksExtended" (1.3.2) by Fujikura (6.63s) - StacksExtended.cs
  63 "Steam Groups" (0.3.8) by Wulf/lukespragg (0.95s) - SteamGroups.cs
  64 "Tebex Donate" (1.0.0) by Tebex (2.39s) - TebexDonate.cs
  65 "TimeOfDay" (2.3.2) by Fujikura (0.00s) - TimeOfDay.cs
  66 "Tipster" (1.4.3) by FuJiCuRa (0.38s) - Tipster.cs
  67 "TruePVE" (0.9.0) by ignignokt84 (48.61s) - TruePVE.cs
  68 "Tsunderellas horse plugin" (1.5.1) by Tsunderella (17.03s) - TsunHorse.cs
  69 "UberTool" (1.3.15) by FuJiCuRa (18.31s) - UberTool.cs
  70 "UpLifted" (1.2.6) by FuJiCuRa (11.33s) - UpLifted.cs
  71 "Vanish" (0.7.2) by nivex (21.73s) - Vanish.cs
  72 "Waypoints" (1.1.2) by Reneb (0.00s) - Waypoints.cs
  73 "ZLevelsRemastered" (2.9.5) by FuJiCuRa (3.48s) - ZLevelsRemastered.cs
  74 "ZoneManager" (2.5.15) by Reneb / Nogrod (50.49s) - ZoneManager.cs
(09:37:59) | 8723480[70391067/8723480] was killed by Henrynight[64708959/76561198120521795]

plugins that I have pulled: Botspawn and ZombieHorde(paid)

Posted 7 years ago

Last updated 6 years ago

#11

In response to krazy ():
I'm having a similar problem, the ram on my server stays around 8 -9 + gb all the time Memory: 8.92...

What host are you using?