Plugin reverting specific config valuesSolved

Hey there,

we've recently had to deal with performance issues which lead to frequent crashes. One of the main issues which heavily contributed was this plugin. Despite its already heavy load, the pluhin seems to have reverted one specific config value from true to false.

Starting at line 44:
{
      "Entities Allowed To Drop Loot": {

        "Despawn These Dropped Loot Bags When Base Despawns": false, <-----------
        "Auto Turrets": true,
        "Flame Turret": true,
        "Fog Machine": true,
        "Gun Trap": true,
        "SAM Site": true
      }

The setting value keeps on being reset to false leading to lootbags piling up and not just lowering server fps but also clogging memory map count which eventually leads to 
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3dc00000049
Obtained 20 stack frames.
#0 0x007f10bce5a050 in (Unknown)
#1 0x007f10bcea8eec in (Unknown)
#2 0x007f10bce59fb2 in gsignal
#3 0x007f10bce44472 in gsignal
#4 0x007f10b8e6d3d2 in abort
#5 0x007f10b8e6d32a in abort
#6 0x007f10b8e6f090 in GC_merge_unmapped
#7 0x007f10b8e6f2f5 in GC_merge_unmapped
#8 0x007f10b8e6e445 in GC_unmap_old
#9 0x007f10bdd30b88 in GC_unmap_old
#10 0x007f10bdaf76d3 in GC_finish_collection
#11 0x007f10bdaf785e in GC_finish_collection
#12 0x007f10bdb5a9b7 in GC_collect_a_little_inner
#13 0x007f10bdb4bd97 in GC_collect_a_little_inner
#14 0x007f10bdb4bd51 in GC_collect_a_little
#15 0x007f10bdb4c058 in GC_collect_a_little
#16 0x007f10bddb4327 in (Unknown)
#17 0x007f10bce4524a in (Unknown)
#18 0x007f10bce45305 in (Unknown)
#19 0x0055a8fa170029 in (Unknown)


While i havent been able to confirm if this crash is actually (solely) caused by lootbags, the serverlags/dropping server fps definetly are as reloading the plugin or manually deleting lots of loot does lead to +20-30fps depending on the plugins prior runtime and user activity.

Sincerely,
RustyMain

hi, it's a large plugin with over 600 options so it's going to have a heavy load. this has nothing to do with your loot bag or crash issue though.

the plugin can't revert config settings. that's something on your end. you'll have to look more closely to determine what is happening. it is usually a json error, the file not being saved even when it appears to, bad file permissions or a caching issue with the web panel. I can't recall anything else at the moment, but this will help narrow it down.

your server crash shows that vm.max_map_count is set too low. this crash happens when this limit is exceeded, which can certainly be caused by loot.  some item would have to be set excessively high in the loot tables, in another plugin's config, or spawned by another plugin. a bad copypaste file could cause this. you can rule that out by enabling Empty All Containers option in the profiles. some sellers have base packages and leave excessive loot in their copypaste files. it could be industrial, but nothing you've stated would indicate that.

vm.max_map_count should be 2262144 in /etc/sysctl.conf or any high number really.

apply it with:
sudo sysctl -w vm.max_map_count=2262144

persist hardware reboot by adding it to /etc/sysctl.conf:
vm.max_map_count=2262144

ideally, your host should've done this for you already... you can still exceed the default limit even without this issue, so you should make those changes regardless.

Appreciate your quick reply.

We'd like to mention that we've been using RaidableBases with the same configuration for about half a year now although admittively, issues only just started arising once we scratched 40-50 players for the first time in a wipe. Ever since, those issues have persisted even through low pop wipes with playernumbers as low as 5-15. 

We're running on an i9-9900k with 64Gib of memory in FRA (hetzner). On there, we've got proxmox and multiple vms (thankfully, all unaffected) which one of has pterodactyl and multiple gameservers running. My pterodactyl instance has 48Gib of memory allocated and all other services run just fine. The gameserver in question has 22Gib and 300% cpu available although averaging out at ~12Gib of memory and 120% CPU.
After doing some research lately, we too came to the conclusion of having to up the vm.max_map_count parameter for our kernel (which we haven't done yet). However, increasing the max map count for memory would only delay a server crash in case there's something clogging up ram mappings the way it is at the moment which is the root cause im looking for. Despite RaidableBases, we do have a few other plugins under suspicion atm.

To get back to the main issue:
Editing anything else within the gameserver instance works just fine. Editing the RaidableBases config works/saves fine too regardless of how I do it. Changes do get saved throughout (vm and host-) server restarts aswell.

Any changes within the RaidableBases (well, not sure if really any - we haven't tested every config value) remain saved despite
Ln 44 "Despawn These Dropped Loot Bags When Base Despawns": false,
As of our observation, once the plugin gets (re-)loaded, this value always gets set back to false and we've got no clue why/how. 

Thank you for your time - looking forward for an answer.

Sincerely,
RustyMain

 

 

EDIT:
Funnily enough, after just going through data/RaidableBases/profiles/RaidBases.json and editing
Ln 1761 | "Despawn Dropped Loot Bags From Raid Boxes When Base Despawns": true,
the coherent config value discussed earlier
Ln 45 | "Despawn These Dropped Loot Bags When Base Despawns": true,
automatically changes to true. If Ln 1761 is set to false, so does the boolean in Ln 45.

Not sure what causes this behavior and if it may for some reason truly be on our end though you may wanna look into it.

It's so hard to explain this so it's easily understood, I'll try again...

Merged post

Hi, this is not a new issue for Linux. Plugins can cause it to be exceeded, but it exists on its own too. It's a limit, and it just needs to be increased. It has been this way for years, and affects vanilla servers too. Yeah, it can be so infrequent that it doesn't occur for weeks or months at a time, but it still needs to be increased. It will eventually occur that the limit is exceeded and the server will crash with that error. So it's possible that no plugin is at fault here and you're just hitting that limit quicker than usual, but it's more likely a combination of both the limit being set too low and plugins causing it to be exceeded more frequently. Likely from loot tables containing amount(s) set excessively high, bad copypaste file(s) or another plugin. Under normal conditions, if you increase the limit then it won't be exceeded anymore and that specific crash will not happen again. My advise is to increase the limit to 2262144 and continue your search.

I will look into the despawn options, but it's not related to your crash. If anything, unloading the plugin or using the despawn options is nothing more than a bandaid. When you fix the cause of the issue then any bandaids won't be necessary. 

Merged post

There is an issue with the despawn option in the profile. Not with it reverting to default, but the config option being shared with the profile option as you last said. I missed it in the last update when I fixed this in the paid version. So currently if you make the profile false, both will be false. If you make the profile true, both will be true. Regardless of what you set, the config will use the value from the profile instead. It's shared more than once which isn't intended. 

easy fix so this only happens once more

public bool? SET = false;​

on line 18793 in the .cs, change it to:

public bool? SET = null;​

Hey buddy,

sorry if there's been a bit of a misscommunication here. We didn't mean to blame you(r plugin) for the crashes. We simply wanted to highlight that in case theres any plugin clogging up memory mappings, an increase in max maps would only delay a crash (if the server runs long enough that is)-
Regardless of if this plugin was the case or not, we did want to acknowledge the broken boolean regarding despawning loot as this does clog up server fps after a while regardless of our personal initial issue.

Happy to hear you did find and adress the issue - keep up the good work and pump out many more lovely features/plugins in the future. I wish you best wellbeing for however long you may be around :)

Sincerely,
RustyMain

heya, it still needs to be increased. the default isn't set high enough and it will cause it to crash when its exceeded. it doesnt delay the crash, it prevents it entirely under normal conditions.

it doesnt matter how frequently it crashes, even weeks or months inbetween, it will happen under normal conditions if the limit is not fixed

a plugin pushing it over the limit doesnt mean it shouldnt be fixed. thats all the more reason to do so. if the server isnt crashing then that makes it easier to determine whats going on. monitor counts. again, this will happen on vanilla too, its only a matter of time

i wish you luck fixing your issue and hope you find my advice helpful, but there isnt much more to be said about this specific crash

Locked automatically