For the past few months now my server has been crashing 2 or 3 times a week with the same error. It gives me some stack frames sometimes, but the error code is always the same. Here's the most recent one below:

Saving complete
Caught fatal signal - signo:11 code:128 errno:0 addr:(nil)
Obtained 15 stack frames.
#0 0x007fc88835d980 in funlockfile
#1 0x007fc84ba9da70 in GC_push_next_marked_dirty
#2 0x007fc84ba96a23 in GC_mark_some
#3 0x007fc84ba967d0 in GC_collect_a_little_inner
#4 0x007fc84ba959a1 in GC_collect_a_little
#5 0x007fc889867f58 in (Unknown)
#6 0x007fc88966d483 in (Unknown)
#7 0x007fc88966d60a in (Unknown)
#8 0x007fc8896c1f17 in (Unknown)
#9 0x007fc8896bb2a7 in (Unknown)
#10 0x007fc8896bb262 in (Unknown)
#11 0x007fc8896bb54c in (Unknown)
#12 0x007fc8898e4227 in PlayerMain(int, char**)
#13 0x007fc887f7bc87 in __libc_start_main
#14 0x0055be25829029 in (Unknown)

Seems to happen after a server save, so there at least is no rollback. The server runs on Linux I'm pretty sure. Runs a size 4250 procedural map.

For reference, here is the list of plugins installed on the server:

01 "Abandoned Bases" (2.0.8) by nivex (0.03s) - AbandonedBases.cs
02 "AdminHammer" (1.13.0) by mvrb (0.00s) - AdminHammer.cs
03 "Admin Panel" (1.4.7) by nivex (0.01s) - AdminPanel.cs
04 "Admin Radar" (5.3.2) by nivex (0.08s) - AdminRadar.cs
05 "Auto Broadcast" (1.0.9) by Wulf (0.00s) - AutoBroadcast.cs
06 "Auto Doors" (3.3.10) by Wulf/lukespragg/Arainrr/James/Bushhy (0.04s) - AutoDoors.cs
07 "BGrade" (1.1.5) by Ryan / Rustoria.co (0.00s) - BGrade.cs
08 "Backpacks" (3.13.0) by WhiteThunder (0.08s) - Backpacks.cs
09 "BaseUtility" (1.4.1) by Netch (0.00s) - BaseUtility.cs
10 "Beds Cooldowns" (1.1.4) by Orange (0.07s) - BedsCooldowns.cs
11 "Better Chat" (5.2.14) by LaserHydra (0.03s) - BetterChat.cs
12 "BetterChatFlood" (1.0.5) by Ryan (0.00s) - BetterChatFlood.cs
13 "BetterChatIgnore" (1.0.3) by MisterPixie (0.00s) - BetterChatIgnore.cs
14 "Better Chat Mute" (1.2.1) by LaserHydra (0.00s) - BetterChatMute.cs
15 "Better Chat Filter" (1.7.1) by NooBlet (0.01s) - BetterChatFilter.cs
16 "Better Chat Mute Voice" (1.0.4) by collect_vood (0.00s) - BetterChatMuteVoice.cs
17 "BetterNpc" (1.2.8) by KpucTaJl (0.20s) - BetterNpc.cs
18 "Blueprint Manager" (2.0.6) by Whispers88 (0.01s) - BlueprintManager.cs
19 "Bradley" (1.3.9) by Ts3Hosting (0.23s) - Bradley.cs
20 "Building Grades" (1.0.12) by Default/Arainrr (0.00s) - BuildingGrades.cs
21 "Building Workbench" (1.3.3) by MJSU (0.01s) - BuildingWorkbench.cs
22 "Bypass Queue" (1.0.4) by Orange (0.00s) - BypassQueue.cs
23 "Car Spawn Settings" (2.3.0) by WhiteThunder (0.00s) - CarSpawnSettings.cs
24 "Clear Night" (2.3.6) by Clearshot (0.02s) - ClearNight.cs
25 "Coloured Chat" (2.2.87) by collect_vood (0.01s) - ColouredChat.cs
26 "Convoy" (2.4.0) by Adem (0.00s) - Convoy.cs
27 "Cooking 2.0" (2.0.5) by imthenewguy (0.07s) - Cooking.cs
28 "Copy Paste" (4.1.37) by misticos (0.03s) - CopyPaste.cs
29 "Custom Vending Setup" (2.10.2) by WhiteThunder (0.02s) - CustomVendingSetup.cs
30 "Dance" (1.2.0) by senyaa (0.00s) - Dance.cs
31 "Dangerous Treasures" (2.3.6) by nivex (0.26s) - DangerousTreasures.cs
32 "Discord Core" (2.1.3) by MJSU (0.01s) - DiscordCore.cs
33 "Discord Roles" (2.0.10) by MJSU (0.02s) - DiscordRoles.cs
34 "Duelist" (1.3.1) by nivex (0.60s) - Duelist.cs
35 "EasyVote" (2.0.43) by Exel80 (0.01s) - EasyVote.cs
36 "Excavator Lock" (0.4.26) by Lorenzo (0.01s) - ExcavatorLock.cs
37 "Freeze" (3.0.3) by Wulf (0.10s) - Freeze.cs
38 "Furnace Splitter" (2.5.2) by FastBurst (0.03s) - FurnaceSplitter.cs
39 "Ganja" (2.0.5) by The_Kiiiing (0.11s) - Ganja.cs
40 "Godmode" (4.2.14) by Wulf/lukespragg/Arainrr/dFxPhoeniX (0.01s) - Godmode.cs
41 "GridPower" (1.0.16) by Nikedemos (0.69s) - GridPower.cs
42 "HarborEvent" (2.1.8) by KpucTaJl (0.02s) - HarborEvent.cs
43 "Ignore" (2.1.3) by MisterPixie (0.00s) - Ignore.cs
44 "Image Library" (2.0.62) by Absolut & K1lly0u (0.38s) - ImageLibrary.cs
45 "Ingame Clock GUI" (0.0.7) by deer_SWAG (0.01s) - IngameClockGUI.cs
46 "Inventory Viewer" (4.0.6) by Whispers88 (0.00s) - InventoryViewer.cs
47 "Keycard Notify" (1.2.2) by Cooperkit (0.00s) - KeycardNotify.cs
48 "Loading Messages" (1.0.9) by CosaNostra/Def/klauz24 (0.05s) - LoadingMessages.cs
49 "Loot Bouncer" (1.0.10) by Sorrow/Arainrr (0.02s) - LootBouncer.cs
50 "Loot Defender" (2.1.5) by Author Egor Blagov, Maintainer nivex (0.02s) - LootDefender.cs
51 "Mini-Copter Options" (2.4.1) by Pho3niX90 (0.03s) - MiniCopterOptions.cs
52 "Monument Finder" (3.1.1) by WhiteThunder (0.02s) - MonumentFinder.cs
53 "Monuments Recycler" (0.2.6) by Dana (0.04s) - MonumentsRecycler.cs
54 "NTeleportation" (1.7.7) by nivex (0.20s) - NTeleportation.cs
55 "Night Lantern" (2.0.99) by k1lly0u (0.13s) - NightLantern.cs
56 "NightVision" (2.4.1) by Clearshot (0.03s) - NightVision.cs
57 "No Flykick" (1.3.4) by August (0.00s) - NoFlykick.cs
58 "No Give Notices" (0.3.0) by Wulf (0.00s) - NoGiveNotices.cs
59 "No Tech Tree" (1.2.0) by Sche1sseHund (0.00s) - NoTechTree.cs
60 "No Weapon Drop" (1.2.0) by Fujikura (0.00s) - NoWeaponDrop.cs
61 "NpcRaiders" (1.6.0) by Razor (0.01s) - NpcRaiders.cs
62 "NpcSpawn" (2.6.0) by KpucTaJl (0.83s) - NpcSpawn.cs
63 "Object Remover" (3.0.5) by Iv Misticos (0.00s) - ObjectRemover.cs
64 "PermissionsManager" (2.0.9) by Steenamaroo (0.03s) - PermissionsManager.cs
65 "Personal Heli" (1.1.10) by Egor Blagov (0.04s) - PersonalHeli.cs
66 "PersonalRecycler" (1.0.8) by imthenewguy (0.03s) - PersonalRecycler.cs
67 "Play Time Rewards" (1.2.2) by Khan (0.02s) - PlayTimeRewards.cs
68 "PlayerAdministration" (1.6.9) by ThibmoRozier (0.00s) - PlayerAdministration.cs
69 "Player Challenges" (2.0.52) by k1lly0u (0.86s) - PlayerChallenges.cs
70 "PlayerList" (0.3.2) by Wulf/lukespragg (0.00s) - PlayerList.cs
71 "PlayerRanks" (2.2.6) by Steenamaroo (0.24s) - PlayerRanks.cs
72 "PreventLooting" (1.14.1) by CaseMan (0.01s) - PreventLooting.cs
73 "PrivateMessages" (1.1.11) by MisterPixie (0.00s) - PrivateMessages.cs
74 "PveMode" (1.1.1) by KpucTaJl (0.04s) - PveMode.cs
75 "Raid Tracker" (2.1.1) by Clearshot (0.08s) - RaidTracker.cs
76 "Raidable Bases" (2.7.4) by nivex (0.19s) - RaidableBases.cs
77 "Remover Tool" (4.3.41) by Reneb/Fuji/Arainrr/Tryhard (0.04s) - RemoverTool.cs
78 "RustRewards" (3.1.4) by Steenamaroo (2.61s) - RustRewards.cs
79 "Rustcord" (3.3.2) by Kirollos & OuTSMoKE (0.33s) - Rustcord.cs
80 "Server Info" (0.5.8) by FastBurst (0.08s) - ServerInfo.cs
81 "Server Rewards" (0.4.74) by k1lly0u (0.69s) - ServerRewards.cs
82 "Shipwreck" (1.0.9) by Adem (0.04s) - Shipwreck.cs
83 "Sign Artist" (1.4.1) by Whispers88 (0.00s) - SignArtist.cs
84 "Skill Tree" (1.4.6) by imthenewguy (0.10s) - SkillTree.cs
85 "SkinBox" (2.1.26) by k1lly0u (1.51s) - SkinBox.cs
86 "Sort Button" (2.0.2) by MON@H (0.00s) - SortButton.cs
87 "Stack Size Controller" (4.1.2) by AnExiledDev/patched by chrome (0.08s) - StackSizeController.cs
88 "Tebex Donate" (1.7.0) by Tebex (0.01s) - TebexDonate.cs
89 "Terrain Violation Fix" (1.0.3) by Tryhard (0.00s) - TerrainViolationFix.cs
90 "TimeOfDay" (2.3.4) by FuJiCuRa (0.00s) - TimeOfDay.cs
91 "Trade" (1.2.44) by Calytic (0.00s) - Trade.cs
92 "TruePVE" (2.1.3) by nivex (0.04s) - TruePVE.cs
93 "UFilter" (5.1.2) by Wulf/lukespragg (0.01s) - UFilter.cs
94 "Unburnable Meat" (1.0.1) by S642667 (0.00s) - UnburnableMeat.cs
95 "UpdateChecker" (4.1.2) by tofurahie (0.01s) - UpdateChecker.cs
96 "VIPToken" (1.1.7) by imthenewguy (0.00s) - VIPToken.cs
97 "Vanish" (1.8.0) by Whispers88 (0.01s) - Vanish.cs
98 "Vehicle Deployed Locks" (1.10.0) by WhiteThunder (0.02s) - VehicleDeployedLocks.cs
99 "WaterBases" (1.0.14) by Nikedemos (0.17s) - WaterBases.cs
100 "WaterEvent" (2.1.1) by KpucTaJl (0.05s) - WaterEvent.cs
101 "Welcomer" (2.1.0) by Dana (0.02s) - Welcomer.cs
102 "Wipe Block" (1.0.6) by Orange (0.07s) - WipeBlock.cs
103 "Wipe Timer" (1.1.0) by Rick (0.00s) - WipeTimer.cs
104 "XDCasino" (1.7.34) by DezLife (0.06s) - XDCasino.cs
105 "Zone Manager" (3.1.4) by k1lly0u (2.52s) - ZoneManager.cs
106 "Dynamic PVP" (4.2.17) by CatMeat/Arainrr (2.18s) - DynamicPVP.cs

How ofter you doing rust server restarts?

I'm doing them once a day at 11AM UTC

Shneid

I'm doing them once a day at 11AM UTC

Search threw all your plugins see how many ya got doing things OnServerSave()

Alright so after doing a search, I've found these 23 plugins doing something OnServerSave():

AutoDoors.cs
Backpacks.cs
BGrade.cs
ColouredChat.cs
Cooking.cs
Duelist.cs
DynamicPVP.cs
FurnaceSplitter.cs
Godmode.cs
GridPower.cs
LootDefender.cs
NightLantern.cs
NTeleportation.cs
PersonalHeli.cs
PersonalRecycler.cs
PlayerChallenges.cs
PreventLooting.cs
RaidTracker.cs
RemoverTool.cs
RustRewards.cs
SkillTree.cs
SkinBox.cs
Welcomer.cs

Whats your servers memory usage when its server saving?

Razor

Whats your servers memory usage when its server saving?

There is no spike in memory usage when saving. I just ran a save at 9.6gb and it never changed. But that being said, longer uptime results in higher memory usage, but apart from that, no unusual activity happens.

Shneid

There is no spike in memory usage when saving. I just ran a save at 9.6gb and it never changed. But that being said, longer uptime results in higher memory usage, but apart from that, no unusual activity happens.

Do you got logs enabled whats logs show.

Saved 124,839 ents, cache(1.26), write(0.06), disk(0.02).
Saving complete

This is in the logs from when I run server.save

Shneid

Saved 124,839 ents, cache(1.26), write(0.06), disk(0.02).
Saving complete

This is in the logs from when I run server.save

Ya would need to post full log when crash happenes the log should be in the server root folder if enabled in ya startup file ya should have somthing that shows the log example -logfile "serverlog.txt"

Hmm, well I have logs for each day in a logs folder. I also have a core folder that shows dump files on a server crash. But these are both large files and I don't know if I could post them to the website. Could I possibly add you on discord?

Im not to sure what to look for but was hoping somone else would jump in the convo that knows what to look for in the logs.

Do you read this? Seems to be something about unity?
Caught fatal signal - signo:11 code:1 errno:0 - Unity Forum

omegablue

Do you read this? Seems to be something about unity?
Caught fatal signal - signo:11 code:1 errno:0 - Unity Forum

The code is difrent his is code:128 and theres not mutch info on that code.

Shneid

Hmm, well I have logs for each day in a logs folder. I also have a core folder that shows dump files on a server crash. But these are both large files and I don't know if I could post them to the website. Could I possibly add you on discord?

try and make sure you have -nographics   in your startup script most the signo errors i see on google are related to graphics