uMod - Server Crash mmap - Rust

Posted 4 years ago

Hello UMOD community, 

I have the problem that my server keeps crashing with the following message:

[Economics] Saving balances for players... 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[JPipes] 60 pipes saved 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

mmap(PROT_NONE) failed
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e900002783
Obtained 30 stack frames.
#0 0x007fa014a6a7bb in (Unknown)
#1 0x007fa014a55535 in (Unknown)
#2 0x007fa0142da686 in (Unknown)
#3 0x007fa0142da6f3 in (Unknown)
#4 0x007fa0142dd5f3 in (Unknown)
#5 0x007fa0142dd9c0 in (Unknown)
#6 0x007fa0142ddadf in (Unknown)
#7 0x007fa0142de96c in (Unknown)
#8 0x007fa0142e1a02 in (Unknown)
#9 0x007fa0142e1bdc in (Unknown)
#10 0x007fa0142b4cda in (Unknown)
#11 0x007fa01426ae81 in (Unknown)
#12 0x007fa01426aee8 in (Unknown)
#13 0x000000416570cf in (wrapper managed-to-native) object:__icall_wrapper_ves_icall_array_new_specific (intptr,int)
#14 0x00000041698fba in UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)
#15 0x007fa0140fc8f9 in (Unknown)
#16 0x007fa0142690af in (Unknown)
#17 0x007fa014269f5c in (Unknown)
#18 0x007fa0157f74e2 in (Unknown)
#19 0x007fa0157f58ba in (Unknown)
#20 0x007fa0157d8e6a in (Unknown)
#21 0x007fa0157d8aa5 in (Unknown)
#22 0x007fa01558a292 in (Unknown)
#23 0x007fa0156eec00 in (Unknown)
#24 0x007fa0156e8d3e in (Unknown)
#25 0x007fa0156e8d81 in (Unknown)
#26 0x007fa0156e929a in (Unknown)
#27 0x007fa01584c86e in (Unknown)
#28 0x007fa014a5709b in (Unknown)
#29 0x005622f86fe689 in (Unknown)

does anyone have any idea what the problem could be?

Posted 4 years ago

Last updated 4 years ago

-------------------------------
mmap(PROT_NONE) failed
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e8000012fb
Obtained 14 stack frames.
#0 0x007ff458e6f337 in (Unknown)
#1 0x007ff458e70a28 in (Unknown)
#2 0x007ff4582eb686 in (Unknown)
#3 0x007ff4582eb6f3 in (Unknown)
#4 0x007ff4582ee5f3 in (Unknown)
#5 0x007ff4582ee9c0 in (Unknown)
#6 0x007ff4582eeadf in (Unknown)
#7 0x007ff4582ef96c in (Unknown)
#8 0x007ff4582f2a02 in (Unknown)
#9 0x007ff4582f2bdc in (Unknown)
#10 0x007ff4582c5e3a in (Unknown)
#11 0x007ff45827d07a in (Unknown)
#12 0x007ff458290066 in (Unknown)
#13 0x000000400e06b6 in (wrapper managed-to-native) string:FastAllocateString (int)

-------------------------------
same for me ...crashed twice with the same error when I ban players.. My fix was to remove over 50% old bans ..

But the big problem for my server from the last update is that: the save process is slower and make the game freeze for 2-5 seconds if pop is over 50+ players :
(20:10:09) | Saved 21,911 ents, cache(9.42), write(0.51), disk(0.00).

Also there are some little freezes every 5-10 seconds if you ride a horse , fly minicopter, or in boat . (only a server restart will fix this mini spikes in vehicles for a limited period 10-24 hours)

Is strange because my server is a dedicate machine and runned RUST without problems in the last 3 years with pop over 200+ players in wipe days : vanilla , oxide with admin radar&vanish .. (i7-4770 CPU - 10% used , RAM 8.09 GB used / 30.40 GB total , CentOS Linux 7.4.1708) .
Even with oxide.unload * the game still freezes and all the players complained.
I also tried to change the default values for: server.tickrate, server.updatebatch, server.maxpacketspersecond .. but no positive results.

Something is not verry well optimised with the cache or GARBAGE COLLECT ..

I'm out of solutions and if the next update will be the same I'll quit hosting this game.

Posted 4 years ago

Last updated 4 years ago

Hey, I'm not sure and I think this is causing one of the plugins. 
My memory usage on the server is excessive. 
CPU i7 5820K 8-12% - 27GB RAM used / 128 GB total .. 
the vanilla that runs as a second server also has almost 10GB RAM consumption.

9 months after

Posted 4 years ago

Just started getting the same problem this morning...

mmap(PROT_NONE) failed
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3ea0005a0c8
Obtained 17 stack frames.
#0 0x007f662a0c8bb0 in (Unknown)

Posted 4 years ago

RustNRelax
Just started getting the same problem this morning...

mmap(PROT_NONE) failed
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3ea0005a0c8
Obtained 17 stack frames.
#0 0x007f662a0c8bb0 in (Unknown)

mmap(PROT_NONE) failed
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e800007bae

what plugins are you using? We may be able to find a commonality.

Posted 4 years ago

I figured out the problem on my servers just yesterday.

I use LGSM to schedule a restart every day, but because of the high entity count, it was taking the server longer to start up towards the end of the month. LGSM queries the server with the monitor command every 5 minutes and if the server hasn't completely started then it tries to start it again, but this results in the server doing a full restart before it has completely started up.

At the beginning of the month, with 100,000 entities this is ok, but when you get to 200,000 entities it takes longer to start up and the monitor service thinks the server is still down.

To fix this I changed the querydelay setting in LGSM from 5 mins to 15 mins and it solved the problem.

Posted 4 years ago

RustNRelax
I figured out the problem on my servers just yesterday.

I use LGSM to schedule a restart every day, but because of the high entity count, it was taking the server longer to start up towards the end of the month. LGSM queries the server with the monitor command every 5 minutes and if the server hasn't completely started then it tries to start it again, but this results in the server doing a full restart before it has completely started up.

At the beginning of the month, with 100,000 entities this is ok, but when you get to 200,000 entities it takes longer to start up and the monitor service thinks the server is still down.

To fix this I changed the querydelay setting in LGSM from 5 mins to 15 mins and it solved the problem.

Not sure how this could be related. My server flat out crashes out of nowhere after starting up and being up for hours.

More than enough memory low cpu when it occurs. There are 500,000 entities - not sure if that is the cause.

Everytime it crashes, the last lines in the log are similar to:

mmap(PROT_NONE) failed
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e800007bae

Posted 4 years ago

It must be a different problem with your server, but odd that it's a similar error message. I would try disabling the plugins one at a a time. Sorry I can't help more with that one.

Posted 4 years ago

I never used to get this error before and this March 2021 wipe I have had this error crash my server twice already.

mmap(PROT_NONE) failed Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e800000089

Posted 4 years ago

Last updated 4 years ago

#10

same issue here, on latest version of oxide and server version (linux):
mmap(PROT_NONE) failed
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e800039905

Doesnt not matter of entity or player count, does not matter of server pop. Can crash several times a day or go days with out crashing.

Posted 4 years ago

#11

yes mine started to do this too, right after the trains update, it running on linux as well. it can crash with 1 player, or 30, usually it crashes late at night like 5 am, there is definately something wrong here. even using all the old plugins from before, I get these crashes randomly now.

Posted 4 years ago

#12

Still getting this crash. Started in November and now is happening again. Surely we can get a fix for this somehow??

Caught fatal signal - signo:11 code:1 errno:0 addr:0x6c Obtained 16 stack frames. #0 0x007f81231a9630 in _L_unlock_13 #1 0x007f8124d33551 in physx::PxsNphaseImplementationContext::unregisterContactManager(physx::PxsContactManager*) #2 0x007f8124cd5655 in physx::Sc::Scene::finishBroadPhaseStage2(unsigned int) #3 0x007f8124cd57e8 in physx::Sc::Scene::updateCCDSinglePassStage3(physx::PxBaseTask*) #4 0x007f8124ca5d24 in physx::Cm::Task::run() #5 0x007f8124258d4a in PhysxJobFunc(physx::PxBaseTask*) #6 0x007f8124175949 in JobQueue::Exec(JobInfo*, long long, int) #7 0x007f8124175ad4 in JobQueue::ExecuteOneJob() #8 0x007f8124280890 in CompletionTask::ProcessAllPhysicsTasks() #9 0x007f8124254627 in PhysicsManager::Simulate(int, float) #10 0x007f81241c185e in ExecutePlayerLoop(NativePlayerLoopSystem*) #11 0x007f81241c18a1 in ExecutePlayerLoop(NativePlayerLoopSystem*) #12 0x007f81241c1dda in PlayerLoop() #13 0x007f812432b6de in PlayerMain(int, char**) #14 0x007f8122dee555 in __libc_start_main #15 0x0055beb42af699 in _start

3 weeks after

Posted 4 years ago

#13

This is happening again. Only on my servers that have more than 250,000 entities and only after I perform a scheduled restart.

mmap(PROT_NONE) failed
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e800005021
Segmentation fault (core dumped)

Posted 4 years ago

#14

The only true way to figure this out is if we all start posting our installed plug-ins. It must be a plug-in causing these crashes - just due to the fact there are like 10,000 servers and only 5 of us. 😀

I have not had a crash since my last post.

Posted 3 years ago

#15

@RustNRelax - I'm getting a very similar error - mine is:

mmap(PROT_NONE) failed
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e9002d49c1
Obtained 18 stack frames.
#0 0x007f29bac133c0 in (Unknown)
#1 0x007f29baa5218b in (Unknown)
#2 0x007f29baa31859 in (Unknown)
#3 0x007f29ba4b1cbe in (Unknown)
#4 0x007f29ba4b1d2b in (Unknown)
#5 0x007f29ba4b4c2b in (Unknown)
#6 0x007f29ba4b4e94 in (Unknown)
#7 0x007f29ba4b591d in (Unknown)
#8 0x007f29bb7942a1 in (Unknown)
#9 0x007f29bb62be32 in (Unknown)
#10 0x007f29bb62bfc6 in (Unknown)
#11 0x007f29bb68c551 in (Unknown)
#12 0x007f29bb68685e in (Unknown)
#13 0x007f29bb6868a1 in (Unknown)
#14 0x007f29bb686dda in (Unknown)
#15 0x007f29bb7f06de in (Unknown)
#16 0x007f29baa330b3 in (Unknown)
#17 0x005650328a3699 in (Unknown)
Aborted (core dumped)

What plugins are you running? I've got:

01 "BotSpawn" (2.1.6) by Steenamaroo (35.54s) - BotSpawn.cs
02 "ChestStacks" (1.3.4) by MON@H (1.97s) - ChestStacks.cs
03 "Gathering Manager" (2.2.75) by Mughisi (0.25s) - GatherManager.cs
04 "Hammer Time" (1.0.20) by Shady (0.18s) - HammerTime.cs
05 "Kits" (4.0.10) by k1lly0u (0.08s) - Kits.cs
06 "Night Lantern" (2.0.98) by k1lly0u (1.95s) - NightLantern.cs
07 "Sign Artist" (1.2.6) by Whispers88 (0.01s) - SignArtist.cs

08 "Extension | Bot Spawn Events" (0.1.0) by Reb_ (0.01s) - BotSpawnEvents.cs
09 "Discord Alert | Map Wipe" (0.1.0) by Reb_ (0.00s) - DiscordAlertMapWipe.cs
10 "Discord Alert | Player Banned" (0.1.0) by Rebajas (0.00s) - DiscordAlertPlayerBanned.cs
11 "Discord Alert | Purge warning" (0.1.0) by Reb_ (0.00s) - DiscordAlertPurgeWarning.cs
12 "Heli Queue" (1.2.0) by Reb_ (0.23s) - HeliQueue.cs
13 "Helper | Purge Status" (0.1.0) by Reb_ (0.00s) - HelperPurgeStatus.cs
14 "Logging Data | Log Chat" (0.1.0) by Reb_ (0.01s) - LoggingDataLogChat.cs
15 "Logging Data | Log Event" (0.1.2) by Reb_ (0.27s) - LoggingDataLogEvent.cs
16 "Logging Data | Log Reset" (0.1.2) by Reb_ (0.00s) - LoggingDataLogReset.cs
17 "FNF | Log | Explosives Usage" (2017.7.27) by Rebajas (0.00s) - StandaloneAnnounceExplosivesUsage.cs
18 "Standalone | Bow Master" (0.1.0) by Reb_ (0.01s) - StandaloneBowMaster.cs
19 "Standalone | Bradley Broadcast" (0.1.0) by Reb_ (0.00s) - StandaloneBradleyBroadcast.cs
20 "Standalone | Cargo Broadcast" (0.1.0) by Reb_ (0.06s) - StandaloneCargoBroadcast.cs
21 "Standalone | Chat | Call Admin" (0.1.0) by Reb_ (0.00s) - StandaloneChatCallAdmin.cs
22 "Standalone | Chat | Doing" (0.1.0) by Reb_ (0.00s) - StandaloneChatDoing.cs
23 "Standalone | Chat | Where" (0.1.0) by Reb_ (0.00s) - StandaloneChatWhere.cs
24 "Standalone | Graceful Restart" (0.1.0) by Reb_ (0.00s) - StandaloneGracefulRestart.cs
25 "Standalone | Heli Broadcast" (0.1.0) by Reb_ (0.07s) - StandaloneHeliBroadcast.cs
26 "Standalone | Manage Electricity Output" (0.1.0) by Reb_ (0.62s) - StandaloneManageElectricityOutput.cs
27 "Standalone | Manage Minicopter Decay" (0.1.0) by Reb_ (0.66s) - StandaloneManageMinicopterDecay.cs
28 "Standalone | Periodic Messages" (0.1.0) by Reb_ (0.00s) - StandalonePeriodicMessages.cs
29 "Standalone | Remove Looted Timed Crate" (0.1.0) by Reb_ (0.01s) - StandaloneRemoveLootedTimedCrate.cs
30 "Standalone | Rewards" (0.1.0) by Reb_ (0.01s) - StandaloneRewards.cs
31 "Standalone | Set Time" (0.1.0) by Reb_ (0.00s) - StandaloneSetTime.cs
32 "Web Site Data | Player Count" (0.1.0) by Reb_ (0.01s) - WebSiteDataPlayerCount.cs
33 "Web Site Data | Server Performance" (0.1.0) by Reb_ (0.63s) - WebSiteDataServerPerformance.cs