uMod - Nighlty Server Crash - System.IO.IOException: Too many open files - Rust - Community

Nighlty Server Crash - System.IO.IOException: Too many open files

Posted 4 years ago

I am getting a pretty serious fault on my server. Every morning between 4-5AM (at different times) my server crashes. It is still playable but the mods break and the map wont save. The log is filled with errors, starting with:

System.IO.IOException: Too many open files

at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean isAsync, System.Boolean anonymous) [0x00000] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access) [0x00000] in <eae584ce26bc40229c1b1aa476bfa589>:0

at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess)

at Oxide.Core.Logging.RotatingFileLogger.BeginBatchProcess () [0x0000c] in <0fbce81d88d64454b3e7abb24df7026b>:0

at Oxide.Core.Logging.ThreadedLogger.Worker () [0x0002f] in <0fbce81d88d64454b3e7abb24df7026b>:0

at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00014] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <eae584ce26bc40229c1b1aa476bfa589>:0 (IOException: Too many open files)

at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean isAsync, System.Boolean anonymous) [0x00000] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access) [0x00000] in <eae584ce26bc40229c1b1aa476bfa589>:0

at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess)

at Oxide.Core.Logging.RotatingFileLogger.BeginBatchProcess () [0x0000c] in <0fbce81d88d64454b3e7abb24df7026b>:0

at Oxide.Core.Logging.ThreadedLogger.Worker () [0x0002f] in <0fbce81d88d64454b3e7abb24df7026b>:0

at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00014] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <eae584ce26bc40229c1b1aa476bfa589>:0

at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <eae584ce26bc40229c1b1aa476bfa589>:0

And then the mods will have errors eventually down the line.
I did try toi edit '/etc/security/limits.conf' and install the two lines:
* soft nofile 10000
* hard nofile 10000
But that didn't resolve the issue.

Any ideas as to where to look to get this resolved?

Posted 4 years ago

Which distro does your server run?
Also what is the output of the ulimit -n command?

Posted 4 years ago

The machine is Ubuntu 20.4.3

server@www:~$ ulimit -n
4096

There was no crash last night after making the change yesterday.

Posted 4 years ago

EdmontonsRust
The machine is Ubuntu 20.4.3

server@www:~$ ulimit -n
4096

There was no crash last night after making the change yesterday.

As you can see, your limits are not saved. Try adding this line to the /etc/pam.d/login file:

session  required  pam_limits.so

Posted 4 years ago

For Ubuntu:

echo "session required pam_limits.so" >> /etc/pam.d/common-session
echo "* hard nofile 999999" >> /etc/security/limits.conf
echo "* soft nofile 999999" >> /etc/security/limits.conf
echo "fs.file-max = 999999" >> /etc/sysctl.d/10-file-max.conf
echo "fs.file-max = 999999" >> /etc/sysctl.conf
sysctl -w fs.file-max=999999
sysctl -p

That should definitely set them :D