I want to limit the amount of RAM the server is allowed to use. Is there a parameter or configuration option to force Rust to automatically restart the game server once it reaches a certain threshold set by me? I have an octa-core server with 32 GB of RAM but I don't want the server to consume all of my memory or CPU since I'm trying to run other game server as well.

I'm not familiar with linux in that case, but on windows you have two options - simplest and smartest one - use Process Lasso (or analog) for CPU limiting and use Process Governor (.NET diagnostics toolkit) for RAM limiting. Second way is to launch virtual box that allows limiting resource usage (VMWARE for example), install OS into it (Win server core for example) and launch server inside. As I said - it's pretty ridiculous, so 1st method would be better.

Unfortunately I'm using Ubuntu 18.04, command-line based. No graphical interface to use. So unfortunately the methods for Windows aren't options for me.

Have you tried Google?

Once you've limited the RAM then all you need to do is run a monitoring app to restart the server if it stops. If you're using lgsm you can set a cron job for '~/rustserver monitor'

Well that's the issue. I can't figure out how to limit the amount of RAM that the server uses. I'm trying to figure that out so that I can use the LinuxGSM Rust server monitor.

What I'm saying is that you're probably looking in the wrong place - Oxide users are not going to know much about this, and you're unlikely to be able to limit your RAM via the rustserver settings, or within LinuxGSM. You'll most likely need to limit the RAM of this app from "outside" via Ubuntu configs. My Google link gives you a bunch of suggestions. I've skimmed it and 'cgroups' and 'ulimit' both look hopeful.