Hey there, for multiple wipes in a row now I'm getting major graphical lag when players go towards the center of the map. Has anyone else run into this, and does anyone have any idea how to fix this?

My FPS goes from 150 to 10 for a few seconds.

Map size: 3k Barren

Modded Battlefield server

Serve FPS rarely goes below 220, no spam in logs, I do notice the entity count climbs non-stop after wipe even without many bases being built.



I ran a performance report to get a list of entities and I see an insane amount of weapon attachments. Any idea what could cause that?

I just wiped yesterday morning..... By the end of wipe it has been getting up to like 200k entities with hardly any bases.

Just from yesterday the lasersight.entity went up 3k entities.

I've tried multiple entity cleanup plugins but none of them remove the excess entities.

I tried the object remover plugin as well but it doesn't seem to find these weapon mod entities anywhere near me so I don't know if I should delete them or not...



I have not used the raidable bases plugin this wipe as it's causing server freezes either, I know that one spams code locks.

Thanks

I don't see how I can edit my post, but here are my plugins: https://pastebin.com/x6xaJJE9

1. you are UNDER the map. so your view range is extended.
2. you are at a place where objects go to DIE!!!!!!!!  can you hear all the animals?
3. you are in a place you should not be.
4. you told us NOTHING about the server....

How did you produce the "list"?

I'd suspect the NPC's interacting with a plugin. maybe the stacker & loot plugins.
and yes.... you will hit lag at 0,0,0...
as a say it is where all the server shit goes to die...... ESP.. the shit that is not disposed of correctly....

1. The graphical lag/stutter (it's not network lag) happens above ground, I took a picture of 0,0,0 because I knew people would want to see what was there. It happens as you near the center of the map not just at the exact center.

The list of entities is from https://umod.org/plugins/performance-monitor

Server stats: https://gyazo.com/664dd8597adc8ab26d0baa6db692bf4d

Merged post

I went ahead and used the object remover plugin to remove all weapon mod attachments and it fixed 90%+ of the graphical lag towards center of map, but something is causing them to keep coming back, not sure what yet.

Hi,

1. don't agree..... it is exactly this... but for different reasons...
Test things in the environment that causes problems, is always the best way.. since what you are seeing at  0,0,0 may not be what you think it is..

Had exactly this discussion with someone on Discord 2 days ago... and some really strange shit with multiplying objects.
so much so it took 200 kill int on exactly the same location  to make the problem go away... just like you see.
already filed a bug with FP about what i think is going on...

Agreed removing the object will not solve the problem... it comes back......

Try unloading and blocking the stacking plugin, and also any plugin that allows the items to be above the normal quatity.....

also your server spec looks "virtulised", do you not have a spare PC you can load your server image onto for testing ?
if you you will be able to pindown exactly what is going on with a couple of computers that are local to you.

also now i know the name of the plugin you are using to generate your list... i will take a look at the code to see what it is doing

Hi, I will play around with the stacking plugins in the morning when my pop is low.

The server is bare metal, so it's a dedicated server in a facility somewhere not virtualized at all.

The weird thing is I didn't see any of these weapon mods near 0,0,0, maybe they were inside that red can, but once I deleted them all the performance was instantly better. It took my entities from 50 or 55k down to 20k, and the stuttering if loading in things was gone when approaching the middle of the map.

I used this plugin to remove them: https://umod.org/plugins/object-remover

This is the plugin we are using for stack size: https://umod.org/plugins/stack-size-controller but this wouldn't effect actual entities loaded into the map would it? If for example a player has a stack of 1000 flashlights for example in their inventory, there's not actually 1000 3d flashlights in-game right?

I did look through my alpha loot profiles today and took out anything related to weapon mods but that didn't seem to effect anything.

I do have another server (our old one) that I could spin up and do some testing on with an exact copy of the setup.

Thanks

ok, i had a look at the plugin code.
its not really useful, since it counts anything with Uid 0 as "unowned" which is not correct......
since every item the server ever creates starts out as  Uid 0, including every live server entity.

Also it takes a massive amount of time to run and even worse it cannot be unloaded or cancelled during a run

I might have some idea how this crap is happening....  it might be raidable bases and the stacker routine or some such iteraction . 

say for example you set a loot  with 100,000 sulphur in the raidable base.... you can do this.. but it does not go via the stacker since the objects are not instigated that are prestacked, its just a <list> with a <int> count.

now if you have the real world  stack set to say 100, when the raidable base decays the sulphur will spill out, then it falls into the domain of the stacking algorithm becasue it becomes a real world entity to become visible... at which point it becomes liable to be restacked & combined , but to do that it must be realised in the real world then combined back into the stack size. and at THIS point it becomes a REAL entity, only to be destroyed when it is re-stacked.
or it might be user bases that are decaying with massive prestacks...

if you see where i'm going with this, there maybe situations where you are realising a storage container into decay that dumps loot in values beyond the stacking ... and off it goes

actually that would be an interesting test.. setup a dummy usersbase with 100 of an item then set the stack size to 1 & see what happends when the container decays...

Merged post

seems they fixed it the base decays into  unburied stashes that can be overstacked even when it goes back to vanilla...
there is no longer any validation routines for over stacked items...

Hi, I don't have raidable bases enabled right now though, so it can't be that. I'm really not sure what is creating these weapon attachments as they shouldn't be in any of my loot tables, and I'm not using any sort of base creation plugin or anything.

Just from overnight:
"lasersight.entity": {
"Total": 2488,
"Owned": 0,
"Unowned": 2488
},
"smallscope.entity": {
"Total": 1820,
"Owned": 0,
"Unowned": 1820
},
"holosight.entity": {
"Total": 1465,
"Owned": 0,
"Unowned": 1465
},

Merged post

Another note is that I have despawnconfig plugin with those mods set to despawn in 1 minute. It's like the entities are being attached to something in huge amounts, maybe scientists or something, and not being dropped on the ground? Which doesn't make sense because the lag it's causing is in the center of the map, none of this makes sense to me.

Merged post

For anyone coming across this same issue here is my temporary fix until someone can figure out what's actually causing the root issue:

What I am doing each morning (and sometimes nights) is using this plugin:
https://umod.org/plugins/object-remover to run commands in my client (after setting permissions to my admin group).

object lasersight.entity action remove radius 0 inside false

object smallscope.entity action remove radius 0 inside false

object holosight.entity action remove radius 0 inside false

object flashlight.entity action remove radius 0 inside false

object silencer.entity action remove radius 0 inside false

object muzzlebrake.entity action remove radius 0 inside false

object 8xscope.entity action remove radius 0 inside false

object foundation action remove radius 0 inside false

This is run from the client connected from your F1 console.

KEEP IN MIND: this will remove all of those 3d models on any gun any of your players are currently using, so I try to use this when my population is low in the morning. This doesn't seem to remove anything from bandit/outpost turrets.

This took my entities from 55k to 20k last night and resolved 100% of the lag in the center of the map. This morning I woke up and the entities were back up to 25k, ran it again and took me back down.

I have no clue what is causing this bug but it is extremely annoying.... At least I've found a way to "reset" the lag to some extent each day.

Weapon mods are attached near the weapon, so that could account for their location near the map origin.  Perhaps the weapons are being deleted improperly, leaving the attachments behind, with position coords near 0,0,0.

running any NPC base addons? Skining addons?

Maybe weapon mods for weapons of scientists? I mean it's in the thousands (like 10k a day almost of entities if not more).

I'm not currently running any NPC base addons, I was using raidable bases but not for a few wipes now as since the Rust update it crashes my server. Still waiting on an update from the plugin developer.

Maybe modify ObjectRemover.cs to only target items around 0,0,0 so it does not impact other game play items.
and also modify the object scanner to include the Vector3 co-ords when NOT 0,0,0 to see if there is a Vector3 cluster, that might give it away.

Well another thing that just doesn't make any sense is this: https://gyazo.com/40e6b8996b579864582af537430d5cb6

It shows no entities in any range lol, but without range it finds them.

I just cleared weapon mod entities after wiping at 10am CST this morning and I removed 30,000 freaking entities.....

Merged post

If I run those same commands from the server and not my client I do get results I'd expect, so is it possible that the entities only exist server side and they aren't being synced to the clients, but just the sheer number of them are lagging the server near center of map because the server has to iterate through them to see if they should be sent to the client? I don't actually see the weapon mods anywhere on my client in the center of the map.

Already said this......
Client & server are NOT the same , and I also filed a bug & called FP out about syncing shite at 0,0,0 under the map.
crap that dies on the client goes to 0,0,0 to die and
crap that dies on the server goes to 0,0,0 to die but they don't sync in real time until you go to 0,0,0

Even worse, the server is classed as user ID 0 and every dead object is assigned to 0 to die.
Why the hell not  assign the server user 1, so that  dead & living objects could be clearly seperated for processing and never the two would meet,
but now that would be a massive code re-write, it's just lazy ass programming.

you have a leak in a plugin or you platform is leaking, becasue i'm not seeing the same object generation on my servers.

razorfishsl

Already said this......
Client & server are NOT the same , and I also filed a bug & called FP out about syncing shite at 0,0,0 under the map.
crap that dies on the client goes to 0,0,0 to die and
crap that dies on the server goes to 0,0,0 to die but they don't sync in real time until you go to 0,0,0

Even worse, the server is classed as user ID 0 and every dead object is assigned to 0 to die.
Why the hell not  assign the server user 1, so that  dead & living objects could be clearly seperated for processing and never the two would meet,
but now that would be a massive code re-write, it's just lazy ass programming.