Hi Nivex.
I've been experiencing the same lag that has been mentioned in a couple other threads lately. I've seen the proposed solutions of rebuilding configs from scratch which have worked for some and not others.
So to recap, every minute or so we see a multiple second drop out in network throughput and lag for all users. It doesn't matter how many users are connected. We use maintained events to keep 10 bases spawned at all times. Spawning 10+ bases manually does not cause any lag. This only happens for me with Maintained Events. I haven't any other methods so the issue may or may not exist with those as well.
I tried the rebuilding of configs from scratch and was not able to fix my issue. Our players have built a lot of bases near or on the grid. We also have a large protection bubble for bases because players were attacking from outside the bubble to be safe which is cheating in my mind. Iw ish the plugin ignored any damage from outside the bubble but that's not the reason for this post.
I believe I've traced the issue to the GetEventPosition procedure being called from SpawnRandomBase. I've added a bunch of debugging to the plugin to find out that it seems to try thousands of times to find a suitable location to spawn a base then give up. Likely because of our players bases encroaching the grid and our large protection bubbles (150) and distance between raids (300). This takes up to almost 12 seconds to fail. During the running of this procedure the server stalls. Because we don't reach the intended 10 bases the issue continues until either 10 bases spawn successfully or I shutdown the plugin.
This has nothing to do with the number of plugins or memory or server power as I've disabled all but copypaste and RadiableBases to test this.
I propose that if the plugin finds a location that is not suitable that it removes it from the raidable spawns list (RSL) and the number of tries be drastically reduced. The RSL should only contain valid spawn locations. The checks should just be a backup IMHO.
I hope this helps. In the meantime I'm going to reduce the number of bases that spawn and the protection bubble radius as this should help.
Thanks for the hard work on the development.
-Fire
Found potential cause of lag
aye certainly. i've always tried to improve this method as it is very hungry on performance, but it most certainly depends on server power. some machines can handle it without issue while others can't. either way, it needs addressed as performance is always the number 1 priority for such a hungry plugin that uses more hooks than almost every other plugin lol
the plugin does ignore damage from outside of the bubble if you tell it to, Block Damage Outside Of The Dome To Players Inside and Block Damage Outside Of The Dome To Bases Inside are both in each profile. this is why you request features sir :P save yourself the hassle of having to find some sloppy workaround that you don't like
it tries up to 2000 times for automated (maintained and scheduled)
for buyable events it will try up to 1000 times and iterate again in 50 meter increments, which only worsens your problem, but the point is 2000 for automated at most. this used to be 10000. i will reduce it to 1000 after fixing the below issues
positions are indeed already removed at var rsl = rs.GetRandom(); but it should remove everything in a 50 meter radius as well. that would help
RSL contains valid positions which are validated from IsValidLocation when the grid generates. looking at GetEventPosition you will see why there is such a variance on when they can or cannot be used. it can certainly be improved though. such as moving submerged checks into IsValidLocation rather than checking them each time. the only issue there is if someone messes with the sea levels. i would have to use a hook for this to valid them again
i will have it permanently remove positions that:
- fail terrain check since that's static, it will just require that i rewrite how its checked in IsValidLocation. i will also have to use a global terrain elevation setting rather than per profile as it is now. i don't really see a need for it to be otherwise
- if a TC is suddenly present at the location. this will cause issues with the position not being reusable after a TC is destroyed so I will have to use OnEntityDeath(BuildingPrivilege, HitInfo) to re-add these positions
I believe IsAreaSafe is optimized as best as it can be, as it only checks layers if they're passed as an argument. so World and Water are not passed in GetEventPosition so they aren't checked. this leaves Construction, Deployable and Player_Server where more variance comes into play. I could permanently remove positions that fail based on Construction and Deployable. this means the raid bases would never spawn at an area that contained FOB's for raid bases even if they were removed. so either I'd have to check for their destruction in OnEntityDeath() or require you to reload the plugin to refresh the locations. that would eat up positions on the grid over time otherwise
I think the best option for the time being is to use custom spawn points and put that filename under Spawn File Name in Maintained Events so that events only spawn in these locations. this means the grid will not be used when checking for a location for Maintained Events, and therefore no lag
Merged post
also to note using stopwatches would be a great way to find out what specific is the heaviest on performance in GetEventPosition, i should probably use that find it myself :P
Hi Nivex. Yeah I was trying to profile the procedure but was having issues getting stopwatches working for some reason. I don't use them often so it was probably me. I'm used to using a performance profiler.
I tried the block damage variables and they weren't working. They've changed so I'll try them again and see. Thanks for the reminder.
As I said I tried this on a dev machine with all resources dedicated to the two plugins, so I don't think my exact problem goes away with a "more server power". There are a number of variables that could cause this situation to be better or worse for example. Maximum Elevation, Protection Radius, Player Cupboard Detection Radius to name a few. At some point a combination of these being set to high will render any server laggy.
It is just exhausting the automated search of the grid. Most of the time when it isn't lagging it find a suitable position almost immidiately. It's when it "spins it wheels" that it lags. A spawn file is a good idea, but I like automation :-)
I'll have to dig into it more. If I can help pinpoint the sticking points I'll forward them along. Hopefully I can help further.
-Fire
Merged post
Hey Nivex,
Had a thought. The grid generation procedures don't seem to run in parralel with anything and therefore don't hang the server. What about modifying that to run on a schedule and update the RSL based on the various checks? Would require less change to your code and might reduce the chances this edge condition appears.
Just an idea.
-Fire
Merged post
This just came in from my debugging.
[Raidable Bases] GetSpawns took 0.003 ms
[Raidable Bases] GetEventPosition.Checks took 8194.164 ms
[Raidable Bases] CountTooClose = 710
[Raidable Bases] CountNotSafe = 335
[Raidable Bases] CountTerrain = 86
[Raidable Bases] CountBuildPriv = 1226
[Raidable Bases] CountSubmerged = 2
[Raidable Bases] CountSubmerged2 = 0
[Raidable Bases] GetEventPosition took 8196.896 ms