I have it installed and will keep an eye on it.

Merged post

Found this in the log.

06/19 22:47:09 | *.*.*.*:64895/76561############/Cora disconnecting: closing
06/19 22:47:45 | [Backpacks] Cleaning up backpack for disconnected player 76561############
06/19 22:47:45 | [Economics] Saving balances for players...
06/19 22:47:45 | [Backpacks] Detected unexpected death of backpack container for player: 76561############
06/19 22:47:45 | [Backpacks] Prevented unexpected destruction of backpack container for player: 76561############
06/19 22:47:45 | [Backpacks] Cleaning up backpack for disconnected player 76561############
06/19 22:47:46 | Saved 238,282 ents, cache(0.58), write(0.11), disk(0.04).
06/19 22:47:46 | Saving complete

Thanks for sharing that log. "Detected unexpected death of backpack container" indicates that the backpack container was killed via the combat system, meaning something called entity.Die() not just entity.Kill(). The Backpacks plugin will never do that, so we can rule out whether the Backpacks plugin is the one causing this.

The only other possibilities are vanilla logic and other plugins. Given the timing of people reporting this only after the June force wipe, I was initially inclined to suspect a Rust change. However, when searching Rust code for ways that containers can be death'd, I didn't find anything new or that I think could explain this issue. Also considering that the number of servers experiencing this issue is low, I'm inclined to think the cause is another plugin.

Thanks. I am now wondering which other plugins I have could be causing this. I will look into what I am running and see if I can figure it out.  I wonder now if it might be something like BetterNPC or Raidable bases? They do their own type of clean up.

FYI, 
The strange thing is, It only happenes to admin and moderators on my server.
I'm using the following plugins.

01 "AdminMenu" (2.0.18) by k1lly0u (0.90s) - AdminMenu.cs
02 "ArmoredTrain" (1.3.5) by Adem (1.60s) - ArmoredTrain.cs
03 "Backpacks" (3.11.8) by WhiteThunder (0.18s) - Backpacks.cs
04 "Better Chat" (5.2.12) by LaserHydra (0.00s) - BetterChat.cs
05 "BetterNpc" (1.2.2) by KpucTaJl (0.82s) - BetterNpc.cs
06 "BoxLooters" (0.3.6) by 4seti / k1lly0u (0.88s) - BoxLooters.cs
07 "Building Skins" (1.1.1) by Marat (0.07s) - BuildingSkins.cs
08 "Building Workbench" (1.3.2) by MJSU (0.47s) - BuildingWorkbench.cs
09 "ChaosExtensionDownloader" (1.0.7) by k1lly0u (0.00s) - ChaosExtensionDownloader.cs
10 "Clear Night" (2.3.6) by Clearshot (0.17s) - ClearNight.cs
11 "Copy Paste" (4.1.37) by misticos (0.03s) - CopyPaste.cs
12 "Craft Car Chassis" (1.2.2) by WhiteThunder (0.00s) - CraftChassis.cs
13 "Cupboard Limiter" (1.7.4) by Spiikesan (1.49s) - CupboardLimiter.cs
14 "DiscordMessages" (2.1.8) by Slut (0.02s) - DiscordMessages.cs
15 "Discord Messages Chat" (1.0.4) by Slut (0.00s) - DiscordMessagesChat.cs
16 "Discord Report" (1.2.0) by misticos (0.03s) - DiscordReport.cs
17 "Discord Status" (4.0.1) by Gonzi (0.17s) - DiscordStatus.cs
18 "Economics" (3.9.1) by Wulf (0.05s) - Economics.cs
19 "Entity Owner" (3.4.1) by Calytic (0.02s) - EntityOwner.cs
20 "Eternal Plants" (1.0.1) by 0x89A (3.43s) - EternalPlants.cs
21 "Extended Recycler" (1.2.2) by beee/The Friendly Chap (0.32s) - ExtendedRecycler.cs
22 "Freeze" (3.0.3) by Wulf (1.92s) - Freeze.cs
23 "Fuel Gauge" (0.6.1) by Oryx (0.16s) - FuelGauge.cs
24 "Gather Manager" (2.2.78) by Mughisi (0.33s) - GatherManager.cs
25 "GUIAnnouncements" (2.0.5) by JoeSheep (1.47s) - GUIAnnouncements.cs
26 "Image Library" (2.0.61) by Absolut & K1lly0u (0.03s) - ImageLibrary.cs
27 "Inventory Viewer" (4.0.6) by Whispers88 (0.01s) - InventoryViewer.cs
28 "ItemsLostDurabilityonNPCHit" (1.4.0) by kaucsenta (0.30s) - ItemsLostDurabilityonNPCHit.cs
29 "Kits" (4.4.0) by k1lly0u (0.08s) - Kits.cs
30 "Magic Loot" (1.0.5) by collect_vood (3.28s) - MagicLoot.cs
31 "Monument Addons" (0.14.0) by WhiteThunder (0.11s) - MonumentAddons.cs
32 "Monument Finder" (3.1.1) by WhiteThunder (0.02s) - MonumentFinder.cs
33 "NextGen PVE" (1.5.0) by RFC1920 (29.02s) - NextGenPVE.cs
34 "No Give Notices" (0.3.0) by Wulf (0.00s) - NoGiveNotices.cs
35 "NPC Drop Gun" (2.0.8) by 2CHEVSKII (0.02s) - NPCDropGun.cs
36 "NpcSpawn" (2.5.5) by KpucTaJl (3.81s) - NpcSpawn.cs
37 "NTeleportation" (1.7.5) by nivex (0.49s) - NTeleportation.cs
38 "Personal Heli" (1.1.8) by Egor Blagov (1.46s) - PersonalHeli.cs
39 "PlayerList" (0.3.2) by Wulf/lukespragg (0.02s) - PlayerList.cs
40 "PlayerSkins" (3.0.3) by k1lly0u (0.25s) - PlayerSkins.cs
41 "PocketZipline" (1.0.9) by Adem (0.15s) - PocketZipline.cs
42 "Queue Holder" (1.0.1) by Tryhard (0.00s) - QueueHolder.cs
43 "Raidable Bases" (2.7.2) by nivex (3.75s) - RaidableBases.cs
44 "Recycler Speed" (2.0.2) by Ryz0r/yetzt (0.00s) - RecyclerSpeed.cs
45 "Remover Tool" (4.3.39) by Reneb/Fuji/Arainrr/Tryhard (0.02s) - RemoverTool.cs
46 "RobSpawn" (1.1.1) by Steenamaroo For RobJ (0.32s) - RobSpawn.cs
47 "SAMSiteAuth" (2.4.3) by haggbart (0.00s) - SAMSiteAuth.cs
48 "Server Armour" (1.22.3) by Pho3niX90 (0.24s) - ServerArmour.cs
49 "Server Info" (0.5.8) by FastBurst (0.04s) - ServerInfo.cs
50 "Shop" (1.1.71) by David (0.10s) - Shop.cs
51 "Sign Artist" (1.4.1) by Whispers88 (0.01s) - SignArtist.cs
52 "Simple Kill Feed" (2.1.3) by Krungh Crow (0.02s) - SimpleKillFeed.cs
53 "Simple Splitter" (2.2.5) by Shady14u (2.00s) - SimpleSplitter.cs
54 "Sort Button" (2.0.2) by MON@H (0.01s) - SortButton.cs
55 "Sputnik" (1.2.0) by Adem (0.84s) - Sputnik.cs
56 "Stack Modifier" (2.0.7) by Khan (120.44s) - StackModifier.cs
57 "UpdateChecker" (3.2.2) by tofurahie#4144 (0.01s) - UpdateChecker.cs
58 "Vanish" (1.8.0) by Whispers88 (0.02s) - Vanish.cs
59 "Vehicle Decay Protection" (2.3.0) by WhiteThunder (0.98s) - VehicleDecayProtection.cs
60 "Vending In Stock" (1.1.0) by AVOcoder / rostov114 (0.00s) - VendingInStock.cs
61 "WagonPlus" (1.1.2) by ninco90 (0.34s) - WagonPlus.cs
62 "WaterBases" (1.0.11) by Nikedemos (0.84s) - WaterBases.cs
63 "Zone Manager" (3.1.2) by k1lly0u (2.86s) - ZoneManager.cs
64 "Hotel" (2.0.26) by Shady14u (0.18s) - Hotel.cs

Another debug version, this will log the damage type and amount when the backpack container dies, if applicable, or else it will just log "No HitInfo". Knowing this info may help determine what is causing it to die.
https://raw.githubusercontent.com/WheteThunger/Backpacks/develop/Backpacks.cs

Thank you. I have it installed.

My backpack was gone and I got these log today using V3.11.9.
I'm not sure but I think I was using teleport comand at this time.
Now, I suspect teleportation (instant teleportation like console comand or MapNoteTeleport) but NTeleportation is fine I guess.

15:45 [Info] [Server Armour] Player 76561############ is at steam level 9
15:45 [Info] [Server Armour] IP/CACHE| ID:76561############ ADD:*.*.*.* RATING:0 AGE:
15:48 [Info] [Server Armour] Player 76561############ is at steam level 3
15:48 [Info] [Server Armour] IP/CACHE| ID:76561############ ADD:*.*.*.* RATING:0 AGE: 2023-06-25T06:48:07.089Z
15:48 [Warning] [BoxLooters] Saved Boxlooters data
15:48 [Info] [Economics] Saving balances for players...
15:48 [Warning] [Backpacks] Cleaning up backpack for disconnected player 76561############
15:48 [Warning] [Backpacks] Detected unexpected death of backpack container for player: 76561############. No HitInfo
15:48 [Warning] [Backpacks] Prevented unexpected destruction of backpack container for player: 76561############
15:52 [Info] [Server Armour] Player 76561############ is at steam level 3
15:52 [Info] [Server Armour] IP/CACHE| ID:76561############ ADD:*.*.*.* RATING:0 AGE: 2023-06-25T06:52:18.192Z

Not sure if this will help Identify the culprit. But I had a player's backpack items disappear. No hit info...

07/10 17:10:19 | [Traffic Drivers] Traffic car was sucessfully spawned!
07/10 17:10:44 | [Economics] Saving balances for players...
07/10 17:10:45 | [Backpacks] Cleaning up backpack for disconnected player 76561############
07/10 17:10:45 | [Backpacks] Detected unexpected death of backpack container for player: 76561############. No HitInfo
07/10 17:10:45 | [Backpacks] Prevented unexpected destruction of backpack container for player: 76561############
07/10 17:10:46 | Saved 170,546 ents, cache(1.11), write(0.06), disk(0.03).
07/10 17:10:46 | Saving complete
07/10 17:11:14 | [GridPower] Saving data...

In both of the above cases, it showed "No HitInfo", meaning whatever killed the backpack storage used code like ".Die()" or less likely ".Die(null)". You can search the code of your plugins for such snippets (without the quotes) to see if there are any matches. It's remotely possible that a plugin is destroying the backpack containers without using those functions directly, but let's see what you find first.

Can you also confirm, were the three steam ids in that output the same steam id or different ones? The last two should be the same, but expecting the first one to be different. Either way, the timing seems unlikely to be a coincidence.

Merged post

I have a new theory. From reading the above logs, I get the sense that not only are there two distinct steam IDs, the routine destruction of one player's backpack container entity directly caused the other player's backpack container entity to be destroyed as well. Normally I would estimate that phenomonen to be caused by GroundWatch, a pattern that Rust follows where destruction of one entity can notify nearby entities to check if they still have a valid floor/ground to stand on, and to destroy themselves if they don't. GroundWatch makes sense because it issues entity death via .Die() and because all backpack container entities are in the same world position.

The issue with the GroundWatch theory is that the Backpacks plugin already destroys the GroundWatch and DestroyOnGroundMissing objects that Rust natively attaches to the backpack container entity (which is a coffin by the way). What may be happening is that another plugin is recreating those objects and attaching them to the coffin. One plugin in jcverrett's list that has good reason to modify GroundWatch behavior is Chest Stacks, which had an update on May 1st which specifically called out changes related to coffins, weeks before this thread was created. Since the issue only happens to moderators on Kinya's server, I'm guessing maybe the Chest Stacks permissions are only given to moderators.

I've reached out to Supreme for help troubleshooting, but in the mean time, if you want to help troubleshoot, you can try reproducing the issue by making sure you have the Chest Stacks permission, then opening your backpack, then having another player open their backpack, then have that player disconnect, then you trigger a server.save (to cause that player's backpack to be cleaned up because they are not connected). My guess is that if you have the Chest Stacks permission, the GroundWatch object attached to your backpack container by that plugin will detect the other player's backpack entity being destroyed and will cause itself to be destroyed as a result, which you'll see immediately in your logs. If you can reproduce the issue with this process (or maybe with another process you figure out since I'm trobuleshooting basically blind here), then you can try revoking the Chest Stacks permission or removing the Chest Stacks plugins (and reloading Backpacks in case of any residual changes that Chest Stacks might have left over on them) and follow the same process to see if the issue can still be reproduced.

The log I posted from 2 days ago was all the same user.

I was trying to search with Notepad++ using the CTRL F function and did not find either AddComponent<GroundWatch> or AddComponent<DestroyOnGroundMissing>. 

Search "AddComponent<DestroyOnGroundMissing>" (0 hits in 0 files of 124 searched)
Search "AddComponent<GroundWatch>" (0 hits in 0 files of 124 searched)

Searching for .Die() it found

Search ".Die()" (8 hits in 3 files of 124 searched)
C:\rustserver\oxide\plugins\ZoneManager.cs (4 hits)
Line 400: player.Die();
Line 920: else player.Die();
Line 2025: player.Die();
Line 2044: player.Die();
C:\rustserver\oxide\plugins\PlaneCrash.cs (3 hits)
Line 708: crashPlane.Die();
Line 719: (baseEntity as BasePlayer).Die();
Line 730: baseCombatEntity.Die();
C:\rustserver\oxide\plugins\SignMap.cs (1 hit)
Line 232: sign.Die();
Search ".Die(null)" (0 hits in 0 files of 124 searched)
Search ".Die()" (8 hits in 3 files of 124 searched)

Serching .Die(null)

Search ".Die(null)" (0 hits in 0 files of 124 searched)

Calling 'OnMaxStackable' on 'StackItemStorage v1.0.8' took 139ms [GARBAGE COLLECT]
*.*.*.*:39311/77861194526454258/PlayersName disconnecting: closing

This happened on my server, maybe check this?

Flechen

Calling 'OnMaxStackable' on 'StackItemStorage v1.0.8' took 139ms [GARBAGE COLLECT]
*.*.*.*:39311/77861194526454258/PlayersName disconnecting: closing

This happened on my server, maybe check this?

I don't think that's related.

jcverrett

The log I posted from 2 days ago was all the same user.

I was trying to search with Notepad++ using the CTRL F function and did not find either AddComponent<GroundWatch> or AddComponent<DestroyOnGroundMissing>. 

Search "AddComponent<DestroyOnGroundMissing>" (0 hits in 0 files of 124 searched)
Search "AddComponent<GroundWatch>" (0 hits in 0 files of 124 searched)

Searching for .Die() it found

Search ".Die()" (8 hits in 3 files of 124 searched)
C:\rustserver\oxide\plugins\ZoneManager.cs (4 hits)
Line 400: player.Die();
Line 920: else player.Die();
Line 2025: player.Die();
Line 2044: player.Die();
C:\rustserver\oxide\plugins\PlaneCrash.cs (3 hits)
Line 708: crashPlane.Die();
Line 719: (baseEntity as BasePlayer).Die();
Line 730: baseCombatEntity.Die();
C:\rustserver\oxide\plugins\SignMap.cs (1 hit)
Line 232: sign.Die();
Search ".Die(null)" (0 hits in 0 files of 124 searched)
Search ".Die()" (8 hits in 3 files of 124 searched)

Serching .Die(null)

Search ".Die(null)" (0 hits in 0 files of 124 searched)

Thanks for the info. I'm honestly pretty confused at this point. What seems to be happening is that during backpack cleanup, when destroying the items inside the backpack, one of those items must be triggering something that inadvertently leads to the parent container dying. That doesn't really make sense to me.

I've looked again at the Rust code and there are unfortunately not many places that could trigger an entity to die without providing HitInfo. Besides GroundWatch, all of the other cases are specific to other types of entities and won't apply to storage containers, so GroundWatch seems like the only possibility. I pushed another debug version which will now detect the OnEntityGroundMissing event for backpack storage containers in order to log the event and also block it. If you don't see events related to this in the logs next time a backpack, then we can rule out whether it's a GroundWatch (though if that's the case, will have to go back to the drawing board).

https://raw.githubusercontent.com/WheteThunger/Backpacks/develop/Backpacks.cs

I have downloaded it and will install it.

Had a hit with the new one.

08/05 19:22:44 | [Backpacks] Cleaning up backpack for disconnected player 76561############
08/05 19:22:44 | [Economics] Saving balances for players...
08/05 19:22:44 | [Backpacks] Detected ground missing for backpack container for player: 76561############. Blocking event to prevent item loss.
Found DestroyOnGroundMissing component.
Found GroundWatch component. Position: (1.2, 0.3, 0.0). Layers: 10551296. Radius: 0.1. Whitelist: N/A.
08/05 19:22:45 | Saved 185,476 ents, cache(1.01), write(0.07), disk(0.03).
08/05 19:22:45 | Saving complete

jcverrett

Had a hit with the new one.

08/05 19:22:44 | [Backpacks] Cleaning up backpack for disconnected player 76561############
08/05 19:22:44 | [Economics] Saving balances for players...
08/05 19:22:44 | [Backpacks] Detected ground missing for backpack container for player: 76561############. Blocking event to prevent item loss.
Found DestroyOnGroundMissing component.
Found GroundWatch component. Position: (1.2, 0.3, 0.0). Layers: 10551296. Radius: 0.1. Whitelist: N/A.
08/05 19:22:45 | Saved 185,476 ents, cache(1.01), write(0.07), disk(0.03).
08/05 19:22:45 | Saving complete

Thanks, that confirms the issue is GroundWatch. From the info in the logs, it has the same properties as the GroundWatch component normally attached to coffins by vanilla. Issue is, the plugin is destroying the GroundWatch component, so not sure how it's coming back. I assume it has to be another plugin, but as your previous code search revealed no plugins explicitly adding GroundWatch, it must be added some very indirect way. I've pushed another debug version which should add a lot more checks for GroundWatch to determine whether that component is getting added before spawn, after spawn, or some time later. After running this version for while, you'll want to check the logs for any mention of "GroundWatch".