Server Crashing - Backpacks to blame???Not An Issue
Full log including stacktrace

Hey, I seem to be having an issue with a server of ours crashing. I enabled the stacktrace log on someones advice and it seems to be getting stuck in a loop when gathering (to my untrained eye that is)? Could a "no fuel required" vehicle plugin be causing this issue? or is it something completely different? Here is the log (note, i removed 17k lines that are repeating):

Stacktrace:
at <unknown> <0xffffffff>
at System.Collections.Generic.List1<T_REF>.set_Capacity (int) [0x00022] in <ff4e3fe86250407285eaebda8f185c35>:0
<...>
at System.Collections.Generic.List1<T_REF>.EnsureCapacity (int) [0x00038] in <ff4e3fe86250407285eaebda8f185c35>:0
at System.Collections.Generic.List1<T_REF>.AddWithResize (T_REF) [0x0000b] in <ff4e3fe86250407285eaebda8f185c35>:0
at System.Collections.Generic.List1<T_REF>.Add (T_REF) [0x00036] in <ff4e3fe86250407285eaebda8f185c35>:0
at System.Collections.Generic.List1<T_REF>.FindAll (System.Predicate1<T_REF>) [0x00034] in <ff4e3fe86250407285eaebda8f185c35>:0
at ItemContainer.FindItemsByItemID (int) [0x0001f] in <e835aa7a159e4026a4c988e02103a9af>:0
at Item.MoveToContainer (ItemContainer,int,bool,bool,BasePlayer,bool) [0x00042] in <e835aa7a159e4026a4c988e02103a9af>:0
at Item.MoveToContainer (ItemContainer,int,bool,bool,BasePlayer,bool) [0x002e1] in <e835aa7a159e4026a4c988e02103a9af>:0
at Item.MoveToContainer (ItemContainer,int,bool,bool,BasePlayer,bool) [0x002e1] in <e835aa7a159e4026a4c988e02103a9af>:0
at Item.MoveToContainer (ItemContainer,int,bool,bool,BasePlayer,bool) [0x002e1] in <e835aa7a159e4026a4c988e02103a9af>:0
17k more of these ^^^^^^
at Oxide.Plugins.Backpacks/ItemContainerAdapter.TryInsertItem (Item,Oxide.Plugins.Backpacks/ItemQuery&,int) [0x00051] in <332770a879f947e2887ee12db782d5b4>:0 at Oxide.Plugins.Backpacks/Backpack.TryGatherItem (Item) [0x0017f] in <332770a879f947e2887ee12db782d5b4>:0 at Oxide.Plugins.Backpacks/InventoryWatcher.OnItemAddedRemoved (Item,bool) [0x000df] in <332770a879f947e2887ee12db782d5b4>:0 at (wrapper delegate-invoke) System.Action2<Item, bool>.invoke_void_T1_T2 (Item,bool) [0x00077] in <ff4e3fe86250407285eaebda8f185c35>:0 at ItemContainer.Insert (Item) [0x0004e] in <e835aa7a159e4026a4c988e02103a9af>:0 at Item.SetParent (ItemContainer) [0x00040] in <e835aa7a159e4026a4c988e02103a9af>:0 at Item.MoveToContainer (ItemContainer,int,bool,bool,BasePlayer,bool) [0x00581] in <e835aa7a159e4026a4c988e02103a9af>:0 at PlayerInventory.GiveItem (Item,ItemMoveModifier,ItemContainer) [0x00056] in <e835aa7a159e4026a4c988e02103a9af>:0 at PlayerInventory.GiveItem (Item,ItemContainer) [0x00004] in <e835aa7a159e4026a4c988e02103a9af>:0 at BasePlayer.GiveItem (Item,BaseEntity/GiveItemReason) [0x0005b] in <e835aa7a159e4026a4c988e02103a9af>:0 at Oxide.Plugins.VehicleLicence/BaseVehicleSettings.RefundVehicleItems (Oxide.Plugins.VehicleLicence/Vehicle,bool,bool) [0x0005b] in <332770a879f947e2887ee12db782d5b4>:0 at Oxide.Plugins.VehicleLicence.RefundVehicleItems (Oxide.Plugins.VehicleLicence/Vehicle,bool,bool) [0x00028] in <332770a879f947e2887ee12db782d5b4>:0 at Oxide.Plugins.VehicleLicence.OnEntityDeathOrKill (BaseCombatEntity,bool) [0x0001f] in <332770a879f947e2887ee12db782d5b4>:0 at Oxide.Plugins.VehicleLicence.OnEntityDeath (BaseCombatEntity,HitInfo) [0x00003] in <332770a879f947e2887ee12db782d5b4>:0 at Oxide.Plugins.VehicleLicence.DirectCallHook (string,object&,object[]) [0x00552] in <332770a879f947e2887ee12db782d5b4>:0 at Oxide.Plugins.CSharpPlugin.InvokeMethod (Oxide.Core.Plugins.HookMethod,object[]) [0x00083] in <795304323ac74a298b8ed190a1dfa739>:0 at Oxide.Core.Plugins.CSPlugin.OnCallHook (string,object[]) [0x000e2] in <beb2b64691c64e2b95b99491bd85442c>:0 at Oxide.Core.Plugins.Plugin.CallHook (string,object[]) [0x00063] in <beb2b64691c64e2b95b99491bd85442c>:0 at Oxide.Core.Plugins.PluginManager.CallHook (string,object[]) [0x00091] in <beb2b64691c64e2b95b99491bd85442c>:0 at Oxide.Core.OxideMod.CallHook (string,object[]) [0x0000e] in <beb2b64691c64e2b95b99491bd85442c>:0 at Oxide.Core.Interface.CallHook (string,object[]) [0x00007] in <beb2b64691c64e2b95b99491bd85442c>:0 at Oxide.Core.Interface.CallHook (string,object,object) [0x00016] in <beb2b64691c64e2b95b99491bd85442c>:0 at BaseCombatEntity.Die (HitInfo) [0x00050] in <e835aa7a159e4026a4c988e02103a9af>:0 at BaseCombatEntity.Hurt (HitInfo) [0x00324] in <e835aa7a159e4026a4c988e02103a9af>:0 at BaseVehicle.Hurt (HitInfo) [0x00009] in <e835aa7a159e4026a4c988e02103a9af>:0 at BaseCombatEntity.Hurt (single,Rust.DamageType,BaseEntity,bool) [0x0002b] in <e835aa7a159e4026a4c988e02103a9af>:0 at BaseHelicopter.DelayedImpactDamage () [0x00024] in <e835aa7a159e4026a4c988e02103a9af>:0 at InvokeHandlerBase1<TREF>.DoTick () [0x0013f] in <ed7e8328fe0d407183c87ff89235d60a>:0 at InvokeHandlerBase`1<TREF>.LateUpdate () [0x0000d] in <ed7e8328fe0d407183c87ff89235d60a>:0 at (wrapper runtime-invoke) object.runtime_invoke_void__this (object,intptr,intptr,intptr) [0x0002c] in <ff4e3fe86250407285eaebda8f185c35>:0

Any help would be appreciated

According to the logs, what happened is that a player's vehicle was killed, so Vehicle License collected the items from the vehicle's container(s) and tried to give those items to the vehicle owner. When one of the items was given to the player, Backpacks detected it and tried to gather the item into the Backpack. Backpacks found a suitable page and tried to insert the item. Then somehow Rust's Item.MoveToContainer function got into infinite recursion, leading to a stack overflow.

At this point, I don't think Backpacks or Vehicle License is to blame because they simply tried to collect/move items, and there's nothing in the stacktrace that suggests either of those plugins were involved in the recursion. In other words, a player probably could have triggered this error another way.

Looking at the Rust code, there are 6 places where Item.MoveToContainer can recurse into itself. I can rule out 3 of those places. Of the 3 remaining places, one stands out as suspicious. Basically, when an item is being moved to a destination container, Rust first tries to fill up all existing stacks of that item in the destination container, before removing the remaining amount to an empty slot. Rust achieves this through recursion, by having the Item.MoveToContainer function call itself each time it fills a stack while having some amount of the item leftover.

As far as the vanilla Rust code, Item.MoveToContainer appears to be incapable of infinitely recursing. However, there are two hooks called in the Item.MoveContainer function: CanStackItem and OnItemStacked. If either of those hooks were used by a plugin, in such a way that the item that was filled up had its amount reduced to below the max stack size, the recursive Item.MoveToContainer call would re-find the same item in the destination container and try to stack it again. I was able to simulate this by using the OnItemStacked hook to transfer 1 of the item back every time it was stacked, to keep it consistently below the max stack size. This successfully resulted in infinite recursion and stack overflow. Given this proof of concept, my best guess is that you have a plugin using one of those hooks to reduce an item stack during transfer. For example, if a plugin tries to limit the amount of wood in the backpack to 5000 (or any container for that matter), the process of transferring the items back to the player inventory when they exceed the limit could lead to this issue.

For next steps, I suggest you search the code of all your plugins for CanStackItem and OnItemStacked. If any are found, reach out to the developer to ask if how they are using those hooks can result in such a side effect.

Locked automatically