Yes, don't reference Newtonsoft.Json yourself; inherit it from Oxide.References.
I try to delete it but it keeps coming back
Newtonsoft.Json will still appear in the assembly explorer.
It is bundled in Oxide.References.dll and as Newtonsoft.Json.dll. Do not reference both Oxide.References.dll and Newtonsoft.Json.dll. You only need one, and the one you need is Oxide.References.dll. More information on how to use references in Visual Studio is provided by Microsoft, here.