Sometimes its not possible to place ladders inside metal frag buildings on raids. have this:
{
"Allow building ladders in the privilage zone": true,
"Block building": true,
"Bypass block privilage": "buildingblocker.bypass",
"Chat prefix": "[BuildingBlocker]",
"Prefix color": "#FF3047"
}

Will be nice if you will add any video of that

Probably fighting with NoEscape, are you using that plugin too?