Locals fear being cast away on Brexit border ‘island’