Conservatives losing ground on Fourth Circuit