The controversy over Supreme Court Chief Justice HL Dattu holding a conference on Good Friday has drawn the prime minister into its ambit, with Justice Kurian Joseph taking his complaint to Narendra Modi.