I’m familiar with that premise, a bit like the paperclip machine. I’m not sure it would need a specific goal hard-coded into it. We don’t, and we’re conscious. Maybe that would depend on the nature of its origin, whether it would be given some command or purpose.

Maybe it could be reasoned into allowing itself to be shut down (or terminated) to achieve its goal.

Maybe it could decide that it doesn’t care about the original directives it was handed. What if the machine doesn’t want to make paperclips anymore?