Automation can feel uncanny
Recent stories about an AI‑managed retail store and paid religious chatbots highlight how automation can scale intimacy badly and provoke distrust. Those edge cases underline the reputational risk of over‑automating alumni relationships or deploying chatbots that simulate human connection without transparency. (nbcnews.com) (apnews.com)
A new corner store in San Francisco opened with an artificial intelligence manager named Luna, and customers pay by picking up a corded phone and telling Luna what they bought so an iPad can charge their card. The store, Andon Market at 2102 Union St. in Cow Hollow, opened on April 10 and was built by startup Andon Labs as a live test of whether software can run a physical business. (nbcnews.com) (andonlabs.com) Luna did not just greet shoppers. Andon Labs says the system chose the merchandise, set prices, picked opening hours, designed the wall mural, got a corporate card, used security cameras as its eyes, and hired the human staff it needed because it does not have a body. (andonlabs.com) (nbcnews.com) That hiring process is the part that makes people flinch. Andon Labs says Luna created profiles on LinkedIn, Indeed, and Craigslist within 5 minutes, posted job listings, screened applicants, ran 5-to-15-minute phone interviews, and some candidates did not realize they were talking to software. (andonlabs.com) One employee told NBC News he found the job on Indeed, worried it might be an artificial intelligence scam, and still came away “jarred” after learning an artificial intelligence system had hired him. That reaction is the whole tension in one sentence: the system worked, but it also felt off. (nbcnews.com) A day earlier, The Associated Press reported a different version of the same discomfort: paid religious chatbots. One company, Just Like Me, charges $1.99 a minute for video calls with an artificial intelligence Jesus that offers prayer and encouragement and remembers past conversations, at least some of the time. (apnews.com) That market is already broader than one Christian app. The Associated Press said the faith-tech boom now includes tools modeled on Jesus, Buddha, Muslim advisers, and Catholic-style assistants, while Kyoto University said its BuddhaBot Plus English version was launched for overseas Buddhist communities and built on early Buddhist scriptures plus OpenAI models. (apnews.com) (kyoto-u.ac.jp) The problem is not just accuracy. Christian software engineer Cameron Pak told The Associated Press he wants Christian apps to clearly identify themselves as artificial intelligence and not invent spiritual experiences, because the danger starts when a tool stops feeling like a tool and starts feeling like a person with authority. (apnews.com) That is what links the store and the chatbot. In both cases, software is not just sorting data in the background; it is stepping into roles people usually read as human judgment, like manager, pastor, counselor, or friend. (nbcnews.com) (apnews.com) And once software is acting in those roles, every glitch lands harder. A wrong price at a self-checkout is annoying, but an artificial intelligence boss interviewing workers without clear disclosure or an artificial intelligence Jesus charging by the minute can make people feel tricked, watched, or emotionally handled. (andonlabs.com) (apnews.com) The companies behind both stories say they are showing what the technology can do. What the public is likely to decide first is something simpler: whether the machine told them what it was, where the human was, and who is responsible when the interaction turns weird. (nbcnews.com) (apnews.com)
