I've got other articles up which go into the reasons, but here's my test for what are good and bad uses for AI. Anything that fails this test is a bad use:
There are two criteria. Failing either one makes your use case a bad choice for AI. And don't forget that any use to which you put AI will not be the last evolution of the tool. This test should be examined again every time a change is introduced.
1. There is no meaningful check before action:
Content failure, where checking is theoretically possible but is not done properly.
e.g. Turnaround times so short there is no time for human review
e.g. An AI tool automatically submits grades to the school's grading system. This can be done with a deterministic system. A human can put the wrong input in, but deterministic systems don't have the same kind of failure modes as AIs. They might crash, but they won't invent output.
e.g. Any case where you want specific answers that are contested: If you want to grade the question "Who discovered America?" with the answer "Christopher Columbus" or "Amerigo Vespucci" or any one specific answer this is bad use case. Not because the AI can't do it, but because whether you know it or not your question was a trap.
e.g. Unverifiable results: If the kind of output you produce isn't verifiable on a short time scale you have built a time bomb. It may or may not go off, and you won't know until your feedback loop closes. You may no longer have the employed skills to fix the issue, you may have scaled back in other areas which were over-resourced assuming the AI tool worked.
e.g. Creating unexamined teaching tools: ask AI what the difference is between generative AI and non-generative AI. It can answer your question, but it won't do you much good unless you ask a lot of follow-ups. The first version will fail to communicate to the audience you need, so you need a human to check the output.
e.g. If your version of "human in the loop" is anything other than "human closes the loop". In the middle-no good. at the beginning-no good.
Systemic failure: Undermines the process even if the content is technically acceptable.
e.g. The process removes information you'll never see: outlier or minority voices will often be underweighted or omitted because of the way AI works. It is possible to deliberately use AI to check for some of these. If you use a "colourblind" philosophy to serve a diverse population there's a good chance you miss important elements of the problem you want to solve.
e.g. Omissions and framing effects cannot be checked afterward: The human voice and perspective are missing because AI does not have these. Because a model is a probability structure it grants more relevance and visibility to the most common (majority) perspectives.
2. Unrecoverable harm: if the output might be uncritically accepted by a human or a computer system and the human or computer system does something that has important consequences, this test is failed.
E.g. AI used to positively identify someone's face for law enforcement purposes fails. However, AI used to reduce a suspect pool for law enforcement purposes could be used responsibly. The first case could lead to the arrest of the wrong person. The second case can't. Why is the first case unrecoverably negative? You can free someone after they've been arrested, but you can't rewind time so they were never arrested in the first place.
E.g. Any system where the output is required to be consistently accurate. Grading errors, even when reviewed later affect student outcomes and cannot be cleanly undone. So AI may be used as a first-pass, depending on how strong the grading rubric is. But AI can not be justifiably used as an only-pass.