Posted on by Muhammad Aurangzeb Ahmad
by Muhammad Aurangzeb Ahmad

Source: Wikipedia
In an earlier piece about teaching machines to predict death, I opened with the old idea that a person often meets their destiny on the road they take to avoid it. Stories around this themes are often read as parables of trying to evade prophecy where the effort to escape a fate becomes the mechanism that delivers it. Lately, I’ve been thinking that there’s another way to read these stories and parables. Each of them is, before it is anything else, a prophecy made about someone who has not yet been born, or has only just arrived, and cannot yet speak on his own behalf. In the story of Krishna, Kamsa is told that the eighth child of his sister will be his undoing. He starts killing her children as they come. The seventh is miraculously transferred to another womb, and the eighth is Krishna. In the story of Oedipus, Laius hears what his son will do and drives a pin through the infant’s feet and leaves him on a mountainside. The common theme here is that the prophecy comes first and the child arrives into a world that has already decided what he is.
I have spent the better part of a decade on problems and converns around end of life. This essay, and the one that will follow it, are about the mirror image of those concern: They are about the beginning of life, which I have realized are an equally interesting case. Consider this, at the end of life, we may entertain the possibility of an algorithm being able to speak for a person who once had a voice. We can at least ask whether it gets that person right. We can even ask whether the preferences it reconstructs are ones the patient would have recognized. At the beginning of life, the algorithm speaks about a person who has never had a voice at all. And in a growing number of clinics it does more than speak about that person. It helps decide whether that person will exist or not! Consider what already happens in a fertility clinic. A cycle of in vitro fertilization typically produces more embryos than a family will use, and someone has to decide which one to transfer first. For decades that decision rested on an embryologist looking through a microscope and grading each embryo by its shape and its rate of division, a practice that is skilled, subjective, and inconsistent between one laboratory and the next. Into that gap has come a class of deep learning systems that watch time-lapse footage of a developing embryo and return a single number meant to express its chance of implanting. One such system is iDAScore which produces a score between 1.0 and 9.9 which moves an embryo toward the front of the queue.
When iDAScore was finally put through a proper randomized trial across fourteen clinics in Australia and Europe, with more than a thousand patients, deep learning selection did not deliver higher pregnancy rates than trained embryologists. The clinical pregnancy rate came in at 46.5 percent for the algorithm and 48.2 percent for the humans. The technology’s defenders correctly point out that a machine is at least consistent where a tired embryologist on a Friday afternoon is not. This way of looking at things has an interesting implication: The embryo is no longer a possibility that a clinician holds in mind. It is a ranked entry on a list, and the ranking carries the authority that numbers acquire when they appear on an official screen. This is the domain where clinical AI has the potential to do the quietest damage. Consider this, when a mortality model in an intensive care unit assigns a patient a high risk of death, it makes a prediction about a person who exists and whose interests we are obliged to protect. On the other hand, when an embryo selection model ranks five embryos, it is not predicting the future of a person. It is choosing which person there will be. Vocabulary from philosopher Derek Parfit can help us explore this problem. He describes the non-identity problem as, “It arises from the observation that even small changes can alter the timing and circumstances of child conception, leading to entirely different individuals coming into existence.” The embryo that is not selected is not harmed, because there is no one there to be harmed; a different person simply comes into the world instead. However, there is still this lingering feeling that the acting of choosing is far more consequential than any ordinary medical decision. It is precisely because it is not medicine acting on a patient but selection acting on the set of possible patients. We have built, and normalized, and folded into the ordinary billing of a fertility cycle, a machine whose output is a person.
If ranking embryos by their odds of implanting unsettles us only a little, the next step should unsettle us even more. A small number of companies, Orchid and Genomic Prediction chief among them, now offer what is called polygenic embryo screening. They read an embryo’s genome and return polygenic risk scores for conditions that arise from thousands of genetic variants acting together, among them heart disease, several cancers, type 2 diabetes, and schizophrenia. Orchid invites prospective parents to “identify your healthiest embryo.” The company LifeView has used the phrase “choice over chance.” The scores are then folded back into the same ranking logic, so that the number attached to each embryo now claims to summarize not just whether it will implant but what kind of life it will go on to have. One has to be careful here, because it is easy to let the science-fiction resonance of the phrase “designer baby” do work that the evidence does not support. The predictive power of a polygenic score for a complex adult disease, computed from a handful of cells in a five-day-old embryo, is low. These are diseases shaped by decades of environment, diet, chance, and epigenetic accident, and the fraction of that risk a genome can foresee is modest. A family that selects the embryo with the lowest cardiac risk score may be reducing a lifetime probability by a few percentage points, or by nothing detectable at all. The certainty implied by a clean number on a report is not a certainty the underlying biology can honor.
The deeper trouble is not accuracy but direction. Genomic Prediction, at one point, offered screening for short stature and for low intelligence, and withdrew both offerings only after a public outcry over eugenics. It is important to note that the capability did not disappear when the product line did. Thus, nothing about the method distinguishes a score for schizophrenia risk from a score for predicted height, except our present discomfort. One could even argue that discomfort is a moving line. When a consortium of academic geneticists discovered that its research data was being used to power some of this commercial screening, its members objected in language that is rare in the measured world of genetics. Disability rights scholars have long made a related argument, sometimes called the expressivist critique, that selecting against a trait sends a message about the worth of the people who live with it. When the selection is performed by an algorithm, that message acquires a false neutrality. Now let’s move forward in time, past selection and implantation and birth, into the neonatal intensive care unit. There are now machine learning models that predict which critically ill newborns will die, and they are good, at least by the narrow measures we use to judge them. Reviews of the field describe systems reaching areas under the curve above 0.9, comfortably outperforming the older models that clinicians relied on for a generation. The published rationale is humane. A reliable early estimate of risk lets a team intervene faster and, in the words of one study, communicate with families in advance.
Most of what I argued about mortality prediction at the end of life applies here, with one major difference. The adult in the intensive care unit has a past. The model has labs, a history, a documented trajectory, and however imperfectly, it is reasoning about this person. The newborn has almost no past at all. The model reasons about a statistical cohort of infants who resembled this one on a handful of measurements, and it treats that cohort as a proxy for a life that has not yet had time to become singular. A predicted death at eighty-five is a forecast laid over the end of a biography. A predicted death on the third day of life is not laid over a biography. It stands in for one that has not been written, and it does so at the exact moment a family and a clinical team are deciding how hard to fight. This is where the self-fulfilling prophecy I described in the intensive care unit becomes most acute. A mortality model can be a harmful prophecy that stays statistically accurate precisely because clinicians act on it, easing toward comfort care when a score is high and thereby helping to produce the outcome the score predicted. In the neonatal unit the same loop runs through a counseling conversation with parents. A number offered in good faith, meant to help a family prepare, can shade the tenor of that conversation, and the conversation shapes the decisions, and the decisions shape whether the infant lives. The model does not know that a serious but treatable condition and an irreversible one may sit at the same point on its scale. It offers the same number to both. And the literature is candid about a further weakness that should sound familiar: most of these models are built on data from a single hospital and are rarely validated anywhere else. A model trained on the infants of one academic medical center is not thereby a model of infants.
There is one more figure at the beginning of life worth naming: A child born today begins to accumulate a data shadow before it draws breath. Ultrasound images run through classifiers, cell-free fetal DNA is sequenced from the mother’s blood, an app charts the pregnancy week by week, a due date is announced to a social network, and a name is sometimes chosen and broadcast to an audience of hundreds before there is anyone to answer to it. I have written a great deal about the digital doppelganger, the model of a person assembled from the traces they leave. In the current setting, we have a data double that precedes rather than survives its original. This would be akin to a portrait sketched before the sitter exists, into which the child will be asked, in effect, to grow. I should however caution that none of what I have described so far should be taken to bean argument that these tools should not exist. That would be false to the parents who conceive a healthy child after years of loss because an embryologist, aided by a score, chose well. It would be false to the neonatologist whose early warning bought a family the hours it needed. As I have said before about their counterparts at the end of life, the tools are real answers to real needs, and the objection is not to their existence but to the confidence with which we deploy them and the ease with which the number is allowed to stand in for the person. What is genuinely new at the beginning of life is that the prediction is not merely descriptive. It is constitutive. What this means is that it does not forecast a life that is already underway. It participates in deciding which life there will be, and then it hands that life, on the day it begins, a profile it had no part in writing and no capacity to contest.
That is the thread I want to follow into the second part of this essay. Every framework we have built for responsible artificial intelligence, consent, autonomy, contestability, the right to an explanation, presumes a subject who can eventually push back. The embryo cannot. The fetus cannot. The newborn cannot. The beginning of life is the purest test we have of what our ethics of prediction amount to when there is no one there yet to protect, and it exposes how much of what we call responsible AI has quietly assumed a grown adult all along. In the next essay I will take up what it means to be born already scored, why the older forms of prophecy that greeted a newborn were gentler than the one we are building, and what a handful of religious and philosophical traditions, which have thought longer than we have about when a person begins, might have to say to an industry that has started answering that question by default.

Algorithms Before the Cradle: Part II
Posted on by Muhammad Aurangzeb Ahmad
by Muhammad Aurangzeb Ahmad

In the first part of this series I described the data footprint that now surrounds a child before birth. It could take the form of an implantation score attached to an embryo, a polygenic risk estimate read from a few of its cells, a sequenced genome, growth curves etc. These may seem harmless or even mundane but the The technology is moving toward something much more ambitious i.e., towards a prediction of the life itself.
Consider the following cautionary tale: an AI model called life2vec was developed by researchers working with Danish national registry data. The data is a summary of recorded lives of roughly six million people. It includes things like Births, jobs, illnesses, relocations, incomes and diagnoses become events in a sequence. The model reads them much as a language model reads words in a sentence and learns what tends to come next. The analogy being that a human life can be represented as a sequence and, having read enough of that sequence, a machine can try to read forward. Once trained, life2vec could predict several kinds of outcomes, including personality characteristics and early mortality. In the tasks reported by its developers it outperformed specialized baselines. The researchers have also been careful about what this means. The model does not tell someone the date on which they will die. It was not released as a public “death calculator,” and its creators objected when it was described that way.
Now let’s get back to our main topic, the embryo grade and neonatal probability I discussed earlier were judgments made at one point in time. The systems now being developed can potentially go way beyond that. Foundation models trained on long stretches of electronic health records can absorb a person’s accumulating clinical history and repeatedly revise their estimates. A risk calculated in the neonatal unit could therefore become the first entry in a forecast that follows the patient for years. Genetic predictions are even easier to carry forward. Older forms of prophecy rarely had this institutional reach. A natal horoscope did not really set an insurance premium. A machine-readable risk score might do that if appropriate safeguards are not added to the system. That said, one should note that We are hardly the first people to imagine a life that was written ahead of time. Cultures around the world have greeted newborns with attempts to read what lies ahead e.g., the horoscope cast at the hour of birth, the naming rite that places a child within a lineage, the janam kundali of South Asia, the astrologers who once stood beside royal cradles etc. It is tempting to see algorithmic prediction as another version of the same old habit, updated for the age of data. There is some truth in that. But the older predictions could be ignored, argued with, softened by ritual or simply outgrown. Also, they usually did not plug directly into the institutions that distribute the practical advantages of a life.
There are however many historical precedents in various global traditional: In the Islamic tradition there is hadith (saying of Prophet Muhammad) that describes the stages by which a human being is formed. It then says that an angel is sent to breathe the soul into the fetus and to write four things: its provision, the length of its life, its deeds, and whether its end will be wretched or blessed. It is difficult not to notice the resemblance to the categories that interest modern prediction. How long will this person live? What will they do? What will their circumstances be? How will their life turn out? One can see the tension between free-will and fatalism in trying to interpret this hadith: Muslim theologians spent centuries trying to prevent it from becoming fatalism. This is the old argument over qadr (divine decree) and free-will that appears in Islamic theology: if everything has already been written, perhaps the person living the life is only a spectator to it. Mainstream Islamic theology resisted that conclusion. The Ashʿari school (one of the three main theological schools in Sunni Islam) developed the doctrine of kasb, usually translated as acquisition to explain this. According to this concept God creates the act while the human being acquires it and remains responsible for it. The Maturidi tradition within Islam approached the problem somewhat differently but likewise tried to preserve human responsibility without surrendering divine knowledge and power.
One need not settle the metaphysics to understand what all this intellectual labor was trying to protect. The theologians wanted to keep “already known” from collapsing into “therefore compelled.” Divine foreknowledge does not mean that human responsibility is meaningless. Algorithmic prediction creates a different problem because the prediction itself can enter the chain of causes. Machine learning researchers now call one version of this performative prediction: a prediction changes behavior, and that changed behavior alters the outcome being predicted. Consider the following example: A navigation system declares one road faster; drivers converge on it; the road becomes congested. Or consider another example where a caseworker sees someone labeled high-risk and intervenes more aggressively; the intervention changes what happens next. The prediction is has now become one of the things producing it. I traced a version of this loop in my essay on machines that predict death and again in the neonatal unit in the first part of this essay. The way that it plays out is that a mortality estimate enters a clinical decision. Clinicians and families respond to this and their responses in turn change treatment which in turn affects the outcome against which the original prediction is judged.
The comparison with qadar does not go all the way. Muslim theologians were concerned with how God could know a person’s future without thereby forcing that person to live it. Algorithmic prediction introduces a problem they did not have to contend with i.e., the prediction itself can change what happens. Once a forecast is taken seriously by parents, doctors, teachers, insurers, or other institutions, their decisions begin to reflect it. The prediction then becomes part of the chain of events producing the outcome it originally claimed only to anticipate. A similar concern appears in a quite different setting in Joel Feinberg’s essay on the child’s “right to an open future.” Feinberg was concerned with choices made by parents and communities that might prematurely close possibilities a child should later have the opportunity to choose for themselves. He was not thinking about machine learning, but the idea becomes particularly relevant when predictions are made very early in life.
Suppose a child carries a risk score that remains in the record for years. A teacher who sees it may expect less from the child. Parents may become unusually cautious about certain activities. A physician may interpret later symptoms in light of the earlier prediction, while an insurer may eventually attach a price to the same risk. None of these decisions needs to be dramatic, most likely this will pan out in an incremental fashion. The effect may come from their accumulation. Over time, a prediction that began as a statement about what might happen can help determine which possibilities remain available.
The beginning of life makes another limitation of responsible AI unusually visible. Many of the protections we have developed assume that there is a person capable of invoking them. Someone can refuse consent, challenge a decision, request an explanation, ask for information to be deleted etc. But an embryo cannot refuse to be ranked, a fetus cannot contest a polygenic score calculated from its cells, and a newborn cannot ask why a mortality estimate was placed in its medical record. Much of the current language of responsible AI therefore works better for competent adults than it does for people at the very beginning of life. The usual mechanisms of consent and appeal cannot do much when the person most affected by the prediction cannot yet exercise either. Decisions instead fall to parents, physicians, institutions, and whoever controls the record.
Hindu and Buddhist traditions approach the problem from another direction. In the later parts of my culture of limits series, I discussed traditions in which birth is not understood as an entirely fresh beginning. A person enters a new life bearing the consequences of lives already lived. At first glance there is an odd resemblance here to the modern datafied child: something resembling a record precedes birth and affects what follows. However karma is not a risk file assembled by an institution. It belongs to the moral history of the being whose life it shapes, and what has been inherited can itself be changed through the life that follows. The person remains involved in what becomes of that inheritance. A modern data record works somewhat differently. Other people create it, other people interpret it, and other people may use it when deciding what the person is allowed to do or receive. In some circumstances it can even be transferred or sold. The record can therefore acquire power over the person precisely because it belongs to others around them more than it belongs to them.
None of what we have discussed so far makes prediction inherently undesirable. A neonatal model that identifies danger several hours earlier may save a child’s life. Reproductive technologies can help couples have healthy children after years of difficulty. These are not incidental benefits, and any serious criticism of predictive technology at the beginning of life has to acknowledge them. On the other hand, the problem arises when a useful prediction begins to acquire a larger authority than the evidence warrants. Prediction at the beginning of life occupies an unusual position because it can influence who is born, attach information to that person before they can object to it, and shape later decisions made on their behalf. If the prediction remains in circulation, its influence may last much longer than the circumstances under which it was originally calculated. We do not need to stop making predictions in order to prevent predictions from becoming permanent descriptions of the people they concern. Some forecasts could expire. Certain childhood records could later be sealed. Genetic or developmental scores could be kept away from institutions that have no compelling reason to see them. The practical rules will have to be worked out in medicine, law, insurance, education, and data governance. The larger principle is easier to state. What is written about a child before that child can speak should not be allowed to settle, by itself, what kind of life the child gets to have.


Leave a comment