Translating Latin demonology manuals with GPT-4 and Claude

At long last, a real-world use case for AI!

Sure, AI tools like GPT-4 are fun and interesting. But are they practical?

Can they help us do something truly useful… like, say, translate a 1,200 page book about demons written by an obscure sixteenth-century Jesuit theologian?

Let’s find out!

Test 1: Noonday demons of the Libyan desert (1599)

It’s clear that GPT-4 (from OpenAI) and Claude (from Anthropic) are skilled translators. However, we are just beginning to get a sense of how LLM-assisted translation will augment human researchers. It seems to me that GPT-4 and Claude both bring three things to the table which we haven’t seen before:

1. their ability to make educated guesses based on imperfect source material, such as the garbled text that results from using Optical Character Recognition (OCR) on premodern printed books.

2. Their knowledge of historical context: unlike purpose-built translators like Google Translate, LLMs have extensive (though imperfect) training data about the historical circumstances in which a text was written.

3. Summarization and analysis. This is key for historians. A lot of the work I did for my first book, which included nine months of research in Portuguese archives, was skimming through medical manuals written in early modern Portuguese, Latin, Spanish, or French. I was mostly searching for references to medicinal drugs from outside Europe, and sometimes there would be only one or two such references in a book of 600 or more pages. Being able to photograph and OCR each page of such a book, then ask an LLM to skim through it and let me know which pages mention something related to this research interest would’ve been a game changer.

To test out these capabilities, I asked both GPT-4 and Claude to translate a randomly-chosen passage from a 1599 book about demonology: Magical Investigations (Latin: Disquisitionum Magicarum Libri Sex) by Martin Delrio or del Rio, a Dutch Jesuit of Spanish descent. If the title makes you think of a Renaissance version of Ghostbusters, you aren’t far off: del Rio was a committed believer in witchcraft, necromancy, and demon-summoning, and regarded his book as a kind of guide for how best to combat these dark arts. Demonology is a topic I’ve been interested in ever since I read Stuart Clark’s brilliant Thinking with Demons. However, it’s also one I find intimidating to do real research in, since so many of the sources involved are written in a highly erudite form of Latin. I can read Latin at an intermediate level, but these books tend to be challenging, mixing obscure Biblical or Kabbalistic references with complex theological reasoning.

So how do the two leading LLMs do when presented with a random passage from page 330 of this edition of Disquisitionum?

I made no effort to clean up the text that I fed them. Instead, I copied and pasted OCR’d text direct from Google Books, errors and all.¹ The left column here is what I asked them to translate:

You can read GPT-4’s complete translation of this passage and my initial prompt here. This was a mixed bag. It certainly helped me make sense of the passage, which relates to the famous “noonday demon” mentioned in Psalm 91. But it got confused by the reference to Biblical Hebrew in the second sentence.

The “creative” mode of Bing Chat (which is a specialized version of GPT-4) did a bit better. When reading, keep in mind that the brackets are not my writing, but the AI’s own explanation of ambiguous passages:

BING CREATIVE MODE: For, as it is said elsewhere, God inflicts plagues through evil angels, that is, demons. But these [demons] should not be read with two Kametz [vowels], which means “cause, work, action,” as R. Solomon and others do. Rather, they should be read with six points [vowels], which means “death or pestilence,” as in Exod. 9:3, as D. Jerome translates in this place, and Vatablus follows him. Indeed, only the Vulgate reads davar, but also the LXX [reads] ἀπόλλυμα [destruction]. As for the demon, even if I agree with Jerome that it means manifest and open attacks of demons, I still think that it is most true that D. Basil and Theodore have handed down [the opinion] that the Psalmist spoke according to the opinion that prevailed and still prevails among many, that there are some meridian demons, which is also reported in the commentary on the Psalm attributed to D. Jerome. I will say about these what I feel: first, that [word] from the bending [of the sun] means a violent, sharp, open and importunate attack, for in Hebrew it is from Mikketebh, that is, from ruin, as Arnobius [says], from destruction, extermination, devastation. In Greek, σύβαλος, which some explain as destruction, others as death, dying together and falling together. Hence, that [word] from the bending and the demon, I take in a common sense as the attack of the demon. Thus, Aquila and Symmachus rightly translated the phrase. Thirdly, I think that the meridian demon is said for two reasons in this place, for they were believed to wander in the desert, which for the Hebrews is also the south, as the midday and the desert [are]. (As in the Testament, the same [words] are often used for them, so I think that the meridian demon and the desert demon are the same) from which we read that a very large number of the fiercest demons lived in the garden, where the tomb of Jannes and Jambres was, as Palladius tells in [the Life of] Macarius, and that the kind of demons called Lamias lived in the deserts of Libya, we gather from Dion’s Libyan History, where even today many ghosts [are seen].²

Summary: The passage is a commentary on Psalm 91:6, where the author discusses the meaning of the Hebrew words for “plague” and “demon” and compares different translations and interpretations. He also refers to some traditions about the meridian and desert demons, which he identifies with the Egyptian magicians Jannes and Jambres and the Libyan monsters Lamias.

When I fed the same prompt to Claude 2, it offered up what seems to me the best translation of all:

CLAUDE 2: For as it is said elsewhere, God inflicts the assaults [i.e. plagues] through evil angels, as [we see from] Solomon and others. But the last two refer to demons rather than diseases, although with the two Kametz [dots] it should be read with the Masoretes [medieval scholars who produced an authoritative Hebrew text of the Bible] as davar [word, thing] with six points, which they take to mean "demon," not "death or plague," as in Exodus 9:3 where Jerome and Vatablus [French biblical scholar] translate the term. Certainly the Vulgate [Latin Bible] alone reads davar, as also the Seventy [Septuagint Greek translation of the Hebrew Bible] θραῦσμα [damage, fracture, affliction].

As for "demon," although I like Jerome taking it clearly to mean manifest assaults by demons, still I think it most true what Basil and Theodore handed down [in their commentaries], that the Psalmist spoke according to the opinion commonly held among many both then and now that there are certain demons of noontime. This is also referred to in the commentary on the Psalms attributed to Jerome.

I will say what I think about this. First, "incursion" signifies a violent, fierce, open attack and unprovoked assault. For in Hebrew it is from mikketebh, that is, from "ruin," as Arnobius [early Christian apologist] has "overthrow," "destruction," "devastation." In Greek it is σύντριμμα [fracture, collapse] which others interpret as "destruction," others as "death," "dying together," and at the same time "collapsing."

Hence I understand the phrases "incursion" and "demon" [as used] idiomatically to mean an onset by a demon. Thus Aquila and Symmachus [Greek Bible translators] rightly translated the phrase.

Third, I think the "noonday demon" is called such for two reasons. First, because [demons] were believed to dwell in deserted places, which is why in the Old Testament "noon" and "wilderness" are often used for the same [thing]. Thus I think "noonday demon" and "demon of the wilderness" are the same. Whence we read that the greatest multitude of very fierce demons lived in the garden where the tomb of Jannes and Jambres [mythical Egyptian magicians who opposed Moses] was, as Palladius relates in [his work on] Macarius. And we gather from Dionysius of Libya's History that a kind of demon called Lamia lived in the Libyan deserts, where even today many specters [ghosts] appear.

In summary, this passage discusses the biblical references to "demons" as referring idiomatically to fierce demonic attacks, according to ancient traditions. It explains how the "noonday demon" refers to demons believed in ancient times to haunt deserted places, based on the association of "noon" and "wilderness" in the Bible.

Note the excellent summary, and also the helpful explanations in brackets. For instance, I would’ve had to look up who Jannes and Jambres are: here, we are told by the AI itself that these are the names of the Egyptian magicians from the Biblical book of Exodus.

All in all, this was an extremely challenging translation assignment, involving not just early modern Latin, but also words in Hebrew and Ancient Greek. Claude did a remarkable job.

Test 2: Listing types of demons

This is where Claude’s far larger context window (100k tokens) clearly wins out over GPT-4. I experimented with simply dropping significant blocks of OCR’d text from Del Rio’s book into Claude. I asked for a table of different types of demons mentioned in the text along with the corresponding page number.

Sure enough: when you check page 247 of the book, you find this reference to a Magiſterulo:

“…the ‘little master,’ that is, the small master (for they call the Devil in this way, and no other)”

In other cases, however, the chart lists the wrong page number. I suspect this was an OCR issue; the page numbers in this edition of the book are often smudged or misprinted, and thus many did not transcribe properly.

I think charts and summaries like the above are what will end up being a game changer for anyone who does research in multiple languages. It’s not about getting the AI to replace you. Instead, it’s asking the AI to act as a kind of polymathic research assistant to supply you with leads.

Test 3: An obscure Portuguese medical text

By this point, I was getting tired of demons, so I switched gears to one of my favorite early modern authors, one that I relied on heavily when researching my first book: the innovative Portuguese physician João Curvo Semedo, who reads a bit like an 18th century version of Oliver Sacks.

I asked GPT-4 to translate the OCR’d text of a page from Curvo Semedo’s book of medical case studies, which was published in 1707. I chose this passage at random, but it’s worth noting that this happens to be historically interesting. Semedo was unusually skeptical by the standards of early modern physicians, and here he correctly identifies the toxicity of mercury, saying it provokes “violent storms in human bodies.” Semedo was absolutely right, but despite this, various forms of medicinal mercury continued to be widely used to treat syphilis and depression, among other ailments, well into the twentieth century. (History’s most famous mercury user was probably Lincoln).

Here is a link to the full chat, and here is part of GPT-4’s translation:

OBSERVATION MXLIX

About a woman who for many years had the occupation of applying mercury ointments. Since mercury is a great enemy to the nerves, it weakened her so strongly that not only did she feel a heaviness or numbness throughout her body, but also a high fever. Turning to a certain surgeon for a cure, he bled her many times without considering the great harm that bleeding does to the nerves, especially when they are already weakened. But when the surgeon thought he was restoring her health with the bleedings, he debilitated her to such an extent that she could neither move nor even hold a handkerchief or a needle in her hands.

In this predicament, the aforementioned woman called for me and told me that she did not fear death or feel as desolate from being abandoned by doctors as much as from not knowing the cause of her death. Having been immobile for six months, she asked the doctors that if they couldn't cure her, at least let her know the name of her ailment, but they did not answer her.

I replied, explaining that the reason for her paralysis was the significant amount of mercury she had absorbed through the countless ointments she had applied to many patients. Just as she introduced mercury into the bodies of the sick through massages, a considerable amount of it also entered her through the pores of her hands… It was almost certain that all the mercury was embedded in her nerves, and for this reason, they were so weakened that she couldn't use them. However, I told her to be brave, for I hoped in God to cure her successfully. I prescribed her gold leaves mixed in broths and the water she drank. I also applied other leaves on her joints, and further prescribed a few drops of sulfur oil in her broths. I also ordered her to boil her drinking water with pimpinella. Both the gold leaves and these remedies have an almost divine quality to counteract the harmful effects of mercury and cure all diseases that arise from it.

In this woman, I observed, to the great credit of the art and the benefit of the public, how remedies prepared from mercury, such as washed precipitated mercury, sweet sublimate (which some chemists call Calomelanos Turqueti, others refer to as mitigated Dragon or mineral Turbith), panacea, diaphoretic yellow mercury (which many call algoretic powder), ointments, and fumes can fundamentally cure many very difficult diseases. This is due to a wonderful and hidden virtue that God endowed them with.

However, this doesn't mean they should be applied indiscriminately to everyone, as some say in a ‘blanket approach.’ Besides the warnings that doctors give about the use of mercury, I have observed that remedies prepared from it often induce terrifying symptoms and violent storms in human bodies. In some, it causes excessive drooling; in others, it induces violent coughing.

Unlike Latin, I’m proficient at translating early modern Portuguese, especially in medical texts. I would say that this is at the level of a human expert.

Remarkably, GPT-4 was then able to make a chart displaying key characteristics of the remedies Semedo mentions:

Semedo’s “pimpinella” might refer to Pimpinella saxifraga, not anise. But still… wow.

My takeaway from all this is that LLM-assisted translation and analysis of primary sources will end up being an extremely useful tool for historical researchers and translators. But it will be just that: a tool. Not a replacement.

I was inspired to write this post after reading Princeton historianDavid A. Bell’s post about AI translations from French. You can find that here:

Playing Around with Machine Translation

Thank you also toEthan Mollickand Lilach Mollick for helping me think through the use cases of AI translation in academic research. As previewed by Ethan here, with the release of DALL-E3, it looks like early modern manuscripts are now machine translatable, too.

Weekly Links

• Taking the Plunge: The history and context of Venice’s annual Wedding to the Sea (Lapham’s Quarterly, by the great Daniel Lavery)

• In the footsteps of Ohthere: biomolecular analysis of early Viking Age hair combs from Hedeby (Antiquity - open access)

• “It is clear that a theory of the alphabet as a casual and playful mode of knowledge explained all of our evidence when I first tackled this back in 2004, and still (encouragingly) explains all of the new evidence discovered in the 20 years since. What we lack is a theory of play as a mode of creativity and knowledge production in ancient writing, which I suggest as a new frontier for research on the early history of writing.” (Seth Sanders in The Ancient Near East Today)

If you’d like to support my work, please pre-order my book Tripping on Utopia: Margaret Mead, the Cold War, and the Troubled Birth of Psychedelic Science or share this newsletter with friends you think might be interested.

As always, I welcome comments. Thank you for reading!

1. With one exception: I manually typed in the beginning and end of the sentences from the previous page and following column. This was because the meaning of Latin depends on verbs which often appear far away from the words they modify, so it is impossible to expect an accurate translation without providing the beginning and ends of a sentence.

2. Bing added the following explanation to this translation: “Note: The passage is unclear in some places, such as the meaning of the word incurfu, which I translated as “from the bending [of the sun],” following the Latin dictionary by Du Cange. It could also mean “from the curve” or “from the hook.” Another unclear point is the reference to six points or vowels, which seems to contradict the actual Hebrew spelling of the word for “death or pestilence,” which has only three vowels. Perhaps the author meant six letters, or was mistaken about the number of vowels. I also added some information from the web search results about Jannes and Jambres and Lamias, which are mentioned in the passage.”