RLP 334: Transkribus: Handwritten Text Analysis with AI

Nicole (1s):
This is Research Like a Pro episode 334 Transkribus Handwritten Text Analysis with AI. Welcome to Research Like a Pro a Genealogy Podcast about taking your research to the next level, hosted by Nicole Dyer and Diana Elder accredited genealogy professional. Diana and Nicole are the mother-daughter team at FamilyLocket.com and the authors of Research Like a Pro A Genealogist Guide. With Robin Wirthlin they also co-authored the companion volume, Research Like a Pro with DNA. Join Diana and Nicole as they discuss how to stay organized, make progress in their research and solve difficult cases. Let’s go.

Nicole (42s):
Today’s episode is sponsored by Newspapers.com. Well, hello everyone. Welcome to Research Like a Pro.

Diana (50s):
Hi Nicole, how are you doing today?

Nicole (52s):
I’m doing great. I’m really having a fun time working on my kinship determination project for BCG certification and I’m getting close to wanting to submit part of my portfolio. If I can finish it by the end of this year, I don’t know if I can, but trying. Anyway, I’ve been doing a lot of research in city directories and it’s been really fun to try to track my ancestor through the various places he lived in the early 1900s and just, you know, looking at different repositories that have city directories using the Ancestry City directory database, which is very good. But also it, I noticed that it had some weird indexing issues with one of the city directories I was using.

Nicole (1m 34s):
And you know, I always kind of think that, oh, like the indexing on these printed typed records are, is so much better and there’s less errors. But no, this one had an unrecognizable indexed entry for our ancestor and he was going by his first and middle initials with his surname, but the second, the middle initial had been indexed incorrectly, then they had used the post office name that was listed after each person as the person’s middle name. So it was just completely unrecognizable as anything close to our ancestor’s name. And I didn’t find it until I browsed the image and I was actually working on locality research and found that there was a city directory and went to it in a different place.

Nicole (2m 19s):
And I wondered maybe this is like searchable if I go to Ancestry and it was, but I couldn’t find it in the index. And so I just searched by the last name only and still didn’t recognize the name. So I finally just went to the original images and was browsing through those and looking, you know, it is alphabetized, so it’s not that hard and found him that way. And then when I clicked on the entry and Ancestry database, I saw that it was just indexed in such a weird way. And then I knew his brother-in-law was also living there and I looked for him and his entry was actually skipped, so every other person on that page was indexed except for him. So just beware that the indexing isn’t always perfect, even when we think it should be because it’s an easier type of record to index.

Nicole (3m 5s):
So I wondered if this was maybe machine indexed or maybe AI was used for part of the indexing or whatever. So who knows? But just be aware that these city directories are there and what I like to do now is just browse through them by clicking the location. You know, I want it to be in Texas and in this particular county and then you can see the list of years that are available for that county. So anyway, that was kind of a fun thing to work on over the weekend.

Diana (3m 31s):
I love that. And city directories are fabulous. And you’re right, they are completely alphabetized and so it’s not that hard to do the browsing. I think that really does point to machine learning, especially if it was just really easily read and the indexing was so off,

Nicole (3m 49s):
Right? I think if a human had actually done that, they wouldn’t have messed it up unless the person just really didn’t understand how the city directory worked and didn’t bother to read the header.

Diana (3m 59s):
Yeah, there you go.

Nicole (4m 1s):
So who knows? It really helps to understand the source you’re looking at. And it actually, this particular county was using a list of taxpayers in the county as the directory, so it just listed their name and their post office.

Diana (4m 16s):
Hmm.

Nicole (4m 16s):
So that was cool.

Diana (4m 18s):
Yeah, always. So fun. Well, let’s do some announcements for the day. We have our Research Like a Pro Webinar Series wrapping up for 2024. And I will be presenting on Tuesday December 10th at 11:00 AM Mountain time, I will be talking about Who Was Clemsy Cline’s Father? DNA and Indirect Evidence Provide a Candidate in this Burned County Case Study. So I’m excited to really show how it took three research projects to come up with this, but it is a wonderful thing to have figured out a good hypothesis and have DNA confirm that when years ago we had absolutely no idea who Clemsy Cline’s father could be.

Diana (4m 60s):
And now to have her family identified is amazing. So we will be talking about Burned Counties, Arkansas, Tax Records, Federal Land Records, Cluster Research, Boundary Changes, Census Records, DNA, Indirect Evidence. Well the next Research Like a Pro DNA study group begins February, 2025, specifically February 5th to May 14th. And if you are really ready to start adding DNA to your research process, we invite you to join us. You can do an easy project to get started. You can do a more advanced project if you’ve been working with DNA for a bit, but want to make progress on something difficult like I did with my Clemsy Cline project.

Diana (5m 44s):
We have students of all levels, so we would love to work with you if you have been using DNA for a while and gone through our Research Like a Pro DNA process, then we have our peer group leader application on our website and you would receive complimentary registration and then work with a small group throughout the study group. We invite you to join our newsletter for coupons and to find out what we are doing and what’s going on with FamilyLocket. And we are excited to see, we hope many of you at the next RootsTech Conference, which will be March 6th through 8th, and we will be presenting and have a booth there.

Diana (6m 24s):
And you’re always really excited to connect with our listeners and our friends.

Nicole (6m 31s):
Today we’re talking about Transkribus and we’re excited to talk about this topic. And this is something we have done some research and testing with to try to see if this program can be helpful for us. But basically, if you would like to automate the transcription of handwritten documents, this might be a tool that you’ll want to try also, if you want to quickly search for a particular name in a large probate or pension application file without having to transcribe the whole thing, this is also a tool that you can use for that. This Transkribus is an innovative platform that harnesses the power of artificial intelligence to make deciphering old handwriting faster. Whether you’re tracing your family tree or researching historical figures, Transkribus can save you a lot of hours.

Nicole (7m 17s):
And this is because it automatically transcribes handwritten documents into searchable text. So today we’re going to talk about how Transkribus works and its benefits for genealogy research and how you can get started with using it.

Diana (7m 30s):
Alright, so let’s talk a little bit about Transkribus. It was created by scholars at the University of Innsbrook and has been further developed by Read Co-Op COOP, I’m guessing is how you would say that, since 2019. READ CO-OP stands for recognition and enrichment of archival documents, Co-op, and it is a nonprofit organization with over 20,000 members, many from academia. Their mission is to make historical documents accessible to everyone and Transkribus is a key tool in achieving that goal. The platform is designed to be user friendly even for those who are not tech savvy, and it offers a range of features that make it valuable for us Genealogists and also historians.

Nicole (8m 19s):
Yes, agreed. And I think it’s interesting to see who uses it and kind of what the original thought was about who it would be useful for. And I don’t think it was originally thought of as a tool for Genealogists, but of course as Genealogists, we look at old handwriting all the time and sometimes we do actually have a really big file that we need help with. Sometimes we’re just transcribing one page of a deed book and so that doesn’t take us too long. But when we get those really long pension files from the national archives and we have a mix of printed and handwritten texts and letters and all different hands, that can just be daunting to think about processing all by ourselves. So that’s a one instance where I found that uploading the entire PDF file to Transkribus was a really good idea.

Nicole (9m 4s):
Well, let’s go through the transcription process. It’s fairly straightforward. First you upload your PDF or image files. So if you’re using a PDF, that’s straightforward or image files that might work, include a JPEG or a PNG file. Before uploading them you’ll want to rotate any sideways images. When you import a PDF into Transkribus the program separates each page into separate pages for you, which is really helpful if you want to do that and then have image files because then you can, even if that’s all you use the program for, now you have separate image files from a PDF, but then you can also organize any uploaded images and PDF pages into collections.

Nicole (9m 46s):
And a collection would be similar to folders on your computer. When you do the automatic text recognition, the first thing that happens is a layout recognition. And the layout recognition finds text regions, baselines, and words and it just tries to understand the order of the text. So if your document has a complex layout such as tables with multiple columns or like a newspaper, you may want to run the layout recognition separately and manually correct any errors before proceeding. So this might include tax records, inventories of estates and newspapers with many columns, but usually you don’t need to do the layout recognition separately.

Nicole (10m 28s):
So you upload the pages, then you click automatic text recognition, and then it’s gonna ask you to choose the appropriate language and model for your document. Transkribus offers a wide variety of models and they have been trained on different languages and time periods. So you’ll select a model that goes along with the time period and language that your document has. Once you’ve selected the model, you can start the text recognition process, which will automatically transcribe the text in your document. You can actually recognize the text in several images at once, or you can just do one image at a time, so you can leave and come back and you can look at the jobs table to see the progress on your jobs.

Nicole (11m 8s):
After the transcription is complete, you can then review it and make any necessary corrections. Finally, you can save, export or search your Transkribus document. There are several different options you can use for exporting or getting the text out of Transkribus, including the easiest, which is to just highlight the transcribed words and paragraphs and then just copy and paste them into whatever you’re using, whether it’s Airtable or a Word document. You can even export it as a PDF, export it as a Word file, export it as a Text file, and so forth. For 18th century deeds, which can contain challenging handwriting, it’s crucial to select the right model and carefully review the transcription for accuracy.

Nicole (11m 53s):
So Transkribus gives tips and guidance on choosing the best model for your specific document type and language. And so it’s good to read those and the number of models actually increases all the time. But let’s go through some of the models as of this year, 2024, and we’ll give you some ideas for choosing the right model.

Diana (12m 11s):
One thing you want to do is really take a look at your document and think about it. So think of the type of text. Is it handwritten, printed, or is it a mix of both? We will often see something like a pension have a mix. Then we think of a language. What language is it written in? Does it contain multiple languages? Think of the training size, how many words was the model trained on? You’ll notice that there will be some that have larger training models and those sets generally lead to better accuracy. Time period is something else to consider. When was your document created? Models trained on documents from specific time periods may perform better on similar documents.

Diana (12m 54s):
Then you’ll want to think of the character error rate, which is abbreviated CER. This indicates the percentage of characters transcribed incorrectly and a lower CER means higher accuracy. And so after you carefully consider those factors, then you can choose the model that is most likely to produce an accurate transcription of your document Transkribus does give you all the detailed information about each model, including its training, set size, the time period, and the CER or character error rate. And that will help you to make an informed decision about which model to choose.

Nicole (13m 30s):
In addition to the basic models, they have some advanced models with advanced features. So let’s go over those. Transkribus has language models, and you’re probably familiar with these because of the explosion of ChatGPT and other models like that which are large language models. While Transkribus has incorporated the use of language models to improve accuracy in these transcriptions, which are really using computer vision to look at the handwriting and try to recognize it, but then incorporating the language model helps to make sure that the words that are transcribed are actually likely to occur within the context of the text.

Nicole (14m 10s):
And this is really helpful for documents with challenging handwriting or faded ink. So it’s kind of like combining the handwritten text recognition with predictive language model that will then help you understand what could be in it. Another useful feature is Smart Search, which stores multiple alternatives for each word, allowing you to find a word even if it wasn’t transcribed correctly. And this is super useful as well, especially when you aren’t necessarily looking for a perfect transcription, but you’re wanting to search through a large document or collection of documents. So both of these features are available as additional options after you select your model.

Nicole (14m 51s):
So you choose a model and then you can check the box for language model and then you can check the box for Smart Search. So depending on what you have and if you would like to use the language model, you can check those boxes. Transkribus also offers super models and these are advanced models that are based on transformer architecture, which excels at natural language processing. So basically it’s using the the language model and incorporating that right into the model itself instead of having it as an extra checkbox later. Supermodels can handle multiple languages and they can also handle old and new forms of language and both handwritten and printed text.

Nicole (15m 35s):
They’re really useful for creating ground truth data for training custom models. Now, ground truth data just means this is what you give to the AI model and say, here is what a good transcription looks like for this type of handwritten text. And so then it learns from that. So if you wanna quickly train your own model, you can start by using a supermodel to create your ground truth data and then manually correct it and make sure it’s very accurate. You can have a free genealogist subscription or individual subscription that that level is free. But if you want to try these supermodels, you’ll have to upgrade and you can just upgrade for one month to try it out. And it’s not too much I, think it’s like $15, but it, it shows the pricing in euros.

Nicole (16m 18s):
So what I did is when I had a pension file that I wanted to create as a searchable pension file, I went ahead and upgraded so that I could use a supermodel. And that’s because I had both printed and handwritten text in that file and I wanted to use the supermodel to make it searchable. So that was the choice that I made at that time. This was a US Civil War Pension Application file for my husband’s second great Uncle Richard Dyer. And it had over 150 images. I had ordered it several years ago and had never taken the time to transcribe the whole thing since it was for a collateral relative. It wasn’t my direct line, but I wondered if there were any helpful clues in that that would help me discover Richard Dyer’s father’s parents.

Nicole (17m 0s):
It didn’t, but it still was a useful thing to order because when I did a DNA research report that included Richard Dyer, I was able to find additional evidence from this pension application file about one of his sons. Well, Transkribus did a great job with his pension file and I started the automatic transcription, then came back later when it was done, you know, an hour later. And I searched that automatic transcription for Robert thinking that it might mention Richard’s father, John Robert Dyer. But instead, I found several mentions of Richard’s son Robert Luster. And as I mentioned, I had been working on a DNA report and proving this line up to Richard Dyer was challenging because the son who the DNA tester was descended through had a different last name, which didn’t make sense to me and I couldn’t figure out why.

Nicole (17m 51s):
But when I contacted the test taker, they said that their ancestor was part of a polygamous family. And so Richard actually had two wives who were both alive at the same time. He was in polygamous relationship in Tennessee. That information helped a lot and then pairing that with evidence that Richard’s son, Robert Luster was mentioned in the pension application file of Richard Dyer helped cement the evidence that he was his son or else why would he be his executor and and that kind of thing. So he was listed as the executor in the pension file after Richard died. That was really helpful way to use Transkribus and hopefully that practical application helps you see how you might use it as well.

Diana (18m 33s):
Thanks for taking us through that I. think it gave us a lot of ideas. Well now let’s have a word from our Sponsor. Today’s episode is sponsored by Newspapers.com. Break down genealogy brick walls with a subscription to the largest online newspaper archive. Did you know Newspapers.com has over 1 billion pages of digitized newspapers dating back to 1690? Their growing collection includes papers from the US, UK, Canada, Australia and beyond. Discover birth and marriage announcements, obituaries and everyday stories about your ancestors in seconds. Newspapers.com can help you fill in the gaps between vital records and reveal details about your ancestors’ lives that you can’t find anywhere else.

Diana (19m 14s):
Their easy to use search feature lets you filter your results by date, locations specific paper and more. When you find something interesting, Newspapers.com makes it a snap to share it with family and friends. You can even save it directly to your Ancestry tree. Come explore 1 billion pages and make infinite discovery today on newspapers.com. Use promo code Family Locket for a 20% discount on your subscription.

Nicole (19m 37s):
Well, let’s talk about training a custom model at Transkribus. And when I first learned about this, I thought it sounded very intriguing and like a fun project. So I’ll talk about how you do it and then whether or not I decided to do it myself. So if you have a large collection of documents with unique handwriting, penmanship or language, you might consider training your own AI model in Transkribus. This can lead to improvements in the accuracy compared to using other pre-trained models. So to do it, you just need to manually transcribe between 25 and 75 pages of your documents or more if the documents have very diverse handwriting or types of information the way that it was written and the accurate transcriptions you provide in these 25 to 75 pages become the ground truth for your custom model.

Nicole (20m 27s):
Ground truth is the target for training an AI model. You can speed up the process of transcribing these 25 to 75 pages by using a similar base model or a supermodel as a starting point and then correcting the transcriptions manually to ensure the best possible results. It’s crucial to make sure your training data is accurate and consistent. You can also check baselines and add more data to further improve the character error rate of your model. Another thing you can do as you’re creating the ground truth is to use textual tags. Textual tags enhance your transcriptions by tagging specific words like abbreviations, places and people and adding additional attributes.

Nicole (21m 8s):
So to do that you can just click on the tag icon, select the text, and then use keyboard shortcuts to apply tags. So for example, for a superscript, you would select the text that you want to be super scripted and then control and period. And when you press those two buttons together, then it applies the tag for superscript so that the model understands that this is super scripted, which can be really important for 18th century, 19th century documents that use a lot of these super scripting. Well, after learning about the custom models a lot and thinking about what I had, I decided that there were plenty of good models for English information, English documents in the 1800s that I didn’t need to train my own model because I thought it was pretty close to good enough.

Nicole (21m 57s):
I did consider creating one that was unique for 19th century deeds in a particular location like in Tennessee or in a county, but I ultimately decided it wasn’t worth the time and so I didn’t do it. But if you have a language that isn’t well represented and you want to make a model that works with that language, I would highly recommend that. But a lot of the people using Transkribus have large archives that were all written by the same person. So they have thousands of pages by one scribe. And in that case, I think it makes sense to make your own custom model.

Diana (22m 29s):
Well, it would certainly make it so you wouldn’t have to do so much correction page by page by page if you just did that for the first 25 pages and then hopefully from that it would’ve learned enough about the handwriting that it would do very accurate transcriptions for those thousands of pages.

Nicole (22m 44s):
Right, especially if the scribe had consistent handwriting.

Diana (22m 48s):
Exactly. And I’m thinking about the journal that we have of our ancestor, William Henry Kelsy. I have a transcription that was done in the eighties, but I have noticed some inaccuracies and I would love to really go through and and check that out. And I have images of the actual diary, which is down at the BYU Special Collections, Brigham Young University. So I’m going to have to look into whether it would be worthwhile to do a custom model or just have it use the supermodel because it is, you know, 1850s, 1860s and the handwriting is fairly good. so I maybe wouldn’t even, wouldn’t even have to do with a training, a custom model project.

Diana (23m 29s):
Well let’s talk next about Transkribus for printed documents. So it’s primarily known for the handwritten text recognition, but it also excels at transcribing printed documents. In fact, it outperforms traditional OCR or optical character recognition software, especially if you are looking at a faded or problematic image. For example, Transkribus can accurately transcribe an obituary with faded text or a challenging newspaper article from the 1800s. If you have an illegible sentence like the start of a bankruptcy article for example, the model will have a hard time reading it. So you can type directly into the transcription area to correct it.

Diana (24m 13s):
Or you can try copying and pasting the transcription into a large language model like ChatGPT to ask for suggestions on fixing it. This can work really well for deeds and wills that have narrative and lots of common boilerplate phrases, but for something like a bankruptcy article, it would just be easier to correct it yourself in Transkribus.

Nicole (24m 38s):
Well, let’s talk about some tips and tricks for using Transkribus and getting the most out of it. The first is about fixing regions and lines. So if the automatic layout recognition doesn’t accurately identify the text regions and lines in your document, you can actually go in and manually edit them. There is a menu, like a three dots menu that you can click on or you can use keyboard shortcuts. And this will ensure that the transcription process runs smoothly and produces accurate results. Another tip is for when you have mixed text. So a document that has both handwritten and printed text, like that civil war pension application I mentioned transcript’s, supermodels are the best option for those type.

Nicole (25m 20s):
So if you have any type of document with mixed handwriting and printed text, make sure you try the supermodel. They’re specifically designed to handle that and they can significantly improve the accuracy compared to using separate models. And it saves you time because you don’t have to try to split it up and say, okay, this is for handwritten, this is for printed and like running separate jobs. And then the last tip is about searching Within transcriptions

Diana (25m 44s):
You can also search within transcriptions. And once you’ve transcribed the document, then you can easily search for specific words or phrases using the search function. And so this can be so helpful if you’re looking for specific names, dates, or locations within a large collection of documents. And I know for one of our recent client projects, which was using records in the 1800s in Germany, the researcher put in an entire collection of marriage records and looked for some specific names and really quickly found the right one for the project, which was so exciting because it saved hours and hours of searching through all those documents.

Diana (26m 24s):
So I think we need to start experimenting and then we can think of use cases for our own research. Well some additional resources to learn more about Transkribus and its features, Transkribus has a help center and help center tells you all about getting started, subscriptions and credits, uploading and downloading, text recognition, layout recognition and more. So that would be a good place to start if you have not really started working with it. And then re co-op website also has a lot of information and that gives you some background on who is creating this, why it was created and I think it is always helpful.

Diana (27m 6s):
Then there’s also the Transkribus YouTube channel and as you know, I really like using YouTube to help get started with something that seems a little bit challenging. Navigating a website. And this is a perfect use for a YouTube video to take you through all the different things you can do with Transkribus. And a lot of these are done specifically by the team at Transkribus. So you can use that to get you going and hopefully you can really have a wonderful experience. I know I used it for a pension document to transcribe for my great-great-grandmother. It was 57 pages and I just couldn’t bring myself to do the transcription, but I put it into Into Transkribus and then I used that and combined with ChatGPT or Claude, I used both of them to correct and so I had Transkribus do the first pass and then I had the large language model do the correction and it went pretty fast.

Diana (28m 1s):
Within about three or four hours, I had the entire 57 pages transcribed. So it’s really fun to think of ways we can use these tools. Well, hopefully now that you’ve had a little bit of an introduction to Transkribus, you realize that it can really be a significant time saver as you’re transcribing and searching your documents. Its AI powered transcription capabilities can help you find the evidence you need in those long image files. So we encourage you to give it a try and see how it can help in your research process. All right everyone, thanks for listening and we’ll talk to you next time.

Nicole (28m 36s):
Alright, bye-Bye

Diana (28m 38s):
Bye-Bye

Nicole (28m 38s):
Thank you for listening. We hope that something you heard today will help you make progress in your research. If you want to learn more, purchase our books, Research Like a Pro and Research Like a Pro with DNA on Amazon.com and other booksellers. You can also register for our online courses or study groups of the same names. Learn more at FamilyLocket.com/services. To share your progress and ask questions, join our private Facebook group by sending us your book receipt or joining our courses to get updates in your email inbox each Monday, subscribe to our newsletter at FamilyLocket.com/newsletter. Please subscribe, rate and review our podcast. We read each review and are so thankful for them. We hope you’ll start now to Research Like a Pro.