How to get the errorRate of a model using the API

[lferrazzini95]

Hi,

I am currently trying to retrieve the errorRate of a specific model using the TranskribusAPI.
Using the documentation:

https://transkribus.eu/TrpServer/Swadl/wadl.html

I found the command '/recognition/errorRate' which I think should do exactly what I need. But the parameters are defined as ref and key and I am not sure how I can define those two parameters to access the correct model and document on which I want to evaluate the model .
Thanks a lot for your help!

Best,
Luca

[DRRV]

Sorry this function is not covered by the python api. you should contact the Transkribus team for that. I assume that you need to provide 2 transcription IDs (the reference and the one you want to compare with)

[lferrazzini95]

Okai I will do that.
Thank you for your answer!

[lferrazzini95]

So I just wanted to post my solution for my problem, but I also got another problem resolving from it.
In order to get the error Rate of a model one needs to pass the Key for the transcriptionof the model and the Ground truth.
I get those keys from the following function:

def getDocumentR(sessionId, colid, docid):
        r = requests.get("https://transkribus.eu/TrpServer/rest/collections/{}/{}/fulldoc?JSESSIONID={}".format(colid, docid, sessionId))
        if r.status_code == requests.codes.ok:
            return r.json()
        else:
            print(r)
            print("documentID or collectionID invalid?")
            return None

def getErrorRate(sessionId, key, key_ref):
        """
            This gets the wer and cer for a specific model on a specified document
        """
        r = requests.get('https://transkribus.eu/TrpServer/rest/recognition/errorRate?JSESSIONID={}&key={}&ref={}'
                     .format(sessionId, key, key_ref))
        
        #extract wer and cer from the content
        wer = float(et.fromstring(r.content)[0].text)
        cer = float(et.fromstring(r.content)[1].text)
        return wer, cer

def getKeys(colId, docId, 'GT'):
      doc = .getDocumentR(colId, docId)['pageList']['pages']
      
      keys = []
      #go through all pages
      for page in len(doc):
          #go through all available transcriptions
          for ts in doc[page]['tsList']['transcripts']:
              #if we are looking for the GT we need to check the status 
              if toolName == 'GT':
                  if ts['status'] == 'GT':
                      keys.append(ts['key'])
                      break
              #if we are looking for a model transcription we need to compare the toolname
              else:
                  try:
                      if ts['toolName'][-len(toolName):] == toolName:
                          keys.append(ts['key'])
                          break
                  except:
                      pass
      return keys

keysGT = getKeys(exampleColId, exampleDocId, 'GT')
keysModel = getKeys(exampleColId, exampleDocId, 'ExampleModel')

cer, wer = getErrorRate(keysGT, keysModel)

Note that you need to have the ground truth transcription saved as GT in Transkribus. And you need a sessionId from the Transkribus client.

I hope this will help others. (if something doesn't work please let me know. I copied and chenged the code from my progam which had some more features and therefore it is possible that I forgot something.)

But now to my problem:
I would like the CER and WER to be case sensitive or insensitive. Is this possible to implement?
I know that in Transkribus this is possible. It would be great, if we could have this feature for the API.