A familiar AI player, Hugging Face, has entered the (productized) GPT race to provide a free, open-source interface that can chat. This alternative to ChatGPT, called HuggingChat, is the product of the machine learning collaborative that started in 2016 and has raised over USD 160m from investors to date.

The HuggingChat user interface is familiar and simple: enter a prompt and get an answer. Unlike in ChatGPT, there is no need to log in or authenticate your user rights (or pay for a premium version).

HuggingChat will be able to respond to all kinds of prompts. But we narrowed down our testing to translation, a feature also found in OpenAI’s ChatGPT with a high level of accuracy from the start.

For HuggingChat the initial translation results are quite different from those obtained with ChatGPT. Translations can be great, or they may not be rendered at all. Perhaps a matter of functionality and not of how the large language model was trained. For now, however, translation performance based on our tests can be considered inconsistent.

HuggingChat is using the OpenAssistant LLaMa 30B SFT 6 version of LLaMA (Large Language Model Meta AI). Meta AI announced the LLaMA LLM on February 24, 2023. It made the model freely available under a noncommercial license for what it stated should be a focus on research and on solving known problems like bias and hallucinations.

Testing, Testing

The first prompt we used was “Translate ‘Few analyses have rendered conclusive data’ into French,” to which the model gave a long answer (in French after being prompted in English), with a criticism of the prompt as “poorly formulated and not very natural” to explain its own inability to translate the phrase (“Je suis désolé, je ne suis pas certain de pouvoir vous traduire cette phrase en français correctement. Cette phrase semble mal formulée et peu naturelle dans n’importe quel contexte. Pouvez-vous reformuler votre requête pour que je puisse vous donner une réponse plus précise?”).

We then gave HuggingChat a second chance with the same phrase, and this time it rendered a translation, but it was incorrect (“Quelques analyses ont fourni des données concluantes.” = “A few analyses have rendered conclusive data,” which does not convey the same meaning).

Perhaps it was a matter of the prompt not being explicit enough, but it is worth noting the LLM’s behavior of being prompted in one language and replying with an explanation in another language, instead of rendering a translation.

On the Other Hand …

In two subsequent full-paragraph tests for which we used English into French and English into German, the engine performed somewhat better and produced fairly fluent output. The translations still contained a number of odd wordings, errors, and inaccuracies, however. The prompts this time were mixed, one specifying in English that the source language was English (into French), and the other specifying only the target text (into German).

So, in short, the model is capable of translation to some degree. However, sometimes HuggingChat will not translate at all and spew puzzling replies, such as “Envoyer la réponse” (Send the response), or amusing pop-ups when given a second chance at the same prompt and translation, such as “Translation not available for some reason.”

Also, when prompted to translate a longer piece of English source content made up of three paragraphs, HuggingChat stopped working altogether — a bug (or capacity-limiting feature?).

Admittedly, this is HuggingChat’s infancy and our testing is exceedingly limited. But at least in translation and at this really early stage, it’s not yet ready to compete with ChatGPT.