Once I was requested to write down a publish, I made a decision to share with the NLP group how studying a programming language may also help Computational Linguists to optimize their on a regular basis work.
Since I began working as a Computational Linguist at Bitext, daily I’ve to take care of massive quantities of knowledge and knowledge and I needed to share my favourite instrument to course of, analyze, and extract info from massive recordsdata: Python.
Python, as a few of you would possibly know, is a freely accessible programming language and it provides lots of functionalities for NLP and information evaluation.
I favor it over different programming languages as a result of it’s broadly utilized in academia, analysis, and business, and it additionally offers the consumer entry to libraries like NLTK, which may be very helpful when the information you might be analyzing is linguistic.
Let me present you a pair features of Python for information evaluation I exploit each day, you will notice that even some Python fundamentals may also help us vastly.
Data Retrieval:
A Python script may also help you to extract from a giant file the particular info that you’re all for. This eases the method, reduces the time, and avoids tedious handbook work.
For example, let’s think about I’ve a giant textual content file in Russian. With simply over 10 strains of straightforward code, I can extract all of the masculine singular types of the previous participle energetic of Russian verbs (not counting irregular verbs).
This script would iterate by tens of millions of strains and provides again a end in seconds, optimizing the method and saving us lots of time.
Textual content processing:
Think about that within the above-mentioned file, we uncover that these varieties now we have extracted weren’t meant to be masculine (e.g: ‘слышавший’), however as an alternative they need to be the female type (e.g: ‘слышавшая’). With this straightforward Python script, we are able to change the masculine varieties into female ones.
In addition to the quick processing time, utilizing a script ensures that there won’t be handbook errors (if we be sure that now we have the right code, after all). The script will undergo the entire file and alter all of the cases in the identical means.
Textual content normalization:
That is very helpful after we get a file stuffed with nice info, however to really use it we have to give it a distinct format. This formatting could also be essential to make it usable by another software program or just to make it extra readable.
For instance, let’s think about we get the flexion of Spanish verbs from a supply that has this messy format:
+o;bailar;1sg;current;indicative;-ar
However with the intention to use it we wish it offered following this format:
Conjugated-form Lemma Particular person Quantity Tense Side
This Python script will learn any variety of strains and return them within the desired format.
Any such change can be not possible to make with most textual content editors if the file is a number of million strains lengthy, however with a Python script, we alter the entire file uniformly in seconds.
These are solely easy examples of what you are able to do with Python to course of and analyze your information, nonetheless, they present the good potential this programming language has.
At Bitext, we use Python to simply work together with Language Processing APIs, take care of large language databases, generate or course of the inflection and derivation of any language and lots of different duties. In case you are all for extra advanced examples full the shape and we’ll ship to you a presentation!
