Dictionary creation using Word Index - William Shakespeare Thesaurus cont.

In the previous part we expanded the Shakespeare dictionary with words like unwash'd But there are may other words in the *missing.csv file that could be added. Let’s talk about Word index. When you load a dictionary into the Editor, there is a whole section of functions on the head-word part that works with something called an Index.

What is Index? Those are the words you may find at the end of a book that point somewhere into the book text where such word was used. What does it have to do with a dictionary? Isn’t dictionary in fact an index? Yes it is. But a good thesaurus could have 50 thousand entries. Each entry has many words inside. Just think of word said and all the synonyms like uttered, stated, or even words as such (said, can be used in that meaning as well)

And we could have 50 thousand of such bundles of related words, or even more. That gives us easily millions of words. Yes, many will be duplicates, but that’s how dictionaries are. The point is, if we want to do some operation, such as locating related words on the web, this would then take a huge amount of time. If the software let’s you do it on the entire dictionary then after a few days, you will get pretty bored looking at the screen fetching data, and then you would just stop it. Of course you will get some enhancement to the dictionary but you would have no idea what they were. These could be random words that you don’t even care about. That’s why the Index is there. The index point’s to a very small part of the dictionary, but also the part we are interested in the most. So what is the Index and how do you get one? Well, it could be a few words or it could be a few thousand words. And you don’t need to prepare such index manually. Loading an entire book will populate the index with a few thousand of unique words. (A normal novel could have 7000-9000 unique words)

I think you may see the point already. Working on a thematic thesaurus, such as Shakespearean Thesaurus, we can load the work of William Shakespeare as the Index and thus limit the operations to the words we are really interested.

But let’s not get that far. We already have a lot of dictionary done for us with the Dictionary Synthetizer, which does more complete shakedown of the words. We still have the *unknown* words text file. As you remember, there are two parts to it, No Synonyms part and Unknown words part. In the previous chapter we talked about the unknown words part - the words that no dictionary could give you a good answer. But there was a good chunk of other words, that are missing from our Thesaurus.

We can look at the unknown.csv file again, import it in the Wordsheet. This time, let’s select the bottom part (unknown/misspelled words) and delete it so we will keep only the top part.

The top part has words that are recognized, but no good synonyms were found. This could be a great candidate for our dictionary index! The bottom part has words that the lexicon didn’t recognize at all. We could use that part as well, in theory, but the chance of fetching synonyms are slim. Hence we will work on the top part. Export it to a new csv file.

With our combined dictionary (from the first part) loaded in the Dictionary Editor, load the doctored csv file as our index.

With the Index loaded, we can now use the Previous/Next buttons to browse through the index. And to see it works, the dictionary should give us no result for these words - they were thrown out after all.

Now let’s see what we can do. Press the Lookup Single entry button. This will fetch results from various places on the web. But do not save this!

Before you do anything, let’s just click Delete Entry button a few times to delete all the entries we just got. (Or use delete Head-word) Why? Because this is entirely not what we want to do! We don’t want to put swooning (Shakespearean word) as the head-word and then fainting (non Shakespearean word), as synonyms. That would be a translation dictionary not a thematic thesaurus. We want just the opposite!

One of the function on the toolbar says Reverse Index Lookup. And that is exactly what we need. It grabs word from the Index, then find synonyms and reverse them. It puts those synonyms as head- words (fainting) and the Index word (‘swooning’) as the synonym entry.

In general, this requires a bit of leap of faith. We assume that those synonyms and head-words are reversible. This would not be always true for a general terms, but here, the words we loaded as Index are words that were already thrown out as not very common. This is where the function works best - on an index that is very specific to the theme. If you use it on a very generic index you would then get very questionable entries since for generic terms the head- word and synonym are not always reversible, giving sometimes ridiculous results. Let’s stay on the word swooning and use this option and then answer no when the question will pop up about rewinding the index, because we wan’t to work only on this word. After the word is processed, stop the fetching other words and then go back to the ‘swooning’ by searching for it.

There is a generic Clean Up button that would take care of multiple entries, but leaving swooning in both verbs and nouns. Or we can use Deep Clean, that would also clean multiple entries across the type, leaving us with only with a single entry (in this case).

Swooning is right there, and not just once. Now we need to clean it.

It is quite possible that you created a grandiose Thematic thesaurus, only then ruining it by accident with entries that do not belong there. And suddenly William Shakespeare gives youi suggestions like ‘electronic’ or iPhone. No problem, you don’t have to start all over.

It did happen a few times to us too. “Remove all synonyms that are NOT in index” is just the exact tool to fix that. In case of our Shakespeare Thesaurus, we would load the entire “Complete works of William Shakespeare” as our index and let it proceed. It would take a bit of time, but it would remove any wrong words, yet still leave it searchable with modern terms head- words.

To be continued…

Wait, it is not there! And that’s a good thing. Because we didn’t want swooning as head-word. Now search for fainting.

With this knowledge, we could let the Reverse Index lookup run on the entire index. It will take time to fetch all the data. The best time would be probably let it run overnight.

Knowing that you could fix over-zealous thesaurus later, you could refine certain parts of the dictionary without too much planning. At the end you could always cut off the wrong words.