Dictionary creation using CSV - William Shakespeare Thesaurus

This is intended for word-geeks who want to look more in-depth at creating a specific dictionary (thesaurus) using CSV format. Let’s just do it by following an example. We are going to make a better Shakespearean thesaurus than we get by just loading TXT book. First, we will start by downloading the The Complete Works of William Shakespeare from project Gutenberg in Plain Text UTF-8 format. Open the txt file, it could be in notepad or in CQuill (Cquill will add it to the project) and remove any text that is not part of the book - that is the beginning “This eBook is for the use of anyone anywhere…” because we don’t want Shakespearean dictionary to know words like ebook, or ‘electronic’ and then also from the end which is a long paragraph about project Gutenberg. “*** END OF THE PROJECT GUTENBERG EBOOK”, because again, we don’t want Will to talk like 21st century lawyer. Now save the file (or export from CQuill to txt (Document Export As…) The standard, easy method is to go to Dictionaries - Synthetize Thesaurus and we will start with that. This would automatically fill the dictionary with as many words as it could guess.

At this moment, there is not much to it, just load the *.txt book you just cleaned in previous step and wait a few minutes. The system will churn the words, and when it is finished, look at the bottom. Check the Save Unknown words to txt file, and then hit Save and name your dictionary. For example myShakespeare.dic. The AI word cruncher created 51 thousand entries with still yet about 12 thousand unknown words. (Your numbers may be different, as there may be improvements in the dictionary over time)

Saving will create the dictionary *.dic file and also save a file called MyShakespeare_unknown.csv.txt This is a txt file formated as standard csv - a comma separated table. You can open it in notepad, or hit the Open button on the dialog and it will open the file for you.

The “unknown” text file consist of two parts No Synonyms part and Unknown/misspelled words following it. The ‘no synonyms’ part are words that we know for sure to be English words. Some are names and they would not have synonyms. Other are a various archaic forms. But none of those are our concern right now. The bottom part of the TXT file does list words that are completely unknown to Vocabulary Synth, either misspelled, or in this case many would be specific to how Shakespeare wrote. Like unwash’d or anchor’d. The weird format you see is the CSV format. Except it is missing the head word, hence it starts with a comma. So line ,(),anchor’d means: missing_head_word,(missing_POS_type),anchord’d

Note: The second item in parenthesis is POS type. JJ means Adjective. In general, POS types for the dictionary can be: • (noun) or (NN) - Noun • (verb) or (VB) - Verb) • (adj) or (JJ) - Adjective • (adv) or (RB) - Adverb In order to fill the missing head-word we would have to write in the document: unwashed,(VB),unwash'd You can certainly do it in the notepad, but CQuill has a pretty awesome spreadsheet document. So why don’t we fire it up! It will make things much easier. Create new Wordsheet, then go to menu Document and use Import CSV. Alternatively you can just copy and paste the lines from the notepad to the Wordsheet and it will format them as a table.

So here we have a huge table (12000 or so lines - yours may vary). I don’t think we are going to manually fill all the missing entries, but we can at least fill some by search and replace.

There seems to be quite a few shortened words where instead of -ed, are ‘d, like unwash’d That’s easy to fix- but remember, we don’t want to replace words on the right side, These are the Shakespearen words that we WANT. We are missing English head-words so we need to create them in the first column. Let’s use the power of Wordsheets.

First select the whole column, by clicking on its header [C]

Now hit Ctrl+C to copy all the cells bellow. We need them in the first column, but for now, let’s just paste them temporarily to some free column on the right side. Just select another empty column and hit CTR+V. Be patient, you are working with 20k lines!

So now we have a column where we can mess with the words and turn them into today’s English. With the [D] column still selected go to menu Wordsheet and select Replace Selection.

We are going to replace ‘d with ed Note the Delete cell check box. This indicates that if there is no match (the word has no ‘d in it) we also want to delete that cell. Now run it. The text in column D will disappear, but wait! Scroll down to find the replaced items.

These are our new head words. Now we can copy the column D over to the column A where it belongs (again select the very top header [D], then CTR+C, , select header [A], CTRL+V), then delete the column [D] as we don’t need it anymore (select the header [D] and press DEL)

Let’s just explain what we had done. The first column is the head word, that is the word you are searching the dictionary for. In Shakespearean Thesaurus, these would be English words. The second is the POS file - the 0 you see in the table is not zero, these are empty () parenthesis. The few lines we successfully replaced would mean that if we search our thematic thesaurus for word like renewed, the thesaurus will offer us renew’d

As you go through the lines you may see that replacing ‘d was not enough. In general Shakespeare seemed displeased with e, but there are also other short versions like: whatsoe’er

We will go about it the same way, copy the entire column C into column D as a temporary place (you can also copy it to a new Wordsheet if this is what you prefer!) and then replace let say ‘red with ered. But after that, how do we merge the newly fixed column [D] with the already somehow populated column [A]? That’s easy, just paste it over as before. If the cell doesn’t exist (and we instructed the replace command to Delete cells that do no match) it won’t be pasted over. A deleted cell does not have an empty string - it has nothing, it doesn’t exist, so it cannot be pasted over anything. Just try it, it works! The same way we can replace ‘ry, ‘st, ‘red, ‘ring, ‘er one at the time, then merge the column with the A column. Now if you look at the A column, there are still some words that may have ‘ in them. If you are going to replace those directly in the column A, make sure you uncheck the Delete cell, so you won’t delete the words you already fixed before! You can manually change things here and there, delete entries that are completely useless and then just save the CSV file by going to Document - Export CSV. You may have noticed that we didn’t care about the missing POS type and that’s because if it is not specified (or a bogus entry, like empty parenthesis we have right now), the import process will try to determine the type itself. You could also experiment trying to replace eth with es. For example accuseth > accuses, but you should look at the results, because it would also replace words like Elizabeth and mess up with other words.

Let’s go back to our Synthetize Thematic Thesaurus. Here we will use the Import CSV button, but before that, let’s talk about the options:

Unlike the TXT import, the CSV has a few peculiar options that don’t say much. To be honest, they were mostly added when we were building some of our thesauri (plural of thesaurus apparently). If nothing else, it could be a note for future ourselves.

CSV Import options: Normal It creates the headword, and fill the entry with the other words on the line in CSV file - so this option creates one entry per line in the dictionary. No trickery. The CVS line: happy,(JJ), jolly, merry Creates: • happy,(JJ), jolly, merry Symmetrical self-reference The idea is that if the word happy has synonym jolly and merry, then both merry and jolly should have synonym happy. This works in many cases, unless the entry is a bit free spirited. For example the 1911 Roget’s thesaurus that we used as part of our General Thesaurus has only some 1400 entries, but each entry has ten, twenty or sometimes more words in them. Symmetrical reference would magically create 40000 entries out of it. Sadly, many would be pretty big stretch, mostly because the 1911 thesaurus takes synonyms more as “somehow” similar words, but not entirely. Hence, this option should be used if you can vouch for the word’s in entry (or for example if there is only one entry like in our case. Both whatsoe’er and whatsoever are obviously interchangeable. The CVS line: happy,(JJ), jolly, merry Creates: • happy,(JJ), jolly, merry • jolly,(JJ),happy • merry,(JJ),happy Cyclical Reference Now, you may be saying: shouldn’t jolly also have merry as synonym? Yes and no. In this case it would work fine and the entry : jolly,(JJ), happy, merry would work. But there are many cases where this wouldn’t be true. The words in the entry are synonym to the head-word, sometimes even symmetrical, but not all of them will be synonyms to each other. We are not going to look through 40 thousand entries, are we? This option would creates gigantic cross-referenced thesaurus with hundred thousand of words in it, but it would be a pretty sorry thesaurus, giving you synonyms that would require a suspension of belief. Now this option still exist, because there is a valid reason, for example if you are building a Rhyming dictionary. If the headword rhymes with the entry words, so should all the entries in some way rhyme with each other. Normally doing something like this would create a huge file (we are talking about Gigabytes), and that would be of no use, so there is another trick - Pointer Reference. It basically creates only one entry per line, but then it goes word by word through the entries and creates a fake - pointer only - headword that points to the original entry. The CVS line: happy,(JJ), jolly, merry Creates: • happy,(JJ), jolly, merry • jolly,(JJ), >[see happy] • merry,(JJ),>[see happy] The file will be very small, but hugely criss-cross-referenced. The downside is that editing would be nearly impossible as most of the entries just point somewhere else. There isn’t much more use besides the Rhyming dictionary, to be honest. Reverse dictionary This is like the symmetrical self reference, except it never creates the original entry: The CVS line: happy,(JJ), jolly, merry Creates: • jolly,(JJ),happy • merry,(JJ),happy Why we would ever want something like that? Yes we do, and a lot of times! Imagine you are building a thematic thesaurus, and let’s take Shakespeare as an example: • haggard,(JJ), wild, unmanageable, untrainable Now you see, building thematic thesaurus with CSV like that actually makes zero sense. If we search for haggard, we will get synonyms wild, unmanageable, untrainable, but that is not what we want at all. That would be a translation from Shakespearean to English. We want to type ‘wild’ and see Shakespearean ‘haggard’. So Reverse Dictionary. In fact we actually don’t want word unmanageable or untrainable as the synonyms in the dictionary at all, because Shakespeare never used these words (He did use ‘wild’ though with many other meanings). Hence the first entry where wild, unmanageable and untrainable is on the right side, like written above will and should NEVER be created in a thematic dictionary, and if you need to have wild there, then you need to define it like so: wild,(JJ),wanton, flighty, frivolous and use Reverse Dictionary option. AI Expand Head Word Here is a bit of trickery, let’s go back to our original CSV file and pick one entry • imprisoned,(JJ),imprison'd The thing is, there are many other words that word imprison’d is synonym to (like jailed) which we don’t have in the CSV. But using this option will create them anyway! It will create entries like: • imprisoned,(JJ),imprison'd • jailed,(JJ),imprison’d • incarcerated,(JJ),imprison’d Granted, we are using a bit of leap of faith here, but in general it works, and creates much more robust thesaurus for free. Let’s just try Normal at first with our CSV file:

Click the Import from CSV and select the file we created in previous step. So we got good 2719 entries (head-words) from our big CSV. As you remember, most of the lines were empty without the head-word so they don’t get imported.

This time it took a bit longer but we somehow created 24 thousand entries. What!? In fact the process created 145 thousand entries, but then many entries were merged and cleaned. We do use leap of faith here, hoping that the synonyms to the head-words are reasonable. In most cases they are, but there could be questionable items. You can’t have free lunch though.

Let’s try AI Expand Head Word option

Ok, let’s test test it. We can do it still in the same window.

Just type word “horrible” into the box and hit search. As you can see, we have a pretty interesting hits. If you tried the same word in the previous (Normal) option, you will get zero hits. Of course we never had word ‘horrible’ in CSV file. So we’ve got a lot for a very little using some trickery. You may try some other words - mostly verbs as that is what we have in the CSV file. Yes, there will be misses here and there, but we’ve got an entire thesaurus build from almost nothing. Wait, that’s actually not our thesaurus! Didn’t we use the Complete Works of William Shakespeare to create the thesaurus? Because we are playing right now only with the unrecognized words!

Hit the save button and select CQuill Partial Dictionary File *.dcc as the type. The reason for *.dcc instead of *.dic is that dcc files won’t be seen by CQuill as viable dictionaries to show in the Thesaurus selection. And that’s what we want, this is still a partial dictionary. Otherwise, they are the same file format.

Let’s sum up where we are: We have myShakespeare.dic file created from the Complete works of William Shakespeare. These are words that the system recognized. Then we took the *unknown file and exctracted a few thousand words that we could fix quickly without going through them manually and we created shakespeare-extra.dcc Now we need to put them together. At this moment I would suggest to make copy of the myShakespeare.dic as it may come in handy. For the next part we use our Dictionary Editor

Fine, let’s save it

Once there, hit the Load button and load the myShakespeare.dic

Now we can hit Merge button and then locate the dcc file we created from the CSV words to Merge the CSV created dictionary with this one.

Now our dictionary grew with additional entries and much more words in those entries. The same way we can add other words, filling the missing entries in the *unknown file, or creating a whole new CSV file from words we can find online.

The result dictionary will be created with William Shakespeare words, but only the ones we recognize today. The process will omit any unknown words (and in the case of Shakespeare, that would be a good 40%!)

Tip: To scroll faster through 20k lines, hold down Shift while turning the mouse wheel.

This dictionary has already a lot’s of words. Let’s try a few:

Now, let’s try again. And here we go - the entry ‘funny’ got a few new words like crook’d.

Despite the fact that we didn’t fill much of the 20K unknown words, the Shakespearean thesaurus is already shaping up. But of course we can fix it even further by trying to locate the missing keywords from web.

To be continued…. We will try to refine the dictionary using Index, Web Crawler in the next chapter

We could already test the dictionary and type a few words in the search entry to see the suggestions. These would be all words that Shakespeare would use, but they are no surprises. This is because the Vocabulary Synth doesn’t fully understand the way Shakespeare wrote and as you can see, a good chunk (12k) words were thrown in the Unknown file. These would be the typically peculiar words, that we could, at least partially, sneak in back to the thesaurus.