The eInk reader bug recently bit me again and I tried a Kobo Glo HD. Nice, but too small so I bought a Kobo Aura One. It just arrived yesterday – these are my first impressions.

That screen. is. so. gorgeous. The 7.8″ size complements the great contrast and resolution, and the excellent font rendering algorithm. This is the best eInk screen I have ever seen. I could read this for hours. The bigger size screen is almost perfect for reading technical documents having charts & pictures. I say almost because it lacks color and the PDF experience is terrible.

It’s lighter and thinner than my 8″ tablet, easier to hold with a grippy back texture, and the screen is matte not glossy reducing reflections.

It synchronizes beautifully with Calibre, just like the Glo HD – topics become collections, syncing annotations, etc. And the software is open – you can side-load your own fonts and it renders them beautifully, and there are hacks available to open some closed features.

The touch screen sucks. There’s no other way to describe it. Too frequently it fails to register my touches. I don’t have this problem with phones, tablets, or the Kobo Glo HD. I’ve been trying to use it slower, but that’s a real pain when typing notes into my books.

The auto-dimming backlight is mysteriously broken. It’s fine in a dark room – evenly lit. Then when you turn on the room lights, the backlight gets brighter!? Just when you need less backlight, it gives you more? I set it for 2-3% in a dark room, then when I turn on the lights it goes up to 10%. If I hold it up close to the light it goes up to 30%. Also, when I put it to sleep, the backlight stays on! This is completely backward and wastes battery. And the auto-illumination level is unstable. As I read, every few moments the backlight subtly and slowly pulses dimmer & brighter. This is a distracting waste of battery.

It may be this auto-illumination is lighting up in a lighted room intentionally to give maximum contrast. But that’s such a waste! I can read the screen fine in a lighted room, so I don’t want to waste battery with unnecessary illumination. Especially since the battery is not all that impressive – see below.

You can slide a finger up and down the left side to adjust the brightness. Nice gesture, but it doesn’t work very well. Sometimes it fails to register the gesture (perhaps due to the touch screen insensitivity?), and even when it does, the brightness changes in big chunks, not smoothly, and with a lag. The only way to get just the right level of brightness is to use the menu.

Hue adjustment is a nice feature and works OK, but not great. At full brightness, the colors are even. But I never use full brightness. Below about 30% brightness, which is still pretty bright, the colors are uneven. A distracting greenish shard appears in the lower left of the screen. All that said, this greenish shard is almost unnoticeable when set at a neutral color between blue & red, and fixed there – disable the automatic hue shift.

The battery is under-whelming. It drops 6-8% per hour while reading, with backlight entirely off or set low (less than 5%). This makes for about 14 hours of reading – less than 1 week for me. That battery life is only a little better than the latest Android tablets. If you bring this on vacation, you will have to bring a charger. E-Ink readers should have battery life at least twice this.

Also, there is a problem with auto-sync killing battery even faster. Yesterday morning it was at 100% having charged all night. I unplugged it and put it to sleep. When I got back home from work 11 hours later it was at 35%. It consumed 70% of battery sitting on my nightstand asleep! I believe this is a bug with auto-sync. Yesterday I disabled auto-sync, charged it all night, it is sleeping all day off the charger, so I’ll find out this evening.

Update: I did a factory reset, which wipes the tablet, reloads the OS and updates firmware. This improved the battery. It now drops 1% in 12 hours while sleeping, and 4-5% per hour while reading, with auto-illumination. MUCH better. From what I read in  forums, many – but not all – users are experiencing this problem, and factory reset fixes it most of the time. Yet even after this “fix” the battery still occasionally drops 10-15% suddenly over a 5  minute period, giving 15-20 hours of reading on a full charge. That’s still not enough for an e-reader – if you go on vacation, you’ll have to take a charger with you.

I confirmed this reader is indeed waterproof. However, water does get inside! So after it’s been immersed, water slowly leaks out from the inside, mainly through the corners and the USB port, for the next 30 minutes or so.

So it’s an imperfect device. Is it worth keeping? I need more time with it to decide. Kobo’s terms give 30 days.

Kobo Aura One Summary:

Pros:

• Beautiful Screen
• Sync with Calibre

Cons:

• Insensitive touch screen misses touches, frustrating to use
• Limited battery life
• Problematic auto-illumination
• Uneven illumination hue

General note: E-Reader vs Tablet

There’s one simple reason why I’m not sure this is a keeper: for the same price I can get a good Android tablet. Same size screen and resolution, with color, that can do far more things. Even with reading, an e-ink reader can never be one device for it all. I read my news online using a mix of Feedly and Browser. An e-reader can’t do that. The best an e-reader can do is support pocket, so I can use a tablet or browser to find the news I want to read, push to pocket, then read it on my e-reader. But that’s cumbersome. Why not do it all with one device?

Indeed – why not? Here are the e-reader advantages:

• eInk screen: visible in direct sunlight
• Annotations: Calibre preserves them
• Battery life: 1 week or more instead of 1 day

Here are the disadvantages:

• No color – black & white only
• PDF experience is poor compared to tablet
• Buggy touch screen & laggy response compared to tablet
• Night reading mode – not as nice as tablet, which does white on black
• Single-function – cannot read news, watch video, check email, play games
• One extra device to carry around, keep charged and loaded with books

I can think of 4 modes of human cognition:

• Abstract Reasoning
  • Logic: how all humans universally know that  proof by contrapositive or induction is compelling.
• Causal Understanding
  • Cause and effect: an infant learns in only a handful of trials that something he drops will hit the floor. Without causal understanding, this would require far more trials to learn.
• Spatial Intuition
  • Bigger, smaller, inside, outside, etc. We visualize shapes and interactions in our minds without having to see from every angle or test every combination.
• Pattern Recognition
  • The classic “which of these doesn’t belong?”. Humans are so good at pattern recognition we often find patterns that don’t even exist!

I don’t claim this list is exclusive, but I can’t think of any others. Every form of cognition seems to fall into one of these categories. Yet all forms of AI currently in practical use – neural nets, decision trees, etc. rely on pattern recognition alone. We simply don’t understand the other modes of cognition well enough to formalize them into algorithms.

We don’t even know if they can be formalized into algorithms! It’s possible – though by no means proven – that they might lie outside what Turing Machines can do. The reverse also may be possible – these forms of cognition might boil down in their fundamental elements to a single form of cognition expressible formally, for example as Turing Machine instructions. If so, the fact that we perceive them as different would be a mental illusion.

Either way, the fact remains that we don’t understand these other forms of cognition well enough to formalize into algorithms. Thus, all forms of AI are essentially based on pattern recognition.

Combining these forms of cognition is extremely powerful. A toddler learns to identify cats with only a handful of examples, yet Googe’s best image recognition AI requires millions. By combining spatial intuition with causal understanding and pattern recognition, the toddler learns much more quickly. Relying on pattern recognition alone severely handicaps AI in two critical ways:

• It learns slower, requiring orders of magnitude more training examples.
• Properly trained AI can out-perform humans on specific tasks, yet still performs poorly on tasks that are open-ended or outside its training.

A human toddler is still light-years ahead of AI. Will we find a way to make AI better despite these limitations? Will we break through these limitations – find a way to formalize other modes of cognition into algorithms?

These questions and the topics of human cognition, AI, and Turing Machines intrigue me.

I recently bought a Kobo Glo HD, mainly out of curiosity to see how it compares with Kindle’s e-readers – specifically, the Kindle 4 and Kindle Touch. I bought it new from Japan for $100, which is cheaper than the Kobo web site price of $130. Now that I’ve been using the Kobo for a couple of weeks, here’s what I’ve found.

First I should list my e-reader priorities:

• Screen: contrast, resolution, layout, font rendering. Unlike a tablet that does many things, an e-reader does only one. It should do it in the absolutely best way. The screen should be easy to read for hours: high contrast, high resolution, smoothly rendered fonts, optimal spacing, with the maximum possible user customization: margins, line spacing, font typeface, rendering, etc.
• Book Formats: part of doing only one thing in the best way, an  e-reader should be able to read all popular book formats: at least the big three: MOBI, EPUB, PDF.
• Reference Lookups: sometimes you want to look up a word in the dictionary or encyclopedia.
• Calibre Integration: I use Calibre to manage my e-books. Pushing books to the e-reader from Calibre should be seamless. Automatically synchronizing collections, tags, highlights and annotations is a big plus.
• Free and Open: I mean free as in liberty, not free as in beer. Books are available from many sources; the e-reader should not lock me into any single vendor’s ecosystem, devices or apps.
• Ecosystem: the e-reader’s native ecosystem – selection, pricing, reviews, etc.

Now here’s how the Kobo Glo HD compares with the Kindle 4 and Kindle Touch.

Screen: overall, Kobo is better than Kindle – it has a nicer-looking easier to read page with a more customizable appearance and layout.

Size: Kobo and Kindle both have 6″ screens. The Kobo is smaller in overall dimensions, and lighter.

Resolution: Kobo is much higher resolution: 1448×1072 which makes 1.55 M pixels. A 6″ diagonal screen is about 4.25″ per side, which makes about 18 square inches, which makes 86k pixels per square inch, linear 294 dpi. This is the same as the Kindle Voyage, which I don’t have and didn’t compare. The Kindle 4 and Touch are both 800×600 which makes 480 K pixels, about 27k pixels per square inch, linear 163 dpi. The Kobo has much higher resolution.

Contrast: about the same. I didn’t observe any significant difference in contrast.

Layout: Kobo has far more layout options. Margins and linespacing can be adjusted over a wider range, and with more fine steps. Kobo spaces fonts and full justification more evenly. Kobo has a fullscreen option that eliminates the header & footer on each page, Kindle does not.

Fonts: Kobo has 7 built-in fonts and can render embedded publisher fonts. Kobo can also use any font files you copy to it, so its fonts are unlimited. Kindle has 3 fonts and you cannot add new ones. Kobo also has a wider range of font sizes with finer sizing steps so you can always get the “just right” size. Kobo has a font weight adjustment the Kindle entirely lacks, so you can get each typeface to the right level of light-dark-contrast to suit your eyes. This weight adjustment works only with Kobo’s built-in fonts, not user-added fonts. Kobo also renders fonts more smoothly – both the characters and kerning.

Lighting: Kobo has built-in lighting whose intensity you can adjust by sliding a finger up and down the left side of the screen. It turns off completely at the dimmest setting (slide downward). This blue-ish cast backlight illuminates the screen evenly. Kindle has no backlight. You can get Kindle cases with built-in lights, but they are expensive, run down the battery quickly, and light the screen unevenly.

Page Turns: Kobo lets you set how often it does a full screen refresh on page turns, where Kindle is fixed at 6. Kindle page turns are slightly faster than Kobo – probably because the Kobo has more than 3 times as many pixels.

Graphics: Many books have graphics and images. The Kindle can explode them full-screen for a better view. Kobo can do this only for books in the KEPUB format. This is not much of a problem because Calibre can convert any book to  KEPUB, and when the book is in KEPUB it opens other Kobo features.

Storage: Kobo has 4 GB of space and just over 3 GB is available for storage. Kindle 4 has 2 GB and Kindle Touch has 4 GB. None of them accept SD cards, but e-books don’t take up much space. You can store thousands of books on any of these devices.

Book Formats: The key difference here is that Kobo supports both MOBI and EPUB while Kindle supports only MOBI. Kobo supports EPUB, MOBI, PDF and KEPUB, Kobo’s proprietary format based on EPUB. Kindle supports MOBI, PDF and AZ3, Amazon’s proprietary DRM locked format based on MOBI. Neither has a good PDF experience – better to use a tablet or computer for PDFs.

Kobo has extra features with books in its native KEPUB format. Graphics can be zoomed and panned, it tracks detailed reading stats like your rate of reading and how long it will take to finish the current chapter or book. Also, the Kobo is more responsive, less sluggish and more accurate when highlighting and annotating text in KEPUB books. This is not a problem, since Calibre can convert any book into the KEPUB format, which does not have to be DRM-locked like Amazon’s proprietary AZ3 format.

Reference Lookups: Kobo and Kindle both have a built-in dictionary and link to Wikipedia. Kobo has dictionaries for additional languages and can translate foreign language words. Both have links to Facebook to share with friends what you are reading, but I haven’t used this feature (I don’t have a FB account).

Calibre Integration: both integrate with Calibre, but Kobo is more seamless in two ways. First, Calibre can automatically manage collections on the Kobo based on each book’s Calibre tags (or other Calibre properties). Thus, when you push your books to Kobo, they are automatically organized the same way they are in Calibre. With Kindle you must manage your collections separately. Second, Calibre synchronizes annotations – highlights and annotations you make on the Kobo appear in Calibre as notes in the book details screen. And if you delete the book from Kobo then re-send it from Calibre, your highlights & notes go with it. This is the killer feature for me – I use highlights & notes in all my books and it’s so nice to have them synchronized and stored. Kindle lets you manually copy your annotations as text files, but there’s no easy way to load them into Calibre attached to your books.

Free & Open: Kobo and Kindle both have a DRM-protected bookstore and ecosystem (apps, devices). Yet Kobo is not locked into this as strongly as Kindle is. Kobo has built-in support for Adobe Digital Editions accounts, making it easy to read library books from Overdrive, which most libraries use. As mentioned above, Kobo also reads both EPUB and MOBI files. Kobo’s software is also more open, and independent software developers have used this to create a more seamless Calibre experience. You can easily edit Kobo’s config files to enable a full screen reading mode, add fonts, and other things.

Ecosystem: here is Kindle’s advantage. Amazon’s selection of books is wider and pricing, while often the same as Kobo, is sometimes lower. Amazon has more readers and more reviews which helps when selecting books.

However, people like me who use Calibre to manage our e-books don’t depend on any single vendor’s ecosystem. I get books anywhere – Kobo, Google, Amazon, library, etc. Copy them to Calibre and organize them by topic so the source is on my PC, not out in someone else’s DRM-locked ecosystem or cloud. I convert them to any format I want, to read on any device I want.

Conclusion: If want the best reading experience and you maintain your books in Calibre, Kobo is your e-reader. The one area where Kindle beats Kobo is in the ecosystem. If you don’t want to manage your e-books yourself, if you prefer the speed and simplicity of single-click reading and you don’t mind being locked into a single vendor’s ecosystem, Amazon is the biggest and cheapest.

One-Device Alternatives: I had a Kindle years ago but for the past few years I read on my tablet because it gives one-device to rule them all – from aviation charts to movies to books to browsing, email and games, I can do everything on one device: my 8″ tablet. My tablet reader is Mantano which is a great app, and I synchronize with books on Dropbox. But this reading system has 2 key drawbacks. First, organizing books on my tablet. I’ve tagged all my books in Calibre but this doesn’t carry over to my tablet, where I have to re-organize everything. Second, my highlights and notes in Mantano cannot appear anywhere else. They don’t sync to Calibre and they are fragile – unless I’m careful I can lose them simply by changing where on the device I store the book. Kobo fixes both of these drawbacks. That could be enough for me to switch back to a dedicated e-reader.

While the Kobo suits my needs better than Kindle, the Kobo isn’t perfect. Here’s what I’d like to see Kobo do to make it better:

• Enable it to display images in books to full screen, and zoom & pan – like Kindle does.
• The default system font is too small and not resizeable. Make it bigger. Better yet, add a system font size setting that applies to the home screen, menus, etc.
• Make it faster / more responsive. It’s a bit laggy pretty much everywhere. Kindle is laggy too, but Kobo is a bit more laggy than Kindle.
• Keep it open – don’t lock it down with DRM. This is the Kobo’s key differentiating feature from Kindle.

Cyanogenmod (CM) is the most popular Android ROM supporting the greatest number of devices. Along with CWM and TWRP recovery, I’ve installed CM on 3 phones and 3 tablets, currently run it on 1 phone and 2 tablets. Here are the practical reasons why:

• No bloatware from carrier or manufacturer
• Clean, well organized – Android as it was meant to be. In comparison, stock ROMs like TouchWiz are a hot mess.
• Faster performance, equal or better battery life.
• All Android features supported, even ones your carrier might disable in their ROM (like tethering).
• Root is built-in, turn on or off with a checkbox.
• Easy automatic updates, just like a stock ROM.
• More frequent updates – most daily builds are stable enough for daily use.

I used to list the following, but removed it:

• Long term support – no planned obsolescence.

Originally I listed this because thanks to CM I’m running Android 6.0.1 on my 4-year-old Note 2 phone, long after Samsung & Verizon abandoned it to Android 4.3. However, I removed this point when I recently learned that CM stopped development for the Note 4. I realized that long term support is always fragile. The most you can expect from the manufacturer is a couple of years. With CM it may go longer, but you depend on developers with professional skills working on it for free as a hobby. At least with CM, if it ever is abandoned there’s at least a chance it will get picked up again, if it’s a popular device with developers.

There are also philosophical reasons I run CM:

• If I own a device, I should be able to run any software I want. I applaud carriers and manufacturers for the work they do building reliable software, yet I object to them locking down the software. Customers should have the choice to run any software they want, so long as it plays nicely on the carrier’s network.
• The above point is especially true when a carrier or manufacturer abandons a device. One could argue that as long as they’re providing software, they can prevent customers from running anything else on their network. I disagree, but there’s at least a thread of merit to that argument. But once a carrier abandons a device, they lose any right to tell customers they can’t run their own software.
• Suggestion: carriers like Verizon and AT&T who lock the bootloaders, should issue their final software release on any device with an unlocked bootloader.
• All human institutions are fallible, but when it comes to security and privacy I trust the open source community and transparency more than I trust any single company. Even for companies like Apple who have earned a public reputation supporting security and privacy, it is reasonable to wonder what goes on privately. What backdoors might exist in their ROM, what private data are they collecting and sharing, because some government agency forced them to do it? Since their ROMs are proprietary and close-source, we’ll never know. Prior to 2013, this would sound like paranoia, but events since then have proved otherwise.

Practical tips before buying any phone or tablet

• Check to see if CM is available and how current it is.
• Make sure it has an unlocked bootloader. This depends on the device and the carrier. For example, the Galaxy Note phones are unlocked on T-Mobile, but locked on Verizon or AT&T.
• Remember – “unlocked” is ambiguous – don’t get confused.
  • Carrier unlocked: carriers are required by law to unlock on request any phone that is not on contract.
  • Bootloader unlocked: there is no law requiring this – carrier’s discretion.

Update: not so fast. Today I picked up my phone and certain icons were missing from the home screen. apps stored on the SD card randomly, intermittently disappear from the home screen (even without rebooting, just waking from sleep). And on booting, I get a system modal error dialog and their icons disappear. The apps are still installed and I can run them if I wait a few moments after booting. Turns out it has the same problem my tablet did before. Adoptable storage is still not working as designed.

On top of that, WiFi mysteriously stopped working and could not be enabled, even after rebooting. And the MAC address reverted to 02:00:00:00:00. Looks like CM 13 is not yet ready to use on the Galaxy Note 2. I reverted to my CM 12.1 backups and may try CM 13 again in a few months.

—

A while ago, I had this to say about Android 6 adoptable storage. Android 6 is great, but this feature just didn’t work. No big deal, I simply kept using the SD card the same old way I’ve been using it for years.

That was a custom port of CM 13 to my old 8″ Galaxy Tab 3. Recently, Cyanogenmod released nightly builds of version 13 for my Galaxy Note 2. Yeah, it’s an old phone. I’ve been using the same phone for almost 4 years. But why not? It’s a fantastic phone, still works like new with great performance and amazing battery life.

I decided to give adoptable storage another try, this time with an official build (albeit a nightly build). Long story short: it works perfectly having none of the problems I encountered before.

• The SD card mounts to /storage/emulated/0. This is the same location where internal storage used to be mounted.
• Because of this, apps like the AOSP camera store their data on the SD card, even when set to “internal”. They’re storing data to the same location, not realizing the SD card is mounted there.
• Same with the system Downloads directory – nice!
• Apps set to prefer external storage install to the SD card, as well as additional data they download. All automatically.
• From the system Apps menu, you can change some apps between external and internal storage.
• For generic shareable folders like music and videos, I’ve had no problems with permissions.
• I’ve had no problems with Folder Sync and other apps that read & write the same files across several apps.
• When you plug the phone into a computer via USB, only 1 directory shows up. It’s the SD card. Previously, 2 directories showed up: internal and external.
• File Managers like Solid Explorer still detect the SD card and use it correctly.

Simple advice:

• Use a high quality class 10 or faster card. Cards are so cheap now, the speed and reliability is definitely worth it.
• Use external storage for apps that accumulate lots of data: mapping apps like Sygic and Droid EFB that download entire states or areas, Evernote which can store GB of data locally, Folder Sync which syncs Dropbox and Box to your local device, Titanium Backup, etc.

Overall, it works automatically and seamlessly, simplifying storage space on the phone. I have not had to do any tweaks.

Ubuntu 16 was released a few weeks ago, the latest version and a LTS release, which means long term support (5 years). All even numbered releases are LTS. I’ve been running Ubuntu since version 10 and updated three machines to Ubuntu 16. A year or two ago I switched to the XUbuntu variant because I don’t like the Unity interface and XFCE is faster and lighter on CPU and RAM. My advice is to stick with Ubuntu 14, if that’s what you’re already running. At least for now.

First, if you have a laptop that needs support for power management, you need the version 4 Linux kernel and must already be running Ubuntu 15. Just keep running it. If you have a desktop, you’re probably running Ubuntu 14, which is a solid release and still supported.

Second, Ubuntu 16 has few practical improvements or upgrades that you might notice. The only difference I’ve noticed is that the Openconnect VPN script is fixed; Ubuntu 15 required a route command after connecting; Ubuntu 16 is fixed and does not. Ubuntu 14 never had this bug.

Third, the Ubuntu 16 upgrader is broken and crashes, so if you try to update you’ll have to fix and complete it manually.

Fourth, Ubuntu 16 has a serious bug: a memory leak in the Xorg process. Previously it used 50 – 100 MB of RAM. On Ubuntu 16 it slowly but constantly grows, after a couple of days reaching a couple of GB, until the system starts swapping. You need to log out to kill the Xorg process and start a new one. This bug occurs only on my desktop using open source video drivers. The other desktop with Nvidia binary drivers, and laptop with Intel HD graphics do not have this bug.

Details: I updated two desktops and a laptop, all 64-bit. One desktop has a generic video card using open source drivers. The other has an Nvidia Quadro K600 using the Nvidia binary driver from the Ubuntu repo. The Laptop is a 2015 Thinkpad Carbon X1 with Intel HD graphics. All three were running Ubuntu 15.10, fully up-to-date, before upgrading, which was running great on all of them.

In all cases, I ran do-release-upgrade from a command prompt. It crashed – didn’t fail, but actually crashed – about halfway through, leaving my machine reporting itself as Ubuntu 16, but with apt-get in a broken state. To complete the install, I ran the following sequence of commands:

apt-get -f install
apt-get update
apt-get dist-upgrade
apt-get autoremove

I repeated this sequence until apt-get stopped reporting errors. This completed the install – at least I think so. All the repos seem to have updated, the system is running kernel 4.4.0, system reports itself as Ubuntu 16 and is running fine.

It’s nice to have VPN work without needing an extra route command. But unless you have a burning need to get on Ubuntu 16, I advise waiting for Canonical to fix the upgrader before installing.

When you’re a data scientist…

This is what people think you do:

This is what you spend most of your time actually doing:

Lately I’ve been working on natural language processing, learning as I go. My first project is to discover topics being discussed in a set of documents and group the docs by these topics. So I studied document similarity and discovered a Python library called GenSim which builds on top of numpy and scipy.

The tutorial starts by mapping documents to points in a vector space. Before we map we do some basic text processing to reduce “noise” – for example stripping stop words. The tutorial casually mentions removing all words that appear only once in the corpus. It’s not obvious to me why one would do this, and the tutorial has no explanation. I did a bunch of googling and found this is commonly done, but could not find any explanation why. Then I thought it about a little more and I think I know why.

We map words into a vector space having one dimension per distinct word in the corpus. A document’s value or position along each dimension (word) is computed. It could be the simple number of times that word appears in that doc – this is the bag of words approach. It could be the TF-IDF for that word in that document, which is more complex to compute but could provide better results, depending on what you are doing. However you define it, you end up with a vector for each document.

Once you have this, you can do lots of cool stuff. But it’s this intuitive understanding of what the vector space actually is, that makes it clear why we would remove or ignore all words that appear only once in the corpus.

One way to compute document similarity is to measure the angle between the vectors representing documents. Basically, are they pointing in the same direction? The smaller the angle between them, the more similar they are. This is where cosine similarity comes from. If the angle is 0, cosine is 1: they point in the exact same direction. If the angle is 180, cosine is -1: they point in opposite directions. Cosine of 0 means they are orthogonal. The closer to 1 the cosine is, the more similar they are.

Of course, no matter how many dimensions the vector space has, the angle between any 2 vectors lies in a 2-D plane – it can be expressed as a single number.

Let’s take a 3-D example: we have 3 words: brown (X axis), swim (Y axis), monkey (Z axis). Across our corpus of many documents, suppose only 1 doc (D1) has the word monkey. The other words each appear in several documents. That means the vectors for every document except D1 lie entirely in the X-Y plane – their Z component is 0. D1 is the only document whose vector sticks out from the X-Y plane.

Now it becomes easy to see why the word monkey does not contribute to similarity. Take any 2 vectors in this example. If both are in the X-Y plane then it’s obvious that the Z axis has no impact on the angle between them. If only one is in the X-Y plane (call it Dx), it means the other (not in the X-Y plane) must be D1. Here, the angle between D1 and Dx is different from the angle between Dx and the projection or shadow of D1 onto the X-Y plane. But, it doesn’t matter because this is true when comparing D1 to every other vector in the set. The relative differences between D1 and each other vector in the set are the same whether we use D1 or the projection of D1 onto the X-Y plane. In other words, using cosine similarity they still rank in the same order nearest to furthest.

Another way to see this is to consider the vector dot product between D1 and D2. As a reminder, the dot product is the sum of the products of each vector’s components in each dimension. Any dimension that has a value of 0 in either vector contributes nothing to the dot product. And of course, every vector except D1 has a 0 for the Z dimension, so the Z component of the dot product will always be 0. The cosine of the angle between any 2 vectors Dj and Dk is equal to their dot product divided by the product of their magnitudes. If we normalize all vectors to unit length, the denominator is always 1 and cosine is the dot product.

Because of this, any word that appears exactly once in the corpus can be ignored. It has no effect on the similarity of documents. But we can actually make a stronger statement: any word that appears in a single document can be ignored, no matter how many times it appears in that document. This is a misleading – yet not incorrect – part of the tutorial. It removes words that appear only once in the corpus. It could go further and remove words that appear in only 1 document, even if they occur multiple times.

I haven’t found any explanation for this at all, so I can’t confirm my explanation. But I suspect this is why once-occurring words are often ignored. In fact, sometimes people get better results by ignoring words that occur in more than 1 document, if they occur only in a very small number of docs. The reasoning seems to be that words appearing in a handful of docs from a corpus of thousands or millions, have negligible impact on similarity measures. And each word you ignore reduces the dimensionality of the computations.