Offline Stack Overflow Access

I’m proud to announce that I’m working on bringing offline Stack Overflow access to Dash, based on the Stack Exchange Data Dump.

The upcoming Stack Overflow docset will let you:

  • Search all questions directly from within Dash, just like any other docset
  • View comments and answers for any question, without any Internet access
  • Waste around 20GB of disk space

If you’re short on disk space, don’t worry! I’m also working on an online-only version of the docset. This version will let you search Stack Overflow using Dash’s blazing fast search engine, but the content will come from stackoverflow.com instead of your drive.

Beyond Stack Overflow: I’m also working on bringing support for all of the Stack Exchange sites, especially my personal favorite – cooking.stackexchange.com.

This is NOT an April Fools’ joke. If all goes well this will come with the next update of Dash.

Dash 2 Now Available

Dash 2 has been released and is now available for free to all Dash users.

What’s new?


  • Make awesome cheat sheets and contribute them to the cheat sheet repo
  • Install docsets from package managers:
  • Lovely new icon, thanks to Reda Lemeden
  • Dash will modify the Alfred workflow on-the-fly so that it includes docset and search profile keywords. This means you can now type php {query} in Alfred instead of dash php:{query}
  • Added Grunt, Mongoose and Rust docsets
  • Lots of minor bug fixes and lots of new bugs!

What’s next?


  • Short term: A few updates to fix any bugs that might come up and add a few new docsets.
    • Note: I’m not currently aware of any unfixed bugs, so if you find any please contact me.
  • Long term: Annotations!

Thank you!

A very warm thank you to all Dash users. Your support is highly appreciated and have helped make Dash awesome!

Please do not hesitate to contact me regarding anything, I reply to all emails as soon as I get them.

Dash for iOS, Android, Windows or Linux

I get asked a lot to bring Dash to other platforms. That won’t happen, because:

  • I’ve got a lot to add to Dash on OS X and I can’t focus on any other platform
  • I’m a complete novice when it comes to developing for any other platform, so I wouldn’t do a great job

Looking for devs to bring Dash to other platforms

I am actively looking for developers of other platforms (iOS, Android, Windows or Linux) that would like to work on a Dash-like app, as their own project and for their own profit.

Later edit: I have found Android and Windows devs and Dash-like apps for these platforms should be released sometime in 2014.

What you’ll make:

  • An API documentation browser app for your favourite platform
  • You can give it your own personal twist or base it on Dash as much as you want
  • Release it as commercial, free or keep it for personal use, I don’t care

What you get:

  • My help, as much as I can
  • Free access to all of Dash’s docsets to be used by your app
    • This includes docset updates and I’m also covering the hosting costs

What you won’t get:

  • Access to Dash’s source code

What I want in return:

  • A link to Dash, on your app’s presentation website and inside the app

Some notes:

  • Don’t start working on this without contacting me first
  • I’ll give exclusivity to Dash’s docsets to whoever looks most promising
  • This is not a weekend project
  • Making an awesome documentation browser takes time

In case you’re considering making this a commercial project:

  • You really should. Dash is my only source of income
  • I really think this is a huge opportunity for indie devs
  • I have no idea how much money you’ll make
  • Dash made around $100 / day during its first versions
  • Dash now makes anywhere between $400 – $1000 / day (avg. is around $600)

Indie App Development as a Job

About a year ago I graduated from the University of Birmingham. Instead of getting a job I chose to make OS X apps instead. I’m very happy with my decision so far.

I struggled for the first few months until I had a change of perspective: I started viewing my apps less as a startup and more as a job.

My guidelines

  • Think of each app you make as a lifelong career and not as a one-time thing.
  • Don’t make apps to get rich. Most likely you’ll never hit the jackpot.
  • Make an app that adds value to your own life. If it fails, you still get to use it.
  • Make an app that you’ll love working on.
  • Build a steady income. Don’t rush it.

It’s a bit weird not to try to hit it big, but you’ll end up being free to make something you love, which in turn might end up being a recipe for success.

Top API Documentation Sets

So far, 157188 users have downloaded API documentation sets while using Dash, my documentation browser app. The chart below shows the unique download counts of individual docsets.

The data is collected by analysing the server logs and counting the unique IPs that have subscribed to a docset feed. As Dash is an OS X-only app, this data is just a slight indicator of what OS X developers need documentation for and nothing else.

One thing surprised me: over 60% of Dash users had a need for web development docsets at some point in time.

Note: The OS X and iOS docsets are missing from this chart because they are not hosted on my servers (Apple hosts them).

The Cheap App Mentality

There’s a lot of debate regarding App Store pricing and most of it revolves around the idea that developers should try to spread awareness about how hard it is to make an app. I think that’s wrong.

The main problem is that instead of evaluating the actual product, some users evaluate how much they think the developer is worth for his work. These users are not the majority, but they are the users that bring prices down and affect the sustainability of otherwise great apps.

Knowing how hard it is to make an app is not the issue.

Dash is an app that is targeted at developers. Developers are my customers. Despite of that, some of them complain that the app is overpriced at $20 and that I should go for $5 instead.

It’s hard to believe that even a novice developer doesn’t know how much work goes into an app, so lack of app-making knowledge should not be the reason of complaint in this case.

Later clarification: users that complain are definitely not the majority, Dash is currently doing great and is sustainable.

Where it all started

The App Store made a lot of indie devs insanely rich, and some of them didn’t deserve it (e.g. fart app devs).

These insane success stories are the ones that reached the news and some users started changing their mentality from “is this app worth it” to “is this dev guy worth being rich”. The answer is often no.

The general consensus is that if you have an app on the App Store, you’re rich.

The reality is that indie devs rarely make it big and most of the time they pour their heart and soul into an app that won’t even cover the electricity bill.

My app is fairly successful (constantly in the Top 200 Grossing of the Mac App Store). However, I’d make quite a bit more money if I’d just get employed. I don’t want to, because I enjoy what I do and I’m grateful to my users.

How do we fix it

We can’t. It will only get worse.

Linux Man Pages in Dash

Dash works great as a man page browser, but I sometimes get requests to make extra docsets containing the man pages of various flavours of Linux.

I’ve decided not to pursue these requests, because:

  • Updates for these docsets would be a nightmare, as man pages change a lot, individually.
  • I’d have to choose which man pages to include and which not to. I’d never be able to guess which obscure man page a user might want.

The current Man Pages docset solves these issues by indexing the man pages that are actually on your Mac.

The workaround

You can copy the man pages from any Linux box to your Mac and Dash will index them as part of the regular Man Pages docset.

Step by step instructions:

  1. Log into your Linux box
  2. Run man -w to list the folders that contain man pages
  3. Copy these folders to your Mac
  4. Optional, but highly recommended: use a batch renamer to rename all of the man page files to have a common prefix. This will help you differentiate between the default OS X man pages and the Linux ones
  5. Move the man pages to somewhere on your MANPATH, or any folder from man -w

That’s it!

A Sneak Peek at Dash’s Future

Disclaimer: This is the current plan for Dash. Plans can change. Please purchase Dash for what it is now and NOT for what it might become.

Tabs

Current status: almost done
Release estimate: soon

Tabs have been implemented and are currently in beta testing. Please join the beta and help test this feature!

Later edit: Tabs have been released with version 1.9.0 of Dash.

New icon

Current status: started
Release estimate: before the end of time

Reda Lemeden is working on a new icon for Dash. The icon is meant to highlight Dash’s two main features: documentation and speed.

Feedback is greatly appreciated! Here’s a preview:

Docset repositories

Current status: planned
Release estimate: 4-10 months

The general idea is for users to be able to search in Preferences > Downloads for their favorite packages and easily install docsets for them.

In the initial release, the following repositories will be supported:

Further repositories will be added incrementally: GoDoc, Haskell’s Hackage, node.js’s npm and Perl PODs.

Annotations

Current status: postponed
Release estimate: before the end of time

Users should be able to extend the documentation pages, publicly or privately, individually or as a team. For example, users might want to add annotations about common pitfalls while using a certain class or method.

Users will be able to create, edit and view annotations and vote for the most useful ones.

Some work has been done for annotations, but currently it has been postponed in order to work on other features with higher priority. I expect to start working on this again after “Docset Repositories” is implemented.

How it will work: a table will be shown on the right side of the documentation page. The table expands when you hover your mouse over it. Scrolling is synced between the documentation page and the annotations table.

That’s it for now

Any and all feedback is highly appreciated. Please join the beta and help test the tabs feature right now!

SQLite FTS contains and suffix matches

SQLite is used by Dash to search through docset indexes. Originally, Dash used LIKE queries which were fast enough, but became increasingly slower as more docsets were added.

SQLite FTS is amazingly fast, but allows only prefix (e.g. query*) matches by default. For Dash, I needed to persuade it to also perform contains matches (e.g. *query*) or suffix matches (e.g. *query).

How it works

It’s simple, for each term I want to be able to search, I store all of its suffixes.

First of all, the table structure:

1
CREATE VIRTUAL TABLE searchIndex USING FTS4(suffixes)

Add the term NSString:

1
INSERT INTO searchIndex(suffixes) VALUES("NSString SString String tring ring ing ng g")

Search using suffix queries:

1
SELECT * FROM searchIndex WHERE suffixes MATCH 'string';

Or contains queries:

1
SELECT * FROM searchIndex WHERE suffixes MATCH 'str*';

Downsides

The only downside I could find was that the database got too large. To avoid this, I compress the data into its actual term.

The compress and uncompress functions behave in this way:

1
2
3
4
5
compress("NSString SString String tring ring ing ng g")
-> NSString

uncompress("NSString")
-> NSString SString String tring ring ing ng g

This compression reduces the database size to what it would be if only the actual terms were added (without all the suffixes).

Speed results

Searching over 1,110,381 terms (in 102 docsets) using contains queries:

1
2
3
4
5
Search for "string" using LIKE:    3.22 seconds
Search for "string" using FTS:     0.18 seconds

Search for "s" using LIKE:         6.87 seconds
Search for "s" using FTS:          0.22 seconds

Alternatives

I chose SQLite FTS because I was already familiar with SQLite and I also needed to work around some Dash-specific edge cases (e.g. how symbols are treated).

Depending on your project, these may be suitable alternatives:

  1. PostgreSQL’s wildspeed module
  2. For OS X or iOS apps: Search Kit

A Poor Man’s CDN

Hosting large and often-downloaded files can be tricky, especially when you want users to have decent download speeds and 100% availability. This is the story of how Dash’s docsets are hosted.

First, some data:

  • At the time of writing, there are 102 docsets hosted
  • The total size of these docsets is 1.5 GB (while archived)
  • Bandwidth requirements are in the range of 5-7 TB / month
  • It would cost about $600 / month to host them in a “regular” CDN (e.g. Amazon CloudFront). In contrast, my hosting only costs $20 / month (thanks to 4 VPSs from DigitalOcean)

Hosting the docsets

Some docsets are somewhat large, so download speeds need to be decent. This is achieved by hosting the files in different data centers:

  • 2 mirrors in New York (for North America)
  • 1 mirror in San Francisco (for North America and Asia)
  • 1 mirror in Amsterdam (for Europe – or at least Western Europe)
  • Extra mirrors can be added in less than 2 minutes to account for spikes

South America, Eastern Europe, Africa and Australia are not directly covered, but should still have alright download speeds, as no one complained yet. More mirrors will be added whenever DigitalOcean opens more data centers.

Load balancing

Dash performs latency tests on all available mirrors by loading a small file. The mirrors are then prioritised based on latency. Whenever a mirror goes down, Dash notices and avoids it. Mirrors that have almost the same latency (±0.03s) are considered equal and are chosen randomly.

This setup results in 100% uptime and really cheap bandwidth costs. I highly recommend you consider a similar setup if you need to host large files for your app.

Hosting the docset feeds

The docset feeds are just small XML files which Dash polls to check for updates. These files are requested a lot, on each Dash launch and every 24 hours afterwards. As each docset has its own feed and most users have more than one docset installed, about 320k HTTP requests are made each day.

These requests are easily handled by a nginx web server on a 512 MB VPS in New York and are also mirrored on GitHub. I tried using Apache but it would sometimes use over 1 GB of RAM while hosting these files and would end up completely failing, while nginx serves requests faster and uses less than 40MB of RAM. I’ll talk about my experiences with nginx in a future post.

Whenever Dash needs to load a feed, it launches 2 threads which race to grab the feed (from kapeli.com or from GitHub), whichever thread finishes first wins and its results are used. Most of the time, the kapeli.com thread wins.

The chances of both kapeli.com and GitHub being unavailable are very very small, so this approach resulted in 100% uptime so far.