Deploying YouTrack on OpenShift for free

How to install YouTrack on Red Hat's OpenShift PaaS service for free

YouTrack is a great issue tracker by JetBrains. Even though the software is proprietary, they allow you to install YouTrack Stand-Alone on your own servers as long as you have at most 10 users.

I attempted to follow this guide to install YouTrack on OpenShift, but it turned out to be trickier than I expected, as I ran into a mysterious bug related to socket binding in OpenShift servers.

So I came up with an easier alternative installation method. It consists in running the YouTrack jar through the JVM without using the Java Service Wrapper by Tanuki Software. The solution is simple and should allow you to have YouTrack up in a few minutes.


1. Sign up for a OpenShift free account and make a DIY app. You can follow the Code Nibbles guide up until section "The repository".


2. Download YouTrack's jar (for Linux) here, should be name youtrack-5.0.1.jar.


3. Place the jar in your local repository, for instance in the root directory of the repo.


4. SSH to your app's server in OpenShift. Type in

~/myrepo$printenv OPENSHIFT_DIY_IP

to discover the IP address that YouTrack should use. Execute also

~/myrepo$printenv OPENSHIFT_DIY_PORT

for the same purpose. Remember these two values, and log out from SSH.


5. Extract or open the YouTrack jar so that we can modify some files in it.


6. Edit the following XML file jetbrains/mps/webr/standalone/runtime/standalone.xml:

...
   **the internal IP address** 
   **the internal port** 
  180000
...

7. Compress all the files that were in the jar. Make sure that the directories structure is preserved. If you don't know what you're doing, I recommend doing steps 5-7 by simply opening the jar file in a common compression tool like 7zip and modifying the files through it, without extracting the jar.


8. Go to directory .openshift/action_hooks in the repository. Set the contents of files start and stop as follows:
start

#!/bin/bash
nohup /usr/lib/jvm/java-1.7.0/bin/java -Xmx512m -Djetty.home=$OPENSHIFT_DATA_DIR -Ddatabase.location=${OPENSHIFT_DATA_DIR}teamsysdata -Djava.awt.headless=true -jar ${OPENSHIFT_REPO_DIR}youtrack-5.0.1.jar ${OPENSHIFT_DIY_PORT} &

stop

#!/bin/bash
APP_PID=`ps aux | grep -v grep | grep java | awk '{print $2}'`
kill -9 $APP_PID > /dev/null 2>&1
exit 0

9. Commit and push the repo to OpenShift:

$git add .openshift/action_hooks/start
$git add .openshift/action_hooks/stop
$git add youtrack-5.0.1.jar
$git commit -m "Deploying youtrack"
$git push

10. When the upload is completed, OpenShift will automatically launch the start script, which means YouTrack is starting up. It takes a while for the initialization to happen, like 2-5 minutes. Grab a cup of coffee and come back to launch your YouTrack installation at the address http://yourapp-yournamespace.rhcloud.com . You should see YouTrack in your browser.

Let me know if this works for you, and I appreciate also improvement suggestions to this method. Enjoy. :)

Easy is not intuitive

We often see developers claiming that a feature is intuitive as in "easy to use". It might be obvious to some, but there is a big difference between "easy" and "intuitive". It is important to recognize the difference because easy is an unprecise term that can be used to label even confusing features.

Stop claiming that your product or its features are intuitive. You must test it first. How do you check whether something is intuitive or easy? Is there a problem not being intuitive? These are some issues we will talk about.

"Intuitive" is something immediately understandable by virtually anyone. And by immediately I mean really immediately. For effortless usability, you want most of your product to be intuitive.

In contrast, "easy" is something that takes a (short) while before the user figures out how it works. Sometimes it is unavoidable to have easy and non-intuitive UI, but you should always prefer the intuitive.

How do you test? One good way I found is to define "intuitive" as understandable even when the user is given only half a second exposure to the subject. Flash something for 500 milliseconds and ask the users if they understood it. Yes means intuitive.

"Easy" is tested in the same way. The thing is easy if users do not understand it within 500 ms of exposure, but they do with at most 5 seconds.

I've made an example test for you. Please play the video without pausing it.

The first picture that flashed is easy, the second is intuitive. Notice that there is nothing intrinsically intuitive about the second, we are just used to see it often to refer to that subject. This is how many things can be intuitive: by convention. There is nothing intrinsically intuitive about the symbol below, but it is intuitive because of convention. It has been used several times with one single meaning.

5 seconds is a long time just to understand some UI in your product. It is bearable, but not desirable. Make sure you check that the UI is intuitive before claiming so, because users could take that as a false promise.

The conclusive hint is:

Go for conventions. Unfamiliar/new elements are rarely intuitive.

HAcid: multi-row transactions in HBase

HAcid is a client library for HBase applications to enable the support for multi-row ACID transactions in HBase. The isolation level of these transactions is Snapshot Isolation, but Serializability is currently supported  as an unstable feature.

HAcid open-source repo at bitbucket.org 

This is the end result of my Master's Thesis at Aalto University. It was inspired by Google Percolator, but is still different in many ways. While Percolator uses locks for controlling concurrency, HAcid is lock-free because it employs optimistic concurrency control.

Using HAcid is straightforward:

import fi.aalto.hacid.*;

Configuration conf = HBaseConfiguration.create();
HAcidClient client = new HAcidClient(conf);

// To enable transactions, use this wrapper instead of HTable
HAcidTable table = new HAcidTable(conf, "mytable");

// Start a new transaction
HAcidTxn txn = new HAcidTxn(client);

// Use HAcidGet instead of HBase's Get
HAcidGet g = new HAcidGet(table, Bytes.toBytes("row1"));
g.addColumn(Bytes.toBytes("fam1"), Bytes.toBytes("qual1"));
Result r = txn.get(g);

// Use HAcidPut instead of Put
HAcidPut p = new HAcidPut(table, Bytes.toBytes("row1"));
p.add(Bytes.toBytes("fam1"), 
      Bytes.toBytes("qual1"), 
      Bytes.toBytes("value"));
txn.put(p);

// Commit the transaction
boolean outcome = txn.commit(); 
// true is "committed", false "aborted"

The algorithms in the HAcid library rely heavily on HBase's single-row transactions, in particular CheckAndPut. One of the tricks employed was explained on this blog already. My Thesis is a complete documentation of the system.

The license is Apache 2.0.
Feel free to comment, ask, fork at bitbucket, etc.

Windows Phone sucks

This post will be focused on my justifiable anger towards this mobile OS.

Microsoft has committed many mistakes with Windows Phone 7 (WP7, or WP7.5, Mango, etc), most of which come from three sources:

• Attempting to recreate/redesign mobile user experience
• Ignoring years and years of good practices in mobile user experience
• Going after Apple's strategies for iOS

While Microsoft has focused a lot on the quickness/simplicity of using WP (and rightfully succeeded in doing so), it has sacrificed many other essential parts of the system. I have tried to enjoy Windows Phone, I even tried to adopt a Nokia Lumia 800, but I soon got frustrated with the dumbed-down works of the system.

The main problem with WP is that they have gone too far trying to make it simple, that it ended up being poor. It feels like the designers understood feature-richness as "complicated user experience" and ended up pruning a bunch of arguably good stuff.

I will start with a popular issue: 

Multitasking and the back button

Windows Phone doesn't really support multitasking. They allow you to have at most 5 apps in a sleeping state. It's not hard to imagine a case when you actually need 5 apps simultaneously open, but this limitation isn't actually the worst part.

The "sleeping state" is clearly bad. One obvious daily example is the following: open Skype (now owned by guess who), start a video chat there with your friend, and then attempt to check a note you wrote in some other note app in the phone. As soon as you press that Home button, you are closing your video chat. Yes, you are terminating it. When you open Skype again, you have to restart the chat. This is a excruciating pain if you want to write down in your notes program something your friend is telling you in the chat, and need to swap back and forth between the two apps.

WP only allows one app at a time to be actually running. The Skype video chat goes to sleeping mode, so it stops the chat and focuses on the note app.

Yes, this sucks.

But it doesn't stop there. The back button just tends to make things really weird, in the sense that no other software's back button works in the same fashion. People would like to feel "at home" with familiar interfaces.

The main problem with the WP back button is that it tries to behave like three different things: a browser's back button, a close button, and a multitask button.
The button even looks like a browser's back button.
If you have, say, 4 apps open, and you repeatedly press back, you will see the apps going backwards in time, in the order you used them. Just like, in a way, your browser does.

And why is this a problem?
Well, because multitasking is actually closer to the tab paradigm than it is to the back paradigm in browsers. And on desktop people use tabs and histories in totally different ways, for different purposes.

You could argue that "it seems so intuitive on the phone, to just click back to the previous thing you had", but the "back paradigm" clearly gets confusing when you are actually swapping back and forth between two apps. Or shall we say... swapping back and back between apps.

To make it worse, you can also do multitasking just with the Windows button, so that you see the home screen every time you want to swap between two apps currently opened. So users can be confused on what is the recommended multitasking style: through the back button or through the Windows button.

And the pain continues because the only way of seeing what is "open" (but unfortunately sleeping) is by holding the back button. What Microsoft designers are telling us is that the back button is responsible for multitasking. Plus, it is also responsible for closing apps. So browser back paradigm + closing functionality + open apps view is all stuffed in a "wannabe intuitive" button. This can get seriously frustrating. 

You have your text message open, half written, and you click back because you want to go back to the other app, but you end up closing your app and going to the home screen. 

Or you want to close your app so you go to the open apps view and you feel terribly disappointed that you can't close apps in that view.

I've asked some friends (familiar with smartphones) to try to close an app given this screen. There's no X button in the corner. You try to swipe an app up and it doesn't close. You swipe it down, it doesn't close. You hold your finger on it, nothing pops up. You just end up quitting. Microsoft could just be telling their customers "hey, just stop worrying, you don't need to close them", but once you get painfully familiar with the 5-open-apps limitation, you will WANT to close some of those, so that the important ones don't end up in limbo.

Or let us say you have the Messaging app on the screen, that you want to keep alive, but for the moment you want to go back to an old open app. Naturally, you try to get the opens app view. If your finger does some mistake of not holding the button for enough time (and fingers will make mistakes), your input will not give the open apps view, and you will end up closing the Messaging app. 

Fear will constantly strike your heart: "what will happen when I press that back button?".

Feature poverty

Among the big players in mobile, WP is probably the most featureless one. 

No bluetooth.

No file system.

No attachments in emails, except photos.

During a call, incoming emails buzz loudly in your ear.

No vibration level settings, it always vibrates really loudly.

No tethering.

No custom keyboard apps.

And the list continues.

But why? 

Probably the WP designers went deep into trying to recreate mobile user experience. But there's also a good portion of blame to put on optimization attempts and the infamous business people at Redmond.

Windows Phone is innovative. Its "home screen" joins icons and widgets into one thing: a tile. They have put a lot of effort into trying to guess what users really want from a mobile phone. In one way, they have succeeded in making a simple mobile OS. On the other hand, it is also too simplistic and naïve.

The problem is that they have mixed up simple user interface with simple functions. You can provide sophisticated procedures using simple user interfaces, but WP designers might have misunderstood that. Sophisticated and necessary functions on mobile have been cut off from the OS in order to try to make the user interface simple. There are some functions that simply all mobile phones should support. For instance, file/media sharing. A mobile OS should make it obvious on how to share a picture or a ringtone to other mobile devices or other devices. 

A designer does not need to be smart to make simple user interfaces for simple functions. But you require a genius designer to make easy user interfaces for sophisticated procedures.

Besides this, it is also possible that Microsoft designers have tried to dictate the preferences of their users. Like Apple does. The difference is that only Apple succeeds in dictating to users how to love the product besides its downsides. Apple makes products so polished that people often don't care if iOS doesn't support bluetooth file sharing. Still, many people around the globe that love iOS want to get rid of Apple's dictatorship, through a magical thing called Jailbreak. Microsoft has forced Bing search down our throats (you can't customize what that search button does) in a similar fashion that Apple forces things. The huge problem for Microsoft is that their products don't have the fame of being worshiped as luxury products, and WP still does not have big enough Jailbreaking communities and softwares.

Windows Phone is more like an attempt from Microsoft to squeeze in their products into the market rather than creating a good mobile operating system. They actually just want people to use Bing, to use Windows and Zune for transferring your pictures to a computer (it's your problem if you use Linux as your OS or you cannot install Zune at the moment), to use Internet Explorer for browsing (everybody knows how bad IE family is), and to store their files in SkyDrive. In Windows Phone, these products are forced, rather than given as an option.

"Screw you if you want an easy and unbiased way to do the things you normally do in mobile phones" is the sad message from Microsoft about Windows Phone.

This is not an operating system, this is Microsoft's self-promoting system.

First and last rows in an HBase table

HBase's Client API is quite limited.
This is because HBase itself must support only simple operations so that the data store can be "cloudy", that is, highly distributed and highly scalable. For this reason the API will probably stay simple.

One interesting question is:
How do I get the last row in an HBase table?

And the disappointing answer is:
You have no direct access to the last row of an arbitrary HBase table.

The word "direct" is important, because technically you can get the last row, in an indirect and inefficient way. Just scan the table starting from any row, and when the scan stops, the last result returned is the last row in the table. This is obviously impractical if your table is even slightly big.

However, if you can afford changing the schema of the row keys of a table, it is efficiently possible.

The strategy is to keep the last row key constant, so you get direct access to it, and to allow the table to grow at the beginning or in the middle. Getting the first row requires a plain scan that stops after returning the first result.

This strategy works well with tables with row keys that are consecutive integers, like timestamps or integers as IDs. Normally, when row keys are integers, the table receives rows in increasing order of the row keys. However, if you can have the table receiving rows always in decreasing order of the row keys, you then have easy access to the first and last rows. This is possible because HBase tables are always sorted by row key. 

The first inserted row should have the highest row key. Subsequent inserted rows are "prepended" to the table.

Essentially, we are turning the table "upside down" with regard to row keys.

Consider an example of a table where rows are comments in a blog post. Row keys are (long) integers, such that the first inserted row has the highest row key, and subsequent inserted rows have the smallest row key when they are inserted. For simplicity, in our example, assume that the largest long value is 20.
The table starts with the first comment inserted:

rowkey   commentmsg     name      date
20       "Nice post"   "Smith"  "May 22"

The next comment added should have a row key that is smaller than the largest row key. For example, the number before 20.

rowkey   commentmsg     name      date
19       "I agree"     "John"   "May 23"
20       "Nice post"   "Smith"  "May 22"

When row keys are integers as IDs, it is convenient to use consecutive decreasing numbers, so essentially we are always prepending the table when inserting a new row.

rowkey   commentmsg     name      date
18       "Cool"        "Roger"  "May 24"
19       "I agree"     "John"   "May 23"
20       "Nice post"   "Smith"  "May 22"

So for retrieving the first row, one just needs to:

Scan scan = new Scan();
ResultScanner scanner = commentsTable.getScanner(scan);
Result firstRow = scanner.next();
scanner.close();

Retrieving the last row is simple, since the row key for the last should be constant:

Get get = new Get(Bytes.toBytes(20));
Result lastRow = commentsTable.get(get);

If you are wondering how to "prepend" for inserting rows, that is the hardest part, but not that hard actually. Basically, one just needs to: (1) retrieve the row key of the first row, and (2) try to insert a row immediately before the existing first row.

Step 1 is simply retrieving the first row, and getting its row key. Step 2 uses checkAndPut to deal with concurrent clients. This is necessary because between step 1 and step 2 another client could have prepended the table, hence changing the knowledge of the first row.

The prepending code should look like:

// Get the first row key (Step 1)
Scan scan = new Scan();
ResultScanner scanner = commentsTable.getScanner(scan);
Result firstRow = scanner.next();
scanner.close();

long prependedRowKey = Bytes.toLong(firstRow.getRow()) - 1;

boolean prependSucceeded = false;
// Try to prepend (Step 2)
do {
    prependSucceeded = commentsTable.checkAndPut(
        Bytes.toBytes(prependedRowKey),
        Bytes.toBytes("colfam"),
        Bytes.toBytes("commentmsg"),
        null,
        new Put(Bytes.toBytes(prependedRowKey).add(
            Bytes.toBytes("colfam"),
            Bytes.toBytes("commentmsg"),
            Bytes.toBytes("I like your post")
        )
    );

    if(!prependedSucceeded) {
        prependedRowKey--;
    }
} while(!prependedSucceeded);

The checkAndPut on line 12 is really important. In one atomic step, it checks whether colfam:commentmsg is null in the row we are trying to create, and writes to colfam:commentmsg if the check was positive. The column you use for checking should be a column that always has a value if the row exists. If you do not have such column in the table, you can just create a flag column called e.g. "exists". The Put that is in the checkAndPut can also be customized according to your needs.


Conclusion:
If your table fits the requirements for this strategy, you have an easy access to the first and last rows of the table. These are really efficient operations, and also safe in concurrent scenarios. The prepending algorithm could suffer from race conditions in extremely concurrent situations, but likely that will not give you large latencies.

Regarding region management and general performance of this approach, there are no apparent problems.

I have been using this strategy in a project I am working on, and it has been unit tested, benchmarked and used without headaches.

From to-do to commit

Hypergrass is a utility for planning your future Mercurial commit messages and for organizing your to-do tasks. This command line tool detects the to-do tasks that you've marked as done, and performs commits using the done tasks as the message.

Personally, I always use a detailed to-do file to guide my development. Often these to-do tasks are small programming tasks that last about 1 or 2 hours to complete. Most programmers that use a DVCS like Mercurial or Git have often felt unsure about what to write in the commit message, reflecting what they have done. They ask themselves "what was it that I just programmed?" then write a couple of imprecise words, and that becomes the commit message. Over time the revision log gets several confusing messages.

With Hypergrass, you avoid maintaining to-do tasks and commit messages as separate things. Normally we have a clear idea what we should program next, and that can be easily written down as a to-do task. You don't need to mentally retrospect to discover what was done. Just mark that task as done and call hypergrass.

It works like this.


1. Make a special file in your repository, the TODO file, 
which should be added to your repository. For detecting your done tasks, Hypergrass requires that the TODO file follows a simple syntax:

+ DONE this is a task that has been completed
- TODO this is a task yet to be done ## this is a comment
- this is a task to be done but not to appear in the commit
>>> 1.1.0 ## 1.1.0 is a version tag

In a nutshell, + or - shows whether the task is completed or not; 'DONE' indicates that the task is shown in the commit message; 'TODO' are tasks that should be eventually marked 'DONE'.


2. Set the Hypergrass TODO file in hgrc
Edit your hgrc to include the following option:

[hypergrass]
todofile = TODO

The value for todofile is the path to the file relative to the root directory of your repository. So if your repository is located at /home/me/myrepo/ and your TODO file is located at /home/me/myrepo/src/TODO, then todofile in hgrc should be set to src/TODO.


3. Edit the TODO file while you develop your code
Keep your TODO file organized, and as soon as a task is done, just mark it with "+ DONE", and save the TODO file.


4. Call Hypergrass
In the terminal, in your repository's directory, just type

~/myrepo$ hypergrass

and your repository will be committed with the done tasks as the commit message.

You can also use the shorter alias, hyg, to do the same.

Hypergrass has a preview option: hypergrass -p
This will show what the commit message will look like, but will not yet commit.

I have been using Hypergrass in my own projects and it has proved to be a good tool to organize tasks, releases, and sprint backlogs. Has worked fine also in Windows command line.

It is currently in alpha, though, so there are no install instructions yet. But these shall come. Take a look at the Bitbucket repository, if you wish!