Monday, May 11, 2015

What the tag line from Highlander can teach us about Exlusion Constraints

If you are trying to get your head wrapped around exclusion constraints via a Google search you may get the impression that the only thing it's good for is solving the double booking problem common to reservation systems. The PostgreSQL documentation expands the possibility a teeny tiny bit by mentioning preventing overlapping circles. But that's it! So here is this truly phenomenal feature that thanks to an absence of varied examples you may not think you have a use for.

I'm here to expand the possibilities by introducing what I call the the Highlander use case. But before I do I want to explain why all the examples you find on Google are about the double booking problem. The fact is prior to the existence of exclusion constraints the double booking problem was a really really really really hard problem to solve in the relational database world. And if you managed to solve it, chances are you really didn't, and it was slow at scale because of all the locking required. So when Jeff Davis brought exclusion constraints to PostgreSQL all the write-ups naturally focused on the fact that there was finally a safe and scalable solution to the double booking problem.

What are exclusion constraints? Before we answer that let's start by explaining what a constraint, in general, is. A constraint is a rule expressed as statements that evaluate to a Boolean value (i.e., true or false). In essence a constraint is simply a Boolean expression. Constraints are normally checked when you try to INSERT a new row into a table or try to update an existing row. If the expression evaluates to true the INSERT or UPDATE is allowed to proceed, otherwise it fails and an error is raised.

The 4 constraints that just about everyone who has created a relational database is familiar with is:
  1. NOT NULL - A column cannot accept NULL as a value.
  2. UNIQUE - A column or combination of columns cannot have duplicates.
  3. PRIMARY KEY - Combination of #1 & #2.
  4. FOREIGN KEY - A column (or columns) in Table1 must have a matching entry in Table2.
An exclusion constraint is just a supped up unique constraint in that it has a lot more expressive power to define exactly what unique means. Now on w/ the use case.

Let's say you sell a system where owners create events and their customers attend the events. One of the features you probably want in this system is the ability for customers to provide feedback on their experience. So basically you want surveys.

Lets assume that there are more than one type of event, therefore you may want to have more than one survey and would like to associate specific surveys w/ specific events. Let's visualize that a tiny bit:
Event_1 -> Survey_1
Event_2 -> Survey_2
Event_N -> Default Survey.

Note Event_N and Default Survey. Event_N is any event that has no explicit survey associated w/ it. And when the system sees one of these it needs to send the default survey to the customers that attended those events. So that means we need a way to mark a survey as the default survey. (Hold on to your hat folks the Highlander reference is almost here!) And since, logically speaking, "there can be only one" default survey it would be great if there was a way to express that at the database level because we can't depend on the application nor people not to violate the rule that there should be only one default survey.

So let's quickly summarize what the Highlander use case is by comparing it to the standard unique constraint. With a standard unique constraint there can be no duplicates, period. But in the Highlander use case, you take one value and make it special by saying you can duplicate anything else, but not this; for this, "there can be only one"!

Exclusion constraints makes addressing the Highlander use case trivial. Let's look at some SQL.
CREATE TABLE surveys (
  id SERIAL PRIMARY KEY,
  name VARCHAR(64) NOT NULL UNIQUE,
  survey_questions JSONB NOT NULL,
  is_default BOOLEAN NOT NULL DEFAULT FALSE

The is_default column solves the identity problem. Next we'll finish the SQL off w/ our exclusion constraint:
...,
  EXCLUDE (is_default WITH =) WHERE (is_default)
);
EXCLUDE (is_default WITH =) is equivalent to having a unique constraint on is_default. But that's not enough, for the same reason that a unique constraint doesn't solve the problem. So we need more expressive power. Exclusion constraints gives us that expressive power by allowing us to specify multiple comma separated criteria in the EXCLUDE block and we can specialize even more by adding a WHERE clause. So that's what we've done here. In layman's terms our exclusion constraint says, "the value of the is_default column of the row being inserted or updated must not equal any other row's is_default column value, but only when is_default IS TRUE". This means we can have a gazillion rows where is_default IS FALSE but only one row where is_default IS TRUE.

If you have a good grasp of SQL's syntax in general and PostgreSQL in particular, then the exclusion constraint syntax should be obvious. But if you are thrown by the double use of is_default let me explain.

The first one is a reference to a column's name. In this case the column that we want to reference is named "is_default". The second one needs to be understood in the context of how WHERE clauses work. All WHERE clauses must be Boolean expressions, and since is_default is defined as a Boolean it's by definition a Boolean expression too. So we take advantage of that fact and make it the entirety of the WHERE clause. But if we wanted to be verbose we could have written the WHERE clause as:
WHERE (is_default IS TRUE)
or as
WHERE (is_default=true)

[UPDATE:]Thanks to an anonymous commenter I've learned that PostgreSQL is even cooler than I gave it credit for because the Highlander use case is better solved w/ a pgified unique constraint. Basically, PostgreSQL allows us to apply a WHERE clause to your basic unique constraint, giving us the same expressive power that I was using an exclusion constraint for, but w/ better performance. The lesson here is simple, if you are coming to PostgreSQL from another database or have been living the lie of using only spec compliant syntax, read the PostgreSQL docs carefully because chances are PostgreSQL improves upon the standard in extremely powerful ways (a.k.a, pgified).

For completeness let's include the new version of the DDL:
CREATE TABLE surveys (
  id SERIAL PRIMARY KEY,
  name VARCHAR(64) NOT NULL UNIQUE,
  survey_questions JSONB NOT NULL,
  is_default BOOLEAN NOT NULL DEFAULT FALSE
);
CREATE UNIQUE INDEX ON surveys (is_default) WHERE (is_default);

Saturday, October 11, 2014

JavaScript and the DOM, The Rent is Too Damn High!

Let x be a HTMLCollection of elements w/ a particular class name. Let's call it foo.

Let y be an Array of elements which need to have foo applied to it.

The algorithm is roughly:
  • For all elements in x remove the class name via Element.classList.remove().
  • For all elements in y add the class name via Element.classList.add().

Now let us assume that there is some overlap between the elements in x and y. It turns out that for some x and y it can be more than twice as fast to compute the intersection of x and y and to use that set to avoid adding and removing foo.

What makes this so weird and interesting is the fact that computing the intersection is O(x.length * y.length), while the algorithm w/o the intersection is O(x.length + y.length). In other words, the version of the algorithm where we don't compute the intersection should be faster than the version where we do. But that's not what actually happens. It's slower, 2x slower in my tests.

The only explanation I can come up w/ is that crossing the JavaScript/DOM divide remains extremely expensive compared to staying on the JavaScript side of the divide. Thus, the rent is too damn high!

Thursday, July 28, 2011

Warped Logic & the Open Web

I really want to shout loudly about what's the most annoying thing to me in Google Chrome that Firefox gets right but won't cause they might fix it then Firefox would [currently] have no usability advantage over Chrome and I want openness/Firefox to win. BTW, winning doesn't mean, let there be one ring to rule them all. Winning does not mean domination by a single corporate interest over the browser and thus access to the web. This must NEVER happen again (I'm talking to you Microsoft/IE6). There MUST always be an open and free (as in speech) kick-ass alternative or two available. The "kick-ass" bit is really important because people behave more pragmatically than idealistically and if the free (as in speech) & open offering(s) is/are not compelling freedom dies and bondage resumes.

Monday, May 16, 2011

Liar! A.K.A This one is for my PGEast 2011 peeps


If you attended PGEast 2011 then you know that a certain company had tons of coffee mugs everywhere. They couldn't give them away even though they were giving them away. It got so bad that the conference organizer thought that maybe the attendees didn't realize that the mugs where for them to take so he made an announcement that the mugs were for us to take.

So let's recap. PGEast 2011 almost 2 months ago & ungodly number of unwanted coffee mugs from NoSQL vendor who will not be named.

So today I start reading this blog post. Then I get to the 15th paragraph and PGEast comes crashing back because I know EXACTLY who he is calling out!

I just loved the way the author called them out w/o naming them, yet absolutely naming them.

Finally, I strongly encourage you not to skip ahead to the 15th paragraph because you need the first 14 to get the full effect of the none name dropping of names in 15.

Monday, June 29, 2009

Breach

I am an IT Consultant. Exactly what I do depends on who I'm working for because I'm capable of doing many things. The reason I'm being explicit about being a consultant is because I want you to understand that I don't have a single employer. I have clients, and depending on the contract sometimes their problems become my problem. This weekend was one of those times.

A system was compromised and for a brief period of time bad things may have happened in a client's name. Though the process of identifying, diagnosing, and fixing the problem took me less than an hour the ramifications were more far reaching than I could have imagined. It could have been a Titanic moment.

I really want to write about the who, what, where, when, and why of this experience but I'm not at liberty to do so. But one day I will be, so I'm putting this out there now as a reminder to myself to finish this conversation.

Monday, June 22, 2009

Eli's Dirty Jokes

The, Eli's Dirty Jokes, videos on YouTube are absolutely hilarious.

Monday, June 15, 2009

Stuck!

Even though the bulk of my professional writing is technical, meaning the data is mostly available I just need to put it into a format understandable by my audience (aka, the people writing the check), I still sometimes encounter writer's block. Like right now. What the hell does that mean exactly? Because here I am writing this blog entry and I'm not having any trouble finding the words to express my frustration. Maybe I'm just naive in my understanding of what technical writing demands of an author. Maybe it requires the same level of access to the creative mind as a non technical work? Who knows? Today I'm not offering answers. I'm just putting pen to paper in an attempt to find my way.

Thursday, June 04, 2009

Quote of the Day, 4 June 2009

Me? I'm going back to running the browser on my UNIX box. It's way too frustrating trying to be a UNIX Engineer via the Windows platform.

Tuesday, June 02, 2009

PGCon2009 Summary

I'm back from PGCon2009. No, I didn't just get back. I've been back for a smidgen over a week now. When I first got there I decided I would blog daily about it, but time didn't permit me to write in any detail. So I decided I would make notes and summarize it all when I got back. The notes thing didn't pan out thanks to twitter. It was simply easier to tweet my thoughts as I thought them than collecting them in little text files and revisiting them later. The summary idea didn't work out either because (a) I didn't have any notes, and (b) all the good stuff was already said on Planet PostgreSQL. But it's been more than a week since the conference and people are still posting summaries so now I feel like I have to say something. Here goes!

Great conference, awesome people, awesome community, and PostgreSQL is really cool technology that I'm confident in trusting my [clients'] data with.

My only gripe with the conference was the keynote address. It's nobody's fault really. The original speaker couldn't make it so they had to slap something together at the last minute. It showed. At the time, my thought was, if the rest of the conference is like this it's going to suck!

...

As I write this I've just realized that the keynote incident is a metaphor for the larger PostgreSQL project. If you attended or watched the "How to Get Your Patch Accepted" talk, it is apparent that quickly slapping something together is not how the PostgreSQL code base is developed or maintained. So I shouldn't have been surprised that the keynote was not a valid indicator about the rest of the conference. I hope I haven't disparaged any of the speakers. They did their best and everyone else in the auditorium enjoyed their keynote.

Wednesday, May 27, 2009

My Apologies to opensolaris.org Blogs

For the last three years all my blog posts have been showing up at http://opensolaris.org/os/blogs/, even though a big chunk of them had nothing to do with Solaris. At the time, I had no idea how to filter just the Solaris specific entries. Now that I think about it, I don't think Blogger even supported tagging back then, and without tags, filtering would have been impossible. Nevertheless, I apologize.

Yesterday I learned how to filter blog posts and today I've finally rectified the problem. There will be no more posts about how to make a great hot dog, book reviews, cloud computing notes, a rant about twitter, and other silliness.

Tuesday, May 26, 2009

PGCon2009 Postscript: Unit Test Your Database!

I started watching some of the PGCon2009 videos that I didn't attend while at PGCon. Last night I watched, Unit Test Your Database!, by David Wheeler. I have had my come to Jesus moment on unit testing years ago, so I'm really happy that there is a solution for testing strictly at the database level without depending on the application layer.

Some developers make the mistake of treating the database as a glorified file system and therefore assume it doesn't need any testing. They are wrong! From the application's point of view, the database should be a black box and application level testing of the database should be limited to the interaction between the application and the interfaces the database exposes, like stored procedures and views. In this development paradigm the database is an independent entity and therefore needs its own set of tests to ensure that it's self consistent. This is where the speaker's own testing framework, pgTAP, excels. It's not limited to just testing the public interfaces the database exposes. It allows you to validate the database itself, i.e., verify the structure of the tables in the database and their relationships, verify triggers, verify the existence of indexes, verify the existence of functions, verify the behavior of functions, etc.

If you are still on the fence about whether unit testing your database is a worthwhile endeavor then watch the video, it may convince you. But if you've already got unit testing religion and are looking for a tool for testing your PostgreSQL database then pgTAP is going to be hard to beat.

Tuesday, May 19, 2009

The Nail in the Coffin

My experimentation with Solaris/OpenSolaris is over. Amid the uncertainty that the Oracle purchase of Sun Microsystems has introduced I got some advice at PGCon2009 today that put the nail in the coffin. I asked someone, what OS was the best for running PostgreSQL? His response was, "The OS you are most familiar with". That OS is GNU/Linux.

In recent years Sun has tried really hard to change the image of Solaris from old Unix to Linux killer (specifically RedHat Linux) to Linux like. To drive the point home, the newer releases of OpenSolaris use bash as its default shell. But it's not enough. The biggest problem Sun had with shaking it's legacy image is that it's still legacy. For example, there are about 5 different flavors of the ps command in a default Solaris install. Sun was fanatic about maintaining backwards compatibility. The problem with that kind of religious fervor is that all the mistakes of the past become a permanent part of the system and haunts it in the present. So if you are a newcomer to Solaris and there is no one there to hold your hand, it is difficult to figure out what is the best way to accomplish a task or the best flavor of a particular tool to use. In essence Solaris newcomers are acutely susceptible to The Paradox of Choice.

So for all the great technology that is in Solaris, the investment in trying to learn it just isn't worth the return right now because I can do everything I need to do in Linux in a fraction of the time and with less frustration (i.e., Solaris still doesn't have a decent package manager). And in all the cases that are important to me, Linux and it's applications run faster than the Solaris equivalent.

But although the nail is in the coffin I'm not going to say goodbye. Who knows, one day I may have to call upon the Cruel Tutelage of Pai Mei and once again embrace the way of the Sun (err ... Oracle).

Choking on Birdseeds

Five days ago I joined twitter. Today I deleted my account. So called micro blogging is just not for me. It just seems totally pointless and I'm way too old to care about being hip. As far as I'm concerned if a technology doesn't make you more productive, entertain you, or help you express yourself, it's a waste of time. Twitter may be great for everybody else, it's just not for me. I'll stick to plain old blogging. It may not be sexy anymore, but it feels just right.

Friday, May 15, 2009

Going To PGCon2009

It's official. I'm going to PGCon2009. I just registered and booked my flight. It will be my first conference in quite some time. About 3 years ago me and a buddy attended a Sun Developer Day event in Atlanta. Loved the city and it's people. The event was okay. The key insight I walked away with was I should give OpenSolaris a serious look.

The fog of uncertainty that has enveloped the MySQL community and its code is the perfect opportunity to look at other solutions. So I'm going to check out what the PostgreSQL community has to offer. Does that make me some sort of database slut?

Thursday, April 02, 2009

Aspect Ratio & You

There are no shortages of libraries and toolkits available to programmers for scaling images. But if you ever find yourself in a position, as I recently have, where you need to roll your own (or maybe you are just curious) I'll explain everything you need to know about maintaining the aspect ratio of a scaled image.

When the issue of scaling images landed on me, the first thing I did was to google it. The search results were not very satisfactory, thus this blog entry.

So what are we talking about when we use the term "aspect ratio"? It's the relationship between an image's height and its width. From a programmatic point of view, the aspect ratio can tell us how much the width of an image should change if the height changes and vice versa. Aspect ratios are normally expressed in the form H:W (i.e., 1:1, 7:3, 4:5, etc). It can also be expressed as a fraction (i.e. 1/1, 7/3, 4/5, etc) and finally, for programmatic purposes, a decimal. The formula for the aspect ratio is:

A = H/W
where A is the aspect ratio, H is the height of the image, and W is the width of the image. Using a bit of algebra we can rewrite the formula to solve for any of the variables. So given that A = H/W then H = A*W and W = H/A.

Lets assume we have an 485px x 1024px image that we need to generate a thumbnail for. The first thing we need to do is determine the aspect ratio of the image:

A = H/W => A = 485/1024 => A = 0.4736328125
Lets also assume that we have this rule that says a thumbnail image must be no more than 140 pixels high. We now have enough information to figure out what the width must be in order to maintain the image's aspect ratio:
W = H/A => W = 140/0.4736328125 => W = 295.587628866
We know the new width maintains the aspect ratio because 140/295.587628866 = 0.4736328125. Now let's look at some code:
/** 
 * Scale <tt>src</tt>'s dimensions to <tt>max</tt> pixels starting w/ the largest side. 
 * 
 * @param image      The source image. 
 * @param max        The maximum number of pixels in each dimension(HxW). 
 * @param heightOnly Indicates that only the image's height should be scaled. 
 * 
 * @return The scaled image. 
 */ 
public static BufferedImage scale(BufferedImage image, final int max, boolean heightOnly) 
{ 
    if (heightOnly) 
        image = scaleByHeight(image, max); 
    else if (image.getHeight() > image.getWidth()) 
    { 
        image = scaleByHeight(image, max); 
        image = scaleByWidth(image, max); 
    } 
    else 
    { 
        image = scaleByWidth(image, max); 
        image = scaleByHeight(image, max); 
    } 
    return image; 
} 
 
/** 
 * Scale <tt>src</tt> by <tt>height</tt>. 
 * 
 * @param image The source image. 
 * @param max   The value to scale the image down to. If the current height of the image is less than <tt>max</tt> then this 
 *              method does nothing. 
 * 
 * @return A (possibly) scaled image. 
 */ 
public static BufferedImage scaleByHeight(BufferedImage image, final int max) 
{ 
    int height = image.getHeight(); 
    if (height > max) 
    { 
        int width = image.getWidth(); 
        final float aspectRatio = height / (float)width; 
        do 
        { 
            height >>= 1; 
            if (height < max) 
                height = max; 
            int k = (int)(height / aspectRatio); 
            if (k > 0) 
                width = k; 
            image = scale(image, height, width); 
        } 
        while (height > max); 
    } 
    return image; 
} 
 
/** 
 * Scale <tt>src</tt> by <tt>width</tt>. 
 * 
 * @param image The source image. 
 * @param max   The value to scale the image down to. If the current width of the image is less than <tt>max</tt> then this 
 *              method does nothing. 
 * 
 * @return A (possibly) scaled image. 
 */ 
private static BufferedImage scaleByWidth(BufferedImage image, final int max) 
{ 
    int width = image.getWidth(); 
    if (width > max) 
    { 
        int height = image.getHeight(); 
        final float aspectRatio = height / (float)width; 
        do 
        { 
            width >>= 1; 
            if (width < max) 
                width = max; 
            int k = (int)(width * aspectRatio); 
            if (k > 0) 
                height = k; 
            image = scale(image, height, width); 
        } 
        while (width > max); 
    } 
    return image; 
} 
 
/** 
 * Scale <tt>src</tt> down to height x width pixels. 
 * 
 * @param src    The source image. 
 * @param height The scaled height. 
 * @param width  The scaled width. 
 * 
 * @return The scaled image. 
 */ 
private static BufferedImage scale(BufferedImage src, final int height, final int width) 
{ 
    int type = src.getType(); 
    if (BufferedImage.TYPE_CUSTOM == type) 
        type = src.getTransparency() == Transparency.OPAQUE ? BufferedImage.TYPE_INT_RGB : BufferedImage.TYPE_INT_ARGB; 
    BufferedImage img = new BufferedImage(width, height, type); 
    Graphics2D gscale = img.createGraphics(); 
    gscale.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BILINEAR); 
    gscale.drawImage(src, 0, 0, width, height, null); 
    gscale.dispose(); 
    return img; 
}

Wednesday, February 11, 2009

Amazon's Cloud: What is EC2?

I'm doing research on Amazon's cloud computing platform. So I'll be jotting down notes here in case this information turns out to be useful to someone other than myself.

EC2 is part of Amazon's cloud computing platform. It enables one to run a full stack (operating system, applications, scripts, etc) on one or more compute nodes. There are different types of compute nodes. Compute nodes are organized into CPU capacity (1-8 CPUs), RAM (up to 15GB), and local storage. The user can programatically start and stop compute node instances to deal w/ increasing or decreasing demand. So in the case where a single compute node may not be sufficient in satisfying demand it is the responsibility of the user to deploy application(s) that are cluster aware.

Data stored via local storage does not persist across restarts. For persistent storage one must use Elastic Block Storage (EBS) or Amazon's S3 storage service.

Links:

Thursday, March 13, 2008

Central New York

I'm in central New York (Syracuse) and I'm having a heck of a time writing anything. My imagination is as frozen as the ground outside. I'm going to the city this week to visit places I haven't been in over 17 years. Maybe I'll be inspired by the trip to jot down a few words.

Monday, February 04, 2008

Finished Reading: The End Of The Alphabet

I contracted the flu/cold during the Christmas holiday and it lasted at least three weeks. And though I'm not a big fan of Christmas [consumerism] it felt ruined. I know what you are thinking but you are wrong. It had absolutely nothing to do with the fact that Santa dissed me again. We've had beef since I was twelve so getting dissed was expected. What I didn't expect, was that his punk ass would try to assassinate me with the flu.

I'M STILL STANDING BITCH!

There will be next year and this time I'll be ready for his punk ass.

BTW, New Years sucked too.

I'm still kinda sick. I have this lingering cough and sometimes I go into these coughing fits that leave me breathless and light headed. My doctor just (Thursday) put me on some killer antibiotics (Avelox) and I need to go get a chest x-ray to see if I have pneumonia. Who knows, maybe Santa will have the last laugh yet.

So for the last three weeks I've been trying to catch up on the work that I didn't do while I was sick, and I've spent the last week and a half working 18+ hour days. The code I worked on was an addendum to an existing system in some places and a bug fix release in others. After four weeks of delays it went live on Friday. Yeah! But what I'm proudest of is during all the chaos I managed to read an entire novel, The End Of The Alphabet, by CS Richardson. How, you say? It's only 113 pages. That's right. He squeezed an entire novel into 113 pages and it took me three whole weeks to read it.

It's a love story, so if you are into that sort of thing, you may enjoy it. Be warned, it's not the, "dude crosses dessert and slays dragon", type of love story. It's simple, efficient, and satisfying. Given that it was a mere 113 pages, I was surprised I found it satisfying. Kudos Mr. Richardson.

My rating for this one is, sweet (sensitive meaning of the word).

Saturday, January 26, 2008

Totally Naked and Loving It!

I just got naked and saved 30+ bucks a month. Yeah me!

Maybe you should get naked too!

Thursday, January 24, 2008

My New Favorite Method

Some days I manage to really amuse myself with my work. Today I've added some flare to a method that may otherwise be really boring. So let me introduce you to my new favorite method:

  /** 
     * Flattens the sub directories of <tt>roots</tt> into a single array. 
     * 
     * @param roots The root directories. 
     * 
     * @return The [sorted] subdirectories of <tt>roots</tt>. 
     */ 
    private static File[] merge(File ... roots) 
    { 
        int x = 0; 
        int k = 0; 
        File[][] forest = new File[roots.length][]; 
        for (File sapling : roots) 
        { 
            File[] tree = sapling.listFiles(); 
            if (null != tree) 
            { 
                int leaves = tree.length; 
                if (leaves > 0) 
                { 
                    forest[x++] = tree; 
                    k += leaves; 
                } 
            } 
        } 
        File[] woods = new File[k]; 
        for (k = 0; --x >= 0;) 
        { 
            File[] tree = forest[x]; 
            int leaves = tree.length; 
            System.arraycopy(tree, 0, woods, k, leaves); 
            k += leaves; 
        } 
        Arrays.sort(woods, chipper); 
        return woods; 
    }

It cracks me up sometimes that I get paid to have this much fun.