Showing posts with label postgresql. Show all posts
Showing posts with label postgresql. Show all posts

Monday, May 11, 2015

What the tag line from Highlander can teach us about Exlusion Constraints

If you are trying to get your head wrapped around exclusion constraints via a Google search you may get the impression that the only thing it's good for is solving the double booking problem common to reservation systems. The PostgreSQL documentation expands the possibility a teeny tiny bit by mentioning preventing overlapping circles. But that's it! So here is this truly phenomenal feature that thanks to an absence of varied examples you may not think you have a use for.

I'm here to expand the possibilities by introducing what I call the the Highlander use case. But before I do I want to explain why all the examples you find on Google are about the double booking problem. The fact is prior to the existence of exclusion constraints the double booking problem was a really really really really hard problem to solve in the relational database world. And if you managed to solve it, chances are you really didn't, and it was slow at scale because of all the locking required. So when Jeff Davis brought exclusion constraints to PostgreSQL all the write-ups naturally focused on the fact that there was finally a safe and scalable solution to the double booking problem.

What are exclusion constraints? Before we answer that let's start by explaining what a constraint, in general, is. A constraint is a rule expressed as statements that evaluate to a Boolean value (i.e., true or false). In essence a constraint is simply a Boolean expression. Constraints are normally checked when you try to INSERT a new row into a table or try to update an existing row. If the expression evaluates to true the INSERT or UPDATE is allowed to proceed, otherwise it fails and an error is raised.

The 4 constraints that just about everyone who has created a relational database is familiar with is:
  1. NOT NULL - A column cannot accept NULL as a value.
  2. UNIQUE - A column or combination of columns cannot have duplicates.
  3. PRIMARY KEY - Combination of #1 & #2.
  4. FOREIGN KEY - A column (or columns) in Table1 must have a matching entry in Table2.
An exclusion constraint is just a supped up unique constraint in that it has a lot more expressive power to define exactly what unique means. Now on w/ the use case.

Let's say you sell a system where owners create events and their customers attend the events. One of the features you probably want in this system is the ability for customers to provide feedback on their experience. So basically you want surveys.

Lets assume that there are more than one type of event, therefore you may want to have more than one survey and would like to associate specific surveys w/ specific events. Let's visualize that a tiny bit:
Event_1 -> Survey_1
Event_2 -> Survey_2
Event_N -> Default Survey.

Note Event_N and Default Survey. Event_N is any event that has no explicit survey associated w/ it. And when the system sees one of these it needs to send the default survey to the customers that attended those events. So that means we need a way to mark a survey as the default survey. (Hold on to your hat folks the Highlander reference is almost here!) And since, logically speaking, "there can be only one" default survey it would be great if there was a way to express that at the database level because we can't depend on the application nor people not to violate the rule that there should be only one default survey.

So let's quickly summarize what the Highlander use case is by comparing it to the standard unique constraint. With a standard unique constraint there can be no duplicates, period. But in the Highlander use case, you take one value and make it special by saying you can duplicate anything else, but not this; for this, "there can be only one"!

Exclusion constraints makes addressing the Highlander use case trivial. Let's look at some SQL.
CREATE TABLE surveys (
  id SERIAL PRIMARY KEY,
  name VARCHAR(64) NOT NULL UNIQUE,
  survey_questions JSONB NOT NULL,
  is_default BOOLEAN NOT NULL DEFAULT FALSE

The is_default column solves the identity problem. Next we'll finish the SQL off w/ our exclusion constraint:
...,
  EXCLUDE (is_default WITH =) WHERE (is_default)
);
EXCLUDE (is_default WITH =) is equivalent to having a unique constraint on is_default. But that's not enough, for the same reason that a unique constraint doesn't solve the problem. So we need more expressive power. Exclusion constraints gives us that expressive power by allowing us to specify multiple comma separated criteria in the EXCLUDE block and we can specialize even more by adding a WHERE clause. So that's what we've done here. In layman's terms our exclusion constraint says, "the value of the is_default column of the row being inserted or updated must not equal any other row's is_default column value, but only when is_default IS TRUE". This means we can have a gazillion rows where is_default IS FALSE but only one row where is_default IS TRUE.

If you have a good grasp of SQL's syntax in general and PostgreSQL in particular, then the exclusion constraint syntax should be obvious. But if you are thrown by the double use of is_default let me explain.

The first one is a reference to a column's name. In this case the column that we want to reference is named "is_default". The second one needs to be understood in the context of how WHERE clauses work. All WHERE clauses must be Boolean expressions, and since is_default is defined as a Boolean it's by definition a Boolean expression too. So we take advantage of that fact and make it the entirety of the WHERE clause. But if we wanted to be verbose we could have written the WHERE clause as:
WHERE (is_default IS TRUE)
or as
WHERE (is_default=true)

[UPDATE:]Thanks to an anonymous commenter I've learned that PostgreSQL is even cooler than I gave it credit for because the Highlander use case is better solved w/ a pgified unique constraint. Basically, PostgreSQL allows us to apply a WHERE clause to your basic unique constraint, giving us the same expressive power that I was using an exclusion constraint for, but w/ better performance. The lesson here is simple, if you are coming to PostgreSQL from another database or have been living the lie of using only spec compliant syntax, read the PostgreSQL docs carefully because chances are PostgreSQL improves upon the standard in extremely powerful ways (a.k.a, pgified).

For completeness let's include the new version of the DDL:
CREATE TABLE surveys (
  id SERIAL PRIMARY KEY,
  name VARCHAR(64) NOT NULL UNIQUE,
  survey_questions JSONB NOT NULL,
  is_default BOOLEAN NOT NULL DEFAULT FALSE
);
CREATE UNIQUE INDEX ON surveys (is_default) WHERE (is_default);

Monday, May 16, 2011

Liar! A.K.A This one is for my PGEast 2011 peeps


If you attended PGEast 2011 then you know that a certain company had tons of coffee mugs everywhere. They couldn't give them away even though they were giving them away. It got so bad that the conference organizer thought that maybe the attendees didn't realize that the mugs where for them to take so he made an announcement that the mugs were for us to take.

So let's recap. PGEast 2011 almost 2 months ago & ungodly number of unwanted coffee mugs from NoSQL vendor who will not be named.

So today I start reading this blog post. Then I get to the 15th paragraph and PGEast comes crashing back because I know EXACTLY who he is calling out!

I just loved the way the author called them out w/o naming them, yet absolutely naming them.

Finally, I strongly encourage you not to skip ahead to the 15th paragraph because you need the first 14 to get the full effect of the none name dropping of names in 15.

Tuesday, June 02, 2009

PGCon2009 Summary

I'm back from PGCon2009. No, I didn't just get back. I've been back for a smidgen over a week now. When I first got there I decided I would blog daily about it, but time didn't permit me to write in any detail. So I decided I would make notes and summarize it all when I got back. The notes thing didn't pan out thanks to twitter. It was simply easier to tweet my thoughts as I thought them than collecting them in little text files and revisiting them later. The summary idea didn't work out either because (a) I didn't have any notes, and (b) all the good stuff was already said on Planet PostgreSQL. But it's been more than a week since the conference and people are still posting summaries so now I feel like I have to say something. Here goes!

Great conference, awesome people, awesome community, and PostgreSQL is really cool technology that I'm confident in trusting my [clients'] data with.

My only gripe with the conference was the keynote address. It's nobody's fault really. The original speaker couldn't make it so they had to slap something together at the last minute. It showed. At the time, my thought was, if the rest of the conference is like this it's going to suck!

...

As I write this I've just realized that the keynote incident is a metaphor for the larger PostgreSQL project. If you attended or watched the "How to Get Your Patch Accepted" talk, it is apparent that quickly slapping something together is not how the PostgreSQL code base is developed or maintained. So I shouldn't have been surprised that the keynote was not a valid indicator about the rest of the conference. I hope I haven't disparaged any of the speakers. They did their best and everyone else in the auditorium enjoyed their keynote.

Tuesday, May 26, 2009

PGCon2009 Postscript: Unit Test Your Database!

I started watching some of the PGCon2009 videos that I didn't attend while at PGCon. Last night I watched, Unit Test Your Database!, by David Wheeler. I have had my come to Jesus moment on unit testing years ago, so I'm really happy that there is a solution for testing strictly at the database level without depending on the application layer.

Some developers make the mistake of treating the database as a glorified file system and therefore assume it doesn't need any testing. They are wrong! From the application's point of view, the database should be a black box and application level testing of the database should be limited to the interaction between the application and the interfaces the database exposes, like stored procedures and views. In this development paradigm the database is an independent entity and therefore needs its own set of tests to ensure that it's self consistent. This is where the speaker's own testing framework, pgTAP, excels. It's not limited to just testing the public interfaces the database exposes. It allows you to validate the database itself, i.e., verify the structure of the tables in the database and their relationships, verify triggers, verify the existence of indexes, verify the existence of functions, verify the behavior of functions, etc.

If you are still on the fence about whether unit testing your database is a worthwhile endeavor then watch the video, it may convince you. But if you've already got unit testing religion and are looking for a tool for testing your PostgreSQL database then pgTAP is going to be hard to beat.

Tuesday, May 19, 2009

The Nail in the Coffin

My experimentation with Solaris/OpenSolaris is over. Amid the uncertainty that the Oracle purchase of Sun Microsystems has introduced I got some advice at PGCon2009 today that put the nail in the coffin. I asked someone, what OS was the best for running PostgreSQL? His response was, "The OS you are most familiar with". That OS is GNU/Linux.

In recent years Sun has tried really hard to change the image of Solaris from old Unix to Linux killer (specifically RedHat Linux) to Linux like. To drive the point home, the newer releases of OpenSolaris use bash as its default shell. But it's not enough. The biggest problem Sun had with shaking it's legacy image is that it's still legacy. For example, there are about 5 different flavors of the ps command in a default Solaris install. Sun was fanatic about maintaining backwards compatibility. The problem with that kind of religious fervor is that all the mistakes of the past become a permanent part of the system and haunts it in the present. So if you are a newcomer to Solaris and there is no one there to hold your hand, it is difficult to figure out what is the best way to accomplish a task or the best flavor of a particular tool to use. In essence Solaris newcomers are acutely susceptible to The Paradox of Choice.

So for all the great technology that is in Solaris, the investment in trying to learn it just isn't worth the return right now because I can do everything I need to do in Linux in a fraction of the time and with less frustration (i.e., Solaris still doesn't have a decent package manager). And in all the cases that are important to me, Linux and it's applications run faster than the Solaris equivalent.

But although the nail is in the coffin I'm not going to say goodbye. Who knows, one day I may have to call upon the Cruel Tutelage of Pai Mei and once again embrace the way of the Sun (err ... Oracle).

Friday, May 15, 2009

Going To PGCon2009

It's official. I'm going to PGCon2009. I just registered and booked my flight. It will be my first conference in quite some time. About 3 years ago me and a buddy attended a Sun Developer Day event in Atlanta. Loved the city and it's people. The event was okay. The key insight I walked away with was I should give OpenSolaris a serious look.

The fog of uncertainty that has enveloped the MySQL community and its code is the perfect opportunity to look at other solutions. So I'm going to check out what the PostgreSQL community has to offer. Does that make me some sort of database slut?