Sunday, September 2, 2012

O/R Modelling in PostgreSQL part 4: Composite Columns and Cross-Table Refs

So far, we have looked at tables as classes and various ways of looking at inheritance within that structure.  However, tables can be nested in PostgreSQL and this has a number of uses, particularly in views, and data structures passed to and from functions.  As PostgreSQL support for things like JSON becomes more important, these approaches will become more important as well.  Composite types in columns however opens up a significant number of gotchas and in general I tend to suggest avoiding this for physical storage.

Composite types, also known as structured data types, allow you to further define interfaces for user defined functions, and add additional sub-interfaces to those.  They can also be used to define complex data in tables although often multiple inheritance provides for a cleaner solution when it comes to actual storage.   However, complex types can be useful in a number of cases where existing data types are inadequate, as an alternative to building the type in C.

In some cases, the logic can be helpful here too.  In general when looking at base storage, my preference is to look at inheritance first, and then failing that to look at in-type storage.  Inheritance can give you all the benefits of in-column storage with fewer gotchas, and greater reusability.  However, there are cases where this breaks down, notably in views.  Also as we noted before tables are composite types (but not every composite type is a table).

We will look at the differences by considering an inheritable table storing address information and see how it behaves differently when used as a compsite type for a column.

Warning:  This is a comparatively new area for PostgreSQL and there are many areas of inconsistent behavior that have not been worked out.  If you are using composite types for column definitions, be prepared to run into unexpected sharp edges and the possibility of behavior changing in the future.  Additionally multiple inheritance (presuming unique column names) provides the same benefits with fewer inconsistencies.  This is really at the edges of the system.

Table Schema:

CREATE TABLE country (
     id serial not null unique,
     name text primary key,
     short_name text not null unique,
     state_nullable bool not null default false,
     requires_mailcode bool not null default true,
     unique(id, state_nullable, requires_mailcode),
);

Interface tables:

CREATE TABLE country_ref (
    country_id int,
);

CREATE TABLE country_ref_ext (
    country_state_nullable bool,
    country_requires_mailcode bool,
    CHECK (country_id NOT NULL)
) INHERITS (country_ref);

And of course, our traverse function:

CREATE OR REPLACE FUNCTION country(country_ref)
RETURNS country 
LANGUAGE SQL AS $$

SELECT * FROM country WHERE id = $1.country_id;

$$;

Ok, so now we have this part set up.

CREATE TABLE address (
    id serial not null unique,
    street_one text not null,
    street_two text,
    street_three text,
    floor text,
    care_of text,
    city text not null,
    state_province text,
    mail_code text,
    foreign key (country_state_nullable, 
                 country_requires_mailcode, 
                 country_id) 
       references country
               (state_nullable, 
                requires_mailcode, 
               id)
) INHERITS (country_ref_ext); 

Of course, now I can take an address and pull the country info off of it by dereferencing the primary key, but the nullable state and mailcode properties are also guaranteed to be equal to what they are in the country table.

Now in this case, we have inherited a specific country reference.  The country is required and so we know there will be a country.

Now in this case, any check constraints we have inherited would be applicable and we could subclass interface classes however we'd like.

Now, suppose we implement the country ref as a database column instead.

So our table looks like this instead:

CREATE TABLE address_noinherit (
    id serial not null unique,
    street_one text not null,
    street_two text,
    street_three text,
    floor text,
    care_of text,
    city text not null,
    state_province text,
    mail_code text,
    country_ptr country_ref_ext
);

Composite Type Behavior and Gotchas

In the first case we can simply do:

SELECT (a.country).name from address a where id = 1;

In the second gase, we would similarly:

SELECT ((a.country_ref).country).name
  FROM address_noinherit a WHERE id = 1;

You can then use this to implement a pointer system in your tables for recurring links.

Except that it doesn't really work that way as well as you would might like.

The NOT NULL check is not checked on a table which is not of the type where it is defined.  Consequently we have to declare that separately:

    CHECK ((country_ref).country_id IS NOT NULL)

Worse, adding a column to the type table can lead to problems.   You cannot add NOT NULL or DEFAULT clauses when adding the type if the table is being used as a type for the column of another table.  To work around this you must add the NOT NULL and DEFAULT constraints after first adding the column.

Except, assuming that runs you into trouble too.   There are exceptions to the above rule, particularly where domains are involved in types.  For example:

A domain for positive, not null integers:
or_examples=# create domain pos_not_null_int as int not null check (value > 0);
CREATE DOMAIN






A table which uses these:
or_examples=# create table rel_examples.domaintest (id pos_not_null_int);
CREATE TABLE



Constraints on the domain are enforced (as a test):
or_examples=# insert into rel_examples.domaintest values (-1);
ERROR:  value for domain pos_not_null_int violates check constraint "pos_not_null_int_check"
or_examples=# insert into rel_examples.domaintest values (null);
ERROR:  domain pos_not_null_int does not allow null values

A table which uses this as a column type:
or_examples=# create table comp_domain_test ( test rel_examples.domaintest);
CREATE TABLE


Despite what I said before, the constraints are enforced here (which they would not be if the initial table had its check constraints declared at the table level):
or_examples=# insert into comp_domain_test values (row(null));
ERROR:  domain pos_not_null_int does not allow null values
or_examples=# insert into comp_domain_test values (row(-1));
ERROR:  value for domain pos_not_null_int violates check constraint "pos_not_null_int_check"

When using complex types in columns the dba is advised not to make too many assumptions.  Given the inconsistent behavior here, I expect that it will likely change in future versions, but in which way one cannot say.

Additionally sub-tuple elements cannot be a part of foreign keys, so you are left with something with all the problems that Oracle's REF implementation does, namely that it gives you a weak foreign key implementation which does not actually enforce the underlying storage.  However it can still be automatically traversed.  In a future post we will look at solutions to foreign key enforcement.

In general, inheritance is cleaner than composite columns and there are far fewer gotchas.  Nevertheless there are some cases where they are an elegant solution to a problem, particularly when looking at nested data storage to enforce constraints which cannot otherwise be expressed.

Composite types in VIEWs

However, where storage is not at issue, composite types, whether based on tables or not, can be used here without the previous issues.  Stored procedures can be used to generate composite types (i.e. without inheritance) or directly from the inheritance tree.

For compsite types which are returned from stored procedures, these can be presented directly in the view, or not.  Methods of previous tables can be called as well (so inventory_item.markup in my previous example might go from a function call to an output column).  Similarly methods associated with types can be called here as well.  However, it may be useful to expose the types directly as well as a way of allowing for additional functionality to be requested.

In the initial address case which uses inheritance, we might:

CREATE VIEW address_view AS 
SELECT * FROM address;

But then we cannot:

SELECT (av.country).* from address_view av where ....;

One way around this is:

CREATE VIEW address_view AS
SELECT id, street_one, city, state_province, mail_code,
       a::country_ref AS country_ref
  FROM address a;

Now we can:

SELECT ((country_ref).country).name FROM address_view
 WHERE city = 'Paris'
GROUP BY country_ref;

Table aliases can be similarly captured for function output that creates complex types and behavior exposed in views this way.

In object-relational design it is critical to consider how a view or interface will be used.  One real problem with including too many heavy O/R features in a table is that it can complicate the use of the data by people running queries.   Nested data structures primarily are helpful when they create interfaces to O/R logic that is needed and would not otherwise be available, or where the application expects the data in this format.  In other words, human and computer interfaces may use varying levels of O/R logic.  This, of course, will be covered by a future item in this series.

Next:  Nested Data Structures and Storage (Uses and Gotchas)
Next Week:  Interlude:  Contrasting to MySQL and Messaging in PostgreSQL

Thursday, August 30, 2012

O/R Modelling 3.2: Set/Subset Modelling with Inheritance

Despite the fact that set/subset modelling is usually not a win for inheritance, sometimes in the cases of more complex logic, it can be.  David Fetter has come up with an example where he tries to solve the problem of single parent/multiple child inheritance on a purely relational model.

In general when you get into constraints this complex, the question is no longer how to avoid complexity but how to manage it.  There are several ways in which refactoring the tables may be helpful, and table inheritance allows us to do just that (no we don't have to lose foreign key management capabilties to do so).  This allows us to look at set/subset management differently.

Of course how we go about managing this complexity may still be a matter of taste and of the exact circumstances involved (the same solution may not be optimal everywhere).  One person may prefer lots of NULLS and complex constraints.  One person may prefer few NULLs, no inheritance, and lots of triggers.  Someone else may prefer a richer toolkit with the pitfalls that provides.  Managing complexity is more an art than a science.  In general though, we don't blame the paintbrush when the artist doesn't know how to use it.

In my experience, using NULLs to indicate type furthermore creates a situation where it is quite possible to end up with misbehavior that takes some time to troubleshoot and so I don't typically like this approach but again, I recognize that the complexity issues remain regardless of the solution.

In general the approach here takes the following steps:

  1. Decompose the table in a relational model
  2. Look for common interfaces
  3. Inherit those interfaces
Fetter's Table Structure:

I am skipping over the check constraints and unique constraints which are relatively complex here.  You should read the full post yourself to see what complexity we are managing.  However for refactoring purposes, I can't really avoid quoting his table field structure:

CREATE TABLE payment (
    payment_id BIGSERIAL PRIMARY KEY,
    payer_id INTEGER NOT NULL REFERENCES payer,
    payment_date DATE NOT NULL,
    amount NUMERIC NOT NULL CHECK(amount > 0),
    currency_id INTEGER NOT NULL REFERENCES currency,
    /* Check */
    check_number TEXT,
    check_date DATE,
    /* Credit Card */\
    cc_id INTEGER REFERENCES locked_down_tight_cc_table,
    /* You wouldn't store an actual CC number here, would you?!? */
    expiry_date DATE,
        CHECK(expiry_date = DATE_TRUNC('month', expiry_date)),
    /* Manila Envelopes are pretty anonymous 
    is_manila BOOLEAN NOT NULL DEFAULT FALSE,
    -- check constraints then follow which are quite complex
    -- and partial unique constraints are added after the fact
); 
/* comments by David Fetter */
-- Comments added by Chris Travers 

Step 1:  Relational Refactoring

We can then factor this out into the following tables:
  • payment_class new, added to categorize payments.
    • Includes an extended storage column to indicate that storage is required for a type.
  • payment (main details go here), adding payment_class_id field
  • payment_check stores check-specific fields
    • payment_class_id set to 1
  • payment_cards stores card-specific fields
    • payment_class_id set to 2

We would then have to create a custom constraint trigger for managing relational integrity between payment and payment_check, payment_cards.  Note that standard foreign keys work one direction so this isn't too bad.

Step 2:  Common Interfaces:

Every payment extension table requires a payment_class_id and a payment_id field which together act as a foreign key in the parent table.  This provides an interface which can go both ways and simplify the interfaces.  This should then be our base table.

Step 3:  Completed redesign

Note that some foreign keys are omitted in order to allow the table to be created in this demo environment.

CREATE TABLE payment_ref (
    payment_id bigint not null,
    payment_class_id int
);

CREATE TABLE payment_class (
    id int not null unique, -- manually assigned, few
    label text primary key,
    extended_storage bool not null    
);

INSERT INTO payment_class (id, label, extended_storage)
VALUES (1, 'check', true),
       (2, 'cards', true),
       (3, 'cash', false);

CREATE TABLE payment (
    payment_id bigserial PRIMARY KEY,
    payment_class_id int NOT NULL references payment_class(id),
    -- above two columns will be primary key
    payer_id int not null, 
    -- foreign key on payer_id omitted due to demo here
    amount numeric not null check (amount > 0),
    currency_id int not null,
    -- foreign key on currency_id omitted due to demo here
    UNIQUE (payment_id, payment_class_id) -- secondary key
);

CREATE TABLE payment_check (
    check_date date not null,
    check_number text not null,
    CHECK (payment_class_id = 1),
    FOREIGN KEY (payment_class_id, payment_id) 
        REFERENCES payment (payment_class_id, payment_id)
        DEFERRABLE INITIALLY IMMEDIATE
) INHERITS (payment_ref);

CREATE TABLE payment_cards (
    cc_info int not null, 
    -- foreign key omitted so table can be created in demo environment
    CHECK (payment_class_id = 2),
    FOREIGN KEY (payment_class_id, payment_id) 
        REFERENCES payment (payment_class_id, payment_id)
        DEFERRABLE INITIALLY IMMEDIATE
) INHERITS (payment_ref);

All that is left is to create the constraint triggers and we are done.  Note inheritance here buys us a consistent interface for joins, plus a consistent interface for the constraint triggers.

In essence the way we are using table inheritance here is as a way to partition additional information to be joined to a table, where foreign keys against the total partition set are relatively uninteresting.  Note however, one could come up with ways here to define an additional unique index (on payment_class(id, extended_storage) which would allow effective foreign keys to be added to the inheritance tree.  The above approach, once constraint triggers are added (and tested) would be easier to understand and maintain in my experience than one using NULLs to indicate type, and would have more modular and maintainable check constraints.

Now the constraint triggers will still have to be created and tested and this adds significant complexity.  This may be a win where we need to maintain extensibility here, for example if we want to be open to adding gift certificates as forms of payment, or international money orders, etc.  On the other hand if we do not have to preserve this, then the inheritance-based approach adds needless complexity regarding foreign keys.

If the relational model is good enough (namely because of the lack of a need for extensibility in payment types) one can simply add an id field to each sub-table, a deferrable foreign key constraint (initially deferred) from payment to the other tables, and check constraints on whether these are NULL based on payment type.  In that case, inheritance just gets us enforcement for identical semantics for foreign keys, and a virtual catalog of payment references.

On the other hand if the purely relational model (fkeys against physical storage tables directly) is not good enough, we can build a system which we use constraint triggers to enforce foreign keys against an object catalog (payment_ref) in this case.

The Moral of the Story

Set/Subset modelling with table partitioning does have some uses, but maintainability requires that the tables be relationally factored as much as possible first, and inherit no more than necessary.  However, sparingly used, inheritance can facilitate (rather than replace!) good relational design.  This works best when one looks at inheritance as a way of doing table partitioning with additional fields per partition.

With 9.2, the addition of NO INHERIT check constraints will allow the top level partition to be used, but before that the top-level needs to be clear of entries.  In general, however, I am not convinced that having the top level partition be used in this case is actually a good idea.  As soon as you do this, a vanilla SELECT and a foreign key will have suddenly very different behavior.  Semantic clarity counsels for using NO INHERIT constraints to forbid all inserts into parent nodes.

Is the complexity tradeoff worth it though?  You be the judge.

Monday, August 27, 2012

PostgreSQL O/R Modelling Part 3: Table Inheritance and O/R Modelling in PostgreSQL

Note:  PostgreSQL 9.2 will allow constraints which are not inherited.  This will significantly  impact the use of inheritance, and allow for real set/subset modelling with records present in both parent and child tables.  This is a major step forward in terms of table inheritance.

PostgreSQL allows tables, but not views or complex types, to inherit table structures.  This allows for a number of additional options when doing certain types of modelling.  In this case, we will built on our inventory_item type by adding attached notes.  However we may have notes attached to other tables too, and we want to make sure that each note can be attached to only one table.

This is exactly how we use table inheritance in LedgerSMB.

Before we begin, however it is worth looking at exactly what is inherited by a child table and what is not.

The following are inherited:
  • Basic column definitions (including whether a column is nullable)
  • Column default values
  • CHECK constraints (cannot be overridden in child tables)
  • Descendants can be implicitly casted to ancestors, and
  • Table methods are inherited 
The following are not inherited:
  • Indexes
  • Unique constraints
  • Primary Keys
  • Foreign keys
  • Rules and Triggers
The Foreign Key Problems and How to Solve Them

There are two basic foreign key problems when using table inheritance on PostgreSQL. The first is that foreign keys themselves are not inherited, and the second is that a foreign key may only target a specific relation, and so rows in child tables are not valid foreign key targets.  These problems together have the same solution, which is that where these are not distinct tables for foreign key purposes, the tables should be factored in such a way as to break this information out, and failing that, a trigger-maintained materialized view will go a long ways towards addressing all of the key management issues.

For example, for files, we might place the id and class information in a separate table and then reference these in on child relations.  This gives a valid foreign key target for the set of all inherited tuples.  Foreign keys can also be moved to this second table, allowing for them to be centrally managed as well, though it also means that JOIN operations must include the second table.  A second option might be to omit class and use the child table's tableoid, perhaps renaming it in the materialized view as table_id.

The Primary Key Problem and How to Solve It

Similarly because indexes are not inherited, we cannot ensure that a specific combination of tuples is unique through a hierarchy.

The most obvious solution is to treat the combination of the natural primary key and the tableoid as a primary key for modelling purposes.   However this becomes only really useful when the primary key is a composite field with mutually exclusive constraints on child tables.  In other words, partitioning is necessary to sane set/subset modelling using table inheritance.

Where Not to Use Table Inheritance


Most of the table inheritance documentation suggests that the primary use of table inheritance is set/subset modelling, where an subset of a set has its own extended properties.  In practical use, however, a purely relational solution is usually cleaner for this specific sort of case and results in fewer key management issues (and therefore less overall complexity).

For set/subset modelling, a much cleaner solution is to use composite foreign keys.

For example, revising the example from the manual for table inheritance, a cleaner way to address the issue where you have cities and capitals is to do something like:

CREATE TABLE cities (
    name text primary key,
    altitude float, 
    population int,
    is_capital bool not null,
    unique(is_capital, name)
);

The unique constraint does nothing to the actual table.  If "name" is guaranteed to be unique, then ("is_capital", "name") is guaranteed to be unique.  What it does, however, is designate that as a secondary key.

You can then create a table of:

CREATE TABLE capitals (
    name text primary key,
    is_capital bool not null,
    state text not null,
    foreign key (name, is_capital) 
        references cities (name, is_capital)
        DEFERRABLE INITIALLY DEFERRED
);


 This approach, where subset membership is a part of a secondary key, allows for greater control regarding set/subset modelling.  You can then create a trigger on cities which checks to see if the city exists in capitals (enforcing an insert flow of "insert into capitals, then insert into cities, then commit").

No primary or foreign key problems occur with this solution to set/subset models.   With multiple subsets you will likely wish to generalize and use a "city_class" field of type int referencing a city_class table so you can control subsets more broadly.  However in general, table inheritance is a poor match for this problem because of foreign key constraint issues.  Primary key management issues are less of an issue however (and becoming significantly easier to solve).


The one use case for set/subset modelling using inheritance is table partitioning.  It has all the problems above, and enforcing foreign keys between two partitioned tables can become very complex, very fast unless materialized views are used to proxy the enforcement.


Table Structure:  Purely Relational Approach

CREATE SCHEMA rel_examples;
CREATE TABLE rel_examples.note (
   note text not null,
   subject text not null,
   created_by int not null,
   note_class int not null,
   created_at timestamp not null,
   id int unique,
   primary key (id, note_class)
);

CREATE TABLE rel_examples.note_to_inventory_item (
   note_id int not null,
   note_class int not null check (note_class = 1),
   item_id int references inventory_item (id),
   foreign key (note_id, note_class) 
      references rel_examples.note (id, note_class)
);

CREATE TABLE rel_examples.note_to_......;

These would then be queried using something like:

SELECT n.* FROM rel_examples.note n
  JOIN rel_examples.note_to_inventory_item n2i 
       ON n2i.note_id = n.id AND n2i.note_class = n.note_class
  JOIN inventory_item i ON i.id = n2i.item_id
 WHERE i.sku = 'TEST123';

This doesn't preclude other Object-Relational approaches being used here.  I could of course add a method tsvector to make full text searches easier, or the like and apply that to note, and it would work just as well.  However it doesn't really apply to the data model very well, and some performance can be gained, since we are only looking at pulling notes by attachment, by breaking up the table into more workable chunks which could be queried independently.

Additionally the global note table is a maintenance  nightmare.  Enforcing business rules in that case is very difficult.

Table Structure:  Object-Relational Approach

An Object-Relational approach starts off looking somewhat different:

CREATE TABLE note (
    id serial not null unique,
    note text not null,
    subject text not null,
    note_class int not null,
    created_by int not null,
    ref_key int not null,
    created_at timestamp not null
);

We'd then probably add a trigger to prevent this table from receiving rows.  Note that we cannot use a check constraint for this purpose because it will be inherited by every child table.  Typically we'd use a trigger instead of a rule because that way we can easily raise an exception.

Then we'd

CREATE TABLE inventory_note (
    CHECK (note_class = 1),
    PRIMARY KEY (id, note_class),
    FOREIGN KEY (ref_key) references inventory_item(id)
) INHERITS (note);

Suppose we also create a tsvector method as such:

CREATE OR REPLACE FUNCTION tsvector(note) RETURNS tsvector
LANGUAGE SQL AS $$
SELECT tsvector($1.subject || ' ' || $1.note);
$$;

Now in my example database, I have an inventory_item with an id of 2, so I might:

INSERT INTO inventory_note
     (subject, note, note_class, created_by, created_at, ref_key)
VALUES
     ('Testing Notes', 'This is a test of the note system', 1, 1, now(), 2);

We know this worked:

or_examples=# select * from note;
 id |               note                |    subject    | note_class | created_b
y | ref_key |         created_at        
----+-----------------------------------+---------------+------------+----------
--+---------+----------------------------
  1 | This is a test of the note system | Testing Notes |          1 |         
1 |       2 | 2012-08-22 01:17:27.807437
(1 row)

Note above that I selected from the parent table and child tables were pulled in automatically.  If I want to exclude these I can use SELECT * FROM ONLY note and none of these will be pulled in.

We can also note that the method is inherited:

or_examples=# select n.tsvector from inventory_note n;
                              tsvector                              
---------------------------------------------------------------------
 'Notes' 'Testing' 'This' 'a' 'is' 'note' 'of' 'system' 'test' 'the'
(1 row)

The tables could then be queried with one of two equivalent approaches:

SELECT n.* FROM note n
  JOIN inventory_item i ON n.ref_key = i.id AND n.note_class = 1
 WHERE i.sku  = 'TEST123';

or alternatively:

SELECT n.* FROM inventory_note n
  JOIN inventory_item i ON n.ref_key = i.id
 WHERE i.sku = 'TEST123';

Both queries return the same results.

Advantages of the Object-Relational Approach


The object-relational model allows us to be sure that every note is attached to something, which we cannot do gracefully in the relational model, and it allows us to ensure that two notes attached to different items are in fact different.  It also allows for simpler SQL.

On the performance side, it is worth noting that in both the query cases above, PostgreSQL will only query child tables where the constraints could be met by the parameters of the query, so if we have a dozen child tables, each of which may be very large, we only query the table we are interested in.  In other words, inherited tables are essentially naturally partitioned (and in fact people use inheritance to do table partitioning in PostgreSQL).

Multiple Inheritance

Multiple inheritance is supported, but if used care must be taken that identically named fields are actually used in both classes in compatible ways.  If they are not, then problems occur.  Multiple inheritance can however be used safely both for interface development (when emulating inheritance in composite types, see below), and where only one of the inheriting relations contains actual columns (the other might contain constraints).

Thus we could still safely do something like:

CREATE note (
    note text,
    subject text,
    ....
);

CREATE table joins_inventory_item (
   inventory_id int,
);

CREATE NOTE inventory_note (
    FOREIGN KEY (inventory_id) REFERENCES inventory_item(id),
    ... -- partial primary key constraints, and primary key def
) inherits (note, joins_inventory_item);

This could then be used to enforce consistency in interface design, ensuring more readable queries and the like, but it doesn't stop there.  We can use joins_inventory_item to be the vehicle for a way to follow a reference.  So:

CREATE OR REPLACE FUNCTION inventory_item
(joins_inventory_item)
RETURNS inventory_item
LANGUAGE SQL AS
$$ SELECT * FROM inventory_item WHERE id = $1.inventory_id $$; 

In this case, then you can use inventory_item as a virtual pointer to the inventory item being joined as such:

CREATE TABLE inventory_barcode (
    barcode text,
    FOREIGN KEY (inventory_id) REFERENCES inventory_item(id)
) INHERITS (joins_inventory_item);

The uses for multiple inheritance however don't end there.  Multiple inheritance gives you the ability to define your database in re-usable chunks with logic following that chunk.    Alternative presentation of sets of columns then becomes manageable in a way it is not with complex types.

A More Complex Example:  Landmarks, Countries, and Notes
Another example may be where we are storing landmarks, which are attached to countries.  Our basic table schema may be:

CREATE TABLE country (
    id serial not null unique,
    name text primary key,
    short_name varchar(2) not null unique
);

CREATE TABLE country_ref (
    country_id int
);

Our traversal function:

CREATE FUNCTION country(country_ref) RETURNS country
LANGUAGE SQL STABLE AS
$BODY$ SELECT * FROM country WHERE id = $1.country_id $BODY$;

Next to notes:

CREATE TABLE note_fields (
    note_subject text,
    note_content text
);

Like country_ref, we probably will never query note_fields themselves.  There is very little of value in querying only contents like this.  However we can add a tsvector derived value:

CREATE FUNCTION note_tsvector(note_fields)
RETURNS tsvector IMMUTABLE
LANGUAGE SQL AS $BODY$

SELECT to_tsvector('english', coalesce($1.note_subject, '') || ' ' ||
coalesce($1.note_content, ''));

$BODY$;

Now, the function name is prefixed with note_ in order to avoid conflicts with other similar functions.  This would allow multiple searchable fields to be combined in a table and then mixed by inheriting classes.

CREATE TABLE landmark (
    id serial not null unique,
    name text primary key,
    nearest_city text not null,
    foreign key (country_id) references country(id),
    CHECK (country_id IS NOT NULL),
    CHECK (note_content IS NOT NULL)
) INHERITS (note_fields, country_ref);

The actual table data will not display too well here, but anyway here are a few slices of it:

or_examples=# select id, name, nearest_city, country_id from landmark;
 id |        name        | nearest_city  | country_id
----+--------------------+---------------+------------
  1 | Eiffel Tower       | Paris         |          1
  2 | Borobudur          | Jogjakarta    |          2
  3 | CN Tower           | Toronto       |          3
  4 | Golden Gate Bridge | San Francisco |          4
(4 rows)

and the rest of the table:

or_examples=# select id, name, note_content from landmark;
 id |        name        |              note_content             
----+--------------------+----------------------------------------
  1 | Eiffel Tower       | Designed by a great bridge builder
  2 | Borobudur          | Largest Buddhist monument in the world
  3 | CN Tower           | Major Toronto Landmark
  4 | Golden Gate Bridge | Iconic suspension bridge
(4 rows)

The note_subject field is null on all records.

SELECT name, (l.country).name as country_name FROM landmark l;

or_examples=# SELECT name, (l.country).name as country_name FROM landmark l;
        name        | country_name
--------------------+---------------
 Eiffel Tower       | France
 Borobudur          | Indonesia
 CN Tower           | Canada
 Golden Gate Bridge | United States
(4 rows)

Demonstrating the note_tsvector interface.  Note that this could be improved upon by creating different tsvectors for different languages.

or_examples=# select name, note_content from landmark l where
plainto_tsquery('english', 'bridge') @@ l.note_tsvector;
        name        |            note_content
--------------------+------------------------------------
 Eiffel Tower       | Designed by a great bridge builder
 Golden Gate Bridge | Iconic suspension bridge
(2 rows)

So the above example shows how the interfaces offered by two different parent tables can be invoked by a child table.  This sort of approach avoids a lot of the problems that come with storing composite types in columns (see upcoming posts) because the complex types are stored inline inside the table.


Base Tables as Catalogs

The base tables in these designs are not generally useful to query for data queries. However they can simplify certain types of other operations both manually and data-integrity wise.   The uses for data integrity will the subject of a future post.  However here we will focus on manual tasks where this sort of inheritance can help.


Suppose we use the above note structure in a program which manages customer contacts, and one individual discovers inappropriate language in some of the notes entered by another worker.  In this case it may be helpful to try to determine the scope of the problem.  Suppose the note includes language like "I told the customer to stfu."  We might want to:


SELECT *, tableoid::regclass::text as table_name
  FROM note_fields nf 
 WHERE nf.note_tsvector @@ to_tsquery([pattern]);


In this case we may be trying to determine whether to fire the offending employee, or we may have fired the employee and be trying to figure out what sort of damage control is necessary.


A test query with the current data set above shows what will be shown:


or_examples=# SELECT *, tableoid::regclass::text as table_name
  FROM note_fields nf
 WHERE nf.note_tsvector @@ to_tsquery('bridge');
 note_subject |            note_content            | table_name
--------------+------------------------------------+------------
              | Designed by a great bridge builder | landmark
              | Iconic suspension bridge           | landmark
(2 rows)


This sort of interface has obvious uses when trying to do set/subset modelling where the interface is inherited.  Careful design is required to make this perform adequately however.  In the case above, we will be having to do somewhat deep inspection of all tables but we don't necessarily require immediate answers, however.


In general, this sort of approach makes it somewhat difficult to store records in both parent and child tables without reducing semantic clarity of those tables.  In this regard, the parent acts like a catalog of all children.  For this reason I remain sceptical whether even with NO INHERIT check constraints it will be a good idea to insert records in both parent and child tables.  NO INHERIT check constraints, however finally provide a useful tool for enforcing this constraint however.


Contrasts with DB2, Informix, and Oracle

DB2 and Oracle have largely adapted Informix's approach to table inheritance, which supports single table inheritance only, and therefore requires that complex types be stored in columns.  With multiple inheritance, if we are careful about column naming, we can actually in-line multiple complex types in the relation.  This provides at once both a more relational interface and one which admits of better object design.

Emulating Inhertance in Composite Types

Composite types do not allow inheritance in PostgreSQL, and neither do views.  In general, I would consider mixing inheritance and views to be dangerous and so would urge folks not to get around this limitation (which may be possible using RULEs and inherited tables).

One basic way to address this is to create a type schema and inherit tables from a table called something like is_abstract as follows:

CREATE TABLE is_abstract (check (false ));

No table that inherits is_abstract will ever be allowed to store rows, but check constraints are only checked on data storage because otherwise many problems arise so the fact that we have a check constraint that, by definition, always fails allows us to deny use of the table for storing data but not use of a relation as a class which could be instantiated from data stored elsewhere.

Then we can do something like

CREATE TABLE types.my_table (
    id int,
    content text
) INHERITS (is_abstract);

Once this is done, types.mytable will never be able to store rows.  You can then inherit from it, and use it to build a logical data model.  If you want to change this later and actually store data, this can be done using alter table statements, first breaking the inheritance, and second droping the is_abstract_check constraint.  Once these two are done, the inheritance tree can be used to store information.

Next:  Inheriting interfaces in set/subset modelling
Next Week:  Complex Types

Friday, August 24, 2012

PostgreSQL OR Modelling Part 2: Intro to Object Relational Classes in PostgreSQL

In the last post we went briefly over the considerations and concerns of object-relational programming in PostgreSQL.  Now we will put this into practice.  While LedgerSMB has not fully adopted this approach I think it is likely to be the long-term direction for the project.

In object relational thinking classes have properties and methods, and sets of objects are retrieve and even manipulated using a relational interface.  While relational systems look at sets of tuples, object-relational systems look at relational manipulation of sets of objects.  A table (and to a lesser extent, a view of a composite type) is hence a class and can have various forms of behavior associated with it the rows, which act as objects.  This means a number of considerations change.  Some differences include:

  • SELECT * is often useful to ensure one receives a proper data type
  • Derived values can be immitated by methods, as can dereferences of keys
  • Common filter conditions can be centralized, as can common queries
In general, understanding this section is important to understanding further sections in this series.

Basic Table Structure

Our current example set will use a very simplified schema for storing inventory.  Consider the (greatly simplified) chart of accounts table:

 CREATE TABLE account (
     id int not null unique,
     control_code text primary key, -- account number
     description text not null
);

Populated by:

INSERT INTO account (id, accno, description)
VALUES (1, '1500', 'Inventory'),
       (2, '4500', 'Sales'),
       (3, '5500', 'Purchase'); 

Of course in a real system the chart of account (and inventory) tables would be more complex.

CREATE TABLE inventory_item (
    id serial primary key,
    cogs_account_id int references account(id),
    inv_account_id int references account(id),
    income_account_id int references account(id),
    sku text not null,
    description text,
    last_cost numeric, -- null if never purchased
    sell_price numeric not null,
    active bool not null default true
);

Now, we'd also want to make sure only one active part can be associated with a sku at any given time, so we'd:

CREATE UNIQUE INDEX inventory_item_sku_idx_u
ON inventory_item (sku) WHERE active IS TRUE;

The create table statement also creates a complex type with the same structure, and thus it defines a data structure.  The idea of relations as data structures in themselves will come up again and again in this series.  It is the fact that tables are data structures which allows us to do interesting things here.

Method 1:  Derived Value called "markup"

The first method we may want to add to this table is a markup method, for calculating our markup based on current sell price and last cost.  Since this value will always be based on two other stored values, there is no sense in storing it (other than possibly in a pre-calculated index).  To do this, we:

CREATE FUNCTION markup(inventory_item) RETURNS numeric AS
$$ SELECT CASE WHEN $1.last_cost = 0 THEN NULL

               ELSE  ($1.sell_price - $1.last_cost)
                     / $1.last_cost

           END;

$$ LANGUAGE SQL IMMUTABLE;

Looking through the syntax here, this is a function named "markup" which receives a single input of the table type.  It then calculates a value and returns it based solely on inputs and returns a value.  The fact that it function always returns the same value when the same input is passed is reflected in the IMMUTABLE designation.  The planner uses this to plan queries, and PostgreSQL will not index function outputs unless they are marked immutable.

Once this is done, we can:

SELECT sku, description, i.markup from inventory_item i;

Note that you must include the table designation in calling the method.  PostgreSQL converts the i.markup into markup(i).

Of course, our table being empty, no values will be returned.  However if you try to omit the i. before markup, you will get an error.

Not only can we include this in the output we can search on the output:

SELECT sku, description from inventory_item i 
 where i.markup  < 1.5;

If we find ourselves doing a lot of such queries and need to index the values we can create an index:

CREATE INDEX inventory_item_markup_idx 
ON inventory_item (markup(inventory_item));

If you use object.method notation, you must add extra parentheses because of create index semantics [see comment added below]:

 CREATE INDEX inventory_item_markup_idx ON inventory_item ((inventory_item.markup));

Instead of adding columns and using triggers to maintain them, we can simply use functions to calculate values on the fly.  Any value you can derive directly from values already stored in the table can thus be calculated on output perhaps even with the values indexed.  This is one of the key optimizations that ORDBMS's allow.

Method 2:  Dereferencing  an Account

The above table is defined in a way that makes it easy to do standard, relational joins.  More specialized Object-Relational-friendly references will be covered in a future posting.   However suppose we want to create a link to the accounts.  We might create a method like:

CREATE OR REPLACE FUNCTION cogs_account(inventory_item) 
RETURNS account
LANGUAGE SQL AS
$$ SELECT * FROM account where id = $1.cogs_account_id $$;

We can now:

or_examples=# select (i.cogs_account).* FROM inventory_item i;

We will get an empty row with the chart of accounts structure.  This gives us the beginnings of a path system for LedgerSMB (a real reference/path system will be discussed in a future posting).

We can even:

select * from inventory_item i 
 where (i.cogs_account).control_code = '5500';

Note that in many of these cases, parentheses are necessary to ensure that the system can tell that we are talking about an object instead of a schema or other system.  This is true generally wherever complex data types are used in PostgreSQL (otherwise the i might be taken to be a schema name).  Unlike Oracle, PostgreSQL does not make assumptions of this sort, and unlike DB2 does not have a separate dereferencing operator.

Warning: Dereferencing objects in this way essentially forces a nested loop join.  It is not recommended when working with large return sets.  For example the previous has a plan (given that I already have one row in this table) as:

or_examples=# explain analyze
or_examples-# select * from inventory_item i where (i.cogs_account).control_code = '5500';
                                                 QUERY PLAN                    
                           
--------------------------------------------------------------------------------
----------------------------
 Seq Scan on inventory_item i  (cost=0.00..1.26 rows=1 width=145) (actual time=0
.174..0.175 rows=1 loops=1)
   Filter: ((cogs_account(i.*)).control_code = '5500'::text)
 Total runtime: 0.199 ms
(3 rows)

Note that the filter is not expanded to a join.  This means that for every row filtered upon, it is executing a query to pull the resulting record from the account table.  Depending on the size of that table and the number of pages containing rows referenced, this may perform badly.  For a few rows, however, it will perform well enough.

Method 3:  Computer-discoverable save method

One application for object-relational modeling is to provide a top-level machine-discoverable API which software programs can use, creating more autonomy and such between the application and the database.  We will create here a machine-discoverable save function which only updates values that may be updated, and returns the values as saved back to the application.  We can consider this sort of loose coupling a "dialog" rather than "remote command" interface because both are assumed to be autonomous and try to provide meaningful responses to the other where appropriate.

Our founction looks like this (not using writeable CTE's for backward compatibility):

CREATE FUNCTION save(inventory_item)
RETURNS inventory_item
LANGUAGE PLPGSQL STABLE AS
$$
  DECLARE out_item inventory_item;

 BEGIN
 -- we don't want to allow accounts to change on existing items
 UPDATE inventory_item
    SET sku = in_item.sku,
        description = in_item.description,
        last_cost = in_item.last_cost,
        sell_price = in_item.sell_price,
        active = in_item.active
  WHERE id = in_item.id;

 IF FOUND THEN
     SELECT * INTO out_item FROM inventory_item
      WHERE id = in_item.id;
     RETURN out_item;
 ELSE
 INSERT INTO inventory_item
                 (cogs_account_id,
                  inv_account_id,
                  income_account_id,
                  sku,
                  description,
                  last_cost,
                  sell_price,
                  active)

          VALUES (in_item.cogs_account_id,
                  in_item.inv_account_id,
                  in_item.income_account_id,
                  in_item.sku,
                  in_item.description,
                  in_item.last_cost,
                  in_item.sell_price,
                  in_item.active);
 SELECT * INTO out_item
      FROM inventory_item
     WHERE id = currval('inventory_item_id_seq');
    RETURN out_item;

  END IF;

 END;
$$;

This function is idempotent and it returns the values as saved to the application so it can determine whether to commit or rollback the transaction.  The structure of the tuple is also discoverable using the system catalogs and so an application can actually look up how to construct a query to save the inventory item.  However it certainly cannot be inlined and it will be slow on large sets.  If you have thousands of inventory parts, doing:

SELECT i.save FROM inventory_item i;

Will be painful and do very little other than generate dead tuples.  Don't do it.

On the other hand a software program (either at code generation or run-time) can look up the structure of the type and generate a call like:

 SELECT (i.save).* 
   FROM (SELECT (row(null, 3, 1, 2, 'TEST123', 'Inventory testing item', 1, 2, true)::inventory_item).save) i;

Obviously this is a sub-optimal interface for humans but it has the advantage of discoverability for a computer.  Note the "::inventory_item" may be unnecessary in some cases, however it is required for all practical purposes on all non-trivial databases because it avoids ambiguity issues.  We really want to make sure that it is an inventory_item we are saving especially as data types may change.  This then allows us to control application entry points to the data.

Note that the application has no knowledge of of what is actually happening under the hood of the save function.  We could be saving it in unrelated relations (and this may be a good way to deal with updateable views in an O-R paradigm where single row updates are the primary use case).

Caveats:  In general I think it is a little dangerous to mix imperative code with declarative SQL in this way.   Object-relational modelling is very different from object-oriented programming because with object-relational modelling we are modelling information, while with object-oriented programming we are encapsulating behavior.  This big difference results in endless confusion.

The most obvious way around this is to treat all SQL queries as questions and treat the transactional boundaries as imperative frames to the declarative conversation.  A human-oriented translation of the following exchange might be:

BEGIN;  -- Hello.  I have some questions for you.

SELECT (i.save).*
  FROM (SELECT (row(null, 3, 1, 2, 'TEST124', 
                'Inventory testing item 2', 1, 2, 
                true)::inventory_item).save) i;

-- Suppose I ask you to save an inventory item with the following info for me.
-- What will be saved?


  id | cogs_account_id | inv_account_id | income_account_id |   sku   |       desc
ription        | last_cost | sell_price | active
----+-----------------+----------------+-------------------+---------+-----------
---------------+-----------+------------+--------
  4 |               3 |              1 |                 2 | TEST124 | Inventory
testing item 2 |         1 |          2 | t


-- The above information will be saved if you ask me to.

COMMIT; -- Do it.

This way of thinking about the overall framework helps prevent a lot of problems down the road.  In particular this helps establish a separation of concerns between the application and the database.  The application is responsible for imperative logic (i.e. what must be done) while the database answers declarative queries and only affirmatively stores data on commit.  Imperative changes to data only occur when the application issues the commit command.

In this regard object behavior (outside of storage which is an odd fit for the model) really doesn't belong in the database.  The database is there to provide answers to questions and update stored information when told to commit changes.  All other behavior should be handled by other applications.  In a future post we will look at some ways to broaden the ways applications can receive data from PostgreSQL.  Object-relational modelling then moves beyond the question of "what information do I have and how can I organize it to get answers" to "what derivative information can be useful and how can I add that to my otherwise properly relational database?"

Alternate Constructor inventory_item(int)

Now in many cases we may not want to have to provide the whole object definition in order to instantiate it.  In fact we may want to be able to ask the database to instantiate it for us.  This is where alternate constructors come in.  Alternate constructors furthermore can be for single objects or for sets of objects.  We will look at the single objects first, and later look at the set-based constructors.

This constructor looks like:

CREATE OR REPLACE FUNCTION inventory_item(int)
RETURNS inventory_item
LANGUAGE SQL
AS $$

SELECT * FROM inventory_item WHERE id = $1

$$;

If I have an item in my db with an id of two (saved in with the previous method call) I can:

 or_examples=# select * from inventory_item(2);
 id | cogs_account_id | inv_account_id | income_account_id |   sku   |      descr
iption       | last_cost | sell_price | active
----+-----------------+----------------+-------------------+---------+-----------
-------------+-----------+------------+--------
  2 |               3 |              1 |                 2 | TEST123 | Inventory
testing item |         1 |          2 | t
(1 row)

I can also chain this together with other methods.  If all I want is the markup for item 2, I can:

or_examples=# select i.markup from inventory_item(2) i;
         markup        
------------------------
 1.00000000000000000000
(1 row)

I can even:

or_examples=# select (inventory_item(2)).markup;
         markup        
------------------------
 1.00000000000000000000
(1 row)

An application can then use something like this to traverse in-application links and retrieve new objects.

Alternate Constructor inventory_item(text)

Similarly we can have a constructor which constructs this from a  text field, looking up by active SKU:

CREATE OR REPLACE FUNCTION inventory_item(text)
RETURNS inventory_item
LANGUAGE sql AS $$

SELECT * FROM inventory_item WHERE sku = $1 AND active is true;

$$;

We can then:

SELECT (inventory_item('TEST123')).markup;

and get the same result as before.

Warning:  Once you start dealing with text and int constructors, you have the possibility of ambiguity on queries in.  For example, SELECT inventory_item('2') will run this on the text constructor instead of the integer constructor, giving you no results.  For this reason it is a very good idea to explicitly cast your inputs to the constructor.

Set Constructor inventory_item(tsquery)

Not only can this be used to create a constructor for a single item.  Whole sets can be constructed this way.  For example we could create a text search constructor (and this allows the default search criteria to change over time in a centrally managed way):

CREATE OR REPLACE FUNCTION inventory_item(tsquery)
RETURNS SETOF inventory_item
LANGUAGE SQL AS $$

SELECT * FROM inventory_item WHERE description @@ $1;

$$;

This allows some relatively powerful searches to be done, without the application having to worry about exactly what is searched on.  For example, we can:

or_examples=# select * from inventory_item(plainto_tsquery('test')); id | cogs_account_id | inv_account_id | income_account_id |   sku   |      descr
iption       | last_cost | sell_price | active
----+-----------------+----------------+-------------------+---------+-----------
-------------+-----------+------------+--------
  2 |               3 |              1 |                 2 | TEST123 | Inventory
testing item |         1 |          2 | t
(1 row)

Here test has been found to match testing because testing is a form of the word test.

Of coruse if we have more than one item in the table, we get more than one result:

 or_examples=# select * from inventory_item(plainto_tsquery('test'));
 id | cogs_account_id | inv_account_id | income_account_id |   sku   |       desc
ription        | last_cost | sell_price | active
----+-----------------+----------------+-------------------+---------+-----------
---------------+-----------+------------+--------
  2 |               3 |              1 |                 2 | TEST123 | Inventory
testing item   |         1 |          2 | t
  4 |               3 |              1 |                 2 | TEST124 | Inventory
testing item 2 |         1 |          2 | t
(2 rows)


These provide examples, I hope, provide some ideas for how one can take the object-relational concepts and apply them towards building more sophisticated, robust, and high-performance databases, as well as better interfaces for object-oriented programs.

Next Week:  Table Inheritance in PostgreSQL