Wednesday, March 8, 2023

Which is worse when working on production databases? Being drunk or tired?

I have decided to do a series of mini-articles on human factors in database operations.   This is the first, and covers fatigue.

In my talk at the PostgreSQL devroom of Fosdem, I asked a few questions:

1.  How many of you have seen someone work on a production database while drunk?  About half the audience.

2.  How many times does this cause a major incident?  No hands.

3.  How many of you have seen someone cause a major incident by working on a production database while tired?  Half the audience again raised their hands.

As an industry, we do not take fatigue seriously enough.  We appreciate people who come in and work after long disruptive on-call shifts.  We don't tell people they are tired and therefore not safe to work on production systems.

We need to do better.  Every single major mistake in my career that has caused production problems has been caused either by power distance problems or by fatigue.

I am not saying people should come into work drunk.  There are probably a number of contextual aspects to why drunkenness doesn't cause a problem in these cases.  However I am saying that people should not touch production systems under fatigue.

Of course this is easier said than done,  If we are drunk, we can feel it, but with even light stress, we often don't feel our fatigue.  We aren't capable of self-monitoring our conditions in this regard.  Fatigue is thus insidious -- it gradually sneaks up on our, invisible, until we make critical mistakes and bad things happen.

While there are reasons to weigh the balance differently in some areas such as operating motor vehicles (to say nothing about flying an aircraft), the fact is that general brain-intensive can be impaired via moderate fatigue perhaps more than levels of alcohol we consider unacceptable while driving.

If we value production operations, we should adopt the following rule:  friends don't let friends work on the production databases tired.

Friday, March 3, 2023

The Coming Storm: Geopolitics and PostgreSQL in a Changing World

 At PGConf India, I watched Bruce Momjian's excellent talk Future Postgres Challenges.  This talk discusses technical, technological, and project-related challenges Postgres has faced and continues to face.  This immediately lead me to ask a question about efforts at geopolitical disentanglement and how we should try to avoid them in the Postgres community.  While this question immediately gets interpreted through the filter of the ongoing war in Ukraine, it is far broader, as the US has been trying to disentangle itself from China  and efforts there are ongoing.  The problems, however, are not new.

When we as a community had decided to have a code of conduct, there was a lot of concern that the code of conduct would be used in politically motivated ways, in particular regarding culture war topics.  I wrote a piece describing some of the issues which I also gave geopolitical importance to at that time.  Culture war topics, I reasoned, are necessarily cultural, and hence trying to push one group's cultural ideas on the world through an open source project would be very harmful.

I wrote that piece in early 2016, before the Liberal International Order began to visibly fall apart with votes for Brexit and the election of Donald Trump.  But the piece remains relevant because the dangers of allowing our community to be torn apart by current geopolitics are, if anything, even worse than they are in the culture war space.

If, as some would like, we were to shun community members and contributors who are resident in the "wrong" countries, or if we were to treat military contracts to American and Russian or Chinese governments differently, then we would risk the possibility of a fork forming around geopolitical lines.  Nothing good will come out of that for anyone.  And so the task ahead is to make sure that we continue making sure our community is culturally and geopolitically inclusive.

I expect that we will face a lot of pressure in coming years to bifurcate the community due to differing geopolitical perspectives, but such could well be fatal for Postgres as a project.  Let's not let that happen.

Edit:  I am moderating the comments as follows:  What is on topic is a discussion about what is good for the project and community.  Arguments about what is good for community that incidentally argue geopolitics are acceptable.  Efforts at arguing geopolitics untethered from that question will get removed.  I am trying to cultivate constructive discussion on a topic of community importance and sometimes this means pulling weeds (by which I mean deleting comments that detract from that). 

Also all conversations need to be in the spirit of discussing what is right, not who is right.  If you wonder why your comment got deleted, that's why.