Debian 13, Postgres, and the US time zones

20 comments

This does not trace things back far enough. The root is where IANA has long since segregated out a set of timezone file names into a "backward" collection:

* https://data.iana.org/time-zones/tzdb/backward

If one traces references, one finds this connected bug on Launchpad. Amusingly to anyone who has ever seen these sweeping timezone database changes over the years, Launchpad marked it as "This bug affects 1 person.".

* https://bugs.launchpad.net/ubuntu/+source/tzdata/+bug/200807...

The rules for the "backward" file are here:

* https://data.iana.org/time-zones/theory.html#naming

All of the US/* timezone names, such as US/Pacific here, have been backwards compatibility measures in place for the whole of the 21st century and some of the late 20th. The Olson database in the 1980s (mod.sources v08i085, comp.sources.unix v14i030) used these names. But the naming scheme changed somewhen in the 1990s to a continent/city and ocean/city form and backwards compatibility with the old names has been preserved ever since by the "backward" file.

* https://groups.google.com/g/comp.unix.solaris/c/ON_MPZxVdv0/...

Debian has moved into a non-depended-upon package a backwards compatibility measure that is as old as Debian itself.

> Launchpad marked it as "This bug affects 1 person.".

That just means no one has clicked "affects me too" button yet (after logging in).

It's good Debian is making this change now to be prepared when the legacy United States are downed at the end of 2052.

That is the logic for using city-based timezone names rather than countries. Country borders change but cities tend to be more stable.

I can’t tell whether you’re joking. Seems like a distinct possibility now.

2052? I hope the last 2 digits aren't swapped...

This is why Debian users should use apt-listchanges to display the latest NEWS.Debian items on upgrade.

I wouldn't expect to see all the important news of the tens of thousands of packages I don't have installed in the release notes.

I was hit by this using unstable (before they made a NEWS item), but when upgrading my stable machines to trixie I got a proper warning/reminder of this specific thing.

> The worst part about this is that it didn't get so much as a mention in the Debian 13 release notes. I read through that document before going for it and never encountered it. Indeed, even now, you won't find "tzdata" or "zone" in it.

I had another issue during my upgrade to Debian 13 that also wasn't mentioned in the release notes. I filed a bug report, but I was told that the issue was not important enough to put in the release notes, and I should have instead more closely read NEWS.Debian of the package. So I don't really know anymore what the "Issues to be aware of" chapter in the release notes is for, because apparently it's only a small selection of issues you need to be aware of.

Sharing undocumented gotcha: nginx default config (/etc/nginx/nginx.conf) now has `server_tokens off` set with a comment that it's "common practice" (agreed). This is not in upstream's git version of the file[1], I therefore presume it's a Debian maintainer change.

The operational upgrade failure is if you have any existing drop-in config that sets `server_tokens off` already a hard fail Nginx error about duplicated keywords will occur, causing the apt process to exit with failure code(s) during the standard upgrade process.

[1] https://github.com/nginx/nginx/blob/master/conf/nginx.conf

What was your issue ?

The version of pdns-recursor in trixie uses a different configuration file syntax then older versions.

(There is a tool to convert the old configuration syntax to the new, but it requires a working installation. There is a command line option to enable support for the old format, which if nothing else helps to be able to run the conversion tool, but there's no information how to enable that command line option in the way Debian starts pdns-recursor.)

Hang on a second, "(...) in 2023. US/* was moved to tzdata-legacy (...)"

US/* was moved to 'backward' (the file for backward compatibility) in the tz database in 1993(!) and as such was essentially marked as deprecated long enough. https://data.iana.org/time-zones/tzdb/backward

You're telling me you didn't notice ? It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.

Not only did I not notice, I have never known that country prefixes were a thing, having to deal with tzdata since 1999. I wonder if that timezone was typed in manually? I doubt Postgres 15 file contained it to begin with.

> You're telling me you didn't notice ? It was...

In a large fraction of cases in the FOSS world, it comes across that the developers really do want to communicate this sort of thing, but there's no clarity on where or how they should do so. See for example various deprecations in Python packaging tools (and standards).

In this case, they did communicate it, and the aforegiven Vogon reference is a mischaracterization. The naming convention is in the current IANA doco and Eggert copy.

* https://data.iana.org/time-zones/tz-link.html#tzdb

* https://web.cs.ucla.edu/~eggert/tz/tz-link.htm#tzdb

Paul Eggert explained the continent/ocean plus largest city naming convention on a WWW page almost a quarter of a century ago. The WWW page was so well publicized that you can find its URL baked into at least four of the O'Reilly animal-cover books from the early 2000s.

* https://web.archive.org/web/20011023074744/http://www.twinsu...

It was explained on Usenet and on mailing lists prior to that.

You know, the tzdata people are quite haughty. They claim to store all that change, accurately, and yet here we are.

An example of this falsehood? Well, in the 70s my father convinced most of my hometown, at least the portions between Main St and Wharf, that DST was absurd.

For almost an entire year, this was observed.

Do you think they kept record in tzdata? I tried to convince them, but no! I still have some dateplanners my father had printed up, and even a picture of the sign that was out on the road (to alert visitors!).

But no!

Do not trust the tzdata people. As you can see, they are not so accurate.

Once upon a time I stumbled upon a small book of maps showing the time zones in Indiana each year in the mid-20th century. It was fascinating to see how various towns and cities would jump back and forth between Eastern and Central year to year, with little regard for the surrounding rural areas.

I frequently kick myself for losing track of that book.

You are just not providing sufficient proof. Think Wikipedia style proof. The fact that DST is or isn't observed should be published in a real book. A random dude printing some date planners or putting up a sign to be photographed isn't enough proof.

Then you get the reverse. I just upgraded to macOS Sonoma (yes I'm always one major version behind with Apple stuff...), and I was annoyed as heck when I had to click through "Look what's new in Calendar!", "Look what's new in Reminders!", "Look what's new in StripClubs!"... I need to use my software right now, I will not read this. Then I will forget it ever popped up, and will not read it in the future either.

There's a vast difference between Apple showing you all it's useless new AI integrations and developers telling system administrators what they need to know.

And I think Debian sent those notes also via its mail system (exim + mail). I don’t usually care for my workstation, but I do check it for my servers.

NEWS.Debian entries are displayed by apt-listchanges in a pager by default when running apt upgrade, as well as sent by email.

Why are time-zones even prefixed by continent? Country-prefixed time-zones make more sense because they're defined politically.

Cities may find themselves in other countries easier than on other continents.

In addition to what others have said, there are several examples where people disagree about which country a city is “rightfully” in.

Nobody can really find fault with Asia/Jerusalem, whereas either Israel/Jerusalem or Palestine/Jerusalem would be controversial.

I disagree. Country borders can move. I have not heard of a city moving between continents however.

I ran into a similar bug in Git Lab where the v18 upgrade didn’t replace now-unsupported time zones so things like scheduled pipelines simply stopped running and were labeled inactive in the UI without explanation.

https://gitlab.com/gitlab-org/gitlab/-/issues/556779#note_26...

The fix is simply to change them to the non-backwards linked names but it caused some confusing API errors since data which would no longer pass validation was already in the database and didn’t look wrong.

I also ran into this using:

- Debian 13

- Interactive Brokers' Trader Workstation

- Racket's `gregor` date and time library

IBKR still sends the old, deprecated US/* timezones. As noted in the article, the solution is to `apt install tzdata-legacy`.

I feel like I’ve been using America/New_York for a decade at least. Am I misremembering?

That’s one of the normal Continent/City time zones, not one of the weird Country/ZoneName ones that have been deprecated for ages. “America” here refers to the continent or continents[0], not the USA.

0: Note that different cultures disagree on whether there are two continents called North America and South America, or one continent called America.

No, that’s been the standard name for a quarter century.

Ooofff. The two difficult things in software engineering, naming, and timestamps.

This hit me in the early 2000s and now everything I do is in UTC. All dates, timestamps, everything, UTC. If you want to look at a local window in time, convert the window to a utc start and end date and search. When viewing, use a js function to translate the utc date to a local one to print. The mental gymnasium of local to utc to local again…

  > The two difficult things in software engineering, naming, and timestamps.
And off-by-one errors. The two most difficult things in software engineering: naming things, timestamps, and off-by-one errors.

Some times cannot be expressed in UTC.

For example: “this meeting will take place at 10 AM on July 31st, 2026, US Pacific time” cannot be expressed in UTC. You can guess what time UTC that refers to, and you’ll probably be right, but you’ll be wrong if it turns out for example that the US abolishes DST before that date.

In that case it is no really a time, it is a condition.

I don't get how this is a snag, it's right there in the log.

Agreed.

> At the time, I went "WTF?" and just commented it out to get it running again. I had bigger fish to fry... and just kind of forgot about it. Everything seemed fine.

I get this, but now you get to laugh and dust yourself off. If it had silently started doing the wrong thing, that would be a worthy complaint.

I'm still not totally sure why these names were deprecated in the first place...

I mean, the folks who run the tz db definitely know what they're doing, it just never 100% clicked with my thinking.

I always prefer `US/Eastern` over `America/New_York` -- it seems more "canonical" to me. New York is _currently_ the anchor city for ET, but will it always be? The place I live (Boston) is currently on ET, but in the future it might be on Atlantic Time. If there was an `America/Boston`, I would use that to be safe, but since there isn't, it just seems better to be to be specific that I mean "Eastern Time" and not "whatever the time is in NYC"... At least then if Boston switches to a different tz, I could intentionally switch to "Atlantic Time" -- doesn't that make more sense? Versus I guess what I'd have to do, which is switch to `America/Puerto_Rico`? (I had to actually search that one, too bad there's no `US/Atlantic`...)

If New-York shifts its timezone, then it will still be New-York and America. And the new tzdata will reflect that shift so your data will always be correct (as in reflecting the time it is in new york). For local time, you will switch to a new city and the date will still be correct.

Exactly, but that is my point: usually my data is not intended to reflect the time it is in New York. Being tied to a (semi-)arbitrary city changes the actual meaning of the zone slightly. If every city was represented and I could choose "Boston" then that would make sense (for data is intended to reflect the time it is in Boston) but of course that's not entirely practical.

(I'll note that I agree with the general wisdom to store data in UTC; here when I talk about zones I'm either talking about user local machine time or display)

FWIW if Boston switches, it won’t be America/Puerto_Rico, it’ll get a new zone name (probably America/Boston). Tzdb zones express that everywhere in that zone has always been on the same time, since the advent of standard timekeeping, so they always fracture when some subset moves to a different zone.

Always run production systems in the Etc/UTC timezone. This eliminates an entire class of problems while only creating minor inconveniences.

I work for a company with servers, employees, and customers in basically every time zone around the world. And yet every server, internal tool, and workflow uses Pacific time. UTC is used precisely nowhere. Setting aside the issues of DST, I imagine it's convenient for the employees and managers in HQ but absolute madness for everyone else.

Are you at Google? Because they also keep using US/Pacific timezone in every incidents where it affects everyone over the globe

That description could match nearly any tech company in the Bay Area or Seattle and everything along the coast.

I work with a development team who manages data integration and migration for massive datasets, and they have sensibly set a standard that every date/time value they store in their databases will be UTC.

But they explicitly or implicitly have also decided not to store the timezone in the strings, so every single value is technically ambiguous. Absolutely drives me crazy.

Update: since there have been questions, these are strings, not native datetime values.

Postgres always stores date/time values in UTC (if they are timezone-aware)

These are strings. And not Postgres.

> they have sensibly set a standard that every date/time value they store in their databases will be UTC.

Not sensible at all for future date/times.

For servers, i agree 100%. For desktop systems, not so much.

How else could you tell what time it is on the servers?

That's easy, if you're Walmart just use the TZ of your corporate HQ

(That's what I heard anyway)

My non-Walmart startup was expanding from California to other regions many years ago, and there was some debate about whether we’d use UTC or America/Los_Angeles globally. I was part of Team Over My Dead Body.

But on which side? :)

There can be only one!

I work for a different company headquartered in the central time zone and we also do this. On some systems.

Google's internal timestamps are in Pacific Time, too.

Eastern Standard Tribe, represent!

I appreciated the reference.

It also makes sense. Timezone is user specific - if you have users from all over the globe, they will need different settings, so this should be set in frontend.

You can also run in UTC+$YOUR_OFFSET timezone if you don't use DST.

On the other hand, UTCx timezones are not silver bullets if your fleet is a part of a multi-continent federation.

Is there a problem with ISO3166 denoted information in general or is there a specific US issue here? I would think ISO code denoted tzdata was a public good in some sense.

This has nothing to do with ISO 3166 at all. The tzdata database is the work of a single person, Arthur Olson, not a committee (or rather it was, from its birth in the 1980s until 2011 when a company decided to sue him for no reason).

And for most of its life it’s been an explicit policy that timezones are named by continent and representative population center, not by country, to avoid entangling it in territorial disputes and improve naming stability for historical data. The US/* (and Canada/*, etc.) names are deprecated and have been since 1995 (?), but apparently people were still using them because the deprecation wasn’t really apparent unless one was especially into reading release notes.

All the legacy time zones were moved out of the default zoneinfo install. It’s not a us-specific issue but the legacy US/ timezones remain in widespread use, and they stop working on Debian 13 ootb (possibly Ubuntu noble as well?).

Does anyone know why they are still in widespread use?

Config defaults somewhere still using them? Man page examples? Tutorials using them? Or just force of habit?

> Does anyone know why they are still in widespread use?

Because of a lack of things compelling people to change them until it causes a breakage. And then when it does cause a breakage, most people would rather move heaven and earth to complain, research workarounds, etc. rather than just change it. (Institutional structures can also make "just" changing it far harder than that should be.)

Some of this is surely just muscle memory or intertia as well. I remember random config values from when I was trying out linux boxes back in high school that I replicated into files that just don't get touched for decades afterwards.

When was the last time you rebuilt your company's postgres config from scratch?

> When was the last time you rebuilt your company's postgres config from scratch?

Last year, when we upgrade to version 17.

I looked at the example/template configuration, diffed it with our configuration from PG15, and for every change decided whether to keep our version or the new setting.

I didn't use it, but Debian/APT has had a tool to do this sort of comparison for any software upgrade for as long as I can remember.

Do other people just copy the old config and shout "YOLO!"?

Still typing "nano -w filename" each and every time since back around y2k when I was working on Linux for the first time I was told that bad things could happen if I didn't...

I suspect some of it will be because the legacy form is a bit more intuitive than the standard form. You don’t really use continents and cities as a reference to time zones normally, countries and local subdivisions makes more sense, but as other people note, it brings up POLITICS.

You don't use them normally in the US, I've been referring to europe/amsterdam or europe/paris all my life in Linux installers and various equipment. I've never ever encountered netherlands/amsterdam or something like that.

It’s the same in Europe as it is in the US. Normal people refer to Europe/Paris as CET, just like normal people refer to America/New_York as Eastern Time.

From the list of deprecated zones, we could have been using "GB", "Poland", "Portugal", "CET", but that's about it. "Netherlands" didn't exist.

Given the old names were deprecated in 1993, it's hardly surprising that I never before discovered "GB".

There’s definitely inertia but I think it’s also that the US/ names match official usage: nobody, not even residents, says the time zone is New York because the official name is Eastern time.

Exactly this. I still don't know if there's a technical difference between America/New_York, America/Detroit, or America/Indianapolis.

America/Detroit is different from America/New_York on times before 1915 (when Detroit switched from Central to Eastern Time).

I wonder how much of an influence it is that US/Eastern is easier to type than America/New_York

Are they in widespread use? They were deprecated in 1995.

I don't see conflict in those statements.

love the takes on this one blog, but "I'm copying an old conf file for over 20yrs and now got this issue" is kinda weak.

American exceptionalism time zones aren't used since the 90s. even the cpus from that time are already dropped from kernel support. heck even the text encoding is gone.

That’s unkind. I want to do the right thing in such cases, but I’m also learning about this today for the very first time. I’ve never, not once, heard that US/Pacific was a bad idea until this post. If not for this, I still wouldn’t know. I thought it and America/Los_Angeles were semantically identically and just kind of symlinks to PST8PDT or whatever.

If anything, the city TZ always felt off, like I was opting in to that specific city’s strange legal decisions or something.

For general user interfaces, it's for the user interface to show "US Pacific" or "UK" rather than America/Los_Angeles and Europe/London.

The timezone selector in KDE shows "Los Angeles | America/United States of America | Pacific" and "London | Europe/United Kingdom", for example.

Imagine my outrage when America/Montreal was deprecated years ago. “It’s the same as America/Toronto, just use that” they said :)

Have Montreal and Toronto ever been in different timezones?

No they have not. But why not nix Toronto instead? Everybody knows Montreal is better, it should get dibs at naming the time zone.

I’m just joking all the way, by the way :)

Toronto's bigger. Same reason Boston and Washington are "New York" and Seatle/SanFrancisco are Los_Angeles

(Detroit gets its own zone because it was significantly different to new york before 1970, normally it's only difference post 1970 which merit a new zone)

> At the time, I went "WTF?" and just commented it out to get it running again. I had bigger fish to fry... and just kind of forgot about it. Everything seemed fine.

You're running a database system and you just casually comment out the configuration setting the timezone?

In what way did everything "seem fine"? SELECT 1 returned something? No further investigation required??

What sort of monster are you by not defaulting to UTC on systems? (⊙_⊙') /s

[flagged]

The timezone names are defined by the Olson database, not Debian. It is the only sensible system in our ecosystem. It certainly beats Outlook which still wrongly insists that my timezone is GMT just because I live in the UK and still confuses everyone (it isn't during the summer; we use BST over the summer, not GMT).

> It is the only sensible system in our ecosystem. It certainly beats Outlook which still wrongly insists that my timezone is GMT just because I live in the UK and still confuses everyone (it isn't during the summer; we use BST over the summer, not GMT).

I went to both Outlook and Teams to check and I have the option to select both "(UTC) Universal Coordinated Time" and "(UTC+0) Dublin, Edinburgh, Lisbon, London", with the later adapting to changes in the summer; but I do agree it's clunkier than the Olson database, combining multiple regions in a single option while splitting regions with the same timezone rules into different ones.

> ...I have the option to select..."(UTC+0) Dublin, Edinburgh, Lisbon, London"

This is factually wrong already. In summer, London is not UTC+0. They mean "UTC+0 ignoring DST", but that is not useful. If they're going to be specific by specifying a UTC offset, what's the point if it doesn't include DST? How is that useful as an identifier when it's ambiguous? With their history of getting it wrong, this just introduces doubt about its correctness.

Further, if you ask Outlook to show you two timezones at once and do not override labels, it will label BST "UTC+0" (it isn't; it's UTC+1!) while also calling eg. India "UTC+5:30", implying a time difference of 5.5 hours when it is actually 4.5 hours. This isn't just a case of "ah - they actually mean ..."; it's most definitely wrong!

The problem is that it has a very US-centric view of what DST is. You can mostly ignore it in the US when calculating time zones because the entire country changes DST at the same time. This is not the case internationally.

> This is factually wrong already. In summer, London is not UTC+0.

Yeah, the "regular" time is UTC+0, with it changing in the summer. I'm aware it's a really poor implementation, but it is there as a separate option from "UTC" itself preserving the same offset (0) all year.

> The problem is that it has a very US-centric view of what DST is. You can mostly ignore it in the US when calculating time zones because the entire country changes DST at the same time. This is not the case internationally.

Probably the reason they, for some reason, split the setting for "Amsterdam, Bern, Berlin, Rome, Stockholm, Vienna" and "Brussels, Copenhaguen, Madrid, Paris" even though they all follow the same timezone and change simultaneously.

> In summer, London is not UTC+0. They mean "UTC+0 ignoring DST", but that is not useful

This is how 99% of people interpret it

It's not ambiguous as you imply.

Summer time is not the default time

I don't know enough about the India case to know how/why it's wrong though

Tell that to Ireland where we observe Irish Standard Time during the summer (and GMT during the winter).

Tell what? Nobody cares, because everybody knows what is being said

Your TZ doesn't change between summer and winter. What changes is the shift

I literally didn't see anybody getting confused with this in any country (yes, including Ireland) with summer time.

But some people think they're too smart when they nitpick about minor issues

> Your TZ doesn't change between summer and winter. What changes is the shift

My TZ is GMT in winter and BST in summer. I am not in GMT in summer. GMT continues to exist in summer, doesn't shift but my clock doesn't follow it.

The UTC "shift" changes indeed. When I am DST-shifted, calling me "UTC" is absolutely wrong.

The practical issue is that people still use "UTC" and "GMT" interchangeably, which is roughly correct anyway since they remain the same in practice. But then during summer when someone says GMT I don't know if they actually mean BST or UTC. That ambiguity only arises because Outlook (and you, apparently) conflate GMT and BST. It's far more of a problem for those actually living in a UTC-adjacent time zone (do you?)

Any moderately reasonable system would be backwards compatible and/or migrate existing values

> The worst part about this is that it didn't get so much as a mention in the Debian 13 release notes. I read through that document before going for it and never encountered it. Indeed, even now, you won't find "tzdata" or "zone" in it.

It's been backwards compatible for 30 years, you've had plenty of time to migrate

Did it give any noticeable deprecation warning? If a piece of functionality is deprecated but users can't notice, is it really deprecated?

How and where, in principle, do you suppose it should (be able to) do so, given the functionality? It's just localizing timestamps, and the result might bubble through multiple layers of software before being presented to anyone. It's not as if C APIs get designed up front to return structures with a boolean "btw you called this in a nonstandard way that might fuck up in the future" flag (or, say, a pointer to a deprecation-warning structure; and then that interface has to be rock-solid stable to be of use) for every call. And if they did, nobody would ever write code that checks it.

> How and where, in principle, do you suppose it should (be able to) do so, given the functionality?

System logs?

> It's not as if C APIs get designed up front to return structures with a boolean "btw you called this in a nonstandard way that might fuck up in the future" flag (or, say, a pointer to a deprecation-warning structure; and then that interface has to be rock-solid stable to be of use) for every call. And if they did, nobody would ever write code that checks it.

Yeah, fair point, no-one should be using a C API at this point.

> “Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard’.”

The timezones are very clearly marked as deprecated on the lists I looked at, but on the other hand I don't live in the USA so I've never had to look particularly hard. Everything I configure is either Etc/UTC or Europe/London which is nice and easy to remember...

> Any moderately reasonable system would be backwards compatible

It is. Install: tzdata-legacy

Debian is known to have made similar monstrously stupid decisions.

For example, they patch OpenSSH source code in a way that makes defaults behave differently than upstream. In the name of backwards compatibility of course.

I assume this will continue until it doesn't anymore, and the only notification you shall receive from the ivory tower is a cryptic one-liner buried in a changelog somewhere.

> For example, they patch OpenSSH source code in a way that...

Isn't it the same thing with the RedHat downstreams ? (Not necessarily OpenSSH but other packages)

IIRC RedHat do all sorts of things to keep their gov / corp customers happy, also usually in the name of backwards compatability, all of which then end up in the downstreams.

Russ Allbery left over bureaucracy and systemd. It sounds like it's chocked full of people who want power and an excuse to patch downstream to create a cottage industry of quirks, busywork, and codependency.

I prefer real choice and light patches that try to upstream as much as possible, or workaround upstream obstinacy rather than create incompatible idiosyncrasies. One area that isn't well represented in barely a/no distro is init freedom neither married to nor completely divorced from the sprawling octopus.

You are confusing Russ Allbery with me, while at the same time making it sound like I have a problem with systemd, which is not the case. Russ remains a debian developer.

I don't think you can be partly married to Cuthulhu. (Systemd.)

[ eldritch noises intensify ]

As an actual answer, it's not too bad on Debian; we only really use/need: systemd (system and user), -logind, -journald, -udevd. All in all, not too many tentacles but there are a few...