I recently wrote an entry for OmniTI’s Seeds Blog. The article talks about the idea of pattern matching vs. troubleshooting, and how those ideas relate to what people consider a “Tech Guru”. I hope you’ll check it out.
New Blog Rising
Almost 5 years ago (really?) I moved my site from the old “planet postgresql” blog hosting service over to my own server, pointing to my own domain. This server was hosted in OmniTI’s Ashburn data center, and it’s been a good home, but we are in the final stages of a data center evacuation project, which all of our sites, and our clients sites, are moving to either our new Ashburn facility, or our Chicago data center.
My old blog ran on an elderly version of S9Y, a PHP based blogging engine that also supported PostgreSQL. Like all new things, I liked it when I started with it, and even now I’d recommend it to folks looking for a blogging system, but rather than trying to port or upgrade the old system, I thought I would take this as an opportunity to try something new. The code I am most likely to touch these days is probably either PHP or node.js, with perl the outside third, so I figured I would look for something else. I did want to limit myself to a pre-built blog engine; while I think running software in different languages is valueable, I know I don’t have time to commit to maintaining my own system from scratch. You’d think that a blog wouldn’t be that complicated, but it’s harder than you’d think. Anyway, I originally was hoping to find something in Go, but blog options were lacking, and as I understand it, Go support on Illumos is still lacking, so I decided to look elsewhere. In the end, I ended up settling on Octopress/Jekyll. I have to admit, I was a bit resistent to go this route, because it almost seems cliche, but I figured that between the ruby and static page generation, it would be different enough to be interesting.
Some of the guys at OmniTI are doing a bunch of hacking on chef, but I can’t say that I’ve done any real ruby hacking in probably seven years. Still, Octopress is suppose to make everything easy, right?
Well, to be frank, I found the state of ruby’s development ecosystem to be a bit of a mess. I suppose that all the bits and pieces are meant to make things work better, but the need to juggle multiple ruby versions feels pretty painful. I’ll be honest, I tried to bypass it all by just using macports ruby, but it wasn’t up-to-date enoughr. Even when I started looking at the various Ruby environment managers, the version differences coupled with compiler issues on OSX made me question the whole thing, but eventually I ended up going with rbenv, and also making use of rbenv-gem-rehash to get everything to work correctly.
Once I was able to get a working ruby environment going, I proceeded to get a basic blog system up and going. That in and of itself required more wrangling than I expect, but I want to save that for another post, since this post is already getting long. The last thing I want to mention is that, although I originally had planned to move everything to one of our hosted OmniOS systems, in the end I decided to take a different route and host the system on Heroku. Once again, this was mainly in the spirit of learning something new. I know a bunch of the guys working on Heroku Postgres and I’ve always been interested in Heroku services in general, and this perticular site seemed simple enough that hosting it on one of their free hosts seemed pretty doable. I do miss not having tools like mod_rewrite or ATS to handle any black magic I might need, but for now things are running ok.
New Postgres Backup and Restore Book

phpPgAdmin 5.1 Released!
Download
To download phpPgAdmin 5.1 right now, visit: http://phppgadmin.sourceforge.net/doku.php?id=downloadFeatures
- Full support for PostgreSQL 9.1 and 9.2
- New plugin architecture, including addition of several new hooks (asleonardo, ioguix)
- Support nested groups of servers (Julien Rouhaud & ioguix)
- Expanded test coverage in Selenium test suite
- Highlight referencing fields on hovering Foriegn Key values when browsing tables (asleonardo)
- Simplified translation system implementation (ioguix)
- Don’t show cancel/kill options in process page to non-superusers
- Add download ability from the History window (ioguix)
- User queries now paginate by default
Translations
- Lithuanian (artvras)
Bug Fixes
- Fix several bugs with bytea support, including possible data corruption bugs when updating rows that have bytea fields
- Numerous fixes for running under PHP Strict Standards
- Fix an issue with autocompletion of text based Foreign Keys
- Fix a bug when browsing tables with no unique key
Incompatibilities
- phpPgAdmin core is now UTF-8 only
- We have stopped testing against Postgres versions < 8.4, which are EOL
Join the 5%
In the next 48 hours, American all across the country (well, half of them anyway) will head to the polls to cast their votes for President. But what does it mean to have a vote that counts? In the year 2000, sitting in Florida, watching that election unfold, I think I have never been closer to having a vote that counted. For a true cynic, sure, my one vote would not have changed the election. However, with a margin of ~500 people, it wasn’t lost on me that I actually knew enough people that had we all voted together, it could have changed the entire election. You can’t get much closer to a vote that counts than that.
In 2004, the election was not nearly as close. With a margin in Florida of almost 400,000, I certainly didn’t know enough people to swing that one. After that I moved to Maryland, and any illusion of a vote that would change the outcome of an election completely disappeared; Maryland is a state that has voted Democrat by double digit margins for years, with no signs of a change. Regardless of if you are voting Republican or Democrat, the outcome here is fairly certain. Of course, Maryland is not alone.
The above graph lists the “likelihood your state will determine the presidency” (source). If you aren’t in one of those states, the truth is that your vote means very little to the outcome of who becomes president. This isn’t to say you shouldn’t vote; it never hurts to take part in the political process, and to be sure there are always a number of state level initiatives that are worth voting on. Some would look at that and say that for most people, voting for president doesn’t really matter. Normally I’d agree, but this year there is a chance that things could be different.
While I’ve no illusion that they will win the election, this year the Libertarian party has the chance to do something significant: obtain 5% of the popular vote. If that happens, they will be eligible to receive matching funds for 2016. While this isn’t significant to the two majority parties (who have opted out of the program so as to not *limit* their fundraising), for a third party this would be a major milestone. If you’ve been dissatisfied with your party, or you live in a state where the outcome is solid, I’d urge you to join me in voting for Gary Johnson. Even if you don’t agree with all of their policies, you probably agree with some; but whether you do or not, the real issue here is getting the Libertarian party to 5% also means getting a whole slew of issues up for discussion which are sorely lacking from the current two-party system we’re working under. That’s something that would count, and definitly something worth voting for.
Shoot the Automated Failure in the Head
- On Monday, they attempted a schema migration which lead to a load spike.
- The high load triggered an automated failover to one of their MySQL slaves.
- Once failed too, the new master also experienced high load, and so the automated failover attempted to revert back
- At this point, the ops team put the automated failure system into “maintenance mode”, to prevent further failover
Our database team at OmniTI is often asked about what type of process we normally recommend for dealing with failover situations, and we stand by our assessment that for most people, manual failover is typically the best option. It’s not that the idea of automated failover isn’t appealing, but the decisions involved can be very complex, and it’s hard to get that right in a scripted fashion. In their post, the Github team mentions that had a person been involved in the decision, neither of the failovers would have been done.
To be clear, manual failover should not mean a bunch of manual steps. I think many people get confused on this idea. When you do need to failover, you need that to happen as quickly, and as correctly, as possible. When we say “manual” failover, we mean that your goal should be to have the decision to failover be manual, but the process to be as scripted and automated as possible.
Another key factor in setting up scripted failover systems, and one that we see forgotten time and time again, is ye old STONITH concept. While it’s not 100% clear, from the description in the Github post, it seems that not only did their system allow automated failover, but it was also allowed to do automated fail-back. Just like any decision to failover needs to be manual, I always like to have at least one manual step involved after failover that is needed to reset the system as “back to normal”. This is extremely useful because it can act as a clear sign for your ops team that everyone agrees things are back to normal. Before that happens, your scripted failover solution should be unable to perform; why allow failover back to a machine that you’ve not agreed is ready to go back into service?
I know none of this sounds particularly sexy, but it’s battle tested and it works. And if you really don’t think you can wait for a human to intervene, build your systems for fault tolerance, not failover; just be warned that it is more expensive, complicated, and time consuming to implement (and the current open source options leave a lot to be desired in the options available to you).
Wondering about ways to help ensure availability in your environment? I’ll be speaking at Velocity Europe the first week of October, talking about ”Managing Databases in a DevOps Environment”; if you’re going to be attending I’d love to swap war stories. And yes, that’s the week after Surge, which is war story nirvana; if you haven’t gotten tickets for one of these events, there’s still time left; I hope to see you there.
Contents of an Office
- root beer bottles
- hanks (philadelphia, pa) (2 cases, plus a spare)
- appalachain brewing company (harrisburgh, pa) (1 case, plus a few spares)
- jack black’s dead red root beer
- mccutchensons (frederick, MD)
- old soaker (atlantic brewing company, bar harbor, maine)
- aj stephens (boston)
- 1 abita brewing root beer cap
- scotch bottles
- auchentoshan three wood
- copper fox rye whisky (bottled 2011-05-05)
- balvenie 12
- glenlivet 12
- grangestone 12 (two bottles)
- glenkinchie 12
- bunnahabain 12
- chivas regal 12
- willett reserved whisky kentucky bourbon
- 1 empty can of frescolita non-empties:
- 1 bottle of scaldis noel
- 1 2 liter bottle of pennslyvania dutch birch beer
- 1 16oz bottle of pennslyvania dutch birch beer
- books
- scalable internet architectures (2 copies)
- begining php and postgresql 8
- version control by example
- mysql tutorial
- mysql database design and tuning
- unix power tools
- perl best practices
- beautiful data
- head first php & mysql
- sterlings gold
- hats
- omniti
- surge 2011
- opensolaris
- half-dozen or more conference badges
- one bottle chipotle tabasco
- 3 boxes of old busines cards (3 different designs)
- 1 gift bag from client (and friend)
- 3 MTA cards from NYC
- 1 container of jellybeans from Truviso (thanks Greg!)
- a blues clues sticker from my daughter
- oscon data elephant sticker
- busch gardens elephant fact sticker
- codango php elephant squeeze toy
- 1 printed photo of gier magnessun
- surge postcards
- 1 copy of Communications of the ACM
- 1 menu from pudgies
- 1 postgresql banner
- real estate brochures of about 2 dozen area buildings
- 2 sharpies
- 1 highlighter
- 1 omniti pen
- 1 hilton garden inn pen
- 1 pewter elephant bookend
- 1 ceramic statue of Apsara
- 1 plastic balancing jet fighter toy
- 1 organic fruit sticker
- several thank you cards from tech friends
- an old contract proposal, full of highlighted issues
- 1 worksheet on goal driven performance optimization from percona
- 1 old sticky brain lying on the floor
- a billing breakdown for one of our long time customers
- 1 screw
- 1 allan wrench
- 2 whiteboard markers
- 1 whiteboard eraser
- 1 business card for “gas station tacos”
- countless ERD’s for PE2
- schema layout for podshow databases
- several resumes, mostly from people we didn’t hire (sorry Jiraporn)
- fax information for my daughters pre-school (she’s in 4th now)
- 1 random screw
- 1 paper clip