Source blog: Everything Sysadmin
Article: "Split Your Overwhelmed Teams"
After a 2-year break, my column in ACM Queue magazine returns! It has a new name "Operations and Life". For many years I wrote a column in ACM Queue Magazine. It was called "Everything Sysadmin" and covered devops, IT, and basically anything I felt like. I stopped writing in 2020 due to the pandemic and a general lack of motivation. My last column was published Nov 2020. Two years later I finally feel like writing again. The new column is called "Operations and Life". I'm going to write about the intersection of devops and personal life. I believe that most techniques we use at work can apply in our personal life and vice-versa.
Usenix LISA is no more. Here's my retrospective.
The Usenix LISA conference is no more. After 35 years, I have a lot of good (and some not good) memories of the conference. It was a big part of my career and I'm sad to see it go. However I'm proud of what LISA accomplished. I wrote my personal reflections on the conference in a new article published on the Usenix website. Warning: this article includes some over-sharing. Read it here: LISA made LISA obsolete (That's a compliment!)
20 years of The Practice of System and Network Administration!
Twenty years ago the first edition of The Practice of System and Network Administration shipped! Since then there has been a 2nd and 3rd edition (2006 and 2014), plus a sequel book The Practice of Cloud System Administration, and many printings. (see the timeline here) When we started the project we had no idea if it would be a success. There was a real chance it could be a flop. Many people told us that our proposal was illogical: How could you have a book about system administration that is vendor agnostic and talks about process and people instead of specific tools and operating systems?
20 years of The Practice of System and Network Administration!
Twenty years ago the first edition of The Practice of System and Network Administration shipped! Since then there has been a 2nd and 3rd edition (2006 and 2014), plus a sequel book The Practice of Cloud System Administration, and many printings. (see the timeline here) When we started the project we had no idea if it would be a success. There was a real chance it could be a flop. Many people told us that our proposal was illogical: How could you have a book about system administration that is vendor agnostic and talks about process and people instead of specific tools and operating systems?
Hear Tom on The Software Engineering Daily Podcast
https://softwareengineeringdaily.com/2021/07/22/stack-overflow-for-teams-a-centralized-knowledge-sharing-and-collaboration-platform-with-tom-limoncelli/ If you've ever googled a CS or programming question, you likely found an answer (or many) on Stack Overflow. Founded in 2008 and named after a common computing error, Stack Overflow empowers the world to develop technology through collective knowledge. More than 100 million people visit Stack Overflow every month making it one of the 50 most-visited websites in the world. Stack Overflow's products include its market-leading knowledge sharing and collaboration platform, Stack Overflow for Teams, in addition to Stack Overflow Reach & Relevance, which is focused on advertising. Stack Overflow for Teams is a knowledge sharing and collaboration solution that developers and managers already know and trust.
Hear Tom on The Software Engineering Daily Podcast
https://softwareengineeringdaily.com/2021/07/22/stack-overflow-for-teams-a-centralized-knowledge-sharing-and-collaboration-platform-with-tom-limoncelli/ If you've ever googled a CS or programming question, you likely found an answer (or many) on Stack Overflow. Founded in 2008 and named after a common computing error, Stack Overflow empowers the world to develop technology through collective knowledge. More than 100 million people visit Stack Overflow every month making it one of the 50 most-visited websites in the world. Stack Overflow's products include its market-leading knowledge sharing and collaboration platform, Stack Overflow for Teams, in addition to Stack Overflow Reach & Relevance, which is focused on advertising. Stack Overflow for Teams is a knowledge sharing and collaboration solution that developers and managers already know and trust.
Updated BP Texas City Animation
This isn't directly sysadmin-related, but it made me think of how a really good outage retrospective can teach others how to prevent problems in the future. " On the 15th anniversary of the incident, the U.S. Chemical Safety Board is announcing a forthcoming interactive training application based on one of the worst industrial disasters in recent U.S. history--the March 23, 2005, explosion at the BP refinery in Texas City, Texas, which killed 15 workers, injured 180 others, and caused billions of dollars in economic losses. This updated animation will be included in the training, which will focus on OSHA's Process Safety Management standard.
Updated BP Texas City Animation
This isn't directly sysadmin-related, but it made me think of how a really good outage retrospective can teach others how to prevent problems in the future. " On the 15th anniversary of the incident, the U.S. Chemical Safety Board is announcing a forthcoming interactive training application based on one of the worst industrial disasters in recent U.S. history--the March 23, 2005, explosion at the BP refinery in Texas City, Texas, which killed 15 workers, injured 180 others, and caused billions of dollars in economic losses. This updated animation will be included in the training, which will focus on OSHA's Process Safety Management standard.
What to give your loved one for April Fools Day?
April Fools Day is coming up! Time to order your coffee-table book of April Fools RFCs! More info here: https://www.rfchumor.com Makes a great gift for nerds that own coffee-tables! Order it today!
What to give your loved one for April Fools Day?
April Fools Day is coming up! Time to order your coffee-table book of April Fools RFCs! More info here: https://www.rfchumor.com Makes a great gift for nerds that own coffee-tables! Order it today!
Thu, Nov 19 NYCDEVOPS meetup: John Allspaw on "Learning From Incidents"
Don't forget! November's nycdevops meetup speaker is John Allspaw, who will give a talk titled "Findings From the Field: 2 Years of Learning From Incidents". The talk starts at 5pm sharp! (NY is in US/Eastern) Please RSVP! See you there! https://www.meetup.com/nycdevops/events/273826675/ (This is a virtual meetup. Everyone in the world is invited!)
Thu, Nov 19 NYCDEVOPS meetup: John Allspaw on "Learning From Incidents"
Don't forget! November's nycdevops meetup speaker is John Allspaw, who will give a talk titled "Findings From the Field: 2 Years of Learning From Incidents". The talk starts at 5pm sharp! (NY is in US/Eastern) Please RSVP! See you there! https://www.meetup.com/nycdevops/events/273826675/ (This is a virtual meetup. Everyone in the world is invited!)
Thu, Nov 19 NYCDEVOPS meetup: John Allspaw on "Learning From Incidents"
November's nycdevops meetup speaker is John Allspaw, who will give a talk titled "Findings From the Field: 2 Years of Learning From Incidents". The talk starts at 5pm sharp! (NY is in US/Eastern) Please RSVP! See you there! https://www.meetup.com/nycdevops/events/273826675/ (This is a virtual meetup. Everyone in the world is invited!)
Thu, Nov 19 NYCDEVOPS meetup: John Allspaw on "Learning From Incidents"
November's nycdevops meetup speaker is John Allspaw, who will give a talk titled "Findings From the Field: 2 Years of Learning From Incidents". The talk starts at 5pm sharp! (NY is in US/Eastern) Please RSVP! See you there! https://www.meetup.com/nycdevops/events/273826675/ (This is a virtual meetup. Everyone in the world is invited!)
Oct 15 NYC DevOps Meetup: "Introduction to Site Reliability Engineering" by Nathen Harvey
This month's nycdevops meetup speaker is Nathen Harvey of Google, who will give a talk titled "Introduction to Site Reliability Engineering". The talk starts at 5pm sharp! (NY is in US/Eastern) Please RSVP! See you there! https://www.meetup.com/nycdevops/events/272956481/ (This is a virtual meetup. Everyone around the world is invited!)
Oct 15 NYC DevOps Meetup: "Introduction to Site Reliability Engineering" by Nathen Harvey
This month's nycdevops meetup speaker is Nathen Harvey of Google, who will give a talk titled "Introduction to Site Reliability Engineering". The talk starts at 5pm sharp! (NY is in US/Eastern) Please RSVP! See you there! https://www.meetup.com/nycdevops/events/272956481/ (This is a virtual meetup. Everyone around the world is invited!)
Adarsh Shah on "Continuous Delivery for Machine Learning" (September NYCDEVOPS Meetup)
Come one, come all! nycdevops does its first virtual meetup! All are invited! Speaker: Adarsh Shah Topic: "Continuous Delivery for Machine Learning" Time: Thursday, September 17, 2020, 5-630 PM Link: https://www.meetup.com/nycdevops/events/272914068/ Hope to see you there!
Adarsh Shah on "Continuous Delivery for Machine Learning" (September NYCDEVOPS Meetup)
Come one, come all! nycdevops does its first virtual meetup! All are invited! Speaker: Adarsh Shah Topic: "Continuous Delivery for Machine Learning" Time: Thursday, September 17, 2020, 5-630 PM Link: https://www.meetup.com/nycdevops/events/272914068/ Hope to see you there!
Book: Become Ansible by Josh Duffney
My coworker Josh Duffney launches his self-published book on Ansible today! Congrats and I wish great success! Crawl, walk, run, sprint your way through learning Ansible with "Become Ansible"! Visit his website becomeansible.com
Book: Become Ansible by Josh Duffney
My coworker Josh Duffney launches his self-published book on Ansible today! Congrats and I wish great success! Crawl, walk, run, sprint your way through learning Ansible with "Become Ansible"! Visit his website becomeansible.com
Time management for political sysadmins
Can you put me in contact with the "tech team" of a political campaign? I am offering my "time management for sysadmins" training pro-bono to any Dem or anti-Trump digital team, sysadmins, devops team, SRE, etc. Contact me via LinkedIn, DM me on Twitter or email me if you know my email address.
Time management for political sysadmins
Can you put me in contact with the "tech team" of a political campaign? I am offering my "time management for sysadmins" training pro-bono to any Dem or anti-Trump digital team, sysadmins, devops team, SRE, etc. Contact me via LinkedIn, DM me on Twitter or email me if you know my email address.
Come work with me at Stack Overflow!
We're hiring! Come join my team at Stack Overflow, Inc. and help maintain one of the most visited websites on the internet! SREs with a Linux and web hosting focus (load balancers, Redis, web servers, etc.) Windows SRE w/focus on Developer Tools: If you know the Microsoft developer ecosystem, love Azure Pipelines + GitHub Actions, and want to help Stack Overflow build the world, apply today. SREs with Azure experience (or equiv) to help run our SaaS version of Stack Overflow Apply at the links!
Come work with me at Stack Overflow!
We're hiring! Come join my team at Stack Overflow, Inc. and help maintain one of the most visited websites on the internet! SREs with a Linux and web hosting focus (load balancers, Redis, web servers, etc.) Windows SRE w/focus on Developer Tools: If you know the Microsoft developer ecosystem, love Azure Pipelines + GitHub Actions, and want to help Stack Overflow build the world, apply today. SREs with Azure experience (or equiv) to help run our SaaS version of Stack Overflow Apply at the links!
Come work with me at Stack Overflow!
We're hiring! Come join my team at Stack Overflow, Inc. and help maintain one of the most visited websites on the internet! Azure/Windows/Linux SRE Linux/web infrastructure SRE Apply at the links!
Get your DevOpsDays NYC tickets now!
DevOpsDays is about 2 weeks away! Commencing on Tuesday, March 3 and lasting for 2 days, this is the best NYC-area DevOps conference you can attend. The speaker lineup is very impressive (if I say so myself :-) ). Tickets: Eventbrite Website: https://devopsdays.org/events/2020-new-york-city/welcome/ Location: New York Academy of Medicine, 1216 5th Ave, New York, NY 10029 The best part about attending a local conference is that your boss only has to approve the ticket price (no hotel or flights, eh?) . Looking forward to seeing you there! P.S. I'm speaking on the last day. It will be all new material that I haven't presented before.
Get your DevOpsDays NYC tickets now!
DevOpsDays is about 2 weeks away! Commencing on Tuesday, March 3 and lasting for 2 days, this is the best NYC-area DevOps conference you can attend. The speaker lineup is very impressive (if I say so myself :-) ). Tickets: Eventbrite Website: https://devopsdays.org/events/2020-new-york-city/welcome/ Location: New York Academy of Medicine, 1216 5th Ave, New York, NY 10029 The best part about attending a local conference is that your boss only has to approve the ticket price (no hotel or flights, eh?) . Looking forward to seeing you there! P.S. I'm speaking on the last day. It will be all new material that I haven't presented before.
DevOpsDays NYC 2020 Call For Participation!
DevOpsdays NYC is happening March 3-4, 2020. The Call For Participation (CFP) is now open! If you have any ideas that you want to share with the local community please submit your talks using the link below: https://www.papercall.io/dodnyc2020
DevOpsDays NYC 2020 Call For Participation!
DevOpsdays NYC is happening March 3-4, 2020. The Call For Participation (CFP) is now open! If you have any ideas that you want to share with the local community please submit your talks using the link below: https://www.papercall.io/dodnyc2020
November nycdevops meetup: DevSecOps and supply chain attack
Irina Tishelman from Sonatype will present a talk "Automate or Die - DevSecOps in the Age of Software Supply Chain Attacks" at the November 14, 2019 nycdevops meetup. The meetup meets at the Stack Overflow office in NYC. https://www.meetup.com/nycdevops/events/265281451/ You don't want to miss this one!
November nycdevops meetup: DevSecOps and supply chain attack
Irina Tishelman from Sonatype will present a talk "Automate or Die - DevSecOps in the Age of Software Supply Chain Attacks" at the November 14, 2019 nycdevops meetup. The meetup meets at the Stack Overflow office in NYC. https://www.meetup.com/nycdevops/events/265281451/ You don't want to miss this one!
Demo Data as Code
My newest article for acmQueue magazine is called Demo Data as Code: https://queue.acm.org/detail.cfm?id=3355565
Demo Data as Code
My newest article for acmQueue magazine is called Demo Data as Code: https://queue.acm.org/detail.cfm?id=3355565
Next nycdevops meetup: Kubernetes Informers (Wed, June 19)
Robert Ross (a.k.a. Bobby Tables) will be the speaker at the next nycdevops meetup on Wed, une 19, 2019. Full details and RSVP info: https://www.meetup.com/nycdevops/events/261842702/ NOTE: Different day and location! Title: Staying Informed with Kubernetes Informers Speaker: Robert Ross (Bobby Tables) from FireHydrant Date: Wed, June 19, 2019 Location: Compass, 90 Fifth Ave, New York, NY 10011 Kubernetes state is changing all the time. Pods are being created. Deployments are adding more replicas. Load balancers are being created from services. All of these things can happen without anyone noticing. But sometimes we need to notice, however, for when we need to react to such events.
April NYC DevOps Meetup: Building a tamper-evident CI/CD system
The April nycdevops Meetup is Thursday, April 18. Doors open at 6:30pm! https://www.meetup.com/nycdevops/events/260294692/ NOTE: The meetings are now on THURSDAY. Title: How to build a tamper-evident CI/CD system Speaker: Trishank Karthik Kuppusamy, Datadog, Inc TALK DESCRIPTION: CI/CD is critical to any DevOps operation today, but when attackers compromise it, they get to distribute malicious software to millions of unsuspecting users. We present how Datadog used TUF and in-toto to develop, to the best of our knowledge, the industry's first end-to-end verified pipeline that automatically builds integrations for the Datadog agent. That is, even if this pipeline is compromised, users should not be able to install malware.
DevOpsDays-NYC nearly sold out! Register soon!
http://dod.nyc for details. DevOpsDays-NYC is Jan 24/25, 2019. Don't miss it!
Ericsson/O2 outage root cause announced
Was the root cause of the O2 outage really an expired certificate? Why wasn't the "root cause" any of these? Certificate expiration not monitored Certificate renewal process complex so that everyone hopes someone else fixes it Certificate renewal is so rare, we aren't good at doing it Deploying new certificates manual and error-prone Vendor did not document all periodic maintenance requirements Soon-to-expire certs not logged Logging for each component an island onto itself The reason, dear reader, is that there is no such thing as a single "root cause". There are only contributing factors.
Understanding your competition is a culture
Disclaimer: I haven't worked at Google for 5+ years so this kind of story is probably outdated. I mean, how could Google not have fixed this problem in the last 10 years? In 2008 I was on a business trip to Seattle and I had dinner with an old college friend who now worked at Microsoft. I noticed that she had an iPhone. This was when Microsoft was heavily pushing their own phone product, and Android hadn't started shipping. I thought it was odd that a Microsoftie would be using an iPhone and pointed it out. " Oh, it's the opposite.
Taryn's SQL upgrade blog post
Cheers to my coworker Taryn for her blog post about how she did an extremely complex series of 30 Microsoft SqlServer upgrades. If you've seen the film "Apollo 13", there's a scene where they have to get something right in the simulator before they can do it in space. That's basically what she had to do. Read the post here: How we upgraded Stack Overflow to SQL Server 2017 Here's some takeaways: Set up a lab environment to test complex changes. Communicate with your users. Write a detailed playbook. Don't do it alone. Ask for help from all over. Keep a lab notebook.
Google Authenticator tips
Things you might not have known about Google Authenticator: Copy and paste If you press and hold the 6-digit number, it puts it in your cut and paste buffer. Re-order the list If you click the pencil to go into edit mode, you can change the order of the items. I find this particularly important because I now have 12 different systems authenticating with this app, and only 4 fit on the screen of my tiny iPhone SE. I've pushed the ones that I use the most to the top of the list. The Google-related services that generally authenticate via a notification asking "Is this you trying to log in?"
SREs with Windows experience? (NYC-area)
My team at Stack Overflow is looking to hire SREs with Windows experience, particularly administration of Microsoft SqlServer. If you are a system administration looking to move into more of an SRE position, this is an ideal opportunity. Here's the job listing: https://stackoverflow.com/jobs/190514/ NOTE: While we are a remote-first team with team members all over the world, this position will have occasional datacenter work requirements, which means 1-hour travel time to our Jersey City, NJ datacenter is a requirement.
I'll be at All-Day-DevOps: Free registration!
https://www.alldaydevops.com/ All Day DevOps is a global event held on the internet. 24 hours of talks, over 100 speakers, all streaming over the Internet. 17-Oct-2018 Registration is free! I will be presenting my talk Stealing The Best Ideas From DevOps: Applying DevOps Outside Of SDLC More info is at: https://www.alldaydevops.com/
LISA '18: Save on LISA registration!
LISA this year is in Nashville, TN, Oct 29-31, 2018. The full schedule is up! Registration is open! Three things you should know: This year Usenix LISA will be 3 days long, instead of the usual 7. This makes it easier to attend, and more focused. I think this is a really good direction for LISA. The schedule is awesome. I got super excited while reading the schedule. All the talks seemed to be much more focused and a greater emphasis on cutting edge topics and things I want to learn about but haven't had time to study. I have discount codes.
Tom @ nycdevops meetup on Tuesday!
I'll be the speaker at this month's NYC DevOps meetup. My topic is about reforming the operations side of DevOps in a new talk called "My Operations Reform Checklist". Topic: My Operations Reform Checklist Speaker: Tom Limoncelli, SRE Manager, Stack Overflow, Inc When: Tuesday, September 18, 2018, 6:30 PM Where: Stack Overflow HQ, 110 William St, NYC, NY For complete details and to RSVP: https://www.meetup.com/nycdevops/events/fmgjmnyxmbxb/
Financial advice when you get a raise
[This is a "first draft". Comments appreciated.] One of the best pieces of financial advice I ever got was to pretend you didn't get a raise, and send the increment to your retirement or long-term investing or to accelerate debt repayment. Simply wait until your first paycheck with the new amount and see how much higher the take-home is. Set up an automatic deposit or transfer of that amount into your long term investment. The theory is that you are used to living without that money. So, why pocket it? Instead, put it towards something that will help you in the long term.
Stack Overflow in NYC is hiring Windows SREs
Are you a Windows Server sysadmin? Love PowerShell? Have experience with Microsoft SQL Server administration? Looking to step up to an SRE position at a website all your friends love? Apply to work with my team at Stack Overflow in NYC. More info here: https://stackoverflow.com/jobs/190514/
"Continuous Discussion" Video Podcast today!
The next episode of the "Continuous Discussion" video podcast will include a panel discussion about the upcoming DevOps Enterprise Summit in Las Vegas this October. I'll be on the panel. Watch the podcast as it is recorded later today: Tuesday, August 21, 10am PT / 1pm ET https://electric-cloud.com/c9d9-devops-podcast/ You won't want to miss it! See you there!
Google has changed GSuite's SPF records
(DNSControl unrolls your SPF records safely and automatically. Sure, you can do it manually, but at the end of this article I'll show you how to automate it so you can 'set it and forget it'.) Google has changed the SPF records for GSuite. You don't have to make any changes since you still do include:_spf.google.com and the magic of SPF takes care of things for you. However, if you unroll your SPF records to work around the 10-lookup limit, you need to take a look at what you've done and re-do it based on the new SPF records.
I'm hiring SREs/sysadmins and more!
My team at Stack Overflow has a number of open positions on the team that I manage: Windows-focused SRE, New York City or Northern NJ: If you love PowerShell, you'll love this position! If your background is more sysadmin than SRE, we're willing to train. Linux SRE, Denver: This will be particularly exciting because we're about to make some big technology changes and it will be a great opportunity to learn and grow with the company. We have a number of other openings around the company: Junior Technology Concierge (IT Help Desk): New York City Engineering Manager: Remote (US East Coast Time Zone) VP of Engineering: New York City.
SO (my employer) is hiring a Windows SRE/sysadmin in NY/NJ
Come work with Stack Overflow's SRE team! We're looking for a Windows system administrator / SRE to join our SRE team at Stack Overflow. (The downside is that I'll be your manager... ha ha ha). Anyway... the full job description is here: https://stackoverflow.com/company/work-here/1152509/ A quick and unofficial FAQ: Q: NYC/NJ? I thought Stack was an "remote first" company! Whudup with that? A: While most of the SRE team works remotely, we like to have a few team members near each of our datacenters (Jersey City, NJ and Denver, CO). You won't be spending hours each week pulling cables, I promise you.
Launched: Stack Overflow for Teams!
I usually don't use my blog to plug my employer but I'm very excited about Stack Overflow's new "Stack Overflow for Teams" launch this week. How would you like a private Stack Overflow area for your team? Stack Overflow for Teams allows teams of any size to use the Stack Overflow that they already know and love but for all their proprietary information - creating a special private space just for them on stackoverflow.com. It uses the same Q&A format, collaborative editing, and even recognition systems to solve the massive knowledge sharing issues that all teams have. More info is on the overview page and our blog post, plus we got great press about it on VentureBeat and GeekWire.
SRECon: Operational Excellence in April Fools' Pranks
The video of my SRECon talk is finally available! " Operational Excellence in April Fools' Pranks: Being Funny Is Serious Work!" at SREcon18 Americas is about mitigating the risk of "high stakes" launches. The microphones didn't pick up the audience reaction. As a result it looks like I keep pausing for no reason, but really I'm waiting for the laughter to calm down. Really! (Really!) On a personal note, I'd like to thank the co-chairs of SRECon for putting together such an excellent conference. This was my first time being the last speaker at a national conference, which was quite a thrill.
Accelerate NYC Launch Party, Saturday, April 21
The NYC launch event for Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations by Nicole Forsgren, PhD, Jez Humble, and Gene Kim will be held this Saturday from 11am-2pm. All are invited. Space is limited. Please RSVP at EventBrite. I'm super excited by this book for two reasons: (1) It explains the business case for devops in a way that speaks to executives. (2) It is based on real data with statistical correlation that show real cause and effect. I'll be at this event. I hope to see you there too!
ZFS Users Conference, April 19-20, Norwalk, CT
Datto will be hosting the 2nd annual ZFS User Conference featuring ZFS co-creator, Matt Ahrens! The date is April 19-20 at Datto HQ in Norwalk, CT. This conference will focus on the deployment, administration, features, and tuning of the ZFS filesystem. Learn about OpenZFS and network with folks running businesses and interesting projects on ZFS. For more information and registration see http://zfs.datto.com (I won't be attending as I'm not longer using ZFS, but I'm still a ZFS fanboy so I felt like promoting this.)
DevOpsDays New York City 2019: Join the planning committee!
2019 feels like a long way off, but since the conference is in January, we need to start planning soon. The sooner we start, the less rushed the planning can be. I have to confess that working with the 2018 committee was one of the best and most professional conference planning experiences I've ever had. I've been involved with many conferences over the years and this experience was one of the best! I invite new people to join the committee for 2019. The best way to learn about organizing is to join a committee and help out. You will be mentored and learn a lot in the process.
Male Ally Summit 2018 (NYC)
(quoted from the website) The Male Ally Summit comes after a successful event on March 13, 2017 in NYC called, "The Role Male Allies, Advocates, and Mentors have in Retaining Women in Tech", we had over 180 in attendance. We know this year will be just as impactful. This year we bring back some of the same amazing speakers such as David Smith and Brad Johnson who are co-authors of, "Athena Rising: How and Why Men Should Mentor Women"; Evin Robinson (Co-founder, New York on Tech); Matt Wallaert (Chief Behavioral Officer, Clover Health). We add to the agenda, Heather Cabot (Co-author, Geek Girl Rising); Bryan Liles (Software Engineer, Heptio); Avis Yates Rivers (CEO and Author, Technology Concepts Groups International), Dennis Kozak (SVP of Next Generation Portfolio Strategy, CA Technologies) and others!
DevOpsDays New York City 2018: Register now!
DevOpsDays NYC is only a few weeks away: Jan 18-19, 2018! Please register asap. We could sell out this year. With this awesome line-up of speakers, tickets are going fast. https://www.devopsdays.org/events/2018-new-york-city/ Or... this handy shortcut: http://dod.nyc
Save 40-60% on the 3rd edition of TPOSANA
The 3rd edition of "Vol 1: The Practice of System and Network Administration" was nominated as a "2017 Community Favorite". To celebrate, you can get it 40-60% off between now and Jan 8, 2018. Click this link and use code "FAVE" See all the favorites here: http://informit.com/favorites By the way... there haven't been many reviews of this book on Amazon, and none that have mentioned the new content in Section I, II and III. I've you've read the new edition and would like to post a review, we'd love to know your opinion (good or bad).
DevOpsDays New York City 2018 Speakers Announced!
Exciting news from the D-O-D-NYC committee! Speakers announced. Wow! I've never seen such an amazing lineup of speakers! The best of the best. The committee this year was flooded with so many amazing proposals but sadly it is a 2-day conference so they had to be very selective. Who benefits? You! Early bird discount ends on Friday. Register soon and save! DevOpsDays-NYC 2018 is Thu/Fri January 18-19, 2018 in midtown Manhattan. Easy to get to via all forms of public transportation. For more information: http://dod.nyc
Quick! Install that Mac printer now!
Does your friend or significant other have a Mac from work that is locked down so that changes can't be made? Any attempt to make a chance in System Preferences asks for the admin password, which you don't have. Maybe they are a teacher at a school with overly-zealous sysadmins? Maybe they work at an insurance company that... just kidding, no insurance company supports Macs. Someone Who Isn't Me knows someone that has a Mac laptop and can't print to the home printers for exactly this reason. To print at home, they generate a PDF, copy the file to a USB stick, and walk it over to another computer that can print.
Productivity When You Just Can't Even
Ryn Daniels has written an excellent piece about time management and productivity when you are burned out. The advice is spot on. I also do these things when I'm not feeling my best too. "It's one thing to talk about productivity when you're already feeling motivated and productive, but how do you get things done when it's hard to even make it through the day?" https://superyesmore.com/productivity-when-you-just-cant-even-0c73d6a1f086209af420c2ced442a2e8
November NYC DevOps Meetup: Building a hybrid cloud
Taras Lipatov, Principal Engineer at Sailthru, will describe his experience building a hybrid cloud using docker/mesos/consul at the Tuesday, November 14, 2017 nycdevops meetup. More info and to RSVP on https://www.meetup.com/nycdevops/events/241852995/. RSVP soon! See you there!
Operational Excellence in April Fools' Pranks
This issue's column in ACMQueue Magazine is titled, "Operational Excellence in April Fools' Pranks". You can read it on the Queue app: http://queue.acm.org/app/
Registration for NYC DOD 2018 is open!
https://www.eventbrite.com/e/devopsdays-nyc-2018-tickets-39330760363
MicroReview: Usenix LISA 2017
I'm a bit too tired to write a full review of Usenix LISA 2017, so let me just say that the content was excellent and the audience was the most diverse that I've seen at a mainstream sysadmin conference. The best part about the content? If I could sum it up in one word, it would be "new". Tons of new stuff, new technology, new ideas, and new techniques. (Ironically the one training session I taught was my not-new Time Management class, but the room was packed with new attendees.) Best meta-issue? Usenix switched to the "Sched App" which was awesome.
DevOpsDays New York City 2018 Announced!
Save this date! January 18-19, 2018 will be the next DevOpsDays NYC! Save this date! Sadly there won't be a 2017 conference, but the 2018 conference is just around the corner. Mark your calendar! Submit talk proposals today! Registration opening soon! See you there!
Final reminder: NYCDevOps tonight: "Storing Secrets in the Cloud"
Don't miss this meeting tonight! Topic: Storing Secrets in Cloud based Key Management Services Speaker: Dan O'Boyle, Stack Overflow, Inc. Date: Tuesday, October 17, 2017 Time: 6:30-9:30 PM Location: Stack Overflow HQ, 110 William St, 28th floor, NY, NY https://www.meetup.com/nycdevops/events/241803854/ The A/C is fixed! Don't miss this cool event! Full details and RSVP.
Vendor jerks at tech conferences
(I've intentionally delayed posting this so that it wasn't clear which conference I'm talking about.) So... I'm at a conference. I take a break from the talks to walk around the vendor show. While most of the booths are selling products I'm not interested in, I suddenly find myself in front of VENDOR-A (name changed to protect them). VENDOR-A makes a product that has both open source and commercial editions, a common business model. Since the company I work for is a happy user of their open source version, I decide to ask about the commercial version. Maybe there's some benefit to be had.
Final reminder: "DNS as Code" talk at NYCDevOps tonight!
Topic: DNSControl: "DNS as Code" from StackOverflow.com Speaker: Thomas A. Limoncelli, SRE Manager @ Stack Overflow Date: Monday, October 2, 2017 Time: 6:30-9:30 PM (SPECIAL TIME AND LOCATION) Location: Madison Suite, Hilton Midtown, 1335 6th Ave, New York, NY 10019 https://www.meetup.com/nycdevops/events/243369226/ VelocityNYC is in town this week. They've graciously provided space for us to host an additional meeting. Space is limited! RSVP soon! Full details and RSVP. We will be going out for drinks after the talk.
October NYCDevOps: Two meetings! DNSControl and Storing Secrets in the Cloud
We will have two meetings in October. The extra meeting will be on Mon, Oct 2 co-located with the VelocityNYC conference. You don't have to be registered for the conference to come to the meeting. Topic: DNSControl: "DNS as Code" from StackOverflow.com Speaker: Thomas A. Limoncelli, SRE Manager @ Stack Overflow Date: Monday, October 2, 2017 Time: 6:30-9:30 PM (SPECIAL TIME AND LOCATION) Location: Madison Suite, Hilton Midtown, 1335 6th Ave, New York, NY 10019 https://www.meetup.com/nycdevops/events/243369226/ VelocityNYC is in town this week. They've graciously provided space for us to host an additional meeting. Space is limited! RSVP soon! Full details and RSVP.
September NYCDevOps: Habitat in Production (Seth Thomas)
This month's NYCDevOps meetup speaker will be Seth Thomas talking about "Habitat in Production". Date: Tuesday, September 19, 2017 Time: 6:30 PM Location: Stack Overflow HQ, 110 William St, 28th floor, NY, NY Space is limited! RSVP soon! https://www.meetup.com/nycdevops/events/240064742/
Evidence that Go makes it easy to get up to speed
Some recent PRs to the DNSControl Project casually mentioned that this was their first time writing Go code. That's amazing! When was the last time you saw someone say, "here's a major contribution to your open source project... oh and I just learned this language." (and the PR was really darn good!) I think it is pretty rare and one of the special things about Go. Part of Go's original vision was to make it easy for new people to join a project and make contributions. This was important internally at Google, since engineers hop projects frequently. This also benefits open source projects by making it easy to dive in and participate.
Evidence that Go makes it easy to get up to speed
Some recent PRs to the DNSControl Project casually mentioned that this was their first time writing Go code. That's amazing! When was the last time you saw someone say, "here's a major contribution to your open source project... oh and I just learned this language." (and the PR was really darn good!) I think it is pretty rare and one of the special things about Go. Part of Go's original vision was to make it easy for new people to join a project and make contributions. This was important internally at Google, since engineers hop projects frequently. This also benefits open source projects by making it easy to dive in and participate.
Google's BBR fixes TCP's dirty little secret
Networking geeks: Google made a big announcements about BBR this week. Here's a technical deep-dive: http://queue.acm.org/detail.cfm?id=3022184 (Hint: if you would read ACM Queue like I keep telling you to, you'd have known about this before all your friends.) Someone on Facebook asked me for a "explain it like I'm 5 years old" explanation. Here's my reply: Short version: Google changed the TCP implementation (their network stack) and now your youtube videos, Google websites, Google Cloud applications, etc. download a lot faster and smoother. Oh, and it doesn't get in the way of other websites that haven't made the switch. (Subtext: another feature of Google Cloud that doesn't exist at AWS or Azure.
Google's BBR fixes TCP's dirty little secret
Networking geeks: Google made a big announcements about BBR this week. Here's a technical deep-dive: http://queue.acm.org/detail.cfm?id=3022184 (Hint: if you would read ACM Queue like I keep telling you to, you'd have known about this before all your friends.) Someone on Facebook asked me for a "explain it like I'm 5 years old" explanation. Here's my reply: Short version: Google changed the TCP implementation (their network stack) and now your youtube videos, Google websites, Google Cloud applications, etc. download a lot faster and smoother. Oh, and it doesn't get in the way of other websites that haven't made the switch. (Subtext: another feature of Google Cloud that doesn't exist at AWS or Azure.
Four Ways to Make CS and IT Curricula More Immersive
My new column in ACM Queue is entitled, "Four Ways to Make CS and IT Curricula More Immersive". I rant and rail against the way that CS and IT is taught today and propose 4 ways CS educators can improve the situation. The article is free to ACM members. Non-members can purchase an annual subscription for $19.99 or a single issue for $6.99 online or through the Apple or Google stores.
Four Ways to Make CS and IT Curricula More Immersive
My new column in ACM Queue is entitled, "Four Ways to Make CS and IT Curricula More Immersive". I rant and rail against the way that CS and IT is taught today and propose 4 ways CS educators can improve the situation. The article is free to ACM members. Non-members can purchase an annual subscription for $19.99 or a single issue for $6.99 online or through the Apple or Google stores.
Still time to RSVP to NYCDevOps...
This month's NYCDevOps meetup speaker will be Martín Beauchamp talking about "Clos Networks for Datacenters". You don't want to miss this! Date: Tuesday, July 18, 2017 Time: 6:30 PM Location: Stack Overflow HQ, 110 William St, 28th floor, NY, NY Space is limited! RSVP soon! https://www.meetup.com/nycdevops/events/240295361/
Still time to RSVP to NYCDevOps...
This month's NYCDevOps meetup speaker will be Martín Beauchamp talking about "Clos Networks for Datacenters". You don't want to miss this! Date: Tuesday, July 18, 2017 Time: 6:30 PM Location: Stack Overflow HQ, 110 William St, 28th floor, NY, NY Space is limited! RSVP soon! https://www.meetup.com/nycdevops/events/240295361/
Free book: "Expanding Pockets of Greatness"
Companies don't make their "DevOps transformation" over night. Usually there is a small team that adopts devops practices and then, after proving their success, the practices spread throughout the company horizontally. However sometimes their success becomes an island. There is no momentum and the better practices fail to expand around the company. Growing devops practices within a company is not easy. It is especially difficult when it does not have management support, or the advocate does not executive authority. Some techniques for building momentum work, others do not. Earlier this year Josh Atwell, Carmen DeArdo, Jeff Gallimore, and myself sat down to write a list of techniques we've seen succeed.
Free book: "Expanding Pockets of Greatness"
Companies don't make their "DevOps transformation" over night. Usually there is a small team that adopts devops practices and then, after proving their success, the practices spread throughout the company horizontally. However sometimes their success becomes an island. There is no momentum and the better practices fail to expand around the company. Growing devops practices within a company is not easy. It is especially difficult when it does not have management support, or the advocate does not executive authority. Some techniques for building momentum work, others do not. Earlier this year Josh Atwell, Carmen DeArdo, Jeff Gallimore, and myself sat down to write a list of techniques we've seen succeed.
Tom's speaking at Red Hat User Group Tuesday in Woodbridge, NJ
I'll be giving my talk "Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers" at the Northern NJ Red Hat User Group tomorrow. If you are in the area, it would be great to see you there! https://www.meetup.com/NorthernNJRHUG/events/240019682/
Tom's speaking at Red Hat User Group Tuesday in Woodbridge, NJ
I'll be giving my talk "Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers" at the Northern NJ Red Hat User Group tomorrow. If you are in the area, it would be great to see you there! https://www.meetup.com/NorthernNJRHUG/events/240019682/
July NYCDevOps: Clos Networks for Datacenters
This month's NYCDevOps meetup speaker will be Martín Beauchamp talking about "Clos Networks for Datacenters". Date: Tuesday, July 18, 2017 Time: 6:30 PM Location: Stack Overflow HQ, 110 William St, 28th floor, NY, NY Space is limited! RSVP soon! https://www.meetup.com/nycdevops/events/240295361/
July NYCDevOps: Clos Networks for Datacenters
This month's NYCDevOps meetup speaker will be Martín Beauchamp talking about "Clos Networks for Datacenters". Date: Tuesday, July 18, 2017 Time: 6:30 PM Location: Stack Overflow HQ, 110 William St, 28th floor, NY, NY Space is limited! RSVP soon! https://www.meetup.com/nycdevops/events/240295361/
Judging books by their covers
The subtitle of some of my books have recently changed to better reflect the contents. As a result the book covers have been updated. Titles and covers are, essentially, a billboard for the contents. We wanted to make sure they more accurately guide potentially readers to the book.
Judging books by their covers
The subtitle of some of my books have recently changed to better reflect the contents. As a result the book covers have been updated. Titles and covers are, essentially, a billboard for the contents. We wanted to make sure they more accurately guide potentially readers to the book.
The .feedback scam
Do you have feedback you'd like to give to Google, Facebook, StackOverflow, Inc., or Gandi? Now there's a website that will collect that feedback. Or... not. There is a new TLD called ".feedback". It is a scam and ICANN should be ashamed of approving it. The people that run .feedback have pre-registered "for free" 5,000 major companies. As a result you can go to sites like http://www.google.feedback/ and http://www.gandi.feedback/ and http://www.stackoverflow.feedback/ and more. These websites enables people to send feedback about your company and products. Will the company ever receive the feedback? Unlikely. The company probably doesn't know the site exists.
The .feedback scam
Do you have feedback you'd like to give to Google, Facebook, StackOverflow, Inc., or Gandi? Now there's a website that will collect that feedback. Or... not. There is a new TLD called ".feedback". It is a scam and ICANN should be ashamed of approving it. The people that run .feedback have pre-registered "for free" 5,000 major companies. As a result you can go to sites like http://www.google.feedback/ and http://www.gandi.feedback/ and http://www.stackoverflow.feedback/ and more. These websites enables people to send feedback about your company and products. Will the company ever receive the feedback? Unlikely. The company probably doesn't know the site exists.
NYC DevOps meeting tomorrow (Tuesday)
Don't forget to RSVP by 1pm the day of the meeting: http://meetu.ps/39FgcP
NYC DevOps meeting tomorrow (Tuesday)
Don't forget to RSVP by 1pm the day of the meeting: http://meetu.ps/39FgcP
Stackoverflow... but for questions about how to teach CS
My high school had only one computer science teacher. When she had a problem or question, she had no where to turn. I can't imagine how isolating and stressful that must have been. cseducators.stackexchange.com is a Stack Exchange website for computer science educators to ask questions and share successful teaching techniques. It was in private beta until this week. Now everyone can access. If it Are you an educator looking for advice about how to integrate Git into an introductory CS class? Maybe you need to find a better analogy to help a student that doesn't understand 0-indexing? How do you convince students that well-indented code is worth the effort?
Stackoverflow... but for questions about how to teach CS
My high school had only one computer science teacher. When she had a problem or question, she had no where to turn. I can't imagine how isolating and stressful that must have been. cseducators.stackexchange.com is a Stack Exchange website for computer science educators to ask questions and share successful teaching techniques. It was in private beta until this week. Now everyone can access. If it Are you an educator looking for advice about how to integrate Git into an introductory CS class? Maybe you need to find a better analogy to help a student that doesn't understand 0-indexing? How do you convince students that well-indented code is worth the effort?
NYC DevOps meeting next week: Measuring real-world DNS performance at Stack Overflow
Next week's NYCDevOps Meetup speaker is my co-worker, Mark Henderson, on the topic of "Measuring real-world DNS performance at Stack Overflow". An in-depth look at how Stack Overflow records real-world DNS performance, and how you can do it too. You'll learn how we measured DNS performance when picking a DNS vendor, deciding whether or not to set up dual-DNS providers, and more. The meeting is Tuesday, June 20, 2017, 7:00 PM at the Stack Overflow HQ in New York City. For complete information and to RSVP, visit http://meetu.ps/39FgcP. Space is limited. RSVP soon!
NYC DevOps meeting next week: Measuring real-world DNS performance at Stack Overflow
Next week's NYCDevOps Meetup speaker is my co-worker, Mark Henderson, on the topic of "Measuring real-world DNS performance at Stack Overflow". An in-depth look at how Stack Overflow records real-world DNS performance, and how you can do it too. You'll learn how we measured DNS performance when picking a DNS vendor, deciding whether or not to set up dual-DNS providers, and more. The meeting is Tuesday, June 20, 2017, 7:00 PM at the Stack Overflow HQ in New York City. For complete information and to RSVP, visit http://meetu.ps/39FgcP. Space is limited. RSVP soon!
2017 State of DevOps Report
This year's report is fascinating! There is clear evidence that DevOps practices yield remarkable results for IT teams and organizations. Findings are related to transformational leadership, automation practices, continuous delivery, lean product management, and DevOps in not-for-profits and organizations that use off-the-shelf software.
2017 State of DevOps Report
This year's report is fascinating! There is clear evidence that DevOps practices yield remarkable results for IT teams and organizations. Findings are related to transformational leadership, automation practices, continuous delivery, lean product management, and DevOps in not-for-profits and organizations that use off-the-shelf software.
Tom on the Stack Overflow Podcast this week
Three people from the SRE team took over this week's Stack Overflow Podcast. You can hear myself, Mark Henderson and Jason Harvey rant about important topics such as how we got our start, USB Condoms, and my opinionated interpretation of the "First Day Database Destruction" discussion on Reddit. The podcast episode is announced here: (link) You can skip all that and go directly to SoundCloud here: (link) Enjoy!
Tom on the Stack Overflow Podcast this week
Three people from the SRE team took over this week's Stack Overflow Podcast. You can hear myself, Mark Henderson and Jason Harvey rant about important topics such as how we got our start, USB Condoms, and my opinionated interpretation of the "First Day Database Destruction" discussion on Reddit. The podcast episode is announced here: (link) You can skip all that and go directly to SoundCloud here: (link) Enjoy!
How difficult is it to enable HTTPS on a large website?
Very. The larger the site, the harder it becomes. My co-worker Nick Craver just blogged about the multi-year journey to enable HTTPS at StackOverflow.com and all our sub-properties (spoiler alert: there is still more work to do!) It is probably the best detailed description of the process. If your boss ever asks you why HTTPS can't be enabled in just a few hours, this is a good resource. https://nickcraver.com/blog/2017/05/22/https-on-stack-overflow/
How difficult is it to enable HTTPS on a large website?
Very. The larger the site, the harder it becomes. My co-worker Nick Craver just blogged about the multi-year journey to enable HTTPS at StackOverflow.com and all our sub-properties (spoiler alert: there is still more work to do!) It is probably the best detailed description of the process. If your boss ever asks you why HTTPS can't be enabled in just a few hours, this is a good resource. https://nickcraver.com/blog/2017/05/22/https-on-stack-overflow/
May 2017 NYCDevOps Meeting Notes
Here are my notes from last night's NYCDevOps meeting. Title: Using innersourcing to break down organizational barriers Speaker: Aroon Gursahaney, Verizon This was one of the best talks I've seen in a while because it was entirely new material for me. I haven't heard of a company doing anything like this. Last night I learned: You can replace legacy systems by crowdsourcing parts of the project to people around the company in exchange for giving them the opportunity to learn new technologies, tools, and devops practices You can gamify culture change in an organization You cam make crowdsourcing the norm, not the exception.
The Blog March: A personal drumbeat of #resistance
This post is part of the Blog March 2017. Every day this month a different blogger is writing about politics. Please march along with us! A personal drumbeat of #resistance A friend of mine recently commented that she'd like to get more involved in the resistance. She had attended a march or two, but those were occasional and spur of the moment activities. She asked if I had advice about how to turn her occasional activism into something more sustained? In other words, how could she make more of a difference but not quit her job and dedicate her life to the resistance?
3 things that made the 3rd edition better for education
The 3rd edition of The Practice of System and Network Administration was released last September. There are more than 600 new pages of material in this edition. What makes it better for classroom use? Before writing this edition we talked to many university instructors about what they liked or disliked about past editions. We listened! As a result: Shorter chapters. Educators want to assign 10-15 pages of reading each night. Many of our chapters were 40-50 pages long. We split chapters into smaller ones. Top-down organization. We start with best practices and work our way down into the details. Better exercises at the end of each chapter.
Blog March... GO!
Each day in May a different blogger will be writing about something true, important, or moving. I hope you will read along each day this month. Interact and ask questions. Quote, share, comment, and challenge, with respect. Appreciate, learn, and "march" along with us. Links to each day's blog post will appear on The Robin Renée Blog. Robin is a friend and fellow activist who I've known for decades. I'm honored to be part of this movement. My blog post will appear here on EverythingSysadmin.com on Monday 5/15.
Why tech conferences should have Tshirts in women's sizes
Because it is easy to do. Because you complain that you can't think of anything that would make your conference more appealing to women, and this is a tangible thing that you can do to make your conference more appealing to women. It is usually as easy as clicking some extra buttons on the web form when you order the shirts. Because you should be happy that there is something you can fix without having to learn a new skill, spend a million dollars, or form a committee. Its like when someone complains, "Damn! My partner gets so upset about little things like me not taking out the garbage."
DNS as Code
StackOverflow has open sourced the DNS management system we've been using for years. I was the original author, and Craig Peterson has been writing most of the code lately. A blog post went live about the system here: Blog post: Introducing DnsControl - "DNS as Code" has Arrived My favorite part of DNSControl? We can switch between DNS providers faster than cyberthreads can take them down. Check it out if you manage DNS zones, use CloudFlare or Fast.ly CDNs, or just like to read Go (golang) code. Tom
Review: Tivo BOLT+
The newest TiVo model is called the BOLT. I've been using TiVo since the first generation, back when it used a dial-up modem to download the tv guide listings and software updates. My how far we've come! If you have a TiVo already, the BOLT user interface looks and acts the same but everything is faster and better. There is a new feature that automatically skips commercials (if the TV show permits it), an a feature that plays shows at 30% faster speed, with pitch-correction. Everything is faster. This unit has more RAM and a faster CPU than any previous TiVo model, which really shows in the UI.
Hollywood doesn't understand software
Hollywood doesn't understand software. Not, at least, as well as high-tech companies do. This is very frustrating. Bad software keeps wrecking my entertainment experience. I'm currently writing an article and I need to come up with a term that means software that was written by old-school (historically non-technology) companies just so they can say "Look! we made an app! Will you shut up, now?!" as opposed to software that has great fit and finish, gets updated regularly, and stays current. My favorite example of this is the CBS streaming software. It seems like it was written just to shut up people that have been asking to stream NCIS, not because CBS actually wants to be in the streaming business.
New season of Archer starts tonight!
The new season of Archer starts tonight. You're welcome.
Tom speaking at BayLISA meetup, Thu, March 16, 2017
I'll be giving my "Stealing the Best Ideas from DevOps" talk at BayLISA next week. Location will be in Palo Alto (exact location is in flux, RSVP for updates). Hope to see you there! More info and RSVP here: https://www.meetup.com/BayLISA/events/237776292/
February NYCDevOps meetup: Tiny Talks CfP
The NYC DevOps meetup is trying something new in February: Tiny Talks. Tiny talks are short, 5-10 minute talks usually presented by members. They can be basically any topic related to DevOps. The idea is to encourage local members to present at meetings. We'll be doing this for our first time at the February 21, 2017 NYCDevOps meetup. You are encouraged to sign up here (though we won't be strict if you show up with a presentation.) From the website: Tiny talks are 5-10 minute talks on any DevOps-related topic. We want to encourage everyone to submit a proposal.
Tom speaking at NYCDevOps meetup, Feb 21, 2017
I'll be the speaker at the February meeting of NYCDevOps. I'll be presenting my talk "Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers". If you didn't see me give this talk at Usenix LISA, you can catch it here. https://www.meetup.com/nycdevops/events/237543220/ Please RSVP. Seating is limited.
Cascadia IT Conference: Registration is open!
I am pleased to announce that the 7th Annual Cascadia IT conference Registration is open at https://www.casitconf.org/casitconf17/register-now/. This years conference features tutorials on Git, Reusable Infrastructures, Time Management (based on my book), AWS, and Advanced Communication Strategies along with several talks and a keynote by Corey Quinn. Cascadia IT Conference will be held on March 10-11 in Seattle WA. https://www.casitconf.org
New North Jersey Meetup, Wed, Feb 1, 2017
LOPSA-NJ is starting regular meetings in Montclair, NJ. They've asked me to be the speaker at their first meeting. I'm honored to be asked (oh... and I live 5 miles away, so it is difficult to turn down). I'll be giving my talk, "Stealing Best Ideas from DevOps: A Guide for Sysadmins without Developers". LOPSA-NJ/Montclair Wednesday, February 1, 2017 7:00 PM RSVP and other info: https://www.meetup.com/LOPSA-NJ/events/236564846/ I look forward to this new group being a big success! Come join us!
Moving application plists to a new machine on macOS
Today I learned that you can't copy a Mac application's plist by just copying the file. However, you can export the plist and import it on a new machine: Step 1: Exit the app. To make sure the file is stable. Step 2: Export the plist data: $ defaults export info.colloquy ~/info.colloquy.backup To know the name of the plist (info.colloquy in this example) look in ~/Library/Preferences. Use the filename but strip off the .plist suffix. If an app has multiple plists, (I assume you need to) do each of them individually. Step 3: Copy the backup file to the new machine I like to either copy it to Dropbox and wait for it to sync on the other machine, or scp it to my VPS and then scp it down to the new machine.
How I manage my work and personal GitHub accounts
I have two accounts on GitHub: Personal and work. How do I access both from the same computer without getting them confused? I have two different ssh keys and use .ssh/config to do the right thing. Some bash aliases also help. Why? Why is it important to me that I keep these two accounts separate? First, certain repos can only be accessed from one account or the other. Public repos can be accessed from either, but private repos are not so lucky. Second, I want the commit logs for different projects should reflect whether I am "Tom the person" or "Tom the employee".
RIP John Boris
John was active in the LOPSA community. I saw him at nearly every LOPSA-NJ meeting, where he was active in planning and hosting the meetings. He was also on the board of LOPSA (national) where he will be greatly missed. John was also a football coach at the school where he worked in the IT department. It was very clear that his coaching skills were something he applied everywhere, including his helpfulness and mentoring at LOPSA. I had a feeling that when I hugged him at the end of the January LOPSA meeting it might be the last time I saw him.
How Stack Overflow plans to survive the next DNS attack
My coworker did a bang-up job on this blog post. It explains a lot about how DNS works, how the Dyn DDOS attack worked (we missed it because we don't use Dyn), and the changes we made so that we'll avoid similar attacks when they come. How Stack Overflow plans to survive the next DNS attack
NYCDevOps meetup is re-starting!
Hey NYC-area folks! The NYC DevOps meetup is springing back to life! Our next meeting will be Tuesday, January 17, 2017 from 6pm-7:30pm. The meeting is at the Stack Overflow NYC headquarters near the financial district. For complete details, visit https://www.meetup.com/nycdevops/events/236646177/ From the announcement: Please join us on January 17th from 6:00 - 7:30 PM at Stack Exchange for our first annual DevOps Mixer. Our goal is to re-engage with our members for an inaugural meet and greet with our new team of organizers, awesome community members, and of course there will be refreshments! Come socialize with us and talk about your experiences, what's new, what you're working on and what you would like to see from the NYC DevOps Meetup.
Are You Load Balancing Wrong?
I write a thrice-yearly column in acmqueue Magazine. The Dec 20 issue has my newest entry, "Are You Load Balancing Wrong?" You can read it at this URL: http://queue.acm.org/detail.cfm?id=3028689 acmqueue is free for ACM professional members. Non-members can purchase an annual subscription for $19.99 or a single issue for $6.99. Download the app from iTunes or Google Play, or view within your browser. More information here.
Tom speaking at the new LOPSA-NJ Montclair Meetup
I'll be the speaker at LOPSA-NJ/Montclair in February. Hope to see you there! The February meeting will be held at Montclair State University. Topic: Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers Speaker: Tom Limoncelli, StackOverflow.com Date: WEDNESDAY, February 1st, 2017, 7pm (not Thursday) Location: Montclair University, CELS 110, 1 Normal Ave, Montclair, NJ 07043 Talk Description: This talk will present the DevOps principles in terms that apply to all system administrators, and use case studies to explore their use in non-developer environments. DevOps is not a set of tools, nor is it just automating deployments.
Notes on implementing flags for flag flips
Someone recently asked me how the flags in "flag flips" are implemented in actual software. There are a couple different techniques that I've seen. Command-line flags: Normal command line flags. To change a flag, you have to kill and restart the process. These flags are usually implemented in libraries that come with the language, though I have a preference for gflags. A/B flags: When doing A/B testing, a cookie is set with a random number (often a value 0-999). The code then uses the number to partition the tests: A=0-499, B=500-999. To enroll 1 percent of all users, one might use A=0-4 and B=5-9 and everyone else gets the default behavior.
InfoQ's interview with the co-authors of TPOSANA 3rd edition
Ben Linders interviewed Tom, Christine and Strata about the 3rd edition of The Practice of System and Network Administration: See what's new in the 3rd edition of The Practice of System and Network Administration. Learn how to steal DevOps practices for use in enterprise IT and desktop support. What's better than beta testing IT services? Many small batch launches. Learn how bi-directional empathy creates a collaborative IT culture. What can go wrong when launching a new service? Everything! Here's how to avoid surprises. Plus: InfoQ readers can download a book extract with a discount code.
Gall bladder Removal
Today I say goodbye to my gall bladder. I'll miss you, little guy. I scheduled this blog post to appear around the time I should be coming out of surgery. Recovery should be pretty fast (a few days). Here's the story told in cartoon form: Prequel Pancreatitis Gall bladder's Last Day Kidney Hears The News Organ Craft Fair You can read all of The Awkward Yeti's cartoons involving gall bladder here. See you soon!
Another successful LISA!
Congrats to the LISA organizing committee for another successful conference! The Usenix LISA conference was in Boston this year, and ran Sunday to Friday as usual. This was the 30th conference. Highlights: I learned that the coming generation of servers from Dell/HP/etc. will have terabytes of persistent RAM (RAM that retains info between reboots). Operations will have to radically change how they do operations (rebooting won't clear RAM). Software engineers will be re-thinking software designs. (John Roese, Dell EMC) I learned that Homomorphic encryption lets you do math on encrypted data, and you get encrypted results. No need to decrypt the source material.
Solaris being canned.
"Solaris being canned, at least 50% of teams to be RIF'd in short term. Hardware teams being told to cease development. There will be no Solaris 12, final release will be 11.4. Orders coming straight from Larry." https://www.thelayoff.com/t/KBEVoB1 The crazy people that said that Oracle bought Sun just to shut it down and use the patents to sue Google are looking pretty non-crazy now. I guess once the lawsuit was lost (and boy was it expensive) the shutdown was inevitable. Of course, not funding SPARC development so that it could stay competitive with Intel didn't help much either.
TPOSANA on Amazon Lightning Deals today only!
Upgrade to the new edition! Amazon is having a time-limited sale today that includes the new edition of The Practice of System and Network Administration. This is a good way to get the new edition at a deep discount: Follow this link. It should be somewhere on this page. The deal is only available today (Tuesday, 22-Nov) from 12:05 PST to 6:05 PST.
Stealing the Best Ideas from DevOps
Christine Hogan and I will be co-presenting a talk at Usenix LISA '16 entitled "Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers". Full details are on the LISA website. The talk will cover a lot of the devops-y material from our newest book, the 3rd edition of TPOSANA. We'll be doing a book-signing shortly after the talk. In addition, I'll also be teaching two half-day tutorials: "Personal Time Management: The Basics for Sysadmins That Are Overloaded" and "How to Not Get Paged: Managing On-Call to Reduce Outages". Links to all of this here.
Do you teach System Administration?
You do? Then you should attend SESA '16 in Boston, Dec 6, 2016. For more info go to https://www.usenix.org/conference/sesa16 (I'm particularly looking forward to Nicole's opening keynote.)
Pre-Order Kindle edition of 3rd edition of TPOSANA
The Kindle edition of the 3rd Edition of "Volume 1: The Practice of System and Network Administration" can now be pre-ordered. Previously you could only pre-order the paper and PDF versions. The new edition will be available on November 4th, 2016. It includes 22 new chapters, 6 updated chapters, and thousands of updates all over the book. The new edition is nearly twice as long as the previous editions. We worked hard to make this the best edition yet. Here's where you can pre-order the book: Paper: Pre-order from Amazon or InformIT. Kindle: Pre-order from Amazon. PDF (no DRM): Pre-order from InformIT.
Searching random data is not an O(N) problem.
The September/October issue of ACM Queue Magazine has a column I wrote about how to search random data. In theory the best you can do is a linear search. I came up with 10 ways to do better. http://queue.acm.org/detail.cfm?id=2984631 Computer scientists should be upset that I write these things.
Usenix LISA '16 Early Bird Registration Deadline is November 10!
The "Early Bird Registration" deadline is November 10, 2016. After that, the price goes up. LISA is Dec 4-9 and you can register here. I'll be teaching 2 tutorials and Christine Hogan and I will be co-presenting some talks. We also are planning a book signing during the conference. We'll announce the time/location when it gets closer to the event. Presetations: Talk: "Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers" (Tom+Christine) Tutorial: "How to Not Get Paged: Managing On-Call to Reduce Outages" (Tom) Tutorial: "Personal Time Management: The Basics for Sysadmins That Are Overloaded" (Tom) SESA'16 (2016 USENIX Summit for Educators in System Administration) is colocated with LISA.
Watch my Punk Rock DevOps talk live on Friday at 2:30pm PT
My talk "DevOps Where You Wouldn't Have Expected" will be live-streamed from PuppetConf on Friday, Oct 21 at 2:30pm PT / 5:30 ET. The talk happens to summarize the major points of Chapter 1-4 of the new edition of The Practice of System and Network Administration, which is due out on November 4, 2016. You need to pre-register, which takes time. Please preregister early. A full description of the talk is at http://sched.co/6fk7 To register and watch the live-steam visit https://puppet.com/puppetconf/livestream/signup P.S. The unofficial title of the talk is "Punk Rock DevOps".
DevOps Handbook hitting stores Oct 6!
One of the most anticipated DevOps books in years is about to start shipping! DevOps Handbook: How to Create World-Class Agility, Reliability, & Security in Technology Organizations by Gene Kim, Jez Humble, Patrick Debois, and John Willis is the practical guide to doing all the wonderful things that The Phoenix Project talks about. I've received an early copy of the book and it is excellent. It is very down-to-earth, practical, advice. I'll write more next week when I've had time to read the entire thing. You can pre-order it directly from IT Revolution or via Amazon. Check it out!
Why Google should buy Twitter
[Disclaimer: I do not work for Google or Twitter; I have no investments in Google or Twitter. ] Google should buy Twitter. (link to rumors here) Twitter isn't a good "MBA runs the numbers" acquisition. However could be used as a showcase for CGE. It would more than justify itself. In fact, the financial losses might be off-set by the marketing value it provides to CGE. As part of integrating it into the internal Google stack, they should require their engineers to rebuild it on the Google Cloud Engine platform. GCE scales crazy-good. Twitter has a history of scaling problems. If Google could run it on the Google Cloud Engine, and show that it scales, it would be great advertising.
Discounts on The Practice books until Sept 19!
Pearson is doing their annual "Back to Business" sale until Monday, September 19. You can save 35-45%, which is a big deal IMHO. The Practice of Cloud System Administration is 35% off, or 45% off if you buy 2 copies. Buy one for yourself and get a copy for a friend for their birthday. Just use this link to receive the discount. You can also get a copy of the Cloud Administration book plus the new 3rd Edition of The Practice of System and Network Administration (when it ships in November) and save 45% if you use this link and enter offer code "B2B".
TPOSANA launched 15 years ago today!
15 years ago today (or August 24, 2001 depending on who you talk to) the first edition of The Practice of System and Network Administration reached bookstores. We had been working on the book for 2+ years, having first met during a Usenix conference in 1999. Writing it was quite an experience, especially since this was before voice-chat on the internet was common, and we were on different continents (Christine in London and Tom in New Jersey). We collaborated via email, used CVS for our source code repository, and we had monthly phone calls (which Tom dialed from work... Thanks, Lucent!)
Survey: What makes joining a new team difficult/easy?
The last time you joined a new operations/devops/sysadmin team, what make it easy or difficult to get started? For example... if procedures aren't documented, it can be very difficult to know how to perform them. The rest of the team does them by memory, but you are spending your time asking for help, or guessing your way through them. Well-documented procedures (or even tiny bullet lists) make it easy to be self-sufficient quickly. What have you found made assimilating into a new team difficult? What have you seen teams do that made it easy? What did you wish a team had done to make it easy?
USB cables should be "charge-only" by default
Every sysadmin knows that you can protect a server though cryptographic or other means, but if someone has physical access "all bets are off". Right? With physical access they can do physical damage (smash it with a hammer) or pull out a hard disk and read the bits directly. Even security systems that are highly respected (I'm looking at you, Kerberos!) are an "all bets are off"-situation if someone gets the private key through physical access. Sadly we forget this when it comes to smartphones. We'll plug our phones into any darn USB charger we find... especially when we are desperate!
Automating a CI environment for Android Apps
One of the things my team at StackOverflow does is maintain the CI/CD system which builds all the software we use and produce. This includes the Stack Exchange Android App. Automating the CI/CD workflow for Android apps is a PITA. The process is full of trips and traps. Here are some notes I made recently. First, [this is the paragraph where I explain why CI/CD is important. But I'm not going to write it because you should know how important it is already. Plus, Google definitely knows already. That is why the need to write this blog post is so frustrating.]
I'll be speaking at NYC SRE Meetup next week (Wed)
I'll be giving a new talk called "Teaching DevOps to Ops without Devs" at the NYC SRE Tech Talks meetup, Wednesday, August 17, 2016, from 6pm to 8:30pm, at Squarespace's offices in the West Village (8 Clarkson St, 12th Floor). There are 4 speakers during the event. This is a free event. Seating is limited. Info and registration here.
Survey on easy of use for configuration management languages
I'm reposting this survey. The researcher is trying to identify what makes configuration languages difficult to use. If you use Puppet/Chef/CfEngine/Ansible (and more importantly... if you DON'T use any configuration languages) please take this survey. I took it in under 10 minutes. Tom I would like to invite you to take a survey on configuration languages. The survey is about 10 minutes long and does not require any specific skills or knowledge, anyone can participate in it. https://edinburgh.onlinesurveys.ac.uk/configlang The survey is a part of ongoing research at University of Edinburgh School of Informatics on configuration languages. As Paul Anderson has mentioned previously, we are interested in studying the usability of configuration languages and how it relates to the configuration errors.
"The Martian" made me depressed
Have you seen/read The Martian? What's so sad about the movie/book is that it is a reminder of what could have been. Part of the premise is that after the Apollo program, the U.S. continued their plans for landing on Mars. Such plans were dropped for the less ambitious Shuttle program. Think about it. In most science fiction the science is unbelievable. In The Martian, the science was pretty darn accurate and the unbelievable part is that U.S. politicians had the audacity to continue NASA's funding level.
LISA Conversations Episode 11: Russell Pavlicek on "Unleashing the Power of the Unikernel"
Episode 11 of LISA Conversations is Russell Pavlicek, who presented Unleashing the Power of the Unikernel at LISA '15. Watch the Episode here: LISA Conversations Episode #11 with Russell Pavlicek Co-hosts: Lee Damon and Thomas Limoncelli Guest: Russell Pavlicek Recorded Tuesday, June 28, 2016 In this episode we discuss his talk: Unleashing the Power of the Unikernel Recorded at LISA '15 Talk Description You won't want to miss this!
Watch us live today! LISA Conversations Episode 11: Russell Pavlicek on "Unleashing the Power of the Unikernel"
Today (Tuesday, June 28, 2016) we'll be recording episode #11 of LISA Conversations. Join the Google Hangout and submit questions live via this link. Our guest will be Russell Pavlicek. We'll be discussing his talk Unleashing the Power of the Unikernel from LISA '15. The video we'll be discussing: Unleashing the Power of the Unikernel Russell Pavlicek Recorded at LISA '15 Talk Description Watch us record the episode live! Tuesday, June 28, 2016 at 3:30-4:30 p.m. PT LISA Conversations Episode #11 Co-hosts: Lee Damon and Thomas Limoncelli Guest: Russell Pavlicek Join us live! link The recorded episode will be available shortly afterwards on YouTube.
Reminder: Do your homework for next week's LISA Conversations: Russell Pavlicek on "Unleashing the Power of the Unikernel"
This weekend is a good time to watch the video we'll be discussing on the next episode of LISA conversations: Russell Pavlicek's talk from LISA '15 titled Unleashing the Power of the Unikernel. Homework: Watch his talk ahead of time. Unleashing the Power of the Unikernel Recorded at LISA '15 Talk Description Then you'll be prepared when we record the episode on Tuesday, June 28, 2016 at 3:30-4:30 p.m. PT. Register (optional) and watch via this link. Watching live makes it possible to participate in the Q&A. The recorded episode will be available shortly afterwards on YouTube.
Save 20% off Velocity NYC registration!
Don't miss Velocity NYC, Sept 20-22. There are a lot of great talks scheduled... including one that I'll be giving! You can save 20% off registration! Please use this link and then use code AFF20.
New appearances announced: VelocityNYC & PuppetConf!
I'm excited to announce that I'll be speaking at Velocity NYC and (for the first time) at PuppetConf! Velocity NYC, October 19-22, 2016, "Teaching DevOps to Ops without Devs (and so can you!)", Please use this link for information or to register. PuppetConf, October 19-21, 2016 in San Diego, "Doing DevOps Where You Wouldn't Have Expected", Save 35% if you use this link! I look forward to seeing you there!
Next on LISA Conversations: Russell Pavlicek on "Unleashing the Power of the Unikernel"
Our next guest will be Russell Pavlicek. We'll be discussing his talk from LISA '15 titled Unleashing the Power of the Unikernel. Watch live! We'll be recording the episode on Tuesday, June 28, 2016 at 3:30-4:30 p.m. PT. Particpate in the live Q&A by submitting your questions during the broadcast. Pre-registration is recommended but not required. Register and/or watch via this link. Homework: Watch his talk ahead of time. Unleashing the Power of the Unikernel Recorded at LISA '15 Talk Description Watch us record the podcast live! LISA Conversations Episode #11 Co-hosts: Lee Damon and Thomas Limoncelli Guest: Russell Pavlicek Will be recorded: Tuesday, June 28, 2016 at 3:30-4:30 p.m.
NYLUG: Two meetings this month! (Foreman! Unikernels!)
Two meetings this month. Both at different times and locations. Be sure to show up at the right time and place! Tue, June 7, 2016, 7PM Topic: Foreman latest updates: Puppet 4, remote execution and discovery Speakers: Daniel Lobato García and Ivan Necas http://www.meetup.com/nylug-meetings/events/231609565/ and Wed, June 15, 2016, 6:30PM Topic: Unikernels - What Are They Good For? Speaker: Amir Chaudhry http://www.meetup.com/nylug-meetings/events/231555330/
LISA Conversations Episode 10: Clay Caviness and Edward Eigerman on "Managing Macs at Google Scale"
Episode 10 of LISA Conversations is Clay Caviness and Edward Eigerman, who presented Managing Macs at Google Scale at LISA '13. Watch the Episode here: LISA Conversations Episode #10 with Clay Caviness and Edward Eigerman Co-hosts: Lee Damon and Thomas Limoncelli Guests: Clay Caviness and Edward Eigerman Recorded Tuesday, May 31, 2016 In this episode we discuss they talk: Managing Macs at Google Scale Recorded at LISA '13 Talk Description You won't want to miss this!
An Honest Job Advertisement
Imagine if job advertisements were completely honest. Most companies advertising for IT workers would state that the job is mostly great except for twice a year when ``hell month' arrives and everyone scrambles to deploy the new release of some major software system. This month is so full of stress, fear, and blame that it makes you hate your employer, your job, and your life. Oh, and by the way, the software releases are often late, so you can't predict which month will be hell month. As a result, you can't schedule any kind of vacation. Without time off to relax, stress builds and makes your life even worse.
Augeas: Updating of config files without ruining comments and formatting.
Wouldn't it be nice if you could write a program that could reach into an Apache config file (or an AptConf file, or an /etc/aliases file, Postfix master.cf, sshd/ssh config, sudoers, Xen conf, yum or other) make a change, and not ruin the comments and other formatting that exists? That's what Augeas permits you to do. If a config file's format as been defined in the Augeas "lens" language, you can then use Augeas to parse the file, pull out the data you want, plus you can add, change or delete elements too. When Augeas saves the file it retains all comments and formatting.
Future directions for Blackbox
I maintain an open source project called Blackbox which makes it easy to store GPG-encrypted secretes in Git, Mercurial, Subversion, and others. I've written up my ideas for where the project should go in the future, including renaming the commands, change where the keys are stored, add a "repo-less" mode, and possibly rewrite it in a different language: https://github.com/StackExchange/blackbox/blob/master/Version2-Ideas.md Feedback welcome! Tom
Preview a chapter of our new book in acmqueue magazine!
The new issue of acmqueue magazine contains a preview of a chapter from our next book, the 3rd edition of TPOSANA. This issue contains a preview of a chapter from our next book, the 3rd edition of TPOSANA. The chapter is called "The Small Batches Principle". We are very excited to be able to bring you this preview and hope you find the chapter fun and educational. The book won't be out until Oct 7, 2016, so don't miss this opportunity to read it early! ACM members can access it online for free, or a small fee gets you access to it online or via an app.
10 git aliases I can't believe you don't have already
It makes me sad to see people type more than they have to. With these aliases, you reduce the 4 most common commands to 2 letter abbreviations: git config --global alias.co checkout git config --global alias.br branch git config --global alias.ci commit git config --global alias.st status NOTE: This updates your ~/.gitconfig file and adds aliasses "co", "br", "ci", and "st". If you collaborate with others, git pull makes a messy log. Instead, always type git pull --rebase --ff-only. This will make the merge history a lot more linear when possible, otherwise it falls back to the normal pull behavior.
I am Satoshi Nakamoto, inventor of Bitcoin
There is a long and fraught history in Bitcoin of claims and counterclaims about who Satoshi is. I might as well confess that he is me. I come forward at this time because Craig Wright claims to be Satoshi and I can't stand such intentional scammery. If you read any of my pre-Bitcoin books, you'll see there are many pages where the first letter of each line reads "I am Satoshi Nakamoto" and "Someday I will invent Bitcoin". If you can't find the page that contains this, buy more copies of the books. You just haven't found the right one. Please use this link, since it includes my Amazon Associates code.
Have you downloaded the March/April issue of acmqueue yet?
The March/April issue of acmqueue - the magazine written for and by software engineers that leaves no corner of the development world unturned - is now available for download. This issue contains a preview of a chapter from our next book, the 3rd edition of TPOSANA. This issue contains a preview of a chapter from our next book, the 3rd edition of TPOSANA. The chapter is called "The Small Batches Principle". We are very excited to be able to bring you this preview and hope you find the chapter fun and educational. The book won't be out until Oct 7, 2016, so don't miss this opportunity to read it early!
Watch us live today! LISA Conversations Episode 9: kc claffy on "Named Data Networking"
Today (Tuesday, April 26, 2016) we'll be recording episode #9 of LISA Conversations. Join the Google Hangout and submit questions live via this link. Our guest will be kc claffy. We'll be discussing her talk Named Data Networking from LISA '15. The video we'll be discussing: Named Data Networking kc claffy Recorded at LISA '15 Video and Slides Watch us record the episode live! Tuesday, April 26, 2016 at 3:30-4:30 p.m. PT LISA Conversations Episode #9 Co-hosts: Lee Damon and Thomas Limoncelli Guest: kc claffy Join us live! link The recorded episode will be available shortly afterwards on YouTube.
Reminder: Do your homework for next week's LISA Conversations: kc claffy on "Named Data Networking"
This weekend is a good time to watch the video we'll be discussing on the next episode of LISA conversations: kc claffy's talk from LISA '15 titled Named Data Networking. Homework: Watch her talk ahead of time. Named Data Networking Recorded at LISA '15 Video and Slides Then you'll be prepared when we record the episode on Tuesday, April 26, 2016 at 3:30-4:30 p.m. Pacific Time. Register (optional) and watch via this link. Watching live makes it possible to participate in the Q&A. The recorded episode will be available shortly afterwards on YouTube.
Server Hardware Support Rant
[This is a rant. Take it with a grain of salt.] You know what's great about "the cloud"? I don't have you deal with [insert server vendor's name] support process that is so complex and broken that it makes me want to die. If a machine in AWS/GCP/Azure dies I don't have to load a f***ing flash-based web page that breaks on .... oh my god... every browser except one that is 10 years old and runs on an OS that I don't use... and .... god damn it what do you mean my account isn't cleared for that product and...
CU-Boulder remembers Evi Nemeth, April 26, 2016
CU-Boulder will be hosting an event on April 26, 2016, to celebrate the life of Evi Nemeth who passed away three years ago. You may remember Evi from her many books on system administration, her tutorials at Usenix LISA, or many of her other projects that influenced system administration as it exists today. ... we will celebrate the life of retired professor Evi Nemeth, an accomplished sailor who was lost at sea in June 2013. Evi joined the department in 1980 and was one its foundational figures for 20 years. She is best remembered for her rigorous data structures class and for providing a safe haven and confidence-building experiences for a generation of students who didn't fit the typical academic mold.
Easily read UTF16-encoded files from Go
A program I wrote that worked for quite some time started failing. It turns out someone tried to use it to process a file with text encoded as UTF16. The file came from a Windows system and, considering things like UoW, this situation is just going to start happening more frequently. Golang has a great package for dealing with various UTF encodings. That said, it still took me a few hours to figure out how to make an equivalent of ioutil.ReadFile(). I wrapped up what I learned and made it into a module. Everything should just work like magic. Instead of using os.Open(), use utfutil.OpenFile().
Next on LISA Conversations: kc claffy on "Named Data Networking"
Our next guest will be kc claffy. We'll be discussing her talk from LISA '15 titled Named Data Networking. Watch live! We'll be recording the episode on Tuesday, April 26, 2016 at 3:30-4:30 p.m. Pacific Time. Particpate in the live Q&A by submitting your questions during the broadcast. Pre-registration is recommended but not required. Register and/or watch via this link. Homework: Watch her talk ahead of time. Named Data Networking Recorded at LISA '15 Video and Slides Watch us record the podcast live! LISA Conversations Episode #9 Co-hosts: Lee Damon and Thomas Limoncelli Guest: kc claffy Will be recorded: Tuesday, April 26, 2016 at 3:30-4:30 p.m.
LISA Conversations Episode 8: Caskey Dickson on "Why Your Manager LOVES Technical Debt and What to Do About It"
Episode 8 of LISA Conversations is Caskey Dickson, who presented Why Your Manager LOVES Technical Debt and What to Do About It at LISA '15. Watch the Episode here: LISA Conversations Episode #8 with Caskey Dickson Co-hosts: Lee Damon and Thomas Limoncelli Guest: Caskey Dickson Recorded Tuesday, March 29, 2016 In this episode we discuss his talk: Why Your Manager LOVES Technical Debt and What to Do About It Recorded at LISA '15 Video and Slides You won't want to miss this!
Velocity Santa Clara early price ends May 12
Velocity Santa Clara is June 20-23, 2016. Sadly I can't attend this year due to a pre-existing commitment. However you can still register. Do it before the early discount evaporates!
SRE is what DevOps wants to be when it grows up
There are two things you can do if you want to understand the future of system administration. First, if you want to see what DevOps will be like 5-10 years out, you can read the amazing new book, "Site Reliability Engineering: How Google Runs Production Systems". I read a preview copy and it was excellent. Many different Google SRE teams got together to produce a very well-rounded book that covers all aspects of Google's SRE program, which is easily 5-10 years ahead of the industry. (Pre-order from O'Reilly or Amazon) Congrats to the editors Niall Richard Murphy, Chris Jones, Jennifer Petoff, and Betsy Beyer on a great addition to the IT cannon.
Take the 2016 State of DevOps Survey
Whether or not you are in a DevOps environment, please take this survey. The data is useful for helping improve the situation for system administrators of all kinds. http://itrevolution.com/2016devopssurvey/
"How SysAdmins Devalue Themselves" now in CACM
CACM reprinted my article in the April edition. They don't usually publish April Fools articles, but I'll consider this the appropriate place for this article. If you subscribe to CACM, you can read the article online, PDF, Ebook. You can also read it in the original publication, ACM Queue for free.
Archer starts tonight! (and other important TV news)
The new season of Archer starts tonight!! (Mar 31, 2016) at 10pm! The Powerpuff Girls 2016 Reboot starts on April 4th. Set your DVRs now! The Detour starts on April 11. I have high hopes for this show. It is created by Samantha Bee and Jason Jones. Speaking of Samantha Bee, her new weekly news program Full Frontal with Samantha Bee is my new favorite show. I think it shows that she should have replaced Jon Stewart. Silicon Valley's season 3 premieres April 24. If you work at a startup, or just wish you did, you can't miss this show. ProTip: If you DVR can only set up new recordings 14 days in advance, set a reminder in your April 10 todo list.
Epidemic of Vacation Shaming Spreads Across America
This HuffPo article is worth reading: According to an Ernst & Young internal study, for "each additional 10 hours of vacation employees took, their year-end performance ratings improved 8 percent, and frequent vacationers also were significantly less likely to leave the firm". By the way... using your vacation time one day here and a long weekend there is not a "vacation". Generally I find that you're brain doesn't relax until day 3 of a vacation, especially if there is travel involved. If you don't take a long break you never reach that point. If you take a lot of long weekends you end up spending that time doing laundry and you rob yourself of the opportunity to actually relax.
Watch us live today! LISA Conversations: Caskey Dickson on "Why Your Manager LOVES Technical Debt and What to Do About It"
Today (Tuesday, March 29, 2016) we'll be recording episode #8 of LISA Conversations. Join the Google Hangout and submit questions live via this link. Our guest will be Caskey Dickson. We'll be discussing his talk Why Your Manager LOVES Technical Debt and What to Do About It from LISA '15. The video we'll be discussing: Why Your Manager LOVES Technical Debt and What to Do About It Caskey Dickson Recorded at LISA '15 Video and Slides Watch us record the episode live! Tuesday, March 29, 2016 at 3:30-4:30 p.m. Pacific Time (convert) LISA Conversations Episode #8 Co-hosts: Lee Damon and Thomas Limoncelli Guest: Caskey Dickson Join us.
Reminder: Do your homework for next week's LISA Conversations: Caskey Dickson on "Why Your Manager LOVES Technical Debt and What to Do About It"
This weekend is a good time to watch the video we'll be discussing on the next episode of LISA conversations: Caskey Dickson's talk from LISA '15 titled Why Your Manager LOVES Technical Debt and What to Do About It. Homework: Watch his talk ahead of time. Why Your Manager LOVES Technical Debt and What to Do About It Recorded at LISA '15 Video and Slides Then you'll be prepared when we record the episode on Tuesday, March 29, 2016 at 3:30-4:30 p.m. Pacific Time (convert). Register (optional) and watch via this link. Watching live makes it possible to participate in the Q&A.
Wasting one million dollars
"Done" means "launched". It isn't "done" until it is launched. It annoys me to hear people say a project is "done... now I just have to launch it". It isn't done if it isn't in production. There are a few reasons for this: people think that launch is "the last 5 percent of a project" but often 80 percent of your time will be consumed by this last 5 percent. Also, you aren't "done" until other people are benefitting from your work (in business speak... " it is delivering value"). Written code has no business value. Launched code does. You can rig this in your favor.
Maintain Separate GitHub accounts
Someone recently commented that with Github it is "a pain if you want to have a work and personal identity." It is? I've had separate work and personal Github accounts for years. I thought everyone knew this trick. When I clone a URL like [email protected]:TomOnTime/tomutils.git I simply change github to either github-home or github-work. Then I have my ~/.ssh/config file set with those two names configured to use different keys: # TomOnTime / [email protected] Host home-github.com HostName github.com User git IdentityFile ~/.ssh/id_rsa-githubpersonal PreferredAuthentications publickey PasswordAuthentication no IdentitiesOnly yes # tlimoncelli / [email protected] Host work-github.com HostName github.com User git IdentityFile ~/.ssh/id_rsa-githubwork PreferredAuthentications publickey PasswordAuthentication no IdentitiesOnly yes I also have things set up so that if I leave the name alone, my work-owned machines default to the work key, and my personal machines default to my personal key.
Meaningless Anti-Virus Software Features Are Profitable
Tavis Ormandy, Google security expert, is getting press for criticizing Meaningless Antivirus Excellence Awards. This is a good opportunity to mention some thoughts I've had about anti-malware software. I believe that enterprise security defense software (anti-virus, anti-malware, host-based firewall, etc.) should have these qualities: Silent Updating: The software should update silently. It does not need to pop up a window to ask if the new antivirus blacklist should be downloaded and installed. That decision is made by system administrators centrally, not by the user. Hidden from view: The user should be able to determine that the software is activated, but it doesn't need an animated spinning status ball, nor popup windows to announce that updates were done.
Next on LISA Conversations: Caskey Dickson on "Why Your Manager LOVES Technical Debt and What to Do About It"
Our next guest will be Caskey Dickson. We'll be discussing his talk from LISA '15 titled Why Your Manager LOVES Technical Debt and What to Do About It. Watch live! We'll be recording the episode on Tuesday, March 29, 2016 at 3:30-4:30 p.m. Pacific Time. Particpate in the live Q&A by submitting your questions during the broadcast. Pre-registration is recommended. Register and/or watch via this link. Homework: Watch his talk ahead of time: Why Your Manager LOVES Technical Debt and What to Do About It Recorded at LISA '15 Video and Slides Watch live! LISA Conversations Episode #8 Co-hosts: Lee Damon and Thomas Limoncelli Guest: Caskey Dickson Will be recorded: Tuesday, March 29, 2016 at 3:30-4:30 p.m.
LISA Conversations Episode 7: Kris Buytaert on "DevOps: The past and future are here. It's just not evenly distributed (yet)"
Episode 7 of LISA Conversations is Kris Buytaert, who presented DevOps: The past and future are here. It's just not evenly distributed (yet) at LISA '11. Watch the Episode here: Usenix LISA Episode #7 with Kris Buytaert Co-hosts: Lee Damon and Thomas Limoncelli Guest: Kris Buytaert Recorded Tuesday, February 23, 2016 In this episode we discuss his talk: DevOps: The past and future are here. It's just not evenly distributed (yet) Recorded at LISA '11 Talk Description Video and Slides YouTube You won't want to miss this!
How to prepare for April Fools Day
Step 1. Buy this for your boss or coworkers. Step 2. Prepare for hijinks. #yourwelcome #dropthemic
Tom Reacts to Teens React to Windows 95
We're in the process of updating The Practice of System and Network Administration (read the drafts here) and I discovered an old section that was written with the assumption that DHCP was newish and readers would need encouragement to use it. Of course, I ripped it out and replaced it with something more modern. However, I couldn't help but include an explanation for new sysadmins what life was like before DHCP (see The Importance of DHCP). Which leads me to this video of teens reacting to Windows 95 (and associated article). The best quote is, "How do you get Internet without WiFi?"
How does Google manage Oncall?
"Login", the Usenix Newsletter, has an excellent article about how Google manages oncall. Authors Andrea Spadaccini and Kavita Guliani did an excellent job of providing an overview of how Google seeks to balance oncall time with non-oncall time so that engineers have time for actual engineering. While most of the article deals with how to prevent operations people from getting overloaded, they also raise the issue that operations underload is dangerous too. SREs get out of practice if they don't get paged enough. They describe games and simulations that SRE teams do to stay in practice. The article is available for free to Usenix members and newsletter subscribers, or for a nominal charge to everyone else.
Reminder: WebCast on how to "Fail Better": TODAY (Friday, Feb 26)
I'll be giving a talk "Fail Better: Radical Ideas from the Practice of Cloud Computing" as part of the ACM Learning Series at at 2pm EST on Friday, February 26, 2016. The webcast is free. Pre-registration is required. In this talk I explain 3 of the most important points from our newest book, The Practice of Cloud System Administration. The talk applies to everyone, whether or not you are "in the cloud". "See" you there!
Test post for people using RSS readers
We've made some changes to the website. If you read this blog via an RSS reader (or any "feed reader") posts should appear better now. In particular, the author of the post should show up better.
Next on LISA Conversations: Kris Buytaert on "DevOps: The past and future are here. It's just not evenly distributed (yet)"
Our next conversation will be with Kris Buytaert, who presented "DevOps: The past and future are here. It's just not evenly distributed (yet)" at LISA '11. Watch his talk beforehand, and then join us at 11:30 am PST/2:30 pm EST on Tuesday, February 23, 2016, at the Google Hangout On Air. Watch his talk from LISA '15: DevOps: The past and future are here. It's just not evenly distributed (yet) YouTube Slides/Audio/Video ...then watch us interview him live... Tuesday, February 23, 2016 at 11:30 am PST/2:30 pm EST (convert) Watch Live! ...or watch the recorded show shortly after!
WebCast on how to "Fail Better": Friday, Feb 26
I'll be giving a talk "Fail Better: Radical Ideas from the Practice of Cloud Computing" as part of the ACM Learning Series at at 2pm EST on Friday, February 26, 2016. Pre-registration is required. In this talk I explain 3 of the most important points from our newest book, The Practice of Cloud System Administration. The talk applies to everyone, whether or not you are "in the cloud". "See" you there!
How SysAdmins Devalue Themselves
I write a 3-times-a-year column in ACM Queue Magazine. This issue I cover 2 unrelated topics. " How Sysadmins Devalue Themselves" and "And how to track on-call coverage". Enjoy! Q: Dear Tom, How can I devalue my work? Lately I've felt like everyone appreciates me, and, in fact, I'm overpaid and underutilized. Could you help me devalue myself at work? A: Dear Reader, Absolutely! I know what a pain it is to lug home those big paychecks. It's so distracting to have people constantly patting you on the back. Ouch! Plus, popularity leads to dates with famous musicians and movie stars.
A feast of analogies
A few years ago a coworker noticed that all my analogies seemed to involve food. He asked if this was intentional. I explained to him that my analogies contain many unique layers, but if you pay attention you'll see a lot of repetition... like a lasagna. By the way... I've scheduled this blog post to appear on the morning of Wednesday, Feb 10. At that time I'll be getting gum surgery. As part of recovery I won't be able to bite into any food for 4-6 months. I'll have to chew with my back teeth only. Remember, folks, brushing and flossing is important.
ACM Interviews: Thomas Limoncelli
I'm excited to announce that I've been interviewed as part of the ACM Interviews series. Listen to the 1-hour interview or read the summary via this link ACM Interviews are part of the ACM Learning Center (click on Podcasts). Over the last 20+ years Stephen Ibaraki's interviews have included famous computer scientists and innovators like Vint Cerf, Eric Schmidt, Leslie Lamport, and more. (Complete list here.) Stephen is involved in many professional organizations, he frequently addresses the United Nations, and has received numerous honors including being the first and only recipient of the Computing Canada IT Leadership Lifetime Achievement Award. I was quite honored to be asked.
LISA Conversations Episode 6: Alice Goldfuss on Scalable Meatfrastructure (LISA15)
In this episode we talked with Alice Goldfuss about the changes you need to make when growing a DevOps or sysadmin team. Alice also talked about dealing with remote workers, her experience at film school, plus she shares insights about giving your first presentation at a conference. LISA Conversations Episode 6: https://www.youtube.com/watch?v=8bxOstI2r8A Topic: Alice Goldfuss discusses her LISA '15 talk: Scalable Meatfrastructure: Building Stable DevOps Teams You don't want to miss this! For the complete list of LISA Conversations, visit our homepage.
Watch us live today: LISA Conversations: Alice Goldfuss on Scalable Meatfrastructure
Today (Feb 2) at 3:30PM PST we'll be recording this month's episode of LISA Conversations. Our guest will be Alice Goldfuss. We'll be discussing her LISA '15 talk about growing a devops team: Scalable Meatfrastructure: Building Stable DevOps Teams Watch her talk from LISA '15... Scalable Meatfrastructure: Building Stable DevOps Teams ...then watch us interview her live... January 26, 2016 at 3:30 pm-4:30 pm PDT (convert) Watch Live! ...or watch the recorded show shortly after! You won't want to miss this! (NOTE: This recording was rescheduled; our usual time/date is the last Tuesday of the month.)
Watch us live today: LISA Conversations: Alice Goldfuss on Scalable Meatfrastructure
NOTE: Due to illness, today's LISA Conversations is postponed. More info soon. < !-- Today at 3:30PM PST we'll be recording this month's episode of LISA Conversations. Our guest will be Alice Goldfuss. We'll be discussing her LISA '15 talk about growing a devops team: Scalable Meatfrastructure: Building Stable DevOps Teams * Watch her talk from LISA '15... * [Scalable Meatfrastructure: Building Stable DevOps Teams](https://www.usenix.org/conference/lisa15/conference-program/presentation/goldfuss) * ...then watch us interview her live... * January 26, 2016 at 3:30 pm-4:30 pm PDT ([convert](http://www.timeanddate.com/worldclock/converted.html?iso=20160126T1530&p1=224&p2=179)) * [Watch Live!] (https://plus.google.com/events/c4k77vgipedd8e67gqi6nqtq1dc) * ...or [watch the recorded show](https://www.usenix.org/conference/lisa16/lisa-conversations) shortly after! You won't want to miss this!
SHA-1 Certs should cost $10,000
In my previous blog post, "SHA-1 Deprecation: Pro, Con, or Extend?" , I was a bit sarcastic about an anonymous company wanting to keep producing SHA-1 out of lazy greed rather than helping customers. Here's an update by Symantec about their latest actions. Basically, the proposal to extend SHA-1 certs was withdrawn because during the ballot debate, so many new attacks against SHA-1 were revealed that.... oh the embarrassment. So now companies can request SHA-1 certs as long as they expire on Dec 31, 2016. Luckily one good thing happened: non-legacy browsers are removing their trust for the SHA-1 root certs, which will make them more secure and will serve as a canary in the coalmine.
Reminder: Next week's LISA Conversations: Alice Goldfuss on Scalable Meatfrastructure
This weekend is a good time to watch the video we'll be discussing on Usenix LISA conversations. Our guest will be Alice Goldfuss. We'll be discussing her LISA '15 talk about growing a devops team: Scalable Meatfrastructure: Building Stable DevOps Teams Watch her talk from LISA '15... Scalable Meatfrastructure: Building Stable DevOps Teams ...then watch us interview her live... January 26, 2016 at 3:30 pm-4:30 pm PDT (convert) Watch Live! ...or watch the recorded show shortly after! You won't want to miss this!
Next on LISA Conversations: Alice Goldfuss on Scalable Meatfrastructure
On the next episode of LISA Conversations... Our guest will be Alice Goldfuss. We'll be discussing her LISA '15 talk about growing a devops team: Scalable Meatfrastructure: Building Stable DevOps Teams Watch her talk from LISA '15... Scalable Meatfrastructure: Building Stable DevOps Teams ...then watch us interview her live... January 26, 2016 at 3:30 pm-4:30 pm PDT (convert) Watch Live! ...or watch the recorded show shortly after! You won't want to miss this!
BNF meets Bowie
This is floating around teh interwebz and I normally don't post this kind of thing, but since this blog recently discussed the death of Peter Naur, and since David Bowie passed away recently, I thought this was appropriate. This song, Modern Love, was a big hit around the time that I was first getting interested in Bowie. At that time he'd already had more fame and success in the music industry than most could even hope for. As a result, I learned his music in a strange order. First his hits of the day, then going back to his back catalog and learning about his early career and music.
Tonight at BBLISA (Boston/Cambridge)
Just a quick reminder that I'll be the speaker tonight's BBLISA meeting (Wednesday, January 13, 2016 at 7pm). If you are in the Boston/Cambridge area, please stop by! My presentation is titled "Transactional System Administration Is Killing Us and Must be Stopped". This is the same talk I presented recently at LISA, which was very well received. It includes a preview of material from our upcoming 3rd edition of The Practice of System and Network Administration.
NYC SRE Tech talks: A new monthly series
Google NYC has announced a series in monthly tech talks for the Site Reliability Engineering/DevOps community in New York City! The first meeting is January 20th at their Chelsea NYC office and will include a number of short talks by speakers from Google, Dropbox, and StackOverflow.com. I'll be the speaker from StackOverflow. The event will be held on Wednesday, January 20 at Google's campus in Chelsea, at 75 Ninth Avenue. Doors open at 5:30pm, food will be served at 6pm, and talks start at 6:30pm and run until 8pm. RSVPs are required because this is NYC. More info and RSVP information is here at this link.
(Boston/Cambridge) See you this week at BBLISA!
I'll be the speaker at this week's BBLISA meeting (Wednesday, January 13, 2016 at 7pm). If you are in the Boston/Cambridge area, please stop by! My presentation is titled "Transactional System Administration Is Killing Us and Must be Stopped". This is the same talk I presented recently at LISA, which was very well received. It includes a preview of material from our upcoming 3rd edition of The Practice of System and Network Administration. For more information about the talk, directions to the meeting, and so, on, visit the BBLISA website at http://www.bblisa.org/calendar.html
SHA-1 Deprecation: Pro, Con, or Extend?
I read Ryan's article about why SHA-1 should be deprecated faster and why we should veto the proposed extensions. It is an excellent explanation of what's going on. I highly recommend it (and look forward to the complete series when he publishes it): https://medium.com/@sleevi_/legacy-verified-legacy-solutions-15eb688716e4#.pc35r37o1 I feel like the cert provider's reply should be this: Dear Ryan: Screw you. You obviously don't understand the business we are in. We are in the business of PRINTING RANDOM NUMBERS AND SELLING THEM FOR UNGODLY HUGE SUMS. You're naive proposal may help the world, but how does that help us profit? Here's an example, Ryan: 4 See?
(Boston/Cambridge) See you next week at BBLISA!
I'll be the speaker at next week's BBLISA meeting (Wednesday, January 13, 2016 at 7pm). If you are in the Boston/Cambridge area, please stop by! My presentation is titled "Transactional System Administration Is Killing Us and Must be Stopped". This is the same talk I presented recently at LISA, which was very well received. It includes a preview of material from our upcoming 3rd edition of The Practice of System and Network Administration. For more information about the talk, directions to the meeting, and so, on, visit the BBLISA website at http://www.bblisa.org/calendar.html
IPv6 celebrates its 20th birthday
I remember in the 1990s every vendor was saying, "whoa whoa whoa! You have to give us time to roll out silicon that will support this stuff!" and demanding 10 years before deployment. It takes a while to develop silicon, and years to get it into the field. Well, it has been twice your request. No f'ing excuses. IPv6 should be the default protocol on all network equipment. Hey FIOS. Hey Comcast. Hey Time Warner! You have no excuse either. And stop encouraging people to use NAT. That's soooo 1990s. Stateful inspection firewalls do not require NAT. IPv6 celebrates its 20th birthday by reaching 10 percent deployment https://t.co/JX4pS1VSjA— John W.
Peter Naur, RIP, (1928-2016)
Computer scientists Peter Naur has passed away. He is the "N" in "BNF". If you aren't sure what BNF is, you may recognize it as a diagram like this: or this: You can imagine how error prone it was to specify syntax of new languages and systems before this notation was adopted. Imagine explaining either of those diagrams by writing a paragraph in English. Now imagine dozens of people trying to implement the language based on this description and all coming up with slightly different variations, each slightly incompatible. That was the world. You can see BNF notation all over the place.
Best and Worst DevOps songs of 2015
Wait... you didn't know there are songs about DevOps? Hell to the yeah! Best DevOps Song of 2015: Uptown Funk (Mark Ronson ft. Bruno Mars) Uptown Funk is exemplary of good DevOps operations: It encourages being evidence-driven. An important principle of DevOps is that you should base decisions on evidence and data, not lore and intuition. Intuition is great but only gets you so far. With a tiny system is is possible for a single sysadmin to know enough about it to make good guesses. However modern systems are complex enough that we must collect data, analyze it, and base decisions on that data.
Ugh.
My credit union tells me their website will be down Saturday night for upgrades. This not only means that they don't have a good DevOps-style rapid release CI/CD system, but that they have no respect for their IT group who should not have been required to spend this week and the entire weekend planning for the upgrade. They should be spending this weekend at the movie theater watching the force awakens. This is disrespectful of their employees and shows a lack of good management. How could management expect people to focus on a critical upgrade this week? DevOps isn't just a software release methodology.
We can solve the terrorist encryption problem with this one simple solution!
*Lately there has been a renewed debate over the use of encrypted communication. Terrorists could be using encryption to hide their communication. Everyone knows this. The problem is that encryption is required for ecommerce and just about everything on the web. Should encryption be banned? regulated? controlled? Lately there have been a number of proposals, good and bad, for how to deal with this. Luckily I have a solution that solves all the problems! My solution: (which is obvious and solves all problems) My solution is quite simple: Every time a website asks you to create or change a password, it would send a copy to the government.
How can sysadmins devalue themselves?
The new edition of ACM Queue Magazine is out. My column (called "Everything Sysadmin") answers 1-2 questions per issue. This issue's questions are: Q: Dear Tom, How can I devalue my work? Lately I've felt like everyone appreciates me, and, in fact, I'm overpaid and underutilized. Could you help me devalue myself at work? ...and... Q: Dear Tom, We have a very simple on-call schedule, but all the substitutions needed during December make it quite complex. How should we organize it better? For example, our team has a week-long on-call schedule [Monday to Monday]. During November and December, however, there is a flurry of e-mail with people requesting to trade days to accommodate various family and holiday responsibilities.
NYC DevOps: Bridget Kromhout and Casey West - Sometimes you feel like a Docker...
This month's NYCDevOps meeting (hosted at the StackOverflow.com HQ) has special guest speakers Bridget Kromhout and Casey West talking about running Docker images in Cloud Foundry's Elastic Runtime and orchestrating containerized workloads on Lattice. Date: Tuesday, December 15, 2015 Time: 6:30 PM Place: The Stack Overflow HQ (near Wall St.) You must RSVP and bring an ID to get into the building. You should join me at this Meetup. Check it out and RSVP! http://meetu.ps/2QNDCg
New issue of acmQueue is out!
I write a column in ACM Queue magazine called "Everything Sysadmin" (guess where I got the idea for the name?). It appears 3 times a year. The new issue is out and contains a column that answers 2 questions: one is "How can I devalue my work?" and the other is about scheduling substitutions for oncall schedules. Queue is free to ACM members (use your ACM account username/password). You can purchase a 1-year subscription for $19.99 or buy a single issue for $6.99. To read the issue online or via the Queue App (iPhone and Android), go here: http://queue.acm.org/app/landing.cfm
7 signs you're doing devops wrong
Adam Bertram wrote an excellent piece in InfoWorld: 7 signs you're doing devops wrong
Ever run a Novell network?
If you ran a Novell network, especially in the late 80s or early 90s, I hope you watched The Late Show with Stephen Colbert last night when he interviewed Steve Carell and talked about their brief work for Novell.
Cascadia IT Conference CFP Reminder
Call for Participation is open at http://casitconf.org/casitconf16/cfp/ The conference will be in Seattle, WA, on March 11-12, 2106. Submit your proposal by December 25th. If you've never given a presentation at a conference before, consider submitting to a regional conference like Cascadia. It is less intimidating and the audience is very friendly!
Tom speaking at BackBayLISA in January (Boston)
I'll be giving a presentation called "Transactional System Administration Is Killing Us and Must be Stopped" at the January 2016 meeting of BackBay LISA (BBLISA). This is the same talk I presented recently at LISA, which was very well received. It includes a preview of material from our upcoming 3rd edition of The Practice of System and Network Administration. For more information about the talk, directions to the meeting, and so, on, visit the BBLISA website at http://www.bblisa.org/calendar.html
Preview 8 chapters from the next edition of TPOSANA
The 3rd edition of The Practice of System and Network Administration won't be out for another year. However, we've released a set of 8 chapters on SafariBooksOnline.com (SBO). We've taken the previous edition's chapter on "workstations" and expanded it to be an 8-chapter segment on managing a fleet of workstations (laptops and desktops). You'll find we've done something similar for many of the chapters that we're rewriting for the new edition. Part II: Workstation Fleet Management Chapter 4: Workstation Architecture Chapter 5: Workstation Hardware Strategies Chapter 6: Workstation Software Lifecycle Chapter 7: OS Installation Strategies Chapter 8: Workstation Service Definition Chapter 9: Workstation Fleet Logistics Chapter 10: Workstation Standardization Chapter 11: Onboarding New Employees Part III is about Servers.
Why I don't care that Dell installs Rogue Certificates On Laptops
In recent weeks Dell has been found to have installed rogue certificates on laptops they sell. Not once, but twice. The security ramifications of this are grim. Such a laptop can have its SSL-encrypted connections sniffed quite easily. Dell has responded by providing uninstall instructions and an application that will remove the cert. They've apologized and that's fine... everyone makes mistakes, don't let it happen again.
We forget how big "big" is
Talk with any data-scientist and they'll rant about how they hate the phrase "big data". Odds are they'll mention a story like the following: My employer came to me and said we want to do some 'big data' work, so we're hiring a consultant to build a Hadoop cluster. So I asked, "How much data do you have?" and he replied, "Oh, we never really measured. But it's big. Really big! BIIIIG!! Of course I did some back of the envelope calculations and replied, "You don't need Hadoop. We can fit that in RAM if you buy a big enough Dell."
What JJ Abrams just revealed about Star Wars
Last night (Saturday, Nov 21) I attended a fundraiser for the Montclair Film Festival where (I kid you not) for 90 minutes we watched Stephen Colbert interview J.J. Abrams. What I learned: He finished mixing The Force Awakens earlier that day. 2:30am California time. He then spent all day traveling to Newark, New Jersey for the event. After working on it for so long, he's sooooo ready to get it in the theater. " The truth is working on this movie for nearly three years, it has been like living with the greatest roommate in history for too long. It's time for him to get his own place.
Christine's keynote at OpsCon Milano 2015
Christine Hogan gave the keynote presentation at OpsCon Milano 2015 today. Her talk was titled "Learn to Fail Better" and highlighted cultural and technical points from our new book, The Practice of Cloud System Administration. OpsCon had an artist live drawing a summary of the talk, which you can see here: Congrats to Christine on her first conference keynote!
BYOBook Signing at LISA '15
I hadn't planned on doing a book signing at LISA this year but a number of people have asked, so I've set one up. You'll have to bring your own copy as I won't have copies to sell or give away. What: Book signing with Tom Limoncelli Where: The Atrium When: Friday, Nov 13 at 1:30-2pm What about e-books? I have stickers that I will autograph. Where you stick it is up to you. Will you be selling or giving away books? Sadly not this year. That said, feel free to bring books by other authors. I'll sign anything. Your books are too heavy to bring in my luggage.
Automation Should Be Like Iron Man, Not Ultron
Q: Dear Tom: A few years ago we automated a major process in our system administration team. Now the system is impossible to debug. Nobody remembers the old manual process and the automation is beyond what any of us can understand. We feel like we've painted ourselves into a corner. Is all operations automation doomed to be this way? Read my answer in ACM Queue magazine. [Queue Magazine is for computer science practitioners. They asked me to write a column on operations/system administration that would suit that audience. This is the first one. You can read it free online occasionally. Subscribers never miss an issue.
Oursourcing makes sense if you think "software" is a fad
[This piece gets kind of dark. You've been warned.] At the recent DOES15 conference (which was a great conference) many of success stories included the admission that outsourcing had been a big mistake. In some cases outsourcing had nearly sunk the company. What saved them? DevOps, in-sourcing, and vertical integration. If you aren't familiar with the term "vertical integration" it is the MBA term for "if you want something done right, do it yourself." The reason outsourcing had been such a disaster was not the skill of the outsourcing companies or the people. It was the fact that if you don't own your process, you can't control the quality.
When explaining to beginners, use simple examples.
I've always felt that most geeks give examples (to beginners) that are too complex. I believe this is an attempt to be complete. However, beginner examples should be so simple even if you feel like you are committing lies of omission. A recently Slashdot article, Revisiting Why Johnny Can't Code: Have We "Made the Print Too Small"? mentioned that often the examples we give are too complex for the beginners we intend them for. They compare the starting example from Mark Zuckerberg's what-is-coding video to a simple BASIC example. They make a comparison to the book How to Teach Your Baby to Read, the authors explain, "It is safe to say that in particular very young children can read, provided that, in the beginning, you make the print very big."
Watch LISA Conversations Episode 4: Sabrina Farmer!
The Google Hangout with Sabrina Farmer was so amazing we decided to go over-time. If you missed it, watch the video https://www.youtube.com/watch?v=RCgvBZszuwI. We discussed her talk from WiAC '12 titled: Overcoming My Biggest Roadblock, Myself, plus what's it like to work at Google, her recent promotion to Engineering Director, career management tips for women, and much much more! You don't want to miss this episode! (For more info and past episodes visit the Usenix LISA Conversations Homepage.)
Watch us live today: LISA Conversations Episode 4: Sabrina Farmer
Today (Oct 27, 2015) we'll be recording Episode 4 of LISA Conversations. Join the Google Hangout and submit questions live. Our guest will be Sabrina Farmer, who is a SRE manager at Google. We'll be discussing her amazing talk "Overcoming My Biggest Roadblock, Myself" from the 2012 USENIX Women in Advanced Computing Summit (WiAC '12). Watch her talk beforehand, and then join us at 3:30 pm PDT/6:30 pm EDT on Tuesday, October 27, 2015, at the Google Hangout On Air. The video from Sabrina's talk can be found at https://www.usenix.org/conference/wiac12/overcoming-my-biggest-roadblock-myselfÿ Join the hangout: https://plus.google.com/b/108588319090208187909/events/ctqdskbuhuh4fnt1o0f49m5o2ss The talk was brought to my attention when someone described it was being "the talk that brought down the house at WiAC '12".
New column in Queue Magazine: Everything Sysadmin
I've started a column in ACM Queue magazine called "Everything Sysadmin" (guess where I got the idea for the name?). It will appear 3 times a year. The first column is titled, "Automation Should Be Like Iron Man, Not Ultron". Queue is free to ACM members (use your ACM account username/password). You can purchase a 1-year subscription for $19.99 or buy a single issue for $6.99. To read the issue online or via the Queue App (iPhone and Android), go here: http://queue.acm.org/app/landing.cfm
NEW: USENIX Journal of Education in System Administration
If you teach system administration I highly recommend you take a look at USENIX's newest journal: Journal of Education in System Administration (JESA) The journal can be read (for free) online: https://www.usenix.org/jesa/0101. I was honored to be asked to write a piece for the inaugural issue. You can read it online here.
Get ready for LISA Conversations Episode 4: Sabrina Farmer
The next episode of the LISA Conversations video podcast will be a discussion with Sabrina Farmer. We'll be discussing her amazing talk "Overcoming My Biggest Roadblock, Myself" from the 2012 USENIX Women in Advanced Computing Summit (WiAC '12). Watch her talk beforehand, and then join us at 3:30 pm PDT/6:30 pm EDT on Tuesday, October 27, 2015, at the Google Hangout On Air. The video from Sabrina's talk can be found at https://www.usenix.org/conference/wiac12/overcoming-my-biggest-roadblock-myselfÿ The talk was brought to my attention when someone described it was being "the talk that brought down the house at WiAC '12". I watched it and was blown away by her powerful story of self-discovery.
Big discount, 24 hours only!
The Practice of Cloud System Administration is the InformIT "eBook Deal of the Day". You can get it with more than a 40% discount: $24.99. http://informit.com/deals Offer expires October 19th at 11:59 PST.
Usenix LISA: Early Bird Pricing ends Oct 15!
This year LISA is in Washington D.C., from Nov 8-13. If you are on the east-coast, this is a good opportunity to attend the premiere system administration conference. Register now. This year's schedule is packed with amazing talks. I'd like to point out... " Go for Sysadmins" from Chris "Mac" McEniry, Sony Network Entertainment "Neighborly Nagios" from David Josephsen, Librato "systemd, the Next-Generation Linux System Manager" from Alison Chaiken, Mentor Graphics "Software Defined Networking: Principles and Practice" from Nick Feamster, Princeton University "How to Not Get Paged: Managing On-call to Reduce Outages" from Thomas A. Limoncelli, Stack Overflow Register now.
Beyond Blame: Learning From Failure and Success
You're gonna want this book. Pre-order it now. http://bit.ly/beyondblame (Pre-orders are paper right now; it should be available on Kindle soon. Official release date is Oct 25) This is the best book I've ever read about Postmortems and creating a Blameless operations culture. Tom
Can't Make it to PuppetConf? Watch It Live!
I had an interesting conversation with Ryan Coleman, product manager at Puppet Labs. He gave me a preview of some of the things being announced soon and highlighted at PuppetConf next week. If you can't attend, you can livestream the conference for free. In particular, the keynote is on Thurs, Oct 8th at 9am PT (noon ET). How to livestream the entire conference is here: http://info.puppetlabs.com/PuppetConf2015LiveStream.html It isn't too late to grab a ticket and attend in-person! Enjoy!
Interview with Tom Limoncelli on InfoQ
infoq.com interviewed me for their website. We talk about DevOps, automation, and more. Interestingly enough, the person interviewing me was Barry Burd, a professor of mine 20 years ago. View it here: http://www.infoq.com/interviews/limoncelli-devops Enjoy!
Homework for the weekend: Making Push on Green a Reality
Next week's "LISA Conversations" podcast will be a discussion about the LISA '14 talk "Making Push on Green a Reality". We'll be interviewing the presenter, Daniel V. Klein, about the talk and what he has to say about it nearly a year later. " Push On Green" means automatically pushing code to production with no human gates. If all the tests pass, the new code is pushed to production automatically. This enables Google to push code more frequently and with higher confidence than (for example) monthly or weekly code pushes. Watch the video from LISA '14 and get ready to watch us record the podcast live on September 29, 2015, at 3:30pm PDT.
TV Alerts
Monday, Sept 21: Big Bang Theory (new season!) Tuesday, Sept 21: The Muppets (new show! OMG! OMG!)ÿ YOU'RE WELCOME!!! (the links point to Tivo's page to set up 1-step recording for that series)
Next LISA Conversations guest: Daniel V. Klein
We'll be recording Episode 3 of Usenix LISA Conversations on Tuesday, September 29, 2015. Our next conversation will be with Daniel V. Klein who presented "Making Push on Green a Reality" at LISA14. Watch his talk beforehand, and then join us at 3:30 pm PDT/6:30 pm EDT on Tuesday, September 29, 2015, at the Google Hangout On Air. We'll discuss the talk and what he's been doing since. If you miss the live session, you can view the recording on the USENIX YouTube channel. This month's hosts will be Lee Damon and Tom Limoncelli (me!) .
Mr. DeMille, I'm ready for my close-up
After listening to Jon Taffer's interview on The Nerdist Podcast about "Bar Rescue", I'm convinced that I should do a TV show called "IT Rescue" where we visit an IT department that is failing hard and set them up for success. Hollywood... call me!
New reviews of The Practice of Cloud System Administration
I hadn't realized that Google Play permits book reviews. Strata, Christine and I are very please to read these: Ivan Dimitrov wrote: Simplely the best book for system administrators and their managers. Packed with great stuff from first page to the last. If you have to read one chapter - it's the Appendix A :) Adrian Colley wrote: This book covers about 85% of what any programmer needs to know to be a fully competent Google Site Reliability Engineer. It's written like a textbook for a training course, but it serves well as a reference text.
Reorganizing a wiki or documentation system
Someone wrote to me recently asking for advice about how to re-organize his company's documentation stash. Basically they had a directory on a fileserver that had become a free-for-all, collect everything, "cosmic abyss" (his words). Tons of documents. No organizations. Most of it out-of-date or of unknown quality. Did I have any advice that didn't involve complex document control philosophy and best practices? Sure! Here's a strategy I've used at 2 different organizations. It is very simple and low-overhead: Find a way to mark all old docs as "old", then find a way to review docs and mark them as "reviewed".
CfP: USENIX Container Management Summit (UCMS '15)
The 2015 USENIX Container Management Summit (UCMS '15) will take place November 9, 2015, during LISA15 in Washington, D.C. Important Dates Submissions due: September 5, 2015, 11:59 p.m. PDT Notification to participants: September 19, 2015 Program announced: Late September 2015 (quoting the press release): UCMS '15 is looking for relevant and engaging speakers and workshop facilitators for our event on November 9, 2015, in Washington, D.C. UCMS brings together people from all areas of containerization--system administrators, developers, managers, and others--to identify and help the community learn how to effectively use containers. Submissions Proposals may be 45- or 90-minute formal presentations, panel discussions, or open workshops.
CfP: USENIX Release Engineering Summit (URES '15)
Hey all you devops, CI/CD/CD people! Hey all you packagers, launchers, and shippers. Hey all your containers mavins and site reliability engineers! Submissions due: September 4, 2015 - 11:59 pm (quoting the press release): At the third USENIX Release Engineering Summit (URES '15), members of the release engineering community will come together to advance the state of release engineering, discuss its problems and solutions, and provide a forum for communication for members of this quickly growing field. We are excited that this year LISA attendees will be able to drop in on talks so we expect a large audience.
FreeBSD Journal Reviews TPOSANA
Greg Lehey wrote an excellent review of The Practice of System and Network Administration in the new issue of The FreeBSD journal. Even though the book isn't FreeBSD-specific, I'm glad FJ was drawn to reviewing the book. For more about the FreeBSD Journal, including how to subscribe or purchase single issues, visit their website: https://www.freebsdfoundation.org/journal I'm a subscribed to the journal and I highly recommend it. The articles are top notch. Even if you don't use FreeBSD, the articles are a great way to learn about advanced technology and keep up with the industry.
Advice for people that teach system administration?
I've been asked to write an article that will be read by people that teach system administration (and people that research how to best teach system administration). Sadly I'm having writers block. I have too much to say, so I don't know where to start, or how to narrow it down to 1-2 main points. My solution is to crowdsource this a bit. So... What would you tell professors that are studying how to best teach system administration? Or, more importantly: If there is one thing such teachers/researchers should be told, what would it be? Tell in this Google Form (or in the commments, but I'd prefer the form) Thanks!
Usenix LISA Conversations: Episode 2 is up!
Ben Rockwood was the guest of Episode 2. He gave a great talk at LISA'14 and we invited him to discuss it, the reaction it got, and what's new in his thinking since. Ben's a funny guy and had a lot of insightful new things to say. I wasn't able to attend, so Lee Damon cohosted with a great substitute David N. Blank-Edelman. I'll be back next month when Dan Klein will be our guest. More info about ULC including links to past and present videos is available at the homepage: https://www.usenix.org/conference/lisa15/lisa-conversations
LISA Schedule published!
This year's committee has done a bang-up job! See the entire schedule! Register today!
Next week's LISA Conversations guest: Ben Rockwood
We'll be recording Episode 2 of Usenix LISA Conversations on Tuesday (6 days from today). Our next conversation will be with Ben Rockwood who presented "I Am SysAdmin (And So Can You!)" at LISA14. Watch his talk beforehand, and then join us at 3:30 pm PDT/6:30 pm EDT on Tuesday, August 25, 2015, at the Google Hangout On Air. If you miss the live session, you can view the recording on the USENIX YouTube channel. This month's host will be Lee Damon and David Blank-Edelman (substituting for me; I'll return next month).
Why are logs called logs?
Q: Why are logs called logs? A: The name refers to the fact that they are like entries in ship's logbook. Q: Why is it called a logbook? A: Because of an amazing bit of nautical history uncovered by Jeff Reffell of the Designcult blog: The secret origin of "log in" Totally worth the read.
Ethical Obligations in Internet Operations - Questionnaire
[Guest post by Jan Schaumann) Actually, it's about Ethics in Internet Operations. No, seriously, it actually is. As mentioned earlier, somehow Velocity NY accepted my talk on 'Ethical Obligations in Internet Operations'. In order to help me better understand our profession(s) and to prepare a better talk, I've put together a short, anonymous questionnaire for everybody involved in "Internet Operations". " Internet Operations" is the term I use here to combine the various ill- or loosely defined job descriptions relating to the general "tech industry" beyond the job of "Programmer". If you write infrastructure code, you are working in "Internet Operations".
Episode 1 of LISA Conversations
Episode 1 of "LISA Conversations" is up. In our first episode, we spoke with Todd Underwood about his LISA'13 talk about a Post-Ops World. What's next? The next episode will be with Ben Rockwood, where we'll discuss his LISA'14 talk, "I Am SysAdmin (And So Can You!)". You can join the recording live via Google Hangouts On Air on August 25, 2015 at 3:30pm. For more information about the LISA Conversations series, visit the web page: https://www.usenix.org/conference/lisa15/lisa-conversations
Happy SysAdmin Day! (July 31)
I hope you are fully appreciated today and every day. For more info about SysAdmin Day, visit http://sysadminday.com/ If you are in the NYC area, please come to SysDrink's SysAd Day event tonight at 6pm at The Gingerman in mid-town Manhatten. There will be an open bar. This year's event is sponsored by Digital Ocean.
LISA Conversations premieres on Tuesday!
Yes, I've started a video podcast that has a homework assignment built-in. Watch a famous talk from a past LISA conference (that's the homework) then watch Tom and Lee interview the speaker. What's new since the talk? Were their predictions validated? Come find out! Watch it live or catch the recorded version later. The first episode will be recorded live Tuesday July 28, 2015 at 1:30pm PDT.
Save on The Practice of Cloud System Administration
Pearson / InformIT.com is running a promotion through August 11th on many open-source related books, including Volume 2, The Practice of Cloud System Administration. Use discount code OPEN2015 during checkout and received 35% off any one book, or 45% off 2 or more books. See the website for details.
SysAdmin Appreciation Day in New York City
If you are in NYC, there is a SysAdmin Appreciation day event at The Gingerman, 11 E 36th Ave, New York City, NY, on Friday, July 31, 2015, 6:00 PM. This event usually has a big turn-out and is a great way to meet and network with local admins. RSVP here: http://www.meetup.com/Sysdrink/events/223896825/ Thanks to Digital Ocean for sponsoring this event, and Justin, Jay, Nathan and the other organizers for putting this together every year. Hope to see you there!
Schyntax: A DSL for specifying recurring events
There are many ways to specify scheduled items. Cron has 10 8,20 * 8 1-5 and iCalendar has RRULE and Roaring Penguin brings us REMIND. There's a new cross-platform DSL called Schyntax, created by my Stack Overflow coworker Bret Copeland. The goal of Schyntax is to be human readable, easy to pick up, and intuitive. For example, to specify every hour from 900 UTC until 1700 UTC, one writes hours(9..17) What if you want to run every five minutes during the week, and every half hour on weekends? Group the sub-conditions in curly braces: { days(mon..fri) min(*%5) } { days(sat..sun) min(*%30) } It is case-insensitive, whitespace-insensitive, and always UTC.
Usenix Container Management Summit Announced!
The Call for Participation for the new 2015 USENIX Container Management Summit is now online.UCMS '15 will take place November 9, 2015, during LISA15 in Washington, D.C.ÿ
Introducing: LISA Conversations
Step 1: Watch a video from a past Usenix LISA conference. Step 2: Join the Hangout On Air and watch Lee Damon and Tom Limoncelli interview the speaker. Send Q&A during the show. Step 3: Watch and enjoy! Our first 4 are scheduled for the last Tuesday of July/Aug/Sept/Oct. The first one is July 28, 2015 at 1:30pm PDT. We'll be interviewing Todd Underwood about his LISA 2013 talk Post-Ops, A Non-Surgical tale of Software, Fragility, and Reliability. Watch the presentation head of time then join the the Google Hangout On Air. (Want a reminder? RSVP for the event)
Google's "Labs" features are DevOps Third Way
Someone on Quora recently asked, Why did Google include the 'undo send' feature on Gmail?. They felt that adding the 30-second delay to email delivery was inefficient. However rather than answering the direct question, I explained the deeper issue. My (slightly edited) answer is below. NOTE: While I previously worked at Google, I was never part of the Gmail team, nor do I even know any of their developers or the product manager(s). What I wrote here is true for any software company. Why did Google include this feature? Because the "Gmail Labs" system permits developers to override the decisions of product managers.
TOML vs. JSON
[This is still only draft quality but I think it is worth publishing at this point.] Internally at Stack Exchange, Inc. we've been debating the value of certain file formats: YAML, JSON, INI and the new TOML format just to name a few. [If you are unfamiliar with TOML, it is Tom's Obvious, Minimal Language. " Tom", in this case, is Tom Preston-Werner, founder and former CEO of GitHub. The file format is still not reached version 1.0 and is still changing. However I do like it a lot. Also, the name of the format IS MY FREAKIN' NAME which is totally awesome.
Marriage Equality becomes the law of the land in the US
I literally never thought I'd see this day arrive. In 1991/1992 I was involved in passing the LGB anti-discrimination law in New Jersey. When it passed in January 1992, I remember a reporter quoting one of our leaders that marriage was next. At the time I thought Marriage Equality would be an impossible dream, something that wouldn't happen in my lifetime. Well, less than quarter-century later, it has finally happened. In the last few years more than 50% of the states approved marriage equality and soon it became a foregone conclusion. States are the "laboratory of democracy" and with 26 states (IIRC) having marriage equality, its about time to declare that the experiment is a success.
If you change the file format, change the file name
Recently we were having the most difficult time planning what should have been a simple upgrade. There is a service we use to collect monitoring information (scollector, part of Bosun). We were making a big change to the code, and the configuration file format was also changing. The new configuration file format was incompatible with the old format. We were concerned with a potential Catch-22 situation. Which do we upgrade first, the binary or the configuration file? If we put the new RPM in our Yum repo, machines that upgrade to this package will not be able to read their configuration file and that's bad.
My proposals have been accepted at Usenix LISA '15!
My talk and 2 tutorial proposals have been accepted at Usenix LISA LISA Conference! Talk: Transactional system administration is killing us and must be stopped Tutorials: How To Not Get Paged: Managing Oncall to Reduce Outages Introduction to Time Management for busy Devs and Ops The schedule isn't up yet at http://www.usenix.org/lisa15 but Usenix is encouraging speakers to post to social media early this year. See you in Washington DC Nov 8-13, 2015! P.S. You can follow LISA on various social networks: Facebook: https://www.usenix.org/facebook/lisa YouTube: http://www.usenix.org/youtube Google+: https://www.usenix.org/gplus/lisa LinkedIn: http://www.usenix.org/linkedin Instagram: http://www.usenix.org/instagram Twitter: https://www.usenix.org/twitter/lisa
Thanks, QCon New York!
I had a great time at QCon New York last week. It was my first time there and my first time speaking too. The audience was engaged and had great questions. I did a book-signing at the Pearson booth and it was fun meeting readers (and future readers) of our books. Videos of all talks will be available soon. For now you can view the slides.
See you at QCon NYC next week!
I'll be speaking at QCon in their "Architecting for Failure" track. My talk is titled "Fail Better: Radical Ideas from the Practice of Cloud Computing". This conference has a vendor area. I'll be at the Pearson booth signing books on Thursday from 3:50-4:30. Stop by even if you just want to chat! Registration is still open. More about the conference at qconnewyork.com. Hope to see you there!
Design Patterns for being creepy: Playing The Odds
Recently a friend told me this story. She had given a presentation at a conference and soon after started receiving messages from a guy that wanted to talk more about the topic. He was very insistent that she was the only person that would understand his situation. Not wanting to be rude, she offered they continue in email but he wanted to meet in person. His requests became more and more demanding over time. It became obvious that he wasn't looking for mentoring or advice. He wanted a date. She had no interest in that. Unsure what to do, she asked a few other female attendees for advice.
Note to Boeing 787 Dreamliner owners: Reboot every 248 days
If you own a Boeing 787 Dreamliner, and I'm sure many of our readers do, you should reboot it every 248 days. In fact, more frequently than that because at about the 248-day mark, the power system will fail due to a software bug. Considering that 248 days is about 2^31 * 100, it is pretty reasonable to assume there is a timer with 100 microsecond resolution timer held in a 32-bit unsigned int. It would overflow every 248 days. " Hell yeah, I did it! I saved 4 bytes every time we store a timestamp. Screw you. It's awesome.
Two NYC-area Puppet-related events
There are still tickets available for Puppet Camp New York 2015, Friday, May 15, 2015. It is a day of presentations useful for folks from beginner to advanced. I'll be one of the speakers. In my talk I'll be demoing some of the things we do at StackOverflow that make using Puppet safer: Git, CI, Vagrant, and using Puppet Environments. A lot of people at these events aren't using Puppet yet, so the material is usually very introductory. I'll be doing a rehearsal of my talk this Thursday (May 7th) at the NJ LOPSA chapter meeting. They meet in Lawrenceville, NJ (near Princeton).
Backupify: a case study in incompetent marketing
A year or so ago I tried Backupify and then disabled it shortly after. Ever since I've received many emails from them, mostly warning that my disabled account was... umm.. disabled. Thanks for the reminder. On April 17th I complained to them via their Zendesk system and their support agent Adam Deligianis explained the issue and cancelled the account. " I have now processed the cancellation of your account so you will not receive any more emails from us." (link) This week I got more spam from them. This was was pseudo-personal message from Rob May, a SVP of Business Development, that states, "You are receiving this email regardless of your unsubscribe settings because it includes important information that impacts availability of your Backupify account."
A tweet about Git
Best Tweet I've seen in months: git people trying to use hg: "WHY WON'T IT LET ME DO THIS" hg people trying to use git: "WHY DID IT LET ME DO THAT"— @A eevee A@ (@eevee) April 14, 2015 That just about sums it up.
I'll be at NYLUG next week, talking about the new book!
I'll be giving my talk "Radical ideas from The Practice of Cloud System Administration" talk at the NYLUG meeting next week, Wednesday, April 15, 2015 at 6:30 PM. If you haven't seen this talk at DevOpsNYC or other meetups, I hope to see you there!
Usenix LISA has changed a lot in the last 5-10 years
I received an interesting email recently: Did the submissions process for LISA change in recent years? I recall going to submit a talk a couple years ago and being really put off by the requirements for talks to be accompanied by a long paper, and be completely original and not previously presented elsewhere. Now it seems more in line with other industry conferences. Yes, LISA is very different than it was years ago. If you haven't attended LISA in a while, you may not realize how different it is! The conference used to be focused on papers with a few select "invited talks".
The Python Time Travel Debugger
Last year I open sourced my enhancement to Python PDB which lets you rewind time. Sadly I announced it on April Fools Day. Oddly enough, even though I open sourced it, people thought the screencast was a hoax. It isn't at all. It really works. Check out last year's post: http://everythingsysadmin.com/2014/04/time-travel-pdb.html
Usenix LISA 2015 Call For Participation (3 weeks left!)
Only 3 weeks left to submit talk and paper proposals for LISA 2015. This year's conference is in Washington D.C. on November 8-13. This might be a good weekend to spend time writing your first draft! https://www.usenix.org/conference/lisa15/call-for-participation Don't be afraid to submit proposals early. Unsure of your topic? Contact the chairs and bounce ideas off of them.
The State of DevOps Report
Where does it come from? Have you read the 2014 State of DevOps report? The analysis is done by some of the world's best IT researchers and statisticians. Be included in the 2015 edition! A lot of the data used to create the report comes from the annual survey done by Puppet Labs. I encourage everyone to take 15 minutes to complete this survey. It is important that your voice and experience is represented in next year's report. Take the survey But I'm not important enough! Yes you are. If you think "I'm not DevOps enough" or "I'm not important enough" then it is even more important that you fill out the survey.
2015 DevOps Survey
Have you taken the 2015 DevOps survey? The data from this survey influences many industry executives and helps push them towards better IT processes (and removing the insanity we find in IT today). You definitely want your voice represented. It takes only 15 minutes. Take the 2015 DevOps Survey Now
See you tonight at LSPE (Sunnyvale, CA)
See you at 6pm! The meeting is at Yahoo! URL's Cafeteria, 701 1st Ave, Sunnyvale, CA. Please RSVP. http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/221111762/
Just announced: I'll be speaking at the LSPE (Sunnyvale) meeting on Tuesday
I'll be talking about our new book, The Practice of Cloud System Administration, at the SF/Bay Area Large-Scale Production Engineering, which meets at Yahoo! URL's Cafe in Sunnyvale, CA on Tue. Mar 17 at 6:00PM. More info on their MeetUp page. Hope to see you there!
Hiring a network engineer for SRE team (NYC only)
Stack Exchange, Inc. is looking to hire a sysadmin/network admin/SRE/DevOps engineer that will focus on network-related projects. The position will work out of the NYC office, so you must be in NYC or be willing to relocate. If 3 or more of these project sound like fun to you, contact us! Automate Cisco LAN port configuration via Puppet Make our site-to-site VPN more reliable Tune NIC parameters for maximum performance / lowest latency Lead the network design of our global datacenter network deployment strategy Wrangle our BGP configurations for ease of updating and security Establish operational procedures for when ISPs report they can't reach us Sounds interesting?
Balitmore area folks: Come see me this Wednesday!
As previously blogged, if you are in the Baltimore area, check out the Baltimore LOPSA chapter meeting ("Crabby Admins") on Wednesday, March 4th when I'll be talking about my new book, The Practice of Cloud System Administration. Even if you have zero interest in "the cloud", I assure you this talk will be relevant to you. More info here. (The meetings are at the office of OmniTI, in Fulton, MD)
Blackbox now available via "MacPorts"
My open source project BlackBox is now available in the MacPorts collection. If you use MacPorts, simply type "sudo port install vcs_blackbox". There was already a package called "blackbox" so I had to call it something else. Blackbox is a set of bash scripts that let you safely store secrets in a VCS repo (i.e. Git, Mercurial, or Subversion) using Gnu Privacy Guard (GPG). For more info, visit the homepage: https://github.com/StackExchange/blackbox I'm looking for volunteers to maintain packages for Brew, Debian, and other package formats. If you are looking to learn how to make packages, this is a good starter project and will help people keep their files secure.
Usenix SREcon15: Now with more con!
Usenix has announced the schedule for the second SREcon and the big surprise is that it is now 2 days long. The previous SREcon was a single day. I wasn't able to attend last year's conference but I read numerous conference reports that were all enthusiastic about the presentations (you can see them online... I highly recommend the keynote). I'm excited to also announce that my talk proposal was accepted. It is a case study of our experiences adopting SRE techniques at StackOverflow.com/StackExchange.com. The full description is here. I've heard the hotel is nearly full (or full), so register fast and book your room faster.
Just announced: I'll be speaking at the BayLISA (Sunnyvale) meeting in March.
I don't get to California often so I'm excited to announce that I'll be the speaker at the March meeting of BayLISA. For more info check out their MeetUp page: http://www.meetup.com/BayLISA/events/219854117/ I'll be talking about our new book, The Practice of Cloud System Administration.
Bill Nye to Headline NECSS 2015
[This is not directly about system administration but it is of interest to many system administrators.] The Northeast Conference on Science and Skepticism (NECSS) announced that world-renowned science educator Bill Nye will headline NECSS 2015. In addition to giving the conference keynote address on Saturday afternoon, he will be the special guest star of Friday night's SGU Skeptical Extravaganza (a special show open to both conference attendees and the general public) and sign copies of his latest book, Undeniable: Evolution and the Science of Creation. Bill will join the existing speaker lineup, which is quite impressive. NECSS is a four-day celebration of science and critical thinking held each year in New York City.
LOPSA-NJ March meeting: Elastic Search
My co-worker Peter Grace will be the speaker at the Thursday, March 5, 2015 LOPSA-NJ meeting. His topic will be: "Systems Log Aggregation using ElasticSearch/LogStash/Kibana" For more info: http://www.meetup.com/LOPSA-NJ/events/220599343/ The meetings are near Princeton, in lovely Lawrenceville, NJ. Don't forget to RSVP!
New tutorial at Cascadia IT Conference, March 13, Seattle
Registration is open for the Cascadia IT Conference 2015 in Seattle, WA on March 13-14, 2015. Cascadia is a regional conference, but people travel from all over to attend. Why? Because it is worth it. I'll be teaching 2 tutorials on Friday and giving a talk on Saturday morning. The tutorials are: Time Management for Busy Devs and Ops How To Not Get Paged: Managing Oncall to Reduce Outages On Saturday morning I'll be giving a talk called "Live Upgrades on Running Systems: 8 Ways to Upgrade a Running Service with Zero Downtime". This will be a condensed version of what I taught at LISA 2014.
Bucks County DevOps Meetup this Wednesday!
I'll be speaking at the Bucks County DevOps meetup this Wednesday. If you are in the New Hope, PA area, please don't miss this! I don't get out to Pennsylvania very often! http://www.meetup.com/Bucks-County-DevOps/events/206523752/ Tom
17 minute documentary about Grace Hopper
The Queen of Code is a 17-minute documentary about Grace Hopper. It just came out today and I assure you that if you watch it, you'll be glad you did. http://fivethirtyeight.com/features/the-queen-of-code/ On a personal note, Grace Hopper was going to be the graduation speaker when I graduated from Drew University in 1991 but she was ill. She passed away about 6 months later. I wish I could have met her.
Australian sysadmins cop brunt of data-retention burden
SAGE-AU is doing some great media work, making the case that the proposed data-retention law in Australia would create a nightmare for businesses that use computers. They point out that every ill-defined or vague point in the law creates more and more problems. "It's very immature legislation proposal. It's more holes than cheese. There's more questions around it than there are answers," he said. Read the full article here: Australian sysadmins cop brunt of data-retention burden Every country should have an organize that speaks for the IT workers. Go SAGE-AU!
How to document a process?
Dear Tom, I've been asked to document our company's System Integration process. Do you have any advice? Sincerely,A reader. Dear Reader, I get this question a lot whether it is system integration, setting up new computers, handling customer support calls, or just about anything. Documenting a process is an important first step to clarifying what the process is. It is a prerequisite to improving it, automating it, or both. My general advice is to find the process that exists and document exactly how it is done now. Only after that can you evaluate what steps work well and which need to be improved.
Rites of Passage for a modern sysadmin?
Dear readers: I need your help. I feel like I've lost touch with what new sysadmins go through. I learned system administration 20+ years ago. I can't imagine what new sysadmins go through now. In particular, I'd like to hear from new sysadmins about what their "rite of passage" was that made them feel like a "real sysadmin". When I was first learning system administration, there was a rite of passage called "setting up an email server". Everyone did it. This was an important project because it touches on so many different aspects of system administration: DNS, SMTP, Sendmail configuration, POP3/IMAP4, setting up a DNS server, debugging clients, and so on and so on.
How to move PCs to a corporate standard?
Someone asked me in email for advice about how to move many machines to a new corporate standard. I haven't dealt with desktop/laptop PC administration ("fleet management") in a while, but I explained this experience and thought I'd share it on my blog: I favor using "the carrot" over "the stick". The carrot is making the new environment better for the users so they want to adopt it, rather than using management fiat or threats to motivate people. Each has its place. The more people feel involved in the project the more likely they are to go along with it. If you start by involving typical users by letting them try out the new configuration in a test lab or even loaning them a machine for a week, they'll feel like they are being listened to and will be your partner instead of a roadblock.
ACM's new Applicative conf, Feb. 25-27, NYC!
Are you a software developer that is facing rapidly changing markets, technologies and platforms? This new conference is for you. ACM's new Applicative conference, Feb. 25-27, 2015 in Midtown Manhattan, is for software developers who work in rapidly changing environments. Technical tracks will focus on emerging technologies in system-level programming and application development. The list of speakers is very impressive. I'd also recommend sysadmins attend as a way to stay in touch with the hot technologies that your developers will be using (and demanding) soon. Early bird rates through Jan. 28 at http://applicative.acm.org
See you at Bay Bay LISA next week!
Hi Boston-area friends! I'll be giving my "Radical ideas from The Practice of Cloud System Administration" talk at the Back Bay LISA user group meeting on Wednesday, January 14, 2015. Visit bblisa.org for more info.
Fighting Spam Backscatter
Short version: My mailing list server no longer generates bounce messages for unknown accounts, thus eliminating the email backscatter is generates. Longer version: I have a host set up exclusively for running mailing lists using Mailman and battling spam has been quite a burden. I finally 'gave up' and made all the lists "member's only". Luckily that is possible with the email lists being run there. If I had any open mailing lists, I wouldn't have been so luck. The result of this change was that it eliminated all spam and I was able to disable SpamAssassin and other measures put in place.
Predictions for 2015 that you can't live without!
Here are my predictions for 2015: Bloggers who make stupid, attention-getting, predictions will not be held accountable when those predictions don't come true. Windows-only enterprises have started buying Apple laptops to run Windows 10 due to the lower repair rate of the higher quality hardware. This trend will increase and Apple will run a marketing campaign to take advantage of the trend. The battle between Docker and CoreOS to define the container format of the future will stall the industry as it gets more and more nasty. If you thought VHS vs. Betamax was bad, or that AT&T vs. BSD Unix was bad, this will be 100x worse.
ALERT! Two "git" related security vunerabilities
Warning! Upgrade now! There is a security hole in the git client. UNTIL YOU UPGRADE: Do not "git clone" or "git pull" from untrusted sources. AFTER YOU UPGRADE: Do not "git clone" or "git pull" from untrusted sources. THE CODE YOU JUST DOWNLOADED IS UNTRUSTED AND SHOULD NOT BE RUN, YOU FOOL!ÿ
Book review and interview on InfoQ
InfoQ interviewed the authors of The Practice of Cloud System Administration and included it as part of their review of the book. Read it here!
Interview by Win Treese in InformIT
Win Treese interviewed me and my co-authors about the book. An Interview with the authors of "The Practice of Cloud System Administration" on DevOps and Data Security We discussed DevOps in the enterprise, trends in system administration, and at the end I got riled up and ranted about how terrible computer security has become.
Book Excerpt: Capacity Planning
ComputerWorld.com has published an excerpt from our book "The Practice of Cloud System Administration: Designing and Operating Large Distributed Systems Vol 2". The article has a title that implies it is about capacity planning for data centers but it's really about capacity planning for any system or service. Room to grow: Tips for data center capacity planning If you like that it, there's 547 more pages of good stuff like that in the book.
LOPSA-NJ Meeting: Intro to Chocolatey (THURSDAY)
This is LOPSA NJ's birthday meeting. There will be cake! The topic for this month's LOPSA NJ Chapter meeting is Chocolatey. It is a package manager that brings a lot of the benefits that Linux package systems have to the Windows world. Whether you use Windows or Unix, this presentation will be very educational. Topic: Intro to Chocolatey: A Windows Package Manager Speaker: Derek Murawsky Date: Thursday, December 4, 2014 Time: 7:00pm (social), 7:30pm (discussion) Location: Lawrenceville, NJ (near Princeton) For full info about the meeting click here.
LOPSA-NJ Meeting: Intro to Chocolatey (THURSDAY)
This is LOPSA NJ's birthday meeting. There will be cake! The topic for this month's LOPSA NJ Chapter meeting is Chocolatey. It is a package manager that brings a lot of the benefits that Linux package systems have to the Windows world. Whether you use Windows or Unix, this presentation will be very educational. Topic: Intro to Chocolatey: A Windows Package Manager Speaker: Derek Murawsky Date: Thursday, December 4, 2014 Time: 7:00pm (social), 7:30pm (discussion) Location: Lawrenceville, NJ (near Princeton) For full info about the meeting click here.
Book Excerpt: Organizing Strategy for Operational Teams
When Esther Schindler asked for permission to publish an excerpt from The Practice of Cloud System Administration on the Druva Blog, we thought this would be the perfect piece. We're glad she agreed. Check out this passage from Chapter 7, "Operations in a Distributed World". If you manage a sysadmin team that manages services, here is some advice on how to organize the team and their work: Organizing Strategy for Operational Teams
Book Excerpt: Organizing Strategy for Operational Teams
When Esther Schindler asked for permission to publish an excerpt from The Practice of Cloud System Administration on the Druva Blog, we thought this would be the perfect piece. We're glad she agreed. Check out this passage from Chapter 7, "Operations in a Distributed World". If you manage a sysadmin team that manages services, here is some advice on how to organize the team and their work: Organizing Strategy for Operational Teams
20th anniversary of "the big Synopsys downtime"
Paul Evans writes: Next week marks the 20th anniversary of the "big" Synopsys downtime of November 18-20, 1994. It was the most ambitious one we attempted while I was at Synopsys -- a complete reconfiguration of the network in one weekend. It was also the first one we planned and carried out using the flight director and mission control model that Christine and Tom later made known to a wider audience in The Practice of System and Network Administration. Although not all of you were working at Synopsys in 1994, this is probably the best chance I'm going to have to thank all of you for what you contributed to the downtime process.
20th anniversary of "the big Synopsys downtime"
Paul Evans writes: Next week marks the 20th anniversary of the "big" Synopsys downtime of November 18-20, 1994. It was the most ambitious one we attempted while I was at Synopsys -- a complete reconfiguration of the network in one weekend. It was also the first one we planned and carried out using the flight director and mission control model that Christine and Tom later made known to a wider audience in The Practice of System and Network Administration. Although not all of you were working at Synopsys in 1994, this is probably the best chance I'm going to have to thank all of you for what you contributed to the downtime process.
TPOCSA is the "eBook Deal of the Day"
To celebrate Usenix LISA, for 24 hours you can get The Practice of Cloud System Administration at an extra special discount: http://ow.ly/Ea7bH I'll be doing a book signing at LISA on Friday at 10:30 in the LISA Lab. If you have the eBook, I have something special for you! See you there!
TPOCSA is the "eBook Deal of the Day"
To celebrate Usenix LISA, for 24 hours you can get The Practice of Cloud System Administration at an extra special discount: http://ow.ly/Ea7bH I'll be doing a book signing at LISA on Friday at 10:30 in the LISA Lab. If you have the eBook, I have something special for you! See you there!
Introducing Bosun, our new open source monitoring & alerting system
The monitoring system built for Stackoverflow was open sourced today. I've been using it since it was an internal beta and I'm really excited to see it shared with everyone. Congrats to Kyle and Matt on this release! http://blog.stackoverflow.com/2014/11/announcing-bosun-our-new-open-source-monitoring-alerting-system/ and check out: http://bosun.org
Introducing Bosun, our new open source monitoring & alerting system
The monitoring system built for Stackoverflow was open sourced today. I've been using it since it was an internal beta and I'm really excited to see it shared with everyone. Congrats to Kyle and Matt on this release! http://blog.stackoverflow.com/2014/11/announcing-bosun-our-new-open-source-monitoring-alerting-system/ and check out: http://bosun.org
Arrested Devops Episode 23 with Tom Limoncelli
I'm the guest on the new episode of Arrested Devops. I had a lot of fun recording this podcast. I hope you enjoy listening to it! Check it out! http://adevo.ps/23
Arrested Devops Episode 23 with Tom Limoncelli
I'm the guest on the new episode of Arrested Devops. I had a lot of fun recording this podcast. I hope you enjoy listening to it! Check it out! http://adevo.ps/23
Book Signing at Usenix LISA next week
I'll be doing a book signing at Usenix LISA on Friday at 10:30am in the LISA Lab. The first 10 people to arrive will receive a free (printed) copy of the new book The Practice of Cloud System Administration. (I'll also sign other books you bring.) For info about the new book, please attend my talk "Radical Ideas from the Practice of Cloud Computing" on Wednesday at 11:45am-12:30 pm in Grand Ballroom C. I'll also be teaching tutorials and mini-tutorials. Register for LISA today!
Book Signing at Usenix LISA next week
I'll be doing a book signing at Usenix LISA on Friday at 10:30am in the LISA Lab. The first 10 people to arrive will receive a free (printed) copy of the new book The Practice of Cloud System Administration. (I'll also sign other books you bring.) For info about the new book, please attend my talk "Radical Ideas from the Practice of Cloud Computing" on Wednesday at 11:45am-12:30 pm in Grand Ballroom C. I'll also be teaching tutorials and mini-tutorials. Register for LISA today!
2015 speaking gigs: Boston, Pennsylvania, Baltimore
Three new speaking gigs have been announced. January (BBLISA in Cambridge, MA), February (Bucks County, PA), and March (Baltimore-area). See the "see us live" box at the top of http://EverythingSysadmin.com or subscribe to the RSS feed to learn about any new speaking engagements.
2015 speaking gigs: Boston, Pennsylvania, Baltimore
Three new speaking gigs have been announced. January (BBLISA in Cambridge, MA), February (Bucks County, PA), and March (Baltimore-area). The full list is on http://the-cloud-book.com/book-tour.html or subscribe to the RSS feed to learn about any new speaking engagements. The next 3 speaking gigs is always listed on "see us live" box at the top of http://EverythingSysadmin.com.
Apple Pay and CurrentC
I predict one year from today CurrentC won't be up and running and, in fact, history will show it was just another attempt to stall and prevent any kind of mobile payment system in the U.S. from being a success. I'm not saying that there won't be NFC payment systems, just that they'll be marginalized and virtually usess as a result.
Apple Pay and CurrentC
I predict one year from today CurrentC won't be up and running and, in fact, history will show it was just another attempt to stall and prevent any kind of mobile payment system in the U.S. from being a success. I'm not saying that there won't be NFC payment systems, just that they'll be marginalized and virtually usess as a result.
Wait, did you mean Wed the 15th or Thu the 16th?
How many times have you seen this happen? Email goes out that mentioned a date like "Wed, Oct 16". Since Oct 16 is a Thursday, not a Wednesday (this year), there is a flurry of email asking, "Did you mean Wed the 15th or Thu the 16th?" A correction goes out but the damage is done. Someone invariantly "misses the update" and shows up a day early or late, or is otherwise inconvenienced. Either way cognitive processing is wasted for anyone involved. The obvious solution is "people should proofread better" but it is a mistake that everyone makes. I see the mistake at least once a month, and sometimes I'm the guilty party.
Wait, did you mean Wed the 15th or Thu the 16th?
How many times have you seen this happen? Email goes out that mentioned a date like "Wed, Oct 16". Since Oct 16 is a Thursday, not a Wednesday (this year), there is a flurry of email asking, "Did you mean Wed the 15th or Thu the 16th?" A correction goes out but the damage is done. Someone invariantly "misses the update" and shows up a day early or late, or is otherwise inconvenienced. Either way cognitive processing is wasted for anyone involved. The obvious solution is "people should proofread better" but it is a mistake that everyone makes. I see the mistake at least once a month, and sometimes I'm the guilty party.
How to make change when handed a $20... and help democracy
If someone owes you $5.35 and hands you a $20 bill, every reader of this blog can easily make change. You have a calculator, a cash register, or you do it in your head. However there is a faster way that I learned when I was 12. Today it is rare to get home delivery of a newspaper, but if you do, you probably pay by credit card directly to the newspaper company. It wasn't always like that. When I was 12 years old I delivered newspapers for The Daily Record. Back then payments were collected by visiting each house every other week.
How to make change when handed a $20... and help democracy
If someone owes you $5.35 and hands you a $20 bill, every reader of this blog can easily make change. You have a calculator, a cash register, or you do it in your head. However there is a faster way that I learned when I was 12. Today it is rare to get home delivery of a newspaper, but if you do, you probably pay by credit card directly to the newspaper company. It wasn't always like that. When I was 12 years old I delivered newspapers for The Daily Record. Back then payments were collected by visiting each house every other week.
Katherine Daniels (@beerops) interviews Tom Limoncelli
Katherine Daniels (known as @beerops on Twitter) interviewed me about the presentations I'll be doing at the upcoming Usenix LISA '14 conference. Check it out: https://www.usenix.org/blog/interview-tom-limoncelli Register soon! Seating in my tutorials is limited!
Katherine Daniels (@beerops) interviews Tom Limoncelli
Katherine Daniels (known as @beerops on Twitter) interviewed me about the presentations I'll be doing at the upcoming Usenix LISA '14 conference. Check it out: https://www.usenix.org/blog/interview-tom-limoncelli Register soon! Seating in my tutorials is limited!
See you tomorrow evening at the Denver DevOps Meetup!
Hey Denver folks! Don't forget that tomorrow evening (Tue, Oct 21) I'll be speaking at the Denver DevOps Meetup. It starts at 6:30pm! Hope to see you there! http://www.meetup.com/DenverDevOps/events/213369602/
See you tomorrow evening at the Denver DevOps Meetup!
Hey Denver folks! Don't forget that tomorrow evening (Tue, Oct 21) I'll be speaking at the Denver DevOps Meetup. It starts at 6:30pm! Hope to see you there! http://www.meetup.com/DenverDevOps/events/213369602/
Usenix LISA early registration discount expires soon!
Register by Mon, October 20 and take advantage of the early bird pricing. I'll be teaching tutorials on managing oncall, team-driven sysadmin tools, upgrading live services and more. Please register soon and save! https://www.usenix.org/conference/lisa14
Usenix LISA early registration discount expires soon!
Register by Mon, October 20 and take advantage of the early bird pricing. I'll be teaching tutorials on managing oncall, team-driven sysadmin tools, upgrading live services and more. Please register soon and save! https://www.usenix.org/conference/lisa14
Results of the PuppetConf 2014 Raffle
If you recall, the fine folks at Puppet Labs gave me a free ticket to PuppetConf 2014 to give away to a reader of this blog. Here's a report from our lucky winner! Conference Report: PuppetConf 2014 by Anastasiia Zhenevskaia You never know when you will be lucky enough to win a ticket to the PuppetConf, one of the greatest conferences of this year. My "moment" happened just 3 weeks before the conference and let me dive into things I've never thought about. Being a person who worked mostly with the front-end development, I was always a little bit scared and puzzled by more complicated things.
Results of the PuppetConf 2014 Raffle
If you recall, the fine folks at Puppet Labs gave me a free ticket to PuppetConf 2014 to give away to a reader of this blog. Here's a report from our lucky winner! Conference Report: PuppetConf 2014 by Anastasiia Zhenevskaia You never know when you will be lucky enough to win a ticket to the PuppetConf, one of the greatest conferences of this year. My "moment" happened just 3 weeks before the conference and let me dive into things I've never thought about. Being a person who worked mostly with the front-end development, I was always a little bit scared and puzzled by more complicated things.
Tutorial: Evil Genius 101
I'm teaching a tutorial at Usenix LISA called "Evil Genius 101: Subversive Ways to Promote DevOps and Other Big Changes". Whether you are trying to bring "devops culture" to your workplace, or just get approval to purchase a new machine, convincing and influencing people is a big part of a system administrator's time. For the last few years I've been teaching this class called "Evil Genius 101" where I reveal my tricks for understanding people and swaying their opinion. None of these are actually evil, nor do I teach negotiating techniques. I simply list 3-4 techniques I've found successful for each of these situations: talking to executives, talking to managers, talking to coworkers, and talking to users.
Come hear me speak in Denver next week!
On Tuesday, Oct 21st, I'll be speaking at the Denver DevOps Meetup. It is short notice, but if you happen to be in the area, please come! I'll be talking about the new book and how DevOps principles can make the world a better place. I'll have a copy or two to give away, and special discount codes for everyone. The meeting is at the Craftsy Offices, 999 18th St., Suite 240, Denver, CO. For more information and to RSVP, please go to http://www.meetup.com/DenverDevOps/events/213369602/
Tutorial: How To Not Get Paged
Step 1: turn off your pager. Step 2: disable the monitoring system. Or.... you can run oncall using modern methodologies that constantly improve the reliability of your system. I'm teaching a tutorial at Usenix LISA called "How To Not Get Paged: Managing Oncall to Reduce Outages". I'm excited about this class because I'm going to explain a lot of the things I learned at Google about how to turn oncall from a PITA to a productive use of time that improves the reliability of the systems you run. Most of the material is from our new book, The Practice of Cloud System Administration, but the Q&A always leads me to say things I couldn't put in print.
Interview on SpiceWorks.com: Demystifying DevOps with Tom Limoncelli
Holly from SpiceWorks interviewed me while I was in Austin for the SpiceWorld '14 conference. We talked about DevOps from the SMB "IT guy" perspective, Lord of the Rings, Chef vs. Puppet, and my secret desire start a podcast what would be "the Stephen Colbert of DevOps." The interview has been published on their community website: Demystifying DevOps: Q&A with Tom Limoncelli Enjoy!
Tutorial: Live Upgrades on Running Systems
I'm teaching a tutorial at Usenix LISA called "Live Upgrades on Running Systems: 8 Ways to Upgrade a Running Service With Zero Downtime". Ever notice that Google, Facebook and other website aren't down periodically for software upgrades? That's because they're upgrading software on their service while it is live. As a result, they can push new features continuously. In this tutorial I'll describe 8 techniques they use... and so can you. Oh, and here's a secret: I'll have a 9th way to upgrade software... but it requires down-time. That said, it might not require down-time that is visible to users! I'm excited about this tutorial because it covers a lot of the unique topics we cover in The Practice of Cloud System Administration that I haven't talked about publicly before.
Tutorial: Work Like a Team, not a group of individuals
I'm teaching a tutorial at Usenix LISA called "Work Like a Team: Best Practices for Team Coordination and Collaborations So You Aren't Acting Like a Group of Individuals". I'm excited about this class because I'm going to demo a lot of the Google Apps tricks I've accumulated over the years, and combine them with stories about successes (and failures) related to bringing teams together to work on projects. I also get to explain a lot of DevOps culture in ways that make sense to non-DevOps shops (mostly stuff I've been advocating for since before "devops" was a thing). A lot of the material will overlap with our new book, The Practice of Cloud System Administration.
Is TPOCSA a DevOps book?
Quoting from a community forum post on SpiceWorks: It doesn't have "DevOps" in the name, but the new The Practice of Cloud System Administration ... covers a lot of the same concepts, more as "here's some things that have emerged as best practices in the modern world of system administration." Textbook-thick but destined to be a classic like his previous The Practice of System and Network Administration. Thanks to Ernest Mueller for the kind words!
Calling all students and women!
Apply now for a grant to attend LISA14. Submissions are due by Monday, October 13. https://www.usenix.org/conference/lisa14/students-and-grants Are you a student? There are grants available for the general conference and the tutorial program. Are you a woman? As part of its ongoing commitment to encourage women to excel in this field, Usenix is pleased to announce the return of the Google Grants for Women to support female computer scientists interested in attending the LISA14 conference. All female computer scientists from academia or industry are encouraged to apply. Applications are due by October 13. https://www.usenix.org/conference/lisa14/students-and-grants
Concerning PICC
Today, Wednesday, October 8, 2014, we, Matt Simmons and Thomas Limoncelli, resigned from the board of Professional IT Community Conferences, Inc. also known as "PICC". PICC is the New Jersey non-profit business entity that has backed LOPSA-East and Cascadia since 2011. Those two conferences should be unaffected as it was already agreed that they would find new organization(s) to work with for their 2015 conferences. As of June 10, 2014, PICC, Inc. had voted to and was in the process of being dissolved. However we feel this process has become impossible due to the remaining board member's foot-dragging and at times outright deceptive actions.
I'm coming to Europe in November!
I'm honored to be a keynote at NLUUG's Autumn Conference, 20-Nov-2014, in The Netherlands. I don't get to Europe often, so this may be the last chance to see me there for a while. I'm also trying to arrange a book-signing while I'm there. For more info, visit https://www.nluug.nl/events/nj14/ Register now! Registration is limited! Even though the registration page is in Dutch, the talk will be in English. Google translate is your friend.
Tom speaking at NYC DevOps meetup Wednesday!
I'll be the speaker at the Wed, October 8th meeting of the NYCDEVOPS Meetup which meets (I kid you not) at the office of MeetUp, Inc. in New York City. I'll be talking about our new book, The Practice of Cloud System Administration. For more info: http://www.meetup.com/nycdevops/events/208856642/
Cleaning Up Stack Exchange's Puppet Environment
Shane Madden, a coworker of mine, recently re-engineered our Stack Exchange's Puppet environment. It is now full of win. Read about it here: http://shanemadden.net/stackexchange-puppet-cleanup.html
Why women leave tech... because they have good taste
Fortune Magazine published an article called Why women leave tech: It's the culture, not because 'math is hard' TL;DR version: We treat them like shit and are surprised when they leave. So, basically women leave tech because they have self-respect. Good for them. Shame on our industry. A few weeks ago I suggested that there aren't many women in tech because "women have good taste". Every woman that I've said this to has agreed... or at least laughed. However it is an uncomfortable laugh. A laugh that indicates that it is something we all know, but don't know how to talk about.
Tom on DevOps Cafe Podcast
I'm excited to announce that I'm interviewed on the new episode of DevOps Cafe. We talk about the history of system administration leading up to DevOps, recent changes, how the Usenix LISA conference has changed this year, and more.
Apple's livesteam outage was easily preventable: here's how!
The live stream of Apple's announcement of the Apple Watch was marred by technical problems. Users saw messages about "could not load movie" and "you don't have permission to access". As we read Dan Rayburn's excellent technical analysis of what went wrong, we couldn't help but think how easily preventable their problems were. The problem was that Apple introduced a new feature that had unknown resource requirements and (oops!) they didn't have enough resources. For example, suppose a thousand website visitors requires a certain number of computers (resources) to serve the website. Some websites are "heavier" and require the same work to be spread over more computers, others require fewer resources per thousand users.
I'll be at Philly Linux User Group tomorrow (Wednesday)
Hi Philly folks! I will be speaking at the Philadelphia area Linux Users' Group (PLUG) meeting on Wednesday night (Oct 1st). They meet at the University of the Sciences in Philadelphia (USP). My topic will be "Highlights from The Practice of Cloud System Administration" and I'll have a few copies of the book to give away. For more info, visit their website: http://www.phillylinux.org/meetings.html Hope to see you there!
How the internet has affected what books get published?
Someone recently asked me how the rise of the Internet has affected what books get published, specifically related to books about operating systems and other open source projects. This is based on what I've been told by various publishers and is "conventional wisdom". Of course, an actual publisher may disagree or explain it differently, or have counterexamples. This is the email I sent in reply: One way that the internet has changed the book industry that is not well-known outside of publishing circles is that it has lead to the death of the reference book. It used to be for every language or system, someone would make easy money printing a book that lists all the system calls, library calls, or configuration fields in a system.
Brewster Rockit explains network latency.
Sunday's "Brewster Rockit" comic strip explained bandwidth vs. latency better than I've ever seen is some text books: When I interview anyone for a technical position I always ask them to explain the difference between bandwidth and latency. It is an important concept, especially in today's networked world. Years ago most candidates didn't know the difference. Lately most candidates I interview know the difference, but have a difficult time putting it into words. Fewer can explain it in a mathematical or scientific way. Latency is how long information takes to get from one place to another. Bandwidth is how much data per second is sent.
Safari Books Online update
Previously Safari Books Online (the O'Reilly thing... not the Apple thing) had a rough draft of The Practice of Cloud System Administration. Now it has the final version: http://my.safaribooksonline.com/9780133478549 Enjoy!
In NYC for Velocity?
If you are in NYC for Velocity, please check out my tutorial on Monday, "Office Hours" on Tuesday, book signing on Wednesday, or come to my book part on Wednesday night! Or just stop me randomly in the hallway and say "hi!" I love meeting new people! Tutorial: Mon, 3:30pm, Time Management for Busy DevOps Office Hour: Tue, 1:15pm, One-on-one Time Management help (note: the session is 45 minutes long) Book Signing: Wed, 10:45am, [Book Signing] Book Launch Party: Wed, 7pm at the Stack Exchange NYC office
Why is TPOCSA called "Volume 2"?
...because we're re-branding The Practice of System and Network Administration as "Volume 1". Vol 1 == enterprise. Vol 2 == server/service administration. Available as a PDF here.
Configure Appigo Todo Cloud for use with The Cycle
In Time Management for System Administrators I describe a way to manage your todo lists which I call "The Cycle". The book came out before the existence of smart phones and app stores, so it doesn't include some important info. The iPhone app (now available for Android) that I use for The Cycle is Appigo Todo Cloud. It can be configured in a way that makes it easy to do The Cycle. If you recall, in The Cycle you set up a todo list for each day. At the end of the day, you move the remaining items to the next day's list.
NYC Book Launch Party for "The Practice of Cloud System Administration"
Stack Exchange, Inc. (stackoverflow.com / serverfault.com) is hosting the launch party for Tom Limoncelli's newest book, "The Practice of Cloud System Administration." The local DevOps/Sysadmin/Linux user community is invited. Food and beverages will be provided. Date: Wed, Sept. 17, 2014 Time: 7 p.m. until 9 p.m. Location: Stack Exchange NYC HQ, 110 William Street, 28th Floor, NY, NY 10038 RSVP: click here Information about the book: http://the-cloud-book.com If you are in town for Velocity NYC, please stop by!
The ebook is shipping!
The Practice of Cloud System Administration is shipping on Kindle and PDF/Mobi versions are shipping on InformIT. Physical book should start shipping today or Monday. If you get the PDF, I'd love to know the md5 hash of the file. Post in the comments.
Stack Exchange is hiring sysadmins (come work with me!)
The team I'm on at SE is hiring! In particular, we need a system administrator (either a Linux or Windows) that has Cisco network gear experience. Site Reliability Engineer, Networking We'll have another listing for a generalist (someone that knows Linux and Windows) in a few days. We do have opportunities for people that work remotely. For more info about working at Stack Exchange, check out our page: http://stackexchange.com/work-here Oh yeah and... Stack Exchange, Inc. does not discriminate in employment matters on the basis of race, color, religion, gender, national origin, age, military service eligibility, veteran status, sexual orientation, marital status, disability, or any other protected class.
Good Reads, Aug 2014
(The book tour for The Practice of Cloud System Administration has begun! Book signings planned for NY, NJ, Philly, Austin, Denver, Seattle and The Netherlands. Parties planned for NYC on Sept 17 and at my house in NJ on Sept 20. If you are local, please attend. More info on http://the-cloud-book.com) A summary of the interesting articles I've found this month. Mark Burgess on why APIs are bad They aren't idempotent. If you use an API to create an object and crash mid-way through, when you recover from the crash you don't know if the object is in a good state or not.
TPOCSA Book Tour announcement!
I'm excited to announce my "book tour" to promote The Practice of Cloud System Administration, which starts shipping on Friday, September 5! I'll be speaking and/or doing book signings at the following events. More dates to be announced soon. 2014-08-18 New Jersey: DevOps and Automation NJ Group 2014-09-04 New Jersey: LOPSA-NJ Chapter Meeting 2014-09-15 NYC: Velocity Conference NYC (book signing Wed) 2014-10-01 Philadelphia area Linux Users' Group (PLUG) 2014-09-23 Austin, TX: SpiceWorld Conference 2014-09-24 Austin, TX: CloudAustin Meetup 2014-11-09 Seattle, WA: Usenix LISA (attending all week) Soon to be announced: Denver CO (October), Netherlands (November) Still looking for opportunties: SF/Bay Area This book is the culmination of 2 years of research on the best practices for modern IT / DevOps / cloud / distributed computing.
Tom speaking at LOPSA-NJ September meeting (near Princeton/Trenton)
I'll be the speaker at the September LOPSA-NJ meeting. My topic will be the more radical ideas in our new book, The Practice of Cloud System Administration. This talk is (hopefully) useful whether you are legacy enterprise, fully cloud, or anywhere in between. Topic: Radical ideas from The Practice of Cloud Computing Date: Thursday, September 4, 2014 Time: 7:00pm (social), 7:30pm (discussion) This is material from our newest book, which starts shipping the next day. Visit http://the-cloud-book.com for more info.
Save up to 55% off on our new book for a short time
The Practice Of Cloud System Administration is featured in the InformIT Labor Day sale. Up to 55% off! Here is the link Buy 3 or more and save 55% on your purchase. Buy 2 and save 45%, or buy 1 and save 35% off list price on books, video tutorials, and eBooks. Enter code LABORDAY at checkout. You will also receive a 30-day trial to Safari Books Online, free with your purchase. Plus, get free shipping within the U.S. The book won't be shipping until Sept 5th, but you can read a recent draft via Safari Online.
I'm giving away 1 free ticket to PuppetConf 2014!
PuppetConf 2014 is the 4th annual IT automation event of the year, taking place in San Francisco September 20-24. Join the Puppet Labs community and over 2,000 IT pros for 150 track sessions and special events focused on DevOps, cloud automation and application delivery. The keynote speaker lineup includes tech professionals from DreamWorks Animation, Sony Computer Entertainment America, Getty Images and more. If you're interested in going to PuppetConf this year, I will be giving away one free ticket to a lucky winner who will get the chance to participate in educational sessions and hands-on labs, network with industry experts and explore San Francisco.
LISA 2014 OMG
The schedule for 2014 has been published and OMG it looks like an entirely new conference. By "new" I mean "new material"... I don't see slots filled with the traditional topics that used to get repeated each year. By "new" I also mean that all the sessions are heavily focused on forward-thinking technologies instead of legacy systems. This conference looks like the place to go if you want to move your career forward. LISA also has a new byline: LISA: Where systems engineering and operations professionals share real-world knowledge about designing, building, and maintaining the critical systems of our interconnected world.
How sysadmins can best understand Burning Man
So, you've heard about Burning Man. It has probably been described to you as either a hippie festival, where rich white people go to act cool, naked chicks, or a drug-crazed dance party in the desert. All of those descriptions are wrong..ish. Burning Man is a lot of things and can't be summarized in a sound bite. I've never been to Burning Man, but a lot of my friends are burners, most of whom are involved in organizing their own group that attends, or volunteer directly with the organization itself. Imagine 50,000 people (really!) showing up for a 1-week event. You essentially have to build a city and then remove it.
Simple bucket-ized stats in awk
Someone recently asked how to take a bunch of numbers from STDIN and then break them down into distribution buckets. This is simple enough that it should be do-able in awk. Here's a simple script that will generate 100 random numbers. Bucketize them to the nearest multiple of 10, print based on # of items in bucket: while true ; do echo $[ 1 + $[ RANDOM % 100 ]] ; done | head -100 | awk '{ bucket = int(($1 + 5) / 10) * 10 ; arr[bucket]++} END { for (i in arr) {print i, arr[i] }}' | sort -k2n,2 -k1n,1 Many people don't know that in bash, a single quote can go over multiple lines.
New DevOps meetup starting in New Jersey!
As you know, I live in New Jersey. I'm excited to announce that some fine fellow New Jerseyians have started a DevOps Meetup. The first meeting will be on Monday, Aug 18, 2014 in Clifton, NJ. I'm honored to be their first speaker. More info at their MeetUp Page: DevOps and Automation NJ Group Hope to see you there!
TPOCSA Book Tour announcement!
I'm excited to announce my "book tour" to promote The Practice of Cloud System Administration, which starts shipping on Friday, September 5! I'll be speaking and/or doing book signings at the following events. More dates to be announced soon. 2014-08-18 New Jersey: DevOps and Automation NJ Group 2014-09-04 New Jersey: LOPSA-NJ Chapter Meeting 2014-09-15 NYC: Velocity Conference NYC (book signing Wed) 2014-10-01 Philadelphia area Linux Users' Group (PLUG) 2014-09-23 Austin, TX: SpiceWorld Conference 2014-09-24 Austin, TX: CloudAustin Meetup 2014-11-09 Seattle, WA: Usenix LISA (attending all week) Soon to be announced: Denver CO (October), Netherlands (November) Still looking for opportunties: SF/Bay Area This book is the culmination of 2 years of research on the best practices for modern IT / DevOps / cloud / distributed computing.
Online course: An Introduction to Operations Management starting soon
U Penn's Wharton School of Business has one of the best Operations Management classes in the world, and for the last few years it has been available as a MOOC. They've announced a new session starting on September 29 2014. I've been watching some of the video lectures and Christian Terwiesch is an excellent lecturer. What is learned in the class applies to system administration, running a restaurant, or a hospital. I think that system administrators, DevOps engineers, and managers can learn a lot from this class. Sign up for An Introduction to Operations Management or just watch the videos. Either way, you'll learn a lot!
Honest Life Hacks
I usually don't blog about "funny web pages" I found but this is relevant to the blog. People often forward me these "amazing life hacks that will blow your mind" articles because of the Time Management Book. First of all, I shouldn't have to tell you that these are linkbait (warning: autoplay). Secondly, here's a great response to all of these: Honest Life Hacks.
System Administrator Appreciation Day Spotlight: SuperUser.com
This being System Administrator Appreciation Day, I'd like to give a shout out to all the people of the superuser.com community. If you aren't a system administrator, but have technical questions that you want to bring to your system administrator, this is the place for you. Do you feel bad that you keep interrupting your office IT person with questions? Maybe you should try SuperUser first. You might get your answers faster. Whether it is a question about your home cable modem, or the mysteries of having to reboot after uninstalling software (this discussion will surprise you); this community will probably reduce the number of times each week that you interrupt your IT person.
System Administrator Appreciation Day Spotlight: ServerFault.com
This being System Administrator Appreciation Day, I'd like to give a shout out to all the people on ServerFault.com who help system administrators find the answers to their questions. If you are patient, understanding, and looking to help fellow system administrators, this is a worthy community to join. I like to click on the "Unanswered" button and see what questions are most in need of a little love. Sometimes I click on the "hot this week" tab and see what has been answered recently. I always learn a lot. Today I learned: that GRUB has the ability to have a "one time boot" option, that fails to a different boot option; how to diagnose a Windows server that doesn't show all RAM that is installed; the pros and cons of using your ActiveDirectory DCs as your internal DNS servers (there were some suprises); ways to recover from accidentally revoking ...
I'm a system administrator one day a year.
Many years ago I was working at a company and our department's administrative assistant was very strict about not letting people call her a "secretary". She made one exception, however, on "Secretary Appreciation Day". I'm an SRE at StackExchange.com. That's a Site Reliability Engineer. We try to hold to the ideals of what a SRE as set forth by Ben Treynor's keynote at SRECon. My job title is "SRE". Except one day each year. Since today is System Administrator Appreciation Day, I'm definitely a system administrator... today. Sysadmins go by many job titles. The company I work for provides Question and Answer communities on over 120 topics.
50% off Time Management for System Administrators
O'Reilly is running a special deal for Celebrate SysAdmin Day. For one day only, SAVE 50% on a wide range of system administration ebooks and training videos from shop.oreilly.com. If you scroll down to page 18, you'll find Time Management for System Administrators is included. 50% off is pretty astounding. Considering that the book is cheaper than most ($19.99 for the eBook) there's practically no excuse to not have a copy. Finding the time to read it... that may be another situation entirely.
Special deal for Sysadmin Appreciation Day
Today is Sysadmin Appreciation Day! We appreciate all of you! The Practice of System and Network Administration is today's InformIT eBook Deal of the Day. Click on the link to get a special discount.
Tom will be the October speaker at Philly Linux Users Group (central)
I've just been booked to speak at PLUG/Central in October. I'll be speaking about our newest book, The Practice of Cloud System Administration. For a list of all upcoming speaking engagements, visit our appearances page: http://everythingsysadmin.com/appearances/
Tom @ Austin Cloud Meetup
Do you live near Austin? I'll be speaking at the Austin Cloud Meetup in September. They're moving their meeting to coincide with my trip there to speak at the SpiceWorld conference. More info here: http://www.meetup.com/CloudAustin/events/195140322/
IPv6 is real... and mentioned on The Cobert Report
Did you catch Vint Cerf on The Colbert Report last night? Part 1: http://on.cc.com/1wtNMgk Part 2: http://on.cc.com/1wtNYMAÿ He talks about Al Gore's involvement in funding the NSFNET, and the need for IPv6 deployments. Vint handles handles Cobert brilliantly. He literally blows Cobert out of the water in a few places.
Tom @ PuppetNYC Meetup: Tue, July 22
http://www.meetup.com/puppetnyc-meetings/events/194315322/ I'll be the speaker at the July PuppetNYC MeetUp. See you there!
Change-Able Operations or "Could DevOps Save Spiderman?"
[This article first appeared in the SAGE-AU newsletter.] Have you heard about the New York City broadway show Spider-Man Turn Off the Dark? It should have been a big success. The music was written by Bono and the Edge from U2. It was directed by Julie Taymor, who had previously created many successful shows including The Lion King. Sadly, before it opened, the show was already making headlines due to six actors getting seriously injured and other issues. The show opened late, but it did finally open. It ran from June 2011 to January 2014. When the show closed Taymor said that one of the biggest problems with bringing the show to production was that they were using new technology that was difficult to work with.
ANNOUNCEMENT: Our newest website: the-cloud-book.com
http://the-cloud-book.com is up and online! This is our new website dedicated to promoting The Practice of Cloud System Administration. It has a few incomplete pages, but we've decided to start spreading the word now. Check it out! http://the-cloud-book.com
Tom speaking at Puppet NYC Meetup
I'll be speaking at the July NYC Puppet Meetup meeting about the Puppet BlackBox project. If you missed this talk at PuppetCamp NYC, don't miss it here! Tuesday, July 22, 2014 6:30 PM to 8:00 PM The Ladders, 137 Varick St. 3rd Floor, New York, NY Full details are available at meetup. See you there!
Tom @ Puppet Camp NYC, July 8, 2014
Tom will be giving a talk called "Safely storing secrets and credentials in Git for use by Puppet: The BlackBox project" at Puppet Camp NYC. Hope to see you there! http://puppetlabs.com/events/puppet-camp-nyc
Test post. Please ignore.
Testing a new feed mechanism. No need to reply.
Moving the StackExchange/Serverfault NYC Datacenter
We moved our NYC datacenter to a new colo facility last month. We hired a moving company that specializes in moving racks of equipment. They did a great job. Here's a timelapse recording of them packing everything up. These are the computers that run Q&A websites like ServerFault.com (system administration questions) as well as 125 others! Check 'em out!
All chapters available as "rough cuts" on SBO
Safari Books Online now has all chapters of The Practice of Cloud Administration. " Rough Cuts" are pre-editing drafts. You get to see the book with all the typos and misspelled words... but 2-3 months before the real book is available: http://my.safaribooksonline.com/9780133478549 If you want to get some fan-only details about the book and other inside information.
Damn you, LinkedIn
I swear I clicked "cancel" but obviously I have instructed LinkedIn to tell the world I wanted to connect with them them. Every mailing list I'm on seems to have received an invitation. I have a feeling I'm going to get a lot of angry email today. So, if you received an invitation, please ignore it.
Read the (very) rough draft of our new book!
Safari Books now has the (very) rough draft of The Practice of Cloud System Administration online and available to anyone with an account. This book is all new material. How rough is rough? Well, the diagrams are the hand-drawn sketches that will eventually be turned into nicely drawn diagrams. The copyediting hasn't been done yet. There's probably a few other things missing. If you don't have a Safari Books Online account you'll be able to look at the Table of Contents and other materials. With an account, you'll be able to read the entire thing. http://my.safaribooksonline.com/9780133478549 The final release date will be mid-September.
NYC: Day-long Awesome Postmortems workshop!
Yulia Sheynkman and Dave Zwieback are repeating their "Awesome Postmortems" workshop on July 10. https://ti.to/mindweather/awesome-postmortems It's a great way to get the team--and not just ops--offsite to experience a healthier way of dealing and learning from failure. If you are in the NYC-area, this is a great opportunity to learn how to make postmortems an integrated part of how to improve reliability and prevent future outages. When we wrote our "how to do postmortems" section of the upcoming The Practice of Cloud System Administration, we asked Dave for advice because we respect his expertise. Now you can get a full day of training directly from Yulia and Dave!
bash alias of the day: reader
I have some PDFs that have to be reviewed in Adobe Reader, because they include "comments" that OS X "Preview" can't display and edit. This alias has saved me hours of frustration: alias reader='open -a /Applications/Adobe\ Reader.app' Now I can simply type "reader" instead of, say, "cat", and view the PDF: reader Limoncelli_Ch13_jh.pdf For those of you that are unfamiliar with Adobe Acrobat Reader, it is Adobe's product for distributing security holes to nearly every computer system in the world. It is available for nearly every platform, which makes it a very convenient way to assure that security problems can be distributed quickly and globally.
Google SRE University on the east-coast!
Usenix FCW '14 is in Philly June 17-20 and one of the tutorials is the highly rated SRE University--Practical Large System Design. It is taught by actual Google SREs. It is worth going to this conference just to take this class. There are other sysadmin-related classes at FCW '14 this year including Hadoop Operations, Apache Cloudstack, Jenkins CI, and Hands-On Security for system administrators. Check out the full schedule.
Help me explain our new book
Would you help us figure out how to explain our new book? A 30-question survey... only takes a minute.
Good Reads, May 2014
A summary of the interesting articles I've found this month. What is Site Reliability Engineering? An interview with Ben Treynor (Google VP, Site Reliability Engineering) -- SRE isn't just a new name for system administration, it is an entirely new business philosophy. Distributed Systems and the End of the API -- APIs are like assembly language. Nobody programs in assembly language any more. So what's the high-level equivalent? Big Cable says broadband investment is flourishing, but their own data says it's falling -- Remember folks, these are the companies that keep telling the media that people don't want gigabit broadband. The Unreasonable Effectiveness of Checklists -- Checklists are awesome...
Comments on NIST Draft SP 800-160
[I emailed these comments to NIST last week. I've never read NIST standards documents before, so my response may be entirely naive, but since it is my tax dollars at work, I thought I'd put in my two cents.] Subject: Draft SP 800-160 Comments I read with great interest the DRAFT Systems Security Engineering: An Integrated Approach to Building Trustworthy Resilient Systems http://csrc.nist.gov/publications/PubsDrafts.html#800-160 I'd like to comment on two sections, "2.3.4 Security Risk Management" and "Chapter 3: Lifecycle". 2.3.4 Security Risk Management This discusses ways to deal with risk: Avoid, Accept, Mitigate, Transfer. This is a very traditional view of risk.
Product Idea: Real-time re-living the moon landing
I was only 7 months old when Neil Armstrong became the very first man to walk on the moon. I don't remember it very well. Today I was reminded that most of what we see of the moon landings are highlights. 10-second little clips. I would like to know what the entire 8 days were like. I'm sure there are audio and video recordings of the entire thing. All of NASAs recordings are public domain, so they must be available somewhere. Here's my thought for a product. A kit that includes audio and video recordings and other stuff to help you re-live the entire 8 day experience.
"Ops All The Things" Podcast"
I'm the guest on this week's "Ops All The Things!" podcast. We talk about time management and all sorts of things. Check it out! http://www.opsallthethings.com/podcast/006-time-management
I'd like to buy an IP-KVM switch, please.
Hi! I'd like to buy an IP-KVM switch, please. " Sure! We got plenty." Now wait... I have some very specific requirements. " Shoot." First, I want it to connect via some kind of pod or something that I can only buy from you. If there is any interoperability between vendors, I'm going to be very upset. I want full vendor lock-in. " No worries, sir. We have a variety of pods, all highly proprietary. I assure you they won't work with any other vendor. Heck, some of them don't even work with our own products! In fact, if you are switching from another brand we send you a box of bandaids since we know you'll need them after changing all those cables."
My 5-year prediction
I don't make many predictions. However I think two technologies are going to be huge within the next five years. DACs: I'm not saying Bitcoin will be big (though it could be), I'm saying that the underlying technology is revolutionary and may become one the basic data management systems we use in places where today we need a neutral third party. That would be things like: DNS registrations, the stock market, and so on. More info here. CRDTs/CALM: I've been talking about these since 2009, but Chas Emerick's new article makes me confident they're ripe to become very popular very soon.
Mice, Cheese, DevOps, and Job Satisfaction
You've probably seen experiments where a mouse gets cheese as a reward for pulling a lever. If he or she receives the cheese right away, the brain associates work (pulling the lever) with reward (the cheese) and it motivates the mouse. They want to do more work. It improves job satisfaction. If the mouse received the cheese a month later, the brain won't associate the work with the reward. A year later? Fuggedaboutit! Now imagine you are a software developer, operations engineer, or system administrator working on a software project. The software is released every 6 months. The hard work you do gets a reward every 6 months.
Covered costs: $0. Your responsibility: $X billion.
You've probably seen this report: HealthCare.Gov Looks Like A Bargain Compared With State Exchanges. The Federal Healthcare Exchange was able to do the job much cheaper than the state-run exchanges. Ironically the states that benefitted the most were those that refused to participate and therefore were served by the Federal exchange. Personally I think that the insurance companies that got 8.1 million signups should be billed for the cost of those web sites. The bill should include a note saying, "Covered costs: $0. Your responsibility: $X billion." Hilarious, right? (I know, I know... don't quit your day job.) But we, as sysadmins, know the cost-saving power of centralized IT.
Slides from LOPSA-East
I've uploaded my slides from "Top 5 Time Management Tips for SysAdmins" to SlideShare. They apply to developers too. Enjoy.
Tom @ LOPSA-East, New Brunswick, NJ, May 2-3, 2014
I'll be teaching tutorials. I'm also on the organizing committee. More info soon. Visit the conference site for details: http://lopsa-east.org
Good Reads, April 2014
Heartbleed This month was really all about Heartbleed. A lot was written, but I'll highlight the 3 URLs worth reading. Heartbleed The site that broke the news to us all. What Heartbleed Can Teach The OSS Community About Marketing A problem with security is that it is difficult to explain. Here's a case study of doing it right. Please Put OpenSSL Out of Its Misery There was a big call for improving OpenSSL. Poul-Henning Kamp gives a blunt analysis. On a personal note... I think it's a shame OpenBSD's replacement can't be called OpenOpenSSL (literally... the license forbids forks from doing that).
Cisco: top of my list of "difficult to upgrade" things
Heartbleed has reminded me what equipment and products I deal with that are difficult to upgrade. While most people think of DevOps as "rapidly deploying software that your coworkers wrote", it is really about creating a world where we are able to make changes... because change is required to experiment, and innovation requires experimentation... and that means being able to make changes. This includes not just in-house software releases, but all operational changes we do. This includes software and firmware releases we get from vendors. My new(-ish) job at StackExchange has me actually touching hardware instead of living in the virtualized, everything-is-done-for-you, world of Google.
LOPSA-East: If you fly here, don't take the train.
The train station that is at Newark Airport is being repaired and is therefore shut down. The dates of this scheduled maintenance coincide exactly with the conference. Sigh. As a sysadmin, I appreciate the need for scheduled maintenance and appreciate that it was announced in advance. At least this isn't catching us by surprise. If you were planning on flying to Newark Airport, there are 3 ways you can get to the conference: The airport is supplying a shuttlebus to Newark Penn Station (NPS) (not to be confused with New YORK Penn Station). From there take the train. You can take a taxi all the way to the conference, which is expensive.
Getting the courage to do a lightning-talk (LOPSA-East)
LOPSA-East (and many conferences) have a session called "lightning-talks". This is where people do 5-minute talks. The talks range from technical to personal. It's invariably one of the most enjoyable sessions of the conference. You can generally sign up for a 5-minute slot usually right up until the session, though once the space is full it is full. If you have something to say but have been intimidated by the prospect of putting together a 45-minute talk, going through the whole proproposal process, and so on, this is a great way to get your feet wet. The audience is highly receptive to new ideas and new speakers.
Reddit AMA about LOPSA-East
Ask me and the entire planning committee anything.
Time Management training at SpiceWorld Austin, 2014
I'll be doing a time management class at SpiceWorld. Read about my talk and the conference at their website. If you register, use code "LIMONCELLI20" to save 20%. See you there!
Interview with LOPSA-East Keynote: Vish Ishaya
Vish Ishaya will be giving the opening keynote at LOPSA-East this year. I caught up with him to talk about his keynote, OpenStack, and how he got his start in tech. The conference is May 2-3, 2014 in New Brunswick, NJ. If you haven't registered, do it now! Tom Limoncelli: Tell us about your keynote. What should people expect / expect to learn? Vish Ishaya: The keynote will be about OpenStack as well as the unique challenges of running a cloud in the datacenter. Cloud development methodologies mean different approaches to problems. These approaches bring with them a new set of concerns.
Replace Kathleen Sebelius with a sysadmin!
Scientists complain that there are only 2 scientists in congress and how difficult they find it to explain basic science to their peers. What about system administrators? How many people in congress or on the president's cabinet have every had the root or administrator password to systems that other people depend on? Health and Human Services Secretary Kathleen Sebelius announced her resignation and the media has been a mix of claiming she's leaving in disgrace after the failed ACA website launch countered with she stuck it out until it was a success, which redeems her. The truth is, folks, how many of you have launched a website and had it work perfectly the first day?
LISA CFP Deadline Extended to Fri, 4/18!
Whether you are submitting a talk proposal, workshop, tutorial, or research paper, the call for participation submission deadline has been extended to Friday, 4/18! Submit today!
Interview with LOPSA-East Keynote: Elizabeth Krumbach Joseph
Elizabeth Krumbach Joseph will be giving the closing keynote at LOPSA-East this year. I caught up with her to talk about her keynote, source code management, and Star Wars. The conference is May 2-3, 2014 in New Brunswick, NJ. If you haven't registered, do it now! (We'll have an interview with the opening keynote, Vish Ishaya, soon.) Tom Limoncelli: Tell us about your keynote. What should people expect / expect to learn? Elizabeth Krumbach Joseph: Over the past few years there have been a number of high profile incidents and news stories around the subject of women in technology. In my keynote I'll be giving some solid advice for how the technology industry, and each of us, can do a better job of attracting and keeping talent.
How to...
Here's a thought to begin your weekend: How to stop time: kiss. How to travel in time: read. How to escape time: music. How to feel time: write. How to waste time: social media.— Matt Haig (@matthaig1) March 8, 2014
Introducing the Python Time Travel Debugger
Today I'm open sourcing a productivity tool that I've been very excited about: A time-travel extension to the Python Debugger (PDB). Have you ever been using PDB to step through a program and suddenly realize you wish you could jump back in time and know what a variable used to contain? This version of PDB adds the ability to jump back in time to the state of your program as it was in the past. You can examine variables and even continue execution from that point forward (though that is dangerous because it may harm the time space continuum.) How it works: As you know, time is the 4th dimension.
LOPSA-East 2014 is one month away! Register today!
I'll be presenting a few different talks at LOPSA-East, in New Brunswick, NJ, May 2-3, 2014. Tutorials: Introduction to Time Management (half day) Evil Genius 101 (half day) Talks: Sneak peek at my next book: The Practice of Cloud Administration (this is the ONLY conference that will be getting a sneak peek before it is released this September) The Stack at Stack Exchange (how stackexchange.com works) Tom's Top 5 Time Management Tips Hope to see you there! Register today! http://lopsa-east.org/2014/
Velocity Santa Clara best price deals end April 3rd
Good reads, March 2014
A summary of the interesting articles I've found this month. Why Puppet/Chef/Ansible aren't good enough (and we can do better): This is mostly about the Nix package manager and the new linux distro NixOS which is entirely Nix-based down to the bone. I haven't used it yet, but I had to admit this is what I was trying to achieve back in the 1990s with the simple package management system I made... but I didn't go far enough. These people did. I'm looking forward to trying this out. http://dec64.com DEC64 is a new (proposed) floating point format. I fear that most people don't understand how floating point numbers are stored on computers so this will be wasted.
"Evil Genius 101" at LOPSA-East
I'll be teaching my "Evil Genius 101" half-day class at LOPSA-East You want to innovate: deploy new technologies such as configuration management (CfEngine, Puppet, etc.) , set up a wiki, or standardized configurations. Your coworkers don't want change. They like it the way things are. Therefore, they consider you evil. However you aren't evil, you just want to make things better. In this class you will learn how to: Help your coworkers understand and agree to your awesome ideas Convince your manager about anything. Really Turn the most stubborn user into your biggest fan Get others to trust you so they are more easily convinced Deciding which projects to do when you have more projects than time Make decisions based on data and evidence Drive improvements based on a methodology and planning instead of guessing and luck LOPSA-East is a regional sysadmin conference in New Brunswick NJ, May ...
Our newest book: Sneak preview at LOPSA-East
Strata Chalup, Christine Hogan, and I have been working on a new book titled, "The Practice of Cloud Administration". This new book is all new material focused on the design and operation of distributed systems or "cloud" computing. The book is two parts: "Building it" and "Running it". It is the sequel to our enterprise-focused book, "The Practice of System and Network Administration". If you want to see a preview, the only conference where you'll be able to get a sneak peek is at LOPSA-East. Otherwise you'll have to wait until at least September 2014 (we don't have the exact shipping date yet, it could be as late as November).
Isn't it time for an "Intro to Time Management"?
I'll be teaching my "Intro to Time Management for System Administrators" class at LOPSA-East. I haven't taught this class in ages so this is a good opportunity to check it out. The class covers the most important points of my Time Management for System Administrators O'Reilly book. Sysadmins have a time management problem: There are too many projects. Too many interruptions. Too many distractions. This half-day class presents fundamental techniques for eliminating interruptions and distractions so you have more time for projects, prioritization techniques so the projects you do work on have the most impact. It wraps it all into "The Cycle System" which is the easiest and most effective way to juggle all your tasks without dropping any.
Pre-announcement
I'm planning on making a big announcement on Monday. Nothing earth-shattering, but watch this space.
LISA "call for participation" is open: and boy is it different this year!
https://www.usenix.org/conference/lisa14/call-for-participation/ This year the LISA CFP is different both in content and form. This represents a big change for this conference LISA. There is less emphasis on academic talks and instead more emphasis on high-impact, cutting edge talks on what sysadmins need to know about today and in the coming 18-24 months. If you consider the changes over the last few years, soon LISA will be unrecognizable (in a good way) from LISA of the past. This year the focus is on 5 topics: Systems Engineering (Large scale design challenges, Cloud and hybrid cloud deployments, Software Defined Networks (SDN); Virtualization; HA and HPC Clustering; Cost effective, scalable storage; Hadoop/Big Data; Configuration management) Culture (Business communication and capital planning; Continuous delivery and product management; Distributed and remote worker challenges; On-call challenges; Standardization to support automation; Standards and regulatory compliance) Devops (Site reliability engineering; Development frameworks for Ops; Release engineering; API-driven ...
Early Bird Discount Reminder! LOPSA-East SysAdmin Conference, May 2-3 2014, New Brunswick, NJ
Just a reminder to everybody that the Early Bird Discount to LOPSA-East 2014 registrations ends on Sunday, March 23rd at 11:59pm - Register now before it's too late! http://lopsa-east.org/2014/register-for-lopsa-east-14 TALK TO YOU BOSS NOW. SAVE BIG MONEY. LOPSA-East is a regional sysadmin conference in New Brunswick NJ, May 2-3, 2014. Two days of world-class training on a diverse range of topics plus community-selected talks on everything from Active Directory to Code Review for Sys Admins! We have a very exciting lineup of tutorials and talks this year, you can find all of the exciting content at: http://lopsa-east.org/2014/schedule/ You can also take my "Intro to Time Management" half-day class, plus my "Evil Genius 101" half-day class.
Should RelEng also build be responsible for infrastructure?
Someone recently asked me if it was reasonable to expect their RelEng person also be responsible for the load balancing infrastructure and the locally-run virtualization system they have. Sure! Why not! Why not have them also be the product manager, CEO, and the company cafeteria's chief cook? There's something called "division of labor" and you have to draw the line somewhere. Personally I find that line usually gets drawn around skill-set. Sarcasm aside, without knowing the person or the company, I'd have to say no. RelEng and Infrastructure Eng are two different roles. Here's my longer answer. A release engineer is concerned with building a "release".
Should RelEng also be responsible for infrastructure?
Someone recently asked me if it was reasonable to expect their RelEng person also be responsible for the load balancing infrastructure and the locally-run virtualization system they have. Sure! Why not! Why not have them also be the product manager, CEO, and the company cafeteria's chief cook? There's something called "division of labor" and you have to draw the line somewhere. Personally I find that line usually gets drawn around skill-set. Sarcasm aside, without knowing the person or the company, I'd have to say no. RelEng and Infrastructure Eng are two different roles. Here's my longer answer. A release engineer is concerned with building a "release".
Yes, you really can work from HEAD
There is often a debate between software developers about whether it is best to branch software, do development, then merge back into HEAD, or just work from HEAD. Jez Humble and others make the claim that the latter is better. If you make your changes in "small batches" this works. In fact, it works better than branching. When you merge your branch back in the bigger the merge, the more likely the merge will introduce problems. Jez recently tweeted: which caused a bit of debate between various twitterers (tweeters? twits?) Jez co-wrote the definitive book on the subject, so he has a lot of authority in this area.
Thanks, Cascadia IT 2014!
Hey Seattle! This year's Cascadia IT conference has been great! Congrats to everyone that helped put it together and attended. I look forward to seeing you all next year! (And since LISA is also in Seattle this year, I hope to see you all in November!)
LOPSA-East 2014: Full of DevOps and other great stuff!
The LOPSA-East talks schedule was published yesterday. It is broken into 4 tracks: DevOps, Infrastructure, Career Development and "General". I'm impressed! The DevOps Track has a lot of good culture talks, best practices, and big names like Mandi Walls. The Infrastructure Track has case studies as well as talks about how to do it yourself. The Professional/Career Talks Track has a mix of sessions for both junior and senior people. The "General" Track has a huge diversity: network (the hardware kind), networking (the community kind), "lightning talks" and more. There's also a lot of excellent training classes, which I'll write about in another post.
10% off Cascadia IT registration
If you are taking any of my classes (or not) you can use the discount code "CasIT14-Presenter-guest" to get an additional 10% discount on a 2-day registration for the conference. The code expires at midnight, the evening of Saturday, 1 MAR, 2014. You don't have to be taking any of my classes to use the code.
Jumpy, nervous, jittery user interfaces reduce "user trust"
What's with the trend of making user interfaces that hide until you mouse over them and then they spring out at you? How did every darn company hop on this trend at the same time? Is there a name for this school of design? Was there a trendy book that I missed? Is there some UI blog encouraging this? For example look at the new Gmail editor. To find half the functions you need to be smart enough or lucky enough to move the mouse over the right part of the editor for those functions to appear. Microsoft, Facebook, and all the big names are just as guilty.
The FreeBSD Journal: Read it even if you don't use FreeBSD
The first issue of The FreeBSD Journal has finally shipped! I got to read an early draft of the first issue and I was quite impressed by the content. It was a great way to learn what's new and interesting with FreeBSD plus read extended articles about specific FreeBSD technologies such as ZFS, DTrace and more. Even if you don't use FreeBSD, this is a great way to learn about Unix in general and expand your knowledge of advanced computing technologies. The Journal is a brand new, professionally produced, on-line magazine available from the various app stores, including Apple iTunes, Google Play, and Amazon Kindle.
Anti-Pattern: "The big project" that never ships
I was reminded of this excellent blog post by Leon Fayer of OmniTI. As software developers, we often think our job is to develop software, but, really, that is just the means to an end, and the end is to empower business to reach their goals. Your code may be elegant, but if it doesn't meet the objectives (be they time or business) it doesn't f*ing work. Likewise I've seen sysadmin projects that spent so much time in the planning stage that they never were going to ship unless someone stood up and said, "we've planned enough. I'm going to start coding whether you like it or not".
LOPSA-East schedule published!
The schedule of talks and tutorials has been published! Talks: http://lopsa-east.org/2014/lopsa-east-14-talks/ Tutorials: http://lopsa-east.org/2014/lopsa-east-14-training-schedule/ I'm glad to announce that I'll be teaching 2 tutorials and giving 2 talks: "Tom's Top 5 Time Management Tips" and "Book Preview: The Practice of Cloud Administration". My tutorials include "Evil Genius 101", which was standing-room only last year plus "Intro to Time Management for System Administrators" which hasn't been taught at LOPSA-East in quite a few years. Registration opens soon. I look forward to seeing you at this year's conference! Tom
Seattle: CascadiaIT'14 keynote: Æleen Frisch
I'm excited to see that long-time sysadmin and author Æleen Frisch will be the keynote of this year's Cascadia IT conference, Seattle, March 7-8! If you don't recognize her name, check your bookshelf. You probably have a few of her books! http://casitconf.org/ There is still time to register. There are still a few seats left in the tutorials "Evil Genius 101" and "Team Time Management & Collaboration". Don't wait, register today! There are also a dozens other excellent tutorials and talks. Plus, there are a lot of networking opportunities. Hope to see you there! http://casitconf.org/
How not to use Cron
A friend of mine told me of a situation where a cron job took longer to run than usual. As a result the next instance of the job started running and now they had two cronjobs running at once. The result was garbled data and an outage. The problem is that they were using the wrong tool. Cron is good for simple tasks that run rarely. It isn't even good at that. It has no console, no dashboard, no dependency system, no API, no built-in way to have machines run at random times, and its a pain to monitor. All of these issues are solved by CI systems like Jenkins (free), TeamCity (commercial), or any of a zillion other similar systems.
Tool Building Versus Automation
I make a distinction between tool building and automation. Tool building improves a manual task so that it can be done better. Automation eliminates the task. A process is automated when a person does not have to do it any more. Once a process is automated a system administrator's role changes from doing the task to maintaining the automation. There is a discussion on Snopes about this photo. It looks like the machine magically picks and places bricks. Sadly it does not. If you watch this video, you see that it requires people to select and place the bricks. It is a better tool.
Sysadmins that can't script have a choice.
Scripting is becoming more and more important. With everything from computers to networks going virtual, installation is becoming an API call, not a "walk to a rack/desk/whatever and plug it in" call. If you know how to script, you can automate those things. In a few years I can't imagine a system administrator being able to keep their job and/or compete with others if they can't script. There is an exception, of course: People that do desktop/laptop system administration and general in-office IT service. However those jobs are turning more and more into the equivalent of working at a mobile phone store: helping people is basic equipment problems and customer support.
Time Management Tip: The Oscars
Do you marathon through entire seasons of TV shows in a weekend? You might want to check this out. AMC has an event where you can watch all the "best picture" nominations. It is pretty intense but awesome. On the first Saturday you watch 4 of them in a row in a 10-hour session. On the second Saturday you watch the other 5 in a 12-hour session. The following say is the award show. Watching the awards when you've seen 9 of the most nominated films is a different experience. Some of the benefits: If you've been too busy to get to the theaters all year, this is a great way to consolidate a lot of what you missed into 2 days.
Seattle: CascadiaIT'14 hotel discount ends Feb 8!
The hotel discount ends on Feb 8th so book your room as soon as possible! CascadiaIT is an awesome regional conference for sysadmins and devops. If you look at the schedule you're sure to see talks and tutorials you won't want to miss. I'll be teaching "Evil Genius 101" (on how to influence your boss and team) and " Team Time Management & Collaboration". On Saturday I'll be giving a talk about how StackExchange works. While this is a "regional conference" it is drawing people from all over the West coast, Pacific North West, and more. You should be there too.
LOPSA NJ Chapter: Feb meeting is a dinner meetup
LOPSA NJ's February meeting is at two different restaurants, Northern NJ and Southern-ish NJ. The planned discussion topic is "What are some of the most challenging problems that have come up in the last 24 months?" Date: Thursday, February 6th, 2014 Time: 7:00pm Location details: http://www.lopsanj.org/archives/2014/01/lopsa-nj-cluster-meeting-2014.html In the past these "cluster meetings" have been really fun, full of interesting war stories as well as technical info. If you are in the area, I hope see see you there!
We're hiring at StackExchange! SRE with Networking focus
Come join the team that runs ServerFault, StackOverflow, and over 100 other Q&A websites plus "Careers 2.0", the most awesome job site around. We have a great manager (I'm not just saying that because he reads my blog) and cool coworkers!
Seattle: CascadiaIT'14 Registration is OPEN
Tell your friends, tell your neighbors, tell your friends' neighbors and your neighbors' friends! http://casitconf.org/casitconf14/registration-is-now-open/ I'll be teaching "Evil Genius 101" and "Team Time Management & Collaboration" half-day tutorials. Plus I'll be giving a talk on Saturday about "The Stack at StackExchange". The conference is March 7-8, 2014 in Seattle, WA. While it is a regional conference, people come from all over. Hope to see you there!
LOPSA-East: CFP extended to Jan 31!
The LOPSA-East "call for participation" has extended the submission deadline to Fri, Jan 31. You have an extra week to send in your proposed talks. In particular, anything related to cutting edge operational issues ("devops") and new technology (wha t sysadmins should know about "new" things like SSDs, etc). Personally I'd like to see more "culture" talks. If you've done an awesome project in the last year and would like to talk about it, write it up and submit it soon! Interested in presenting? View the CFP: http://lopsa-east.org/2014/ Interested in helping? Email [email protected] Interested in attending? Save May 2-3, 2014 on your calendar!
LOPSA-East - CFP deadline Jan 22, 2014
The call for participation deadline is Wednesday, January 22nd, 2014. LOPSA-East is looking for talks on system-administration related topics especially advanced techniques, DevOps stuff, and etc. I particularly enjoy hearing about project successes... if you have done something exciting where you work, propose a talk about it. That how I got my start! The full CFP is here: http://lopsa-east.org/2014/ If you haven't heard of LOPSA-East, it is our regional Linux/Sysadmin conference; we expect about 150 people. People come from all over the east coast (and often Europe!) . The event is May 2 - 3, 2014, in lovely New Brunswick, NJ, USA.
awk. How I missed you.
awk </dev/null \ 'END { for (i=0; i <13 ;i++) \ { printf("%02d:00-%02d:30\n%02d:30-%02d:00\n", i, i, i, i+1) }}' The output was pasted into a spreadsheet. I don't think this is how the creators of the original spreadsheet imaged things.
DrupalCamp NJ, Princeton, Feb 1, 2014
Feb 1 will be the 3rd annual DrupalCamp NJ on the campus of Princeton University http://www.drupalcampnj.org/. This is the first year with a keynote speaker - Brian Kernighan! Tickets are only $25, which includes coffee, lunch, and an after-party. In addition, the day prior on Jan 31, there are 4 low-cost, full-day training sessions http://www.drupalcampnj.org/training.
bash: Restart bash if old version detected
I write a lot of small bash scripts. Many of them have to run on MacOS as well as FreeBSD and Linux. Sadly MacOS comes with a bash 3.x which doesn't have many of the cooler features of bash 4.x. Recently I wanted to use read's "-i" option, which doesn't exist in bash 3.x. My Mac does have bash 4.x but it is in /opt/local/bin because I install it using MacPorts. I didn't want to list anything but "#!/bin/bash" on the first line because the script has to work on other platforms and on other people's machines. " #!/opt/local/bin/bash" would have worked for me on my Mac but not on my Linux boxes, FreeBSD boxes, or friend's machines.
SSH debugging sucks
How much human productivity is lost every day due to the horrible debugging messages in SSH? I bet it is thousands of hours world-wide. It isn't just sysadmins: programmers, web developers, and many non-technical users are frustrated by this. I'm pretty good at debugging ssh authentication problems. The sad fact is that most of my methodology involves ignoring the debug messages and just "knowing" what to check. That's a sad state of affairs. The debug messages for "ssh -v" should look like this: HELLO! I AM TRYING TO LOG IN. I'VE TOLD THE SERVER I CAN USE (method,method,method). I AM NOW TRYING TO LOG IN VIA (method).
January LOPSA NJ Chapter Meeting Announcement
The January speaker will be Tom Limoncelli on the topic of encouraging technical leadership within your team. LOPSA-NJ Thursday January 2, 2014 7PM Topic: Heroes of Technical Leadership Speaker: Thomas A. Limoncelli, Stack Exchange Date: Thursday, January 2, 2014 Time: 7:00pm (social), 7:30pm (discussion) Location: Lawrenceville, NJ (near Princeton) Pizza and Soda being brought to you by: INetU Please RSVP at http://www.lopsanj.org/rsvp Remember in October when Tom did a draft of his keynote talk from SpiceWorks? Thanks to the feedback it was completely rewritten. Come see what Tom actually presented at the SpiceWorks conference: A good system administrator does their job.
Avoiding the "cost center" mentality.
CIO Magazine 2013 State of the CIO Survey lists five stages of an IT organization from a business stakeholders' view: Cost center Service Provider IT Partner Business Peer Business Game Changer I don't think an IT department needs to start at one phase and work their way forward. However, I do think this list exemplifies the categories of IT organizations I've dealt with. If you think about the book "The Phoenix Project", it really is about how to leap ahead to be the last (best) category. I think that many people don't even know that anything other than "cost center" is a possibility.
How does Bitcoin work?
A brilliant description. It's certainly the best explanation I've ever seen. For the first time I actually understood how it works.http://www.youtube.com/watch?v=Lx9zgZCMqXE
Today I applied to get my LOPSA LPR
https://lopsa.org/LPR You should too. The LOPSA Professional Recognition Program (LPR) is not a certification. It is a recognition that the person in question met or exceeded the standards for professional practice. In particular, it certifies that the person has agreed to abide by the LOPSA Code of Ethics and works to keep their skills current in the last year. I've always been an advocate for some kind of program that would raise the bar among system administrators, encourage professionalism, and spread the word about the Code of Ethics. I'm glad to see LOPSA giving this a try and I think everyone should support it.
Ask Slashdot: Application Security Non-existent, Boss Doesn't Care. What To Do?
I'm really sick and tired of Slashdot doing posts like this, but it isn't slashdots fault. It's our industry's fault. Here's the question: "I am a senior engineer and software architect at a fortune 500 company and manage a brand (website + mobile apps) that is a household name for anyone with kids. This year we migrated to a new technology platform including server hosting and application framework. I was brought in towards the end of the migration and overall it's been a smooth transition from the users' perspective. However it's a security nightmare for sysadmins (which is all outsourced) and a ripe target for any hacker with minimal skills.
LOPSA-East 2014: Call for Participation
[forwarded from Evan Pettrey, this year's LOPSA-East chair] Greetings! LOPSA-East is pleased to announce that we have released our Call for Participation for our 2014 conference. Everybody with a passion for technology and a willingness to share with others in our industry are encouraged to submit! Full details of the CFP can be found on our website at: http://lopsa-east.org/2014/ Important Dates: Deadline for all Submissions - Wednesday, January 22nd, 2014 (midnight EST) Decisions Sent to All Submitters - Monday, February 3rd, 2014 Schedule Published - Monday, February 10th, 2014 Registration Opens - Friday, February 14th, 2014 LOPSA-East '14 Conference - Friday, May 2nd - Saturday, May 3rd We look forward to seeing your submissions!
Tomorrow is Giving Tuesday: Here are the organizations I support (and you should too)
You know Black Friday and Cyber Monday... did you know that tomorrow (Dec 3) is "Giving Tuesday"? Many charities receive most of their donations in December as people rush to donate before the tax year is over. These donations determine if in 2014 they'll be able to grow or will they have to cut back. I'd like to highlight three charities that I think are having a huge impact on our world and encourage you to donate too: USENIX Annual Fund. You probably think of USENIX as the organization that hosts the LISA conference. It is so much more. However what I'd like to point out is that they are on the cutting edge of keeping academic publications "open access".
Deadline for Cascadia IT talk proposals extended to Dec 9:
The proposal deadline for LOPSA's Cascadia IT conference has been extended to 9 DEC. http://casitconf.org/casitconf14/call-for-proposals/
Stop monitoring whether or not your service is up!
99 percent of all monitoring that I see being done is done wrong. Most people think of monitoring like this: Step 1: Something goes down. Step 2: I am alerted. Step 3: I fix the problem as fast as I can. Step 4: I get a pat on the back if I was able to fix it "really fast". (i.e. faster than the RTO) If that's how you think of monitoring, then you are ALWAYS going to have down time. You've got down time "baked into" your process! Here's how we should think about monitoring: Step 1: I get an alert that something is "a bit off"; something that needs to be fixed or else there will be an outage.
Evi Nemeth Update
http://www.katc.com/news/170-days-still-no-sign-of-the-nina
Summary of "Evil Genius 101" Tutorial
Ben Cotton wrote up a summary of my Evil Genius 101 tutorial: https://www.usenix.org/blog/evil-genius-101 Thanks for the great summary, Ben! (Ben Blogs at FunnelFiasco)
How to provide infinite disk storage
A user recently asked for a lot of disk space. Not just a lot of disk space, but growing at an astounding rate per month. (Not big for some places, but bigger than my current employer was used to providing). It was an archive that would start large and grow in leaps and bounds. It had to be actual disk (not tape or other off-line technology) because the data would be accessed constantly. He joked that what he really wanted was infinite disk space. I replied, "I can give you infinite storage." and I wasn't joking. He told me to prove it so I explained: Your data will start large and grow quickly.
Open office hours at LISA
This just in... I'll be having office hours on Thursday from 2-3:30pm at LISA. Stop by for one-on-one time management counseling. It isn't listed yet on the website but will be soon: https://www.usenix.org/conference/lisa13/hack-space
"Evil Genius 101" class via livestream
If you can't make it to LISA this year but want to see my devops-tastic, "Evil Genius 101" class, you can buy the livestream: https://www.usenix.org/conference/lisa13/video/usenix-training-video-stream-half-day-lisa-13-evil-genius-101 You can watch many different LISA presentations livestreamed here: https://www.usenix.org/conference/lisa13/live-streaming
Tom @ Usenix LISA 2013, Washington DC, Nov 3-8, 2013
Tom will be teaching 2 tutorials, doing a book signing, and including the all-new Evil Genius 101 half-day class. Tuesday AM: Half-day tutorial: Advanced Time Management: Team Efficiency Updated! Tuesday PM: Half-day tutorial: Evil Genius 101 New! Thursday, 1-1:30PM: Book Signing in Exhibit Hall C Thursday, 2-3:30PM: "Time Management Office Hours" (one-on-one time management counseling) New! Friday, 9-10:30AM: Guru Session "Time Management for Sysadmins" (Harding Room) Fri, 01 Nov 2013 16:00:00 UTC
Hospitals are the Mainframe of the medical industry
Hospitals are the mainframe of the medical industry. Computers used to be rare and expensive. Every bit, every CPU cycle needed to be carefully groomed, petted, and softly whispered sweet things to, protected and managed. The best way to do this was to make one big computer, the central mainframe, and have everyone worship it like a god, accessed only through 24x80 text-only glass video tubes. Then came PCs. PCs are so cheap you can waste CPU cycles on silly things like... ease-of-use feature, applications that enable communication between people, graphical user interfaces, games, surfing the web, etc. Medical equipment used to be rare and expensive.
Is Cisco finally understanding that SDN is real?
I've been talking about SDN and OpenFlow for a while. It is slowly becoming a reality. This article is one of the warning signs: Here's What Happened When Cisco Lost A $1 Billion Deal With Amazon Let me put the financial impact into more down-to-earth terms. How does Cisco make money? Well, you buy a switch or router and that's good. Then you buy more and that's good too. Then you grow so large that the routing table has gotten too big to be calculated by the CPU/RAM on all the old equipment. Therefore to buy the next device you also have to buy upgrades for all previous devices.
Improved cycle time? What about this technique from the 1990s?
One of the DevOps goals you often hear about is "improved cycle time" for releases. What that means, basically, is speeding up the time from when a developer writes a line of code to when it is in production. The opposite would be writing code for a release that doesn't ship for a year or so (common in shrink-wrapped software). You often hear about teams bring their cycle time from months to days, from days to hours. Etsy brags that they've gotten it down to minutes. The benefits to reducing cycle time are well documented. Well I have a technique that reduces it to a cycle time that is faster than minutes.
Greetings SpiceWorld 2013 Attendees!
Thanks for coming to SpiceWorld and attending my session! I'll have the slides up on this site totomorrow. If you aren't here at the conference you can sign up for the livestream here: http://www.spiceworks.com/begreat
This site now available via IPv6
My hosting company has enrolled this site in their beta for IPv6. All I had to do was ask. If you have a hosted site, I highly recommend that you open a ticket asking for your site to be available via IPv6. If they don't offer it, ask for an arrival date and keep them to it. Enroll in any beta tests and so on. The more demand hosting companies see, the better.
Usenix LISA schedule updated
The schedule of Usenix LISA sessions has been updated with icons that represents categories: DevOps, Cloud System Administration, Coding, Linux, Soft Skills and Women in Advanced Computing. Check it out.
Interested in doing a lightning talk at LISA?
Talks are no more than 5 minutes with no AV (no slides, no videos, no projector). They can be on any topic though we prefer topics related to System Administration. Please keep the content 'professional' in tone. Sign up here. With only 5 minutes to give the talk it is important that you cut to the chase. I've seen some people make the mistake of spending a lot of time on something inconsequential like how to install the software they're talking about (and the talk wasn't about installation techniques). The best talks I've seen start with a solid explanation of the problem (in terms of the pain being caused) then explain the solution.
Evi Nemeth news...
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11140086
LOPSA-East 14: Call for Volunteers
Evan Pettrey just emailed out the LOPSA-East 14: Call for Volunteers. The next conference is May 2-3, 2014. Mark your calendar and join the volunteer team. It is a great way to get involved in the community and meet new people! Evan's complete letter after the bump.
Starting a new habit by linking it to an existing one
I wrote a lot about habits in Time Management For System Administration. If there is something you need to do a lot, turning it into a habit means you are more likely to do it and less mental effort is required when doing it. To paraphrase the Java (programming language) slogan "Write once, run everywhere": Habits let you "Think once, do always." One tip I realized after finishing the book was the power of linking one habit to another. If you need to add a new habit, linking it to an existing habit helps develop that habit. Here's an example: Problem: I forget to put my wedding ring back on after I shower.
Glove and Boots mirror my NYC post!
You may recall Tom's guide for Tourists Visiting NYC (special Sept 11th anniversary edition). Glove and Boots, one of my... no... my favorite YouTube channels has just done a video that is surprisingly similar. Of course, theirs is a lot funnier (and leaves out the Sept 11 Memorial part, understandable). Check it out:
Wrist-watch cell phones
A friend commented to me: Are we really so lazy that we cannot "take the time" to remove a 4 inch device from our pocket to check the next meeting or see who's calling?" A new level of convenience turns something old into something new. Remember the move from dial-up to cable/dsl wasn't so much about speed but the "always on" capability. Wanted to look something up? Your computer was already connected. Friction-free use of the internet made the internet feel more useful. Today can you imagine a 30-60 second delay any time you want to use the internet?
Video: Why Attend LISA '13?
Usenix has released a video with a number of Usenix attendees explaining why they go to LISA. I make a brief appearance. https://www.youtube.com/watch?v=NBJkB8mp3V4
Book signing at LISA
People often tell me they wanted to ask me to sign a copy of my book but "didn't bring it to the conference because they didn't think I'd want to be bothered". The truth is that (nearly) all authors love to be asked to sign their book. I'll be doing a book signing on Thursday at 1pm in the exhibit hall. Other authors such as Mark Burgess and Diego Zamboni will be there to sign books too. I hope to see you there! Tom
Seating at my Usenix LISA tutorials is limited.
Register ASAP before all the seats are gone. This year my classes are all on Tuesday which is the most popular day to attend. That means the seats will fill up even faster. I've totally revamped my "Advanced Time Management: Team Efficiency" tutorial and my entirely new "Evil Genius 101" class is chock full of DevOps goodness plus a lot of traditional "how to sell your evil plan to management" badness. Register today: https://www.usenix.org/conference/lisa13
IPv6 Flashcards
In IPv4 there are a number of things that every sysadmin knows. I bet you recognize the following: 127.0.0.1 10.0.0.0/8 192.168.0.1 /24 /26 /32 255.255.255.255 255.255.255.0 You probably didn't even have to think hard about most of those. So what are the equivalents in IPv6? I don't mean the direct translations, but what is the list of terms and numbers that sysadmins should know? I recently sat down and came up with such a list. I listed things that Unix and Windows sysadmins should know. WAN/LAN network administrators need to know a lot, lot, more. This just covers common knowledge, a lot like the IPv4 list above.
Usenix LISA Early Bird deadline is 1 week away!
The early bird registration and hotel reservation deadlines are both next Tuesday, October 15, 2013. Remember that airfare is a lot cheaper if you book 21 days in advance. If you would like to spread the word about LISA, consider printing this one page flyer and giving it to coworkers. Register today: https://www.usenix.org/conference/lisa13 There are also Facebook, YouTube, Google+, LinkedIn, Lanyrd, and Twitter doohickies to like, plus, follow and so on. SEATS ARE LIMITED for my tutorials. Please sign up ASAP!
Three problems with the DevOps community
People that complain that the enterprise world doesn't get DevOps but don't participate in enterprise conferences. Lack of a "sound bite" definition of DevOps; leads to confusion. I was recently told "DevOps means developers have to carry pagers... that's why our developers don't want anything to do with it." If that's the definition that is getting out, we're in trouble. Engineers thinking that "if something is good, it doesn't need marketing". Tell that to the many inventions of Nikola Tesla that never got turned into products. The "build a better mouse trap and people will beat a path to your door" myth was debunked years ago.
Guest Post: Jennifer Joy's PuppetConf 2013 attendee report
[As you may recall, a few months ago PuppetLabs gave me a few free admission tickets to give away. One of the recipients was Jennifer Joy, who wrote this conference report. -Tom] Conference Report: PuppetConf 2013, by Jennifer Joy It has been a long time since I was in the sysadmin space. Any further clarification would reveal I have been through far too many iterations of technology (I'm pretty sure you can date sysadmins with a swift core sample and counting the rings caused by each swing between centralized and decentralized architectures). The problem of managing large numbers of systems, especially diverse systems, is not new. Having been out of the systems game for over 10 years (ok, now you know, but I'm not telling you when I started!)
Central PA Open Source Conference (CPOSC), October 19, Lancaster, PA
The Central PA Open Source Conference (CPOSC) is a small, low-cost, one-day conference about all things Open Source. It was started in 2008 by a few of the members of the Central PA Linux User Group and the Central PA Ruby Meetup. For more info: http://lanyrd.com/2013/cposc/ http://cposc.org/
Tom @ LOPSA-NJ, Thu Oct 3, 2012, Lawrenceville, NJ (near Princeton)
I'll be speaking at LOPSA-NJ's October meeting. The topic will be "Heroes of IT". It is a rehearsal of a talk I'll be giving at another conference later this month. Full information here: http://www.lopsanj.org
"Soft Skills" instruction at Usenix LISA
The best technical skills are useless if you can't understand the people you work with, communicate with them, and manage your own time. Here is a list of "soft skills" presentations at LISA this year. No, they won't turn you into a manager.
I'll be the speaker at LOPSA NJ Chapter meeting this Thursday.
I'll be rehearsing a talk on "IT Heroes" which I'm writing for a conference later this month. For more information visit the LOPSA NJ website. The meeting is near Princeton, NJ. Hope to see you there!
Assessing Progress with "DevOps Look-for's"
Teams working through The Three Ways need an unbiased way to judge their progress. That is, how do you know "Are we there yet?" Like any journey there are milestones. I call these "look-for's". As in, these are the things to "look for" to help you determine how a group is proceeding on their journey. Since there are 3 "ways" one would expect there to be 4 milestones, the "starting off point" plus a milestone marking the completion of each "Way". I add an additional milestone part way through The First Way. There is an obvious sequence point in the middle of The First Way where a team goes from total chaos to managed chaos.
How to build a private cloud? Do you run one?
I got email the other day asking for advice about building a private cloud. There are plenty of vendors out there that want to help you. There are also a lot of open source solutions. I'm not an expert in all of them, so I can't really give a lot of advice. However there is an impressive number of presentations about building and/or running private and public clouds at Usenix LISA this year. You should consider attending this conference. But here's a little secret about Usenix LISA. The presentations are great, but by just hanging out in the hallway chatting with people (the unofficial "hallway track") you'll get the "inside scoop" that most presentations won't tell you.
Women in System Administration
I keep reading all these horror stories about women being treated badly at technical conferences. I haven't seen a lot of positive stories. I think the conferences that are doing a good job need some recognition. That's why I've made a list of presentations being given by women at the next Usenix LISA conference. Conferences that are doing a good job of inclusion need to be highlighted. This year the conference is in Washington D.C., Nov 3-8.
Where did you learn to debug like that?
Debugging is an important system administration skill. I didn't realize there was so much to know about debugging until I worked at a computer repair shop in high school. PC repair has basically two techniques: Technique 1: remove all the cards and add them back until the system doesn't work. The last part you added was the problem. Technique 2: Remove cards one by one until the system works. The last part you removed was the problem. In system administration the technique is more about coming up with a mental model of how the system is supposed to work and testing each component to see that it is working that way.
Tom @ ACM Webinar on IPv6 and Security
I'll be introducing the speaker Dr. Johannes B. Ullrich of SANS Technology Institute when he talks about "The Security Impact of IPv6". I'll also be moderating the Q&A at the end. You can watch live via the web for free. Resister at http://bit.ly/16qG1Bc Wednesday, September 25, 2013, at noon ET/11 am CT/10 am MT/9 am PT/4 pm GMT
ACM Webinar on IPv6 and Security (Today)
I'll be the moderator of today's ACM Learning Webinar. The topic is "Security Implications of IPv6", the speaker is Dr. Johannes B. Ullrich, SANS Technology Institute. I'll be facilitating the question and answer section at the end. The event is free. To register for this free event, click here. This event is TODAY noon ET/11 am CT/10 am MT/9 am PT/4 pm GMT
System administrators and programming
Someone asked me the other day if I had a "secret of my success". They didn't believe that I got this far on my good looks. (ha ha ha). For most of my career I've been on teams of people where some knew how to code and others didn't. The ones that could code were significantly more productive than the others. Currently I do most of my programming in Python and BASH. There is an excellent full-day tutorial on Python at this year's LISA. There are also full-day tutorials on Puppet, Chef, BASH Shell Scripting ("the command line" is more than just typing commands, eh?)
DevOps at LISA
A shout out to the conference planning committee of Usenix LISA this year. Narayan and Skaar did a great job! The amount of DevOps content is unbelievable. All 6 days have DevOps content that I want to attend from 9am to 5pm. It is going to run me ragged. I've put together a list of all the DevOps content I found in the program. Click here for my list. This year the conference is in Washington D.C., Nov 3-8.
Grants for Women attending Usenix LISA, Sept 30 deadline
Grants are available for women that want to attend Usenix LISA, in Washington D.C., Nov 3-8. This year the LISA '13 Grants for Women are Sponsored by Google. Five women will be selected from the applicants to receive $500 US to apply toward travel/accommodation costs. Apply today! (Sept. 30 deadline) The first time I ever attended a Usenix conference was on a student grant. If I recall correctly I received $80 for round-trip train fare between NJ and Washington D.C. As a student it felt like a million dollars. That was a long time ago.
Usenix LISA 2013: DevOps in every timeslot
There is a devops-related talk in every hour of this year's Usenix LISA conference. Usenix LISA Is a general conference with many tracks going on at any time. A little analysis finds there is always at least one DevOps related talk (usually more than one). This is very impressive. The problem, however, is that many of the talk titles don't make this clear. No worries, I've done the research for you. [I apologize in advance for any typo or errors. Please report any problems in the comments. The conference website has the latest information. Other lists of presentations: Programming, Unix/Linux administration technical skills, Cloud Computing, and Women at Usenix LISA.]
Usenix LISA 2013: Technical Skill-building for Linux/Unix admins
If you are an junior Linux/Unix sysadmin looking to advance your technical skills, here is a list of talks, workshops, and tutorials that you should attend at Usenix LISA 2013. These are skill-building, technical presentations. I only made exceptions for a few "soft topics" talks only if they are for junior sysadmins looking to advance their careers. [I apologize in advance for any typo or errors. Please report any problems in the comments. The conference website has the latest information. Other lists of presentations: DevOps, Programming, Unix/Linux administration technical skills, Cloud Computing, and Women at Usenix LISA.]
Usenix LISA 2013 Cloud system administration presentations
If you run private or public clouds (or want to) here is a list of talks, workshops, and tutorials that you should attend at Usenix LISA 2013. [I apologize in advance for any typo or errors. Please report any problems in the comments. The conference website has the latest information. Other lists of presentations: DevOps, Programming, Unix/Linux administration technical skills, and Women at Usenix LISA.]
Usenix LISA 2013: Women in Advanced Computing
This year's Usenix LISA conference has two exciting events about Women and Computing: Sunday, Nov 3, 2013: 9am-5pm: Workshop: Women in Advanced Computing (WiAC): Recognizing and Overcoming Bias-Ways to Make Your Workplace More Successful and Welcoming Leslie Hawthorn, Red Hat; Sheeri Cabral, Mozilla Format: Half Day Workshop (mixture of presentations and discussion) Thursday, Nov 7, 2013: 11:00 a.m.-12:30 p.m. Panel: Women in Advanced Computing Moderator: Rikki Endsley, USENIX Association; Panelists: Amy Rich, Mozilla Corporation; Deanna McNeil, Learning Tree International; Amy Forinash Format: Panel Participation by women at this year's conference is impressive. Here is a list of talks (I may be missing some, I'm going by first name which is an imperfect algorithm.)
Usenix LISA 2013: Learn to code
If you want to learn to program better, Usenix LISA 2013 has a number of excellent presentations. Usenix LISA 2013 Presentations that teach coding: Sunday, Nov 3, 2013: PowerShell Fundamentals Building Your PowerShell Toolkit Monday, Nov 4, 2013: Core Skills: Scripting for Automation Advanced Shell Programming Wednesday, Nov 6, 2013: Introduction to Chef The Python Programming Language Introduction to Puppet Enterprise Other lists of presentations: DevOps, Unix/Linux administration technical skills, Cloud Computing, and Women at Usenix LISA. [I apologize in advance for any typo or errors. Please report any problems in the comments. The conference website has the latest information.]
Warning sign that your API sucks
People say things like, "Can you just send me a copy of data?" If people are taking your entire database as a CSV file and processing it themselves, your API sucks. (Overheard at an ACM meeting today)
Eiji Toyoda, Promoter of the Toyota Way, Dies at 100
I wonder if he know how much influence he had on DevOps culture. The Three Ways of DevOps are essentially The Toyota Way applied to system administration. Eiji Toyoda, Promoter of the Toyota Way and Engineer of Its Growth, Dies at 100
Tom's guide for Tourists Visiting NYC (special Sept 11th anniversary edition)
Welcome to our fine city! Some say its the greatest city in the world. We love tourists and we want you to visit. NYC has some of the finest theater, museums, shopping, history and dining. I know NYC has a reputation for being unsafe but its actually one of the safest places for tourists to visit. Which brings me to Tom's 4 point guide to visiting NYC: Point 1: Dine well. Say away from the following restaurants: Applebee's, Olive Garden, Hard Rock Cafe, Burger King and McDonalds. Seriously, folks! You are in NYC! Eat someplace you can't find in your own town.
Usenix LISA overlaps election day!
Usenix LISA is early this year. This means two things: It isn't overlapping the December holiday rush (yeah!) but it overlaps with election day (boo!) . New Jersey has an important election this year. I don't want to miss it. Therefore I'm sending away for my New Jersey Application For Vote by Mail Ballot right away. In all states you can vote by Absentee Ballot but you can't do it "same day". You have to write in to apply well in advance and mail it in (depending on the state) far in advance of the real election day. Information on how to do this in your state is available online.
Google StreetView visits my office... literally!
One of the enticements to work at Stack Exchange was that I would be given my own office and it would have a door that closes. What a luxury! Was I special? No, our CEO believes that all engineers should have private offices so they can have peace and quiet when they're trying to get work done. Not that we're unsocial. We use chat rooms and video conferences constantly. Considering that half my team works remotely, we'd be communicating that way anyhow. Shortly after I started my new job I went away for 2 weeks to get married and go on a honeymoon in Maui.
LOPSA NYC: Etsy: Feature Flagging your Infrastructure for Fun and Profit
The first part of the meeting will be about Etsy's deployment infrastructure. The second half of the meeting will be a chance to discuss the talk, brain storm future topic ideas, and hopefully get more presenters. Date? Tuesday, September 10, 2013 When? 7:00 pm Where? 120 West 45th St, 39th Floor, New York, NY 10036 For more info: http://www.lopsa-nyc.org/content/feature-flag
The team I'm on at Stack Exchange is hiring
Here's the job advert. We're looking for someone who will be heavy on the developer side of SRE. If you are interested in OpenTSDB, you'll be very interested in this position. Stack Exchange is the company behind ServerFault.com, StackOverflow.com, and 104 other StackExchange.com sites. I joined the company recently and I really love it here. Check out the advert!
LaTeX dvipdf tip: -dFirstPage/-dLastPage doesn't work
Spoiler alert: it works with Ghostscript 9.09 but not Ghostscript 9.06. I'm writing this mostly to vent. Here's the story. Most of my books are written using the LaTeX formatting system. The latex command outputs a .dvi file, and there are programs that translate dvi to PostScript, pdf, HP LaserJet language, and so on. I'm writing a script that generates one PDF file for each chapter. You'd think this would be easy, and it mostly is. There's some great advice here and here. (Both links are to Stack Exchange sites, btw) My original plan was to call dvipdf then re-process the PDF using pdftk (The PDF ToolKit) or Ghostscript.
Velocity NYC: Stack Exchange will be presenting!
Hey all my NYC peeps! Velocity comes to NYC in October. That's just a few weeks away! My awesome coworkers Steven, Nick, and George will be giving a talk about how ServerFault.com/StackOverflow.com works. It will also be the first public talk about "SE Status" our dashboard. Register today and I'll see you there!
Usenix LISA "Build A Cloud Day", Fri, Nov 8, 2013 in DC (no charge!)
Considering all the security issues raised this year, isn't it time you built a private cloud? Build a Cloud Day will be dedicated to teaching users how to build and manage a cloud computing environment using free and open source software. The program is designed to expose attendees to the concepts and best practices around deploying cloud computing infrastructure. Attendees should expect to learn how to deploy a cloud computing environment using CloudStack and other cloud infrastructure tools that automate server and network configuration for building highly available cloud computing environments. Registration for Build A Cloud Day is free, but space is limited.
LOPSA NJ Chapter meeting: IBM Blue Gene /P, Thu, Sept 5, 2013
It isn't on the website yet, but the September meeting will have a special guest: Title: Anatomy of a Supercomputer: The architecture of the IBM Blue Gene /P. IBM refers to their Blue Gene family of super computers as 'solutions'. This talk will discuss the problems facing HPC that the Blue Gene architecture was designed to solve, focusing on the Blue Gene /P model. To help those unfamiliar with high-performance computing, the talk will begin with a brief explanation of high-performance computing that anyone should be able to understand. Mini-Bio: Prentice Bisbal first became interested in scientific computing while earning a BS in Chemical Engineering at Rutgers University.
Sysadmins/Devops needed for study
I met Jeevitha Mahendiran at Usenix LISA last year. She is studying sysadmins and what we do. She writes: I'm Jeevitha Mahendiran, Graduate Student/Research Assistant Faculty of Computer Science, Dalhousie University, Halifax, Canada. Currently doing a research on "Understanding the Use of Models and Visualization Tools in System Administration Work". The information that the you share regarding your work will be very helpful for my research. We are seeking participants to take part in a study about the tools used by system administrators. Participants will be asked to complete an anonymous and confidential survey that should take about 20-30 minutes to finish.
Evi Nemeth's life-raft spotted?
The search for Evi Nemeth and the others onboard the Nina has been restarted. The the crowd-sourced search of 56,000 satellite pictures appeared to find an orange/yellow object to the west of Norfolk island. The life-raft was orange: Read more: The Nina: Fresh search for missing yacht The project is being funded by donations. To donate visit the Danielle Wright Search Fund.
Puppet Camp DC, Tue, Nov 5, 2013 (free admission!)
If you are sad you can't attend PuppetConf 2013 this week, start planning for Puppet Camp DC. It is co-located with the Usenix LISA conference, which is Nov 6-9, 2013 in Washington D.C. Puppet Camp DC is a community-oriented, regional gathering of Puppet users and developers. You'll have the opportunity to talk to a diverse group of Puppet users, benefit from presentations delivered by prominent community members, and share experiences and discuss potential implementations of Puppet with your peers. Registration for Puppet Camp is free, but space is limited. To continue your "Automation" education USENIX is offering a discount to all Puppet Camp attendees.
Introducing my new tutorial: Evil Genius 101
Topics include: Helping your coworkers understand and agree to your awesome ideas Convincing your manager about anything (really!) Turning the most stubborn user into your biggest fan Getting others to trust you so they are more easily convinced Deciding which projects to do when you have more projects than time Making decisions based on data and evidence Driving improvements based on a methodology and planning instead of guessing and luck The only place you can find this class is at Usenix LISA, Nov 3-8, 2013 in Washington DC. Register TODAY! https://www.usenix.org/conference/lisa13
Usenix LISA training schedule published
https://www.usenix.org/conference/lisa13 The training this year has a lot of advanced topics that will bring a smile from anyone working in a DevOps environment. Tutorials on Jenkins, build-your-own-cloud, and a Googler will teach a class called "SRE University". There are a lot of specific technology tutorials: IPv6, file systems, Puppet, Python and a RaspberryPi class for people that want to move it beyond being a toy. I noticed a bunch of new security tutorials. I'll be teaching my new class 'Evil Genius 101' which is about how to convince your coworkers to get on board with your evil plans for world (or at least network) domination.
Unix/Linux "bash" shell users:
Do not do this at home: https://gist.github.com/solidsnack/4744661 That's just wrong.
So your management fails at IT, huh?
Recently on a mailing list sysadmins were describing horrible management they've experienced. Here is my reply: First, I want to say that my heart goes out to all of you describing terrible working conditions, bad management, and so on. I have huge amounts of sympathy for you all. Health is more important than anything else. If your job is driving you crazy and giving you high BP, my prescription is, 'Try, try, then quit'. Try to change things, talk to management, work to create the workplace you desire. Try again, I'm sure you feel like you've tried a lot, but people aren't mind-readers...
TPOSANA BugHunt Announced!
"The Practice of System and Network Administration" (TPOSANA) is now 7 years old (with some chapters virtually unmodified since the first edition, 12 years ago). We are preparing to update the book and create a 3rd edition but we need your help! We're looking for your input! Yes, you! Our valued readers! We're re-reading all 1,100 pages to find parts that are obsolete or need updating but we need your help. We've decided to crowd-source this part of the project. You are probably a better judge of what is missing, obsolete, or needs updating. Pick a chapter, a page, or a section and file bugs against any issues you find.
The "They know it's a problem"-Syndrome
As a system administrator you hate to see it happen: A user has a problem. They don't report it to you (enter a bug report, file a ticket). They whine to their friends, or suffer in silence. Months later you find out and ask, "Why didn't you file a ticket? I could have fixed it!" They either didn't have time, didn't feel it would do any good, or whatever. Annoying right? What's 100 times more annoying? When sysadmins do it to each other. I've seen it many times. Walking through a process (say... setting up a new machine) and some of the steps require...
Was the search for Evi botched?
http://www.stuff.co.nz/world/south-pacific/8972169/Search-for-missing-yacht-botched
Tom speaking at SpiceWorld conference!
Just announced on the SpiceWorks website! SpiceWorld is a conference held by the fine people at SpiceWorks. Register here: http://www.spiceworks.com/spiceworld/2013/austin/register/
PuppetConf ticket contest: Winners
If your last name is "Cole" or "Joy" you should have email waiting for you with some good news. Congrats! To everyone else that entered, thanks for entering. Please consider going to the conference. PuppetConf.com is a great way to learn about how to get more done by working less. PuppetConf 2013 takes place at the Fairmont Hotel, located in the heart of downtown SF. Puppet Labs has a block of hotel rooms on discount until July 16th, at which time the discount won't be offered anymore. A ton of other social events around the conference are planned. About 2,000 attendees are expected to attend.
I'm giving away 2 free admissions to PuppetConf 2013!
The fine folks at PuppetLabs have given me two tickets to give away! Puppet Labs wants to send two of my readers to PuppetConf this year in San Francisco (happening August 22 - 23rd). The two lucky winners will receive free admission (travel, hotel, meals is for you to provide... though super discounted hotel rates are available if you book by July 16). PuppetConf 2013 is set to host 2,000 attendees this year and include speakers from VMware & RedHat. It will take place at the Fairmont Hotel, located in the heart of downtown SF, where a ton of other social events around the conference are set to take place.
Remembering Evi Nemeth: The woman that saved "sudo"
Technology website The Register called it. With the search called off, we must presume that Evi Nemeth is no longer with us. Their obit, "Godmother of Unix admins Evi Nemeth presumed lost at sea", gives an excellent overview of her life and influence. In the coming months there will be many memorials and articles written about Evi, most by people that knew Evi better than I. That said, I'd like to share something that most people don't know. Evi saved "sudo". Sudo has joined popular culture (or at least popular geek culture) thanks to the famous XKCD cartoon: sudo make me a sandwich.
Evi Nemeth update: stay optimistic
Evi Nemeth's son is still optimistic and so am I. Here's what I glean from this report on 3NewsNZ: The last txt message from Evi wasn't the last txt message. Another txt was sent but not received. The phone company was able to reveal the last txt and its geolocation. The last txt was from Danielle and said "Sails Shredded last night, now bare polies, going 4 knot 310 degrees will update course info at 6pm." Given that info, it should be possible to locate them. However, no update at 6pm tells me we should be prepared for the worst.
Evi Nemeth potentially lost at sea
It is with a heavy heart that I pass on this information. There is a report that the boat Evi Nemeth was sailing on has not been heard from since June 3rd. The New Zealand Herald seems to have broken the story first. Evi co-wrote the groundbreaking book, "UNIX System Administration Handbook". It has been used as a textbook and outside of schools by nearly every Unix/Linux sysadmin I know. It meticulously covers every popular Unix varient of its day. (In the 1990s there was a lot more variation between Unixes). Since its publication there have been many updates and even a Linux-specific version.
Velocity 2013 in Santa Clara
I'll be attending the Velocity conference next week. If you see me please say "hi"!
New article in "login magazine": Ganeti: Cluster Virtualization Manager
The June issue of ";login: Magazine" (the Usenix magazine) is out and it includes a piece I contributed to: Ganeti: Cluster Virtualization Manager by Guido Trotter and Tom Limoncelli If you are a Usenix member you'll receive the issue in the mail soon if you haven't already. Otherwise you can purchase the article or the entire magazine online. Even better: join Usenix and don't miss an issue!
The possibility of cross-site configuration management
Steve Murawski from the ServerFault blog has some interesting notes about the prospect of a single configuration management system that works across Linux and Windows and other things too.
Starting my next chapter
Last week I started my new job at StackExchange.com. You may know SE as the company that runs great Q&A websites like ServerFault.com and StackOverflow.com plus their amazing job site Careers 2.0. I know the company as Joel Spolsky's startup that has been hiring a lot of great people that I know through conferences such as like LOPSA-East, the Cascadia IT Conference, and USENIX LISA. I just joined yet I already feel like everyone is an old friend. StackExchange encourages its employees to be active in the sysadmin community. You'll be seeing me at more events and having more time to write.
First Google Ganeti Conference: GanetiCon 2013
Synnefo has announced the first Google Ganeti Conference: GanetiCon 2013. They will be co-organizers. The announcement was first made on the Synnefo blog. The conference will take place between 3-5 September 2013 in Athens, Greece. The venue and program will be announced soon. Most developers of the Ganeti and Synnefo team will be attending. The first GanetiCon will be a developer oriented conference. Sessions will be a mix of design talks and discussions about new features and future plans. It will also probably feature an advanced Ganeti workshop, depending on user demand. The conference is geared towards people interested in: learning how other companies/institutions use Ganeti checking out how large scale Ganeti deployments look like glimpsing the product roadmap of Ganeti contributing to future design of Ganeti obtaining help with specific Ganeti issues The organizers do not yet have a website.
TodoPro available for Android (beta)
The todo list program that I use on my iPhone is now available on Android. It is a beta. I've been using the earlier betas on my Android tablet and it is looking very good. Previously I hadn't found todo list software for Android that worked well for me, and I had tried many. I'd been doing all my time management on my iPhone because TodoPro worked so well for me. I'm very excited that an Android release is now available. I don't endorse products but I do let people know what I personally use. I think todo list software is very personal...
I feel pain when articles get inaccurate titles
You may have read the Popular Science article: Thieves Stole $45 Million From ATMs Because The U.S. Uses Absurd 40-Year-Old Technology Let me quote: So why is the US so far behind? Infrastructure is a major factor; countries like Japan and the UK are much smaller, so replacing all the old point-of-sale machines and ATMs is easier. Bullshit. Bullshit. Bullshit. Bullshit. The reason is that bank executives had the choice between paying a lot of money to do the right thing or a little money to consultants who would tell them what they wanted to hear. It's a big win for consultants.
WAN Accellerators
What is your opinion of WAN Accelerators? Please post to the comments section. I haven't used or configured a WAN Accelerator but from the reading I've done so far it seems that the older the protocol is, the better a W.A. will help. Older protocols were designed when we were ignorant of the networking realities we have today. More modern protocols tend to do their own compression, caching, don't do stupid things that fail over high latency links, and so on. In particular: Am I right but if you mostly have home-grown protocols, you can tune them better than a W.A.
Zeno's Interview Question
I hope to teach a "how to interview" class at an upcoming conference. Here's one of the points I'll be making. How can one interview question help me understand what the candiate does and doesn't know Unix? Here's the question: What happens with I type this at a shell prompt: telnet www.wikipedia.org 80 RETURN Usually the candiate will explain just the command: "It opens a connection to port 80 on wikipedia". That's a good answer. The follow-up question is, "Please give me more detail". At this point they might explain how DNS works or how TCP network connections work.
"DRM-free" scares the s*** out of me
Today is the International Day Against DRM. As an author, and one that is currently living on unemployment insurance payments, DRM-free scares the shit out of me. Every book I've ever published has been pirated. Some I have even found in the "/tmp" directory of open HTTP servers. Every time I see my books pirated I die a little inside. Writing is very difficult for me. People don't realize how hard it is. How do I stop procrastinating and sit down to write? I eliminate everything else "tempting" from my life for a year or two until the book is done.
Why Google Glass is so important
Every tech blog, news site, magazine and newspaper is writing about Google Glass. Half are saying good things half are saying bad things. But there's one thing they all agree on: Mentioning Google Glass in a headline gets you readers. You're reading this. Right? I bet a whole lot of you don't always read my blog but you are reading this post, right? Writing about the success or failure of Google Glass is fantastic for many reasons: It's low cost. You don't have to spend $1,500 on one. Just read other people's blogs and repeat what they've said, speculate, or just make shit up!
Women in Science and Engineering (WISE) Computing Skills Boot Camp
https://blog.mozilla.org/it/2013/04/30/women-in-science-and-engineering-wise-computing-skills-boot-camp/ Software Carpentry is running a 2-day software skills boot camp in Boston, June 24-25th 2013, for women in science, engineering, medicine, and related research areas. Registration is $20. Boot camps alternate short tutorials with hands-on practical exercises. You are taught tools and concepts you can use immediately to increase your productivity and improve confidence in your results. Topics covered include the Unix shell, version control, basic Python programming, testing, and debugging -- the core skills needed to write, test and manage research software. This boot camp is open to women at all stages of their research careers, from graduate students, post-docs, and faculty to staff scientists at hospitals and in the public, private, and non-profit sectors.
Two questions for LOPSA Board Candidates
The slate of candidates for LOPSA board is up. At "candidate night" here are the questions I'll be asking: Question 1: "I'd like to know about your experience with community-based projects. Please tell us about a project that you took responsibility for seeing through to completion. Please, only projects that are "done" or have reached a self-sustaining mode only. One or two sentences is fine. It doesn't have to be a project where you thought of the idea or even did all the work: just one where you assured it reached the finish line." Question 2: "Surprise!
LOPSA-East is less than a week away!
If you haven't signed up for LOPSA-East, it is this coming Friday and Saturday, May 3-4, 2013 in New Brunswick, NJ. I've finally finished my slides for my "Evil Genius 101" class. I'm very excited about this new class. I hear there are still seats left, but it is filling up fast. To my NYC friends: you can take the train there. The station is 2 blocks away. To my Linux friends: the Linux content is most excellent this eyar. To my Windows friends: Steven Murawski himself is teaching PowerShell classes. Steven FREAKING Murawski! How can you NOT sign up for this?
Balancing Todos and Tickets
A reader asked me: What’s your opinion on merging “to-do” lists with issue trackers in The Cycle? I have a pile of To Do items which aren’t properly “issues”, and a pile of issues. I don’t want to duplicate tickets in the to-do list, but I’d like to look at one place to figure out what to work on next. You are correct in that copying items from your ticket system to your to-do list leads to trouble. They aren’t synced and bad things happen. I have a n-hour to-do item each day called “work on tickets” (where “n” is 1 to 8 hours depending on the requirements of my job).
Today in history: Letter from a Birmingham Jail
50 years ago today Martin Luther King, Jr published "Letter from a Birmingham Jail". It is a beautiful, moving, letter. Everyone should take a moment to read it. (Search for a copy of the letter online here) Many people read it without realizing the joke he put in the first paragraph. Seldom do I pause to answer criticism of my work and ideas. If I sought to answer all the criticisms that cross my desk, my secretaries would have little time for anything other than such correspondence in the course of the day, and I would have no time for constructive work.
Utilizando python para programadores Perl
Mi "Python para programadores de Perl" el artículo está disponible en español gracias a Maria Ramos y Webhostinghub.com: http://www.webhostinghub.com/support/es/misc/python (My Python for Perl Programmers article is now available in Spanish thanks to Maria Ramos and Webhostinghub.com.)
Review: Instant Puppet 3
Instance Puppet 3 by Jo Rhett is 50 pages long and a delight to read. The ebook is available for $9.99 from Packt (pronounced "packed") at http://www.packtpub.com/puppet-3-starter/book For those of you that are unfamiliar with Puppet: Puppet is a system for describing what the configuration of a machine should be and then the "puppet agent" will update a machine to have that configuration. If there is no work to be done, the agent does nothing. If you need to make a change globally you could, in theory, make one change to the description and soon every machine will be updated with the Puppet agent doing the right thing on each machine depending on what operating system it is running.
Today's time management tip
is brought to you by The Joy of Tech: http://tapastic.com/episode/3845
LOPSA-East (formerly PICC) registration time!
If you haven’t registered for LOPSA-East yet, what are you waiting for? LOPSA-East (May 3-4, 2013) is the best regional conference for sysadmins this side of the Mississippi. It is much less expensive than national conferences because you can probably drive instead of fly. The same nationally known speakers you find at the big conferences travel to us instead of the other way around. If you are signing up for my tutorials please remember that space is limited so please register soon! I was thinking about why I like going to conferences the other day and it dawned on me that the answer is simple: I can ask the questions I can’t ask anywhere else.
Get starting with Puppet in 50 pages
My friend Jo Rhett's new book is called "Instant Puppet 3 Starter". In 50 pages he gets you up and running with Puppet3. I haven't read it yet, but if it isn't good let me know and I'll kick his ass. http://www.packtpub.com/puppet-3-starter/book P.S. Just kidding. Jo could kick my ass any day. I'd find some other punishment. Luckily I won't need to because I'm sure it is a great book.
Reminder: LISA '13 Call for Participation deadline (and a request)
Remember that the submission deadline is Tuesday, April 30. Get those proposals in now! If you are submitting a paper, it can be the full (draft) paper or it can be an extended abstract, 4-8 pages in length. See the CFP page for more details. Speaking of which... I have not yet submitted a proposal for an Invited Talk. What would you like to hear me talk about? Invited talks are usually 90 minutes (or 45 minutes for a half-session). What would you like to hear me talk about? (Post a comment)
The mental model mismatch
Humans think in terms of mental models. In IT it is our responsibility to help them form accurate models as well as deal with inaccurate models that exist. Humans use mental models of how things work to fill in context. If we are not given the model, we make one up. This made-up model may be unrelated to how things actually work, but if it is sufficient for us to get our job done then that's "good enough". I think this is evolutionary: we didn't know why the sun rose and fell, but we made up a model that included a god riding across the sky...
For-profit universities are a scam
I’m frustrated with DeVry University, Kaplan University, Walden University, Ashford University, Colorado Technical University, Strayer University, University of Phoenix, Capella University, American Intercontinental University and other businesses. I do not encourage anyone to enroll in these “schools”. Here’s how for-profit “schools” make money: They get students to enroll and help them get government-funded financial aide. The thing about financial aide is that the check gets sent directly to the “school”. The “school” deposits the check. There are no refunds. The student only has to attend one day of classes for this to be legit. So, after the first class students are worked hard in hopes they quit.
Bad Tech Job Interview Questions (and How To Answer Them)
Andy Lester, author of "Land the Tech Job You Love", has an excellent blog post up called Bad Tech Job Interview Questions (and How To Answer Them). It is a good read whether or not you are interviewing. It has good advice if you are on either side of the interview table.
Interview with Chris St. Pierre, UCMS '13 Program Chair
The 2013 USENIX Configuration Management Summit (UCMS '13) call for participation closes Friday, April 5, at 11:59p.m. PDT. In this interview, Chris St. Pierre, UCMS '13 Program Chair, answers questions about the CFP and what to expect at the event, which will take place during USENIX Federated Conferences Week, June 24-28, 2013. https://www.usenix.org/blog/interview-chris-st-pierre-ucms-13-program-chair
Running ChromeOS District Wide
[ This is a guest post from Dan O’Boyle, who I met at a LOPSA-NJ meeting. I asked him to do a guest post about this subject because I thought the project was something other schools would find useful ] I’m a systems engineer for a moderately sized school district in NJ. We own a number of different devices, but this article is specifically about the AcerOne line of netbooks. I was recently tasked with finding a way to breath new life into about 500 of these devices. The user complaints on using these models ranged from “constant loss of wireless connectivity” to the ever descriptive “slow”.
Design Considerations for Faster-Than-Light (FTL) Communication
RFC 6921 has been published today: Abstract We are approaching the time when we will be able to communicate faster than the speed of light. It is well known that as we approach the speed of light, time slows down. Logically, it is reasonable to assume that as we go faster than the speed of light, time will reverse. The major consequence of this for Internet protocols is that packets will arrive before they are sent. This will have a major impact on the way we design Internet protocols. This paper outlines some of the issues and suggests some directions for additional analysis of these issues.
My LOPSA-East tutorial: Evil Genius 101
You want to innovate: deploy new technologies such as configuration management (CfEngine, Puppet, Chef), a wiki, or standardized configurations. Your coworkers don't want change. They like it the way things are. Therefore, they consider you evil. However you aren't evil, you just want to make things better. Learn how to: Brainwash your coworkers into thinking the big change was "their idea". Program people like you program computers: a flowchart for every personality type. How to fix that your "Stormtroopers can't shoot straight". Help your coworkers understand and agree to your awesome ideas. Convince your manager about anything. Really. Turn the most stubborn user into your biggest fan.
Tesla accidentally leaks their secret business plan
Read it before they realize what they've done and take it off their website.
Sheeri Cabral's "When I Moved Abroad"
Over at the Mozilla IT blog is a new post by Sheeri Cabral that every sysadmin in our community should read. Blog post: When I Moved Abroad
Boston Area Sysadmins: BBLISA Looking for Lightning Talks!
Would you like to do a lightning talk at the next BBLISA meeting? [ This message comes from Matt Simmons at the Standalone Sysadmin blog. ] Do you love lightning talks? Because I love lightning talks. When I found out that the DC DevOps group had an entire meeting dedicated to lightning talks, I was jealous. I mentioned the idea to John, Adam, and crew of BBLISA, and they liked it. Of course, when you volunteer an idea, you volunteer /for/ that idea, too, so if you look at the BBLISA Calendar (http://www.bblisa.org/calendar.html), you'll see my name organizing the April meeting.
Stop. Take 5 minutes to save the internet
Hey fellow sysadmins! Please take 5 minutes to make sure your DNS servers aren't open to the world for recursive queries. They can be used as amplifiers in DDOS attacks. https://www.isc.org/wordpress/is-your-open-dns-resolver-part-of-a-criminal-conspiracy/ The short version of what you need to do is here.
Can you make the speed of light faster?
One technical issue that often plagues me is that you can't make the speed of light any faster. Network latency from NYC to Sydney is going to suck no matter what. Helping users understand this is difficult. Often it is equally difficult to make software developers understand this too. Many times people have asked me, sometimes seriously, if we could just make the speed of light faster. There is one obvious way to improve the latency between NYC and Sydney: Tunnel through the earth. A direct route would be much faster. However it looks like scientists are close to a more realistic alternative: use air instead of glass! "
How the Maker of TurboTax Fought Free, Simple Tax Filing
This investigative report by propublica.org is what I thought was going on but had no proof. Basically I've always said that since the IRS gets all the data from our employers and financial institutions electronically, why can't they present our tax forms partially or completely filled out? We should be able to subtract our deductions and that's it. Obviously we should get all the data so we can examine it or hire a tax accountant to examine it. Anytime someone said "yeah, but the people that prepare tax returns would try to stop any legislation like that" I would say, "oh, don't be a conspiracy theory crazyperson".
Today is my last day at Google.
Today is my last day at Google. After 7 years I'm looking forward to doing nothing for a while, writing a book or two (oh yeah, I have a big announcement: I've signed 2 book contracts! More info soon!) , and I'm getting married. Please, no speculation on why I'm leaving. I was at Bell Labs 7 years too. It's just time. (FunFact: I found a draft of a "goodbye message" I wrote. The file's datestamp was Nov 10, 2010.) The annoying thing about job hunting is that usually you have to take random days off from your current job claiming "something came up" or taking vacation days or faking sick days.
The perfect April Fools gift for the geek you love?
There are still a few copies left of the book of April Fools RFCs. http://www.rfc-humor.com They say if you have to explain a joke it wasn't funny. Well, this makes The Complete April Fools RFCs the least funny book in the world. Ok, maybe that's not 100 percent true but you have to be pretty darn technical to get some of these jokes. There are only a few left in stock. Why not pick one up today? Click here to see it on Amazon Tom
What does Stack Exchange do when disaster strikes?
Find out at LOPSA-East (formerly PICC) May 3-4, 2013, New Brunswick, NJ (Early Bird Registration ends April 1st! http://lopsa-east.org Space is limited!) In late October of 2012, Hurricane Sandy was wreaking havoc on the east coast. It was the second costliest hurricane in US history causing widespread power and service disruptions. George Beech, a System Aministrator at Stack Overflow, will be presenting a talk at LOPSA-East 2013 about their successful failover to a backup datacenter and what it took to keep their primary New York City datacenter operational while implementing the Disaster Recovery plan. This talk will focus mostly on Disaster Recovery and migration for a primarily windows based shop.
New video: Ganeti: Your Private Virtualization Cloud "the Way Google Does It"
My 60-minute talk on Ganeti from the Usenix LISA '12 conference has been posted: https://www.usenix.org/conference/lisa12/ganeti-your-private-virtualization-cloud-way-google-does-it Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies such as Xen or KVM and other Open Source software. Ganeti takes care of disk creation, migration, OS installation, shutdown, startup, and can be used to preemptively move a virtual machine off a physical machine that is starting to get sick. It doesn't require a big expensive SAN, complicated networking, or a lot of money. The project is used around the world by many organizations; it is sponsored by Google and hosted at http://code.google.com/p/ganeti.
Google hosted Xen Hackathon, May 16-17, Dublin
The next Xen Hackathon will be hosted by the Ganeti team at Google and takes place on May 16-17, 2013 at Google's offices in Dublin Ireland. I can't make it but many of my coworkers on the Ganeti project will be there. If you use the open source version of Xen and want to get your hack on, please sign up!
Save up to $197 on LOPSA-East registration, really?
Early bird pricing ends April 1st! Make sure you register before then to save up to $197! http://lopsa-east.org/ Best way to save money? Start talking with your boss NOW so all that purchasing department paperwork gets done in time!
Last night I had a nightmare...
...that I got caught in a "spear phishing attack". (A malware attack where they send an email specifically crafted to one or two people.) The email was a receipt from a hotel that I stay at occasionally but it listed the address as being in South Carolina instead of San Francisco. I clicked on the PDF to read it and then realized I was being phished because I haven't been to South Carolina in ages and the invoice mentioned a coworker that I've never traveled with. I started shutting down my computer and made plans to wipe the disks; glad I have good backups but not wanting to go through the pain of being without my laptop until I could do this.
Call For Proposals Extended: Open Source Bridge 2013
The "Call for Proposals" for Open Source Bridge 2013 has been extended 2 weeks (Sat, March 23). The current proposals so far are listed online. The conference itself is June 18-21, 2013 in Portland, Oregon. More info about submitting proposals is here: http://opensourcebridge.org/blog/2013/03/were-extending-our-call-for-proposals/
Cascadia IT Conference 2013 a big success!
Congrats to all involved! Save next year's date: 7 & 8 MAR, 2014. (It's already marked in the Sysadmin Event Calendar: http://everythingsysadmin.com/calendar.html )
Registration open for LOPSA-East 2013! (formerly PICC) May 3-4, 2013, New Brunswick, NJ
Register early and save! http://lopsa-east.org Space is limited! Registration opens at midnight for the 2013 LOPSA-East conference, May 3-4, 2013 at the Hyatt Regency hotel in New Brunswick, NJ. IT professionals from the tri-state area, as well as the entire east coast will be joining us for the most talked about community-driven conference of the year. You can find out more at http://lopsa-east.org LOPSA-East begins Friday with an entire day of training offered by world class instructors. We have half day sessions on Team Efficiency, Configuration Management, Basic and Advanced PowerShell, IPV6 migration, and much more! The conference continues on Saturday, with more half day training sessions along with 45 minute presentations from invited speakers, 5 minute lightning talks, and ‘birds of a feather’ discussions on participant selected topics.
The correct RSS feed for this site
I'm getting reports that some RSS reading software does not see the latest posts from this list. Please be sure to use this URL for your RSS reading pleasure: http://feeds.everythingsysadmin.com/EverythingSysadmin Of course, if you are using an old URL that has gone away, you'll never see this post (so tell your friends!). Thanks!
Why oh why is Google canceling "reader"?
I can not confirm nor deny... @sheeri or it's a conspiracy by @yesthattom to improve TMFSA by removing a huge time sink... ;)— Nahum Shalman (@nahumshalman) March 14, 2013 If that makes you sad, maybe this will cheer you up... Wow. The heads of all major religions (iOS, Android, Windows, and the Catholic Church) have been replaced in a 6 month time period.— Brian Walsh (@thepartycow) March 13, 2013 @thepartycow maybe that's true of those minor religions, but the great religions of Vi and Emacs are eternal.— ghc (@xnomagichash) March 13, 2013
Site Redesign Launched!
www.EverythingSysadmin.com is proud to announce our newly redesigned website! New design and color scheme. After nearly 10 years this new design has a more modern feel. New Feature: The sysadmin events calendar is now a tab for easier viewing. This calendar of events is a joint project with Matt Simmons' Standalone Sysadmin Blog. Updated: Author biographies and book descriptions. New feature: A spinning book carousel in the header! New automation for the "See us live", "Awesome Conferences", and "Best of Blog" boxes. Much improved navigation for older posts. And much much more! We expect to be making minor adjustments over the next few days.
Guidebook for Cascadia IT Conference open to all!
The Guidebook App (available for every smart-phone known to the planet) now lists all the events and talks for the Cascadia IT Conference, scheduled to start this Friday in Seattle, WA. You can download the app whether or not you are attending. I just read through all the talks and they look excellent. I wish I could be there! There is plenty of time to register! If you are local to Seattle there's no excuse. This has got to be the best "bang for your buck" of a conference the region will see all year.
Registration open for LOPSA-East 2013! (formerly PICC) May 3-4, 2013, New Brunswick, NJ
Register early and save! http://lopsa-east.org Space is limited! Registration is open for the 2013 LOPSA-East conference, May 3-4, 2013 at the Hyatt Regency hotel in New Brunswick, NJ. IT professionals from the tri-state area, as well as the entire east coast will be joining us for the most talked about community-driven conference of the year. You can find out more at http://lopsa-east.org LOPSA-East begins Friday with an entire day of training offered by world class instructors. We have half day sessions on Team Efficiency, Configuration Management, Basic and Advanced PowerShell, IPV6 migration, and much more! The conference continues on Saturday, with more half day training sessions along with 45 minute presentations from invited speakers, 5 minute lightning talks, and ‘birds of a feather’ discussions on participant selected topics.
Issue 34 "Hacker News Monthly" issue
My "4 Unix commands I abuse every day" blog post has been published in this month's Hacker News Monthly! Check it out: http://hackermonthly.com/issue-34.html Interestingly enough that post got more hits than any other that I posted last year. It got mentioned on HN (quite an honor) and then the print edition (Hacker News Monthly) contacted me about reprinting it. HNM is a pretty nice deal. If you don't have time to read HN every day, they pick out the best articles of the month and print them as an ebook (multiple formats) and an actual dead-trees printed version too!
How to tell if a site stores passwords in clear-text?
Click on the "I forgot my password" link. If they email you your password, you know they stored it in clear-text somewhere. You should complain. Sadly their first-tier support probably won't understand and will assure you that they take security seriously and you have nothing to fear. Oh well, at least you know and can choose to use a different company or at least use a password you aren't using anywhere else (which, you already do, right?) If they email you a code to reset your password or a temporary password, then either they stored a hash of the password (hopefully they did it right), or they're doing it wrong and their password-recovery system obscures this fact.
Cascadia IT Conference: Discount extended! Don't miss out!
https://plus.google.com/u/0/101281951565093176572/posts/XALRuBSdgqP From the organizers: An impressive number of registrations over the past few days has prompted us to extend early bird pricing through Monday, March 4th. Save as much as $75 over at-the-door pricing by registering before 11:59pm Monday evening! If you're visiting Seattle from out of town, don't forget to make your hotel reservations by phone and be sure to mention the conference to receive a discounted room rate and parking: We also hope you'll join us Thursday, March 14th as the Seattle Area System Administrators Guild (SASAG) hosts a welcome reception sponsored by Silicon Mechanics in the Governors Room at the Hotel Deca.
A Linux equivalent of DaIotFOS?
I often recommend the book The Design and Implementation of the FreeBSD Operating System by Kirk McKusick and George V. Neville-Neil as the best way to learn about Unix. It teaches all the parts of the Unix kernel (process tables, file systems, network stacks, etc) and the algorithms used. A sysadmin gains keen insights into what is going on, which helps them design new systems and debug running systems. It is an excellent textbook and teaches OS theory and concepts along with the narrative of how FreeBSD works. However because it has "FreeBSD" in the title, people often ask if there is a Linux version.
LISA '13 Call for Participation released!
Just seen on Google+: the call for participation is open: https://www.usenix.org/blog/lisa-13-call-participation-opens Extended abstracts, papers, experience reports, and proposals for talks, workshops, and tutorials due: Thursday, April 30, 2013, 11:59 p.m. PDT If you haven't attended LISA before, be sure to check out papers and videos from past LISA events. USENIX members help support open access to conference papers and videos of paper presentations. New in 2013! LISA Labs: New this year is a "hack space" available for informal mini-presentations by seasoned professionals, participation in live experiments, tutoring, and mentoring. This will bring a hands-on component to the conference, where attendees can investigate new technologies, apply what they have learned, and interact with other attendees in a participatory technical setting.
Wikimedia Foundation's system administration
In an effort to help the less technical community understand what Wikimedia Foundation's systems administrators do, Sumana Harihareswara wrote some very interesting blog posts. They're interesting to technical people too. From duct tape to puppets: How a new data center became an opportunity to do things right How the Technical Operations team stops problems in their tracks It is particularly interesting how she expresses the value of what we do to the Wikimedia managers and donors. There's also some information in there about how Wikimedia Foundation Ops uses Puppet, Nagios, and Ganglia. They're both worth reading. Enjoy!
Cascadia IT Conference: Discount about to end!
One week left to get early bird pricing! http://www.casitconf.org/casitconf13 The conference is in Seattle, WA, March 15-16, 2013. Don't miss it!
Are release trains obsolete?
IT systems have many parts. Each needs to be upgraded or patched. The old way to handle this is to align all the individual release schedules so that you can make a "big release" that gets tested as a unit, and released as unit. You can do this when things change at a sane rate. Now more things are changing and the rate is much faster. We also have less control. Operating systems have frequent patches. There are urgent security patches that need to roll out "immediately". Applications have frequent updates, many even upgrade themselves. Our PCs have firmware updates for the BIOS, the keyboard, the IPMI controller, the mouse (yes, my damn mouse needed a flash update recently!)
Reverting in "git"
I'm slowly learning "git". The learning curve is hard at first and gets better as time goes on. (I'm also teaching myself Mercurial, so let's not start a 'which is better' war in the comments). Reverting a file can be a little confusing in git because git uses a different model than, say, SubVersion. You are in a catch-22 because to learn the model you need to know the terminology. To learn the terminology you need to know the model. I think the best explanations I've read so far have been in the book Pro Git, written by Scott Chacon and published by Apress.
Metcalfe's law
Metcalfe's law states that the value of a telecommunications network is proportional to the square of the number of connected users of the system (n^2). Robert M. Metcalfe, the inventor of Ethernet, originally meant it to apply to devices on a network that could communicate with each other. It isn't sufficient to be on the same network if they speak incompatible protocols. It isn't sufficient to speak compatible protocols if they aren't connected. A more plainspoken way to state Metcalfe's law is that every one new user added to a network makes the network more than one unit more useful. A more simple way to understand this law is: "The first person to buy a fax machine was a fool."
Label those datacenter cables!
Matt Simmons of the Standalone Sysadmin blog asked about labeling network cables in a datacenter on the LOPSA-Tech mailing list which brought up a number of issues. He wrote:So, my current situation is that I'm working in a datacenter with 21 racks arranged in three rows, 7 racks long. We have one centralized distribution switch and no patch panels, so everything is run to the switch which lives in the middle, roughly. It's ugly and non-ideal and I hate it a bunch, but it is what it is. And it looks a lot like this. Anyway, so given this really suboptimal arrangement, I want to be able to more easily identify a particular patch cable because, as you can imagine, tracing a wire is no fun right now.
"The Finer Art of Being a Senior Sysadmin"
Sheeri K. Cabral's talk from LCA2013 is now available online: "The Finer Art of Being a Senior Sysadmin" The video is 17 minutes long and makes a lot of references to a blog post I wrote last September. It is a great talk and well worth watching!
Books I've recommended in the last week:
Realizing that I've recommended a lot of books lately. I thought I'd list them here for others to benefit. I'm not saying I've read them all and these are the best, but these are the ones I've read and found useful. Management: If you are getting started as a manager: The One Minute Manager by Kenneth H. Blanchard If you want to be more strategic: The Art of War: Complete Texts and Commentaries If you have to make a big change to your organization (shutting down a project or getting everyone onto a new platform): Managing Transitions: Making the Most of Change by William Bridges Layout and web design: The basics of design, so you can make posters, webpages, and just about anything look more professional: The Non-Designer's Design Book by Robin Williams Website usability: Don't Make Me Think: A Common Sense Approach to Web Usability by ...
Registration is now open for Cascadia IT Conference 2013!
You do not want to miss this conference! http://casitconf.org/ You will learn how to automate the configuration of all your systems when Nathen Harvey teaches you Chef or Garrett Honeycutt teaches you Puppet. You'll stay one step ahead of the game by learning IPv6 from Owen DeLong, the man that teaches IPv6 so well you'll thank him 128 times. The wizard of PowerShell himself, Steven Murawski will teach you how to automate anything in Windows. You'll fix things once and they'll stay fixed after Stuart Kendrick teaches you how to do Root Cause Analysis. You'll learn how to translate "geek" to "manager-speak" and other tips in Navigating the Business World by the internationally recognized experts Nicole Forsgren Velasquez and Carolyn Rowland.
Operational aspects of a system
Users tend to be concerned with what a system does (features, functionality) and sysadmins tend to be concerned with the operational aspects of a system. I just noticed this great Wikipedia page that lists "Non-functional requirements" of a system.Broadly, functional requirements define what a system is supposed to do whereas non-functional requirements define how a system is supposed to be. Functional requirements are usually in the form of "system shall do <requirement>", while non-functional requirements are "system shall be <requirement>". I could see myself using this as a tool for jogging my memory when I'm trying to think of all the aspects of a system that I need to be concerned with either operationally or when writing requirements.
Registration is now open for Cascadia IT Conference 2013!
You do not want to miss this conference! http://casitconf.org/ You will learn how to automate the configuration of all your systems when Nathen Harvey teaches you Chef or Garrett Honeycutt teaches you Puppet. You'll stay one step ahead of the game by learning IPv6 from Owen DeLong, the man that teaches IPv6 so well you'll thank him 128 times. The wizard of PowerShell himself, Steven Murawski will teach you how to automate anything in Windows. You'll fix things once and they'll stay fixed after Stuart Kendrick teaches you how to do Root Cause Analysis. You'll learn how to translate "geek" to "manager-speak" and other tips in Navigating the Business World by the internationally recognized experts Nicole Forsgren Velasquez and Carolyn Rowland.
Ganeti "list" subcommand tips: list output and filtering
If you use the Ganeti command line you probably have used gnt-instance list and gnt-node list. In fact, most of the gnt-* commands have a list subcommand. Here's some things you probably didn't know. Part 1: Change what "list" outputs Unhappy with how verbose gnt-instance list is? The -o option lets you pick which fields are output. Try this to just see the name: gnt-instance list -o name I used to use awk and tail and other Unix commands to extract just the name or just the status. Now I use -o name,status to get exactly the information I need.
Labeling Machines Is A Safety Precaution
Something happened at home today that reminded me of something I used to do when I worked at Bell Labs. My rule was simple. If a machine in the computer room wasn't labeled, I was allowed to power it off. No warning. Click. No power. If I logged into a machine as root and the prompt didn't include the hostname, the only command I was interested in typing was "halt". Both of these rules came from the same source: If sloppy system administration was going to lead to errors and downtime, I wanted that downtime to happen during the day when we can fix it instead of late at night when we should be asleep.
LOPSA-NJ February 7 meeting is a "Cluster Meeting"
This month the NJ chapter of LOPSA's meeting will be something special. Rather than meeting at a library near Princeton, there will be two meetings on the same day: one north and one south. Each meeting will be held at a diner and there will be a suggested discussion topic. Generally someone takes notes at each diner and posts them online. It is interesting to see what each group does with the topic. We've done this a few times before and the discussion is always quite lively. This time the topic is: What's the best new tool you've started using in the last 24 months?
Bug of the day: getent's surprise
What's wrong with this as a way to turn a hostname into a FQDN? FQDN=$(getent hosts "$SHORT" | awk '{ print $2 }') Answer: getent can return multiple lines of results. This only happens if the system is configured to check /etc/hosts before DNS and if /etc/hosts lists the hostname multiple times. There may be other ways this can happen too, but that's the situation that bit me. Of course, there shouldn't be multiple repeated lines in /etc/hosts but nothing forbids it. As a result you can end up with FQDN="hostname.dom.ain hostname.dom.ain which, and I'm just guessing here, is going to cause problems elsewhere in your script.
Cascadia IT Conference 2013 schedule announced!
Cascadia IT Conference 2013 has announced their tutorial lineup and it looks great! If you are in the Seattle area, or can travel there, this is a can't miss conference! Here are some of the tutorial titles: Root Cause analysis -- Intermediate PowerShell Fundamentals Building Your Powershell Toolkit Resolv the World with Chef: An Introduction to Chef for Sysadmins Build A SysAdmin Sandbox An Introduction to Puppet Navigating the Business World for Sysadmins: The Trusted Adviser Navigating the Business World for Sysadmins: Methods IPv6 -- An Introduction The Compassionate Geek: Mastering Customer Service for IT Professionals Over the Edge System Administration The technical sessions will be announced in a few days followed by registration.
Went to Epcot
Mission: Space is still the best ride ever invented. Here's a postcard: from spaaaaace How was your December break?
Jan 1 reminder:
If you use "The Cycle", today is the day you review your "long term goals" list. Cross out obsolete items or items that you now realize only seemed like a good idea at the time. Pick the 1-2 most important ones. Discuss them with your partner/wife/husband/spouse/family and set goals of the year. For each goal, come up with 5-6 milestones that will get you to that goal. Milestones should be measurable. Sprinkle the milestones on the todo lists throughout the next couple of months. (In The Cycle, you have a different todo list for each day of the year; incomplete items slide to the next day.)
All outages are due to a failure to plan
I can't take credit for this, as a co-worker recently introduced me to this point. All outages are, at their core, a failure to plan. If a dead component (for example, a hard drive) failed, then there was a lack of planning for failed components. Components fail. Hard disks, RAM chips, CPUs, mother boards, power supplies, even ethernet cables fail. If a component fails and causes a visible outage, then there was a failure to plan for enough redundancy to survive the outage. There are technologies that, with prior forethought, can be included in a design to make any single component's failure a non-issue.
Review of my LISA '12 half-day tutorial on Time Management for Sysadmins
Ben Cotton write an excellent summary of my half-day tutorial from LISA this year: https://www.usenix.org/blog/time-management-system-administrators-0
Vint Cerf's keynote at LISA '12
Did you miss the Usenix LISA live stream of Vint Cerf's keynote? Video is online: http://ow.ly/g38p7
What's your "LISA '12 moment"?
Every year at Usenix LISA it seems that there is a moment where someone says something that makes me want to jump up and shout, "OMG! Learning that just paid for my entire conference!" It may be something an instructor says at a tutorial, a presenter says at a paper or Invited Talk. Often it is something you learn from the person you just happened to start chatting with while on line waiting for lunch. If you have a "LISA Moment", I encourage you to tweet it with hashtag #lisa12 #moment or post it as a comment to this post.
Book signing at LISA: Taming Information Technology
In the past I've said good things a few different times about "Taming Information Technology: Lessons from Studies of System Administrators" by Eser Kandogan, Paul Maglio, Eben Haber and John Bailey. Eben will be at Usenix LISA next week, in San Diego, doing a book signing during the Wednesday afternoon break on the expo floor. He'll have a limited number of copies for sale at a huge discount (I hear it's $40/book while supplies last). See you there!
Book signing at LISA: Taming Information Technology
In the past I've said good things a few different times about "Taming Information Technology: Lessons from Studies of System Administrators" by Eser Kandogan, Paul Maglio, Eben Haber and John Bailey. Eben will be at Usenix LISA next week, in San Diego, doing a book signing during the Wednesday afternoon break on the expo floor. He'll have a limited number of copies for sale at a huge discount (I hear it's $40/book while supplies last). See you there!
New at LISA: Watch the tutorials via live-streaming!
As you know, I'll be teaching 3 tutorials at LISA this year (Intro To Time Management, Advanced Time Managemente, and Ganeti/Build a private cloud). If you can't attend in person you can still watch over the internet. The cost is about the same as being there, and there will be a chatroom so that you can ask questions just like in-person attendees. However, you save money of travel and hotel. https://www.usenix.org/conference/lisa12/training-program/live-streaming See you there at the conference or via the interwebz!
My Pre-LISA checklist
Get haircut Print out 2-factor "rescue codes" in case my 2-factor fob is lost of dies. De-junk my wallet. Practice slides for the Ganeti tutorial, Time Management tutorials. Reach out to co-workers about coverage while I'm away. Verify flights and hotel information. Pack What's on your pre-LISA checklist? Please post in the comments. I'd like to know! See you in San Diego!Tom
Early-bird discount for LISA'12 ends on Nov 19th!
LISA is coming to San Diego, CA, December 9-14, 2012 and, as always, the committee has put together an amazing schedule of programs. Come for a few days of training, 2-days of technical sessions, or spend an entire week immersed in sysadmin geekery! Take anywhere from 1 to 6 full days of training and create the curriculum that meets your needs. https://www.usenix.org/conference/lisa12/training-program/training-program Take advantage of 47 half- and full-day training sessions from industry leaders, including my highly rated "Intro to Time Management" and "Team Efficiency" tutorials. Take the all-new training class "Build your own cloud with Ganeti Virtual Cluster Manager" co-taught by Guido Trotter and myself.
Why Romney lost today and how Republicans can win in 2016
How to win the most votes? Let me share two datapoints: The Obama campaign was lackluster and just couldn't get momentum. About a month ago he changed his speeches to be pretty hardcore liberal talking points. Suddenly the enthusiasm and polling started doing much better. In the last weeks of the campaign, Romney started stealing Obama's talking points, sounding as liberal as he could be. Suddenly the Romney momentum started building. In fact, if he had kept this up or if the election was a few weeks later, he might have one. Both of these data points indicate that to gain more votes, politicians need to "run to the left".
Pimlical on Android... now on Google Play
Time Management for Sysadmins mentions the Pimlical's DateBk software a lot. It is one of the finest time management software packages around. It was way ahead of its time Sadly it was only available for the Palm series of PDAs. Pimlical's equivalent program for Android is called "Advanced Calendar". Until recently the installation process was a bit... odd. Now it is available on the Google Play app store. As a result, it is much easier to install. Check it out here: https://play.google.com/store/apps/details?id=com.pimlicosoftware.PimlicalA
Google Throws Open Doors to Its Top-Secret Data Center
The photos look like "IBM meets Willy Wonka's Chocolate Factory". For the first time, the company has invited cameras inside its top secret facility in North Carolina. Our tour guide is Google's senior vice president, Urs Hoelzle, who's in charge or building and running all of Google's data centers. 'Today we have 55,200 servers on this floor. Each of these servers is basically like a PC, except somewhat more powerful.' 5 minute video from CBS Morning Show: https://video.google.com/a/?tab=mv&pli=1#/Play/contentId=2e908def456860ab A detailed article by Steven Levy: http://www.wired.com/wiredenterprise/2012/10/ff-inside-google-data-center/
IPv6 Flashcards
IPv6 is an entirely new protocol. It isn't IPv4 with larger addresses. It is new enough that you'll feel like you are starting over on a new planet; one that invented the internet using protocols that remind you of IPv4 but are.... different. I find flashcards are a useful way to learn new terminology. I found these online: General IPv6 Knowledge: High level. Types and Ranges of IPv6 Addresses. Learn those new IP ranges! CCIE IPv6: Lots of technical details. Enjoy! Tom Limoncelli
Mechanical Computer instructional video
US Army 1953 training film on mechanical computers. Gears! Cams! Great animations! http://www.youtube.com/watch?v=s1i-dnAH9Y4
Has the job of a Google SRE changedover the years?
Someone recently asked me if Google's SRE position has change over the years. The answer is 'yes and no'. Yes, the job has changed because there is more diversity in the kind of work that SREs do. Google has more products and therefore more SRE teams. Each team is unique but we all work under the same mission, executive management, and best practices. All teams strive to use the same best practices for release engineering, operational improvements, debugging, monitoring, and so on. Yes, since each SRE team is responsible for a different product with different needs, you'll find each one can be unique priorities.
Homomorphic Encryption Explained
American Scientist has an article that (finally!) explains homomorphic encryption in simple enough terms that even I understand. Homomorphic encryption permits me to send you encrypted data that you can manipulate but never know the contents. You send it back to me, I decrypt it, and see the result. Imagine if a web-based wordprocessor could store your document, edit your document, but never know what your document says. Yes, it sounds crazy but it is theoretically possible. In the last 4 years that theory has been getting closer and closer to reality. I think sysadmins should read this article to get an idea of what crypto might be like in the future.
Book flights to Usenix LISA asap!
Flights are filling up. Book soon. And book your hotel too. One thing I learned from traveling is that it is easier to make a reservation early and cancel/change it than to end up close to the date and find there are no hotel rooms or flights left. This is especially important for hotels. https://www.usenix.org/lisa
Spiceworks interview: part 2
Part 2 of my interview at SpiceWorks is up. Q&A: Tom Limoncelli on the state of IT and junk meetings Q&A Part 2: Tom Limoncelli talks about success, failure and pirates vs zombies Enjoy!
Announcement: I'll be speaking at the Australia IPv6 Summit
I'll be doing a 30-minute talk about how to convince your boss to take IPv6 seriously at the Australia IPv6 Summit. I'll be presenting via video conference. If you are in Australia and/or are concerned with IPv6, please attend this awesome conference. Registration is still open! Info here: http://www.ipv6.org.au/summit/program.php
Coming to the LOPSA-LAX meeting?
There is now a Meetup so you can RSVP: http://www.meetup.com/League-of-Professional-System-Administrators-Los-Angeles/events/84585402/
What makes a sysadmin a "senior sysadmin"?
This came up in discussion recently. Here's how I differentiate between a junior and senior sysadmin: A senior person understands the internal workings of the systems he/she administers and debugs issues from a place of science, not guessing or rote memorization. A senior person has enough experience to know a problem's solution because he or she has seen and fixed it before (but is smart enough to check that assumption since superficial symptoms can be deceiving). A senior person automates their way out of problems rather than "working harder". They automate themselves out of a job constantly so they can be re-assigned to more interesting projects.
DevOps: Google reveals their "DiRT Exercises" (part 3 or 3 in my "disaster preparedness" series)
Weathering the Unexpected by Kripa Krishnan, Google For the first time ever Google discusses our "DiRT" (Disaster Recovery Test) procedure. This is the week of hell where systems are taken down with little or no notice to verify that all the failure protection systems work. Oh yeah... and the funny sidebar at the end was written by me :-) Enjoy! P.S. (I take credit for cajoling Kripa into writing the article. I think she did a bang-up job! Go Kripa!!)
Join me at the October LOPSA New Jersey chapter meeting!
If you live near Princeton, Trenton or New Brunswick and haven't been to the New Jersey chapter of LOPSA meetings then... what are you waiting for? Seriously, folks! They have free pizza! What could be better than pizza, soda, and geekery? I'll be the speaker at the Thu, Oct 4, 2012 meeting. My topic will be: "Deploying IPv6 in the enterprise: How to convince your boss to approve your big plan" The New Jersey chapter is the only LOPSA chapter that hosts its own annual conference. It's a great bunch of folks and I encourage you all to attend.
Spiceworks interview
I've been interviewed on SpiceWorks. Read it here: Check it out! Interesting trivia: The IT department where my S.O. works uses Spiceworks' ticket system.
Join me at the October NYLUG meeting!
I'll be the speaker at the Thursday, October 11, 2012 NYLUG meeting in Manhattan (Chelsea, 14th and 9th ave). http://nylug.org I'll be talking about the Ganeti open source project which I'm involved in. The title of the talk will be: "Ganeti Virtualization Management: Improving the Utilization of Your Hardware and Your Time" If you are in or near NYC, I hope to see you there! Seating is limited. Please RSVP. http://nylug.org Tom
Join me at the October LOPSA Los Angeles chapter meeting!
LOPSA-LA has a dinner on Tuesday, October 16, 2012. I'll be in the area for MacTech and they've asked me to give an after-dinner talk about Time Management. When: Tue, Oct 16, 7pm - 9pm. Location: Sheraton Universal Hotel's Californias Restaurant (333 Universal Hollywood Drive, Universal City, CA 91608) Topic: Time Management for Sysadmins: Impossible or are other people to blame? In this talk I'll explain why the fact that you can't manage your time effectively is everyone else's fault, not yours. I'll blame Darwin, your boss, your users, and maybe even your mom. There are a few solutions, which I'll discuss briefly.
New website! http://www.opsreportcard.com
I've gotten a lot of positive feedback about The Limoncelli Test. So much so, that Peter Grace and I have put all the material on a website called www.OpsReportCard.com. We hope to add resources that help you achieve these 32 points of enlightenment but for now it is mostly the same as The Test. We're also considering making selling an ebook based on the material. Post to the comments section here if you like that idea. We hope you enjoy it! www.OpsReportCard.com. Tom
DevOps: "have more outages" Part 2 of 3: The interview
I moderated a discussion with Jesse Robbins, Kripa Krishnan, John Allspaw about Learning to Embrace Failure. This is the first time you'll see Google reveal what they've been doing since 2006. Read the entire discussion in the new issue of ACM Queue magazine: Resilience Engineering: Learning to Embrace Failure Participants include Jesse Robbins, the architect of GameDay at Amazon, where he was officially called the Master of Disaster. Robbins used his training as a firefighter in developing GameDay, following similar principles of incident response. He left Amazon in 2006 and founded the Velocity Web Performance and Operations Conference, the annual O'Reilly meeting for people building at Internet scale.
Europe officially runs out of IPv4 addresses
Earlier today, the RIPE NCC (Réseaux IP Européens Network Coordination Centre) announced it is down to its last "/8" worth of IPv4 addresses. This means that it is no longer possible to obtain new IPv4 addresses in Europe, the former USSR, or the Middle East, ... http://arstechnica.com/information-technology/2012/09/europe-officially-runs-out-of-ipv4-addresses/ I'll be doing my "Convince your boss to deploy IPv6" talk at the New Jersey chapter of LOPSA meeting next month. That's thursday, oct 4th near Princeton, NJ.
DevOps: To increase reliability you need to have more outages
Here's a good strategy to improve the reliability of your systems: Buy the most expensive computers, storage, and network equipment you can find. It is the really high-end stuff that has the best "uptime" and "MTBF". Wait... why are you laughing? There are a lot of high-end, fault-tolerant, "never fails" systems out there. Those companies must be in business for a reason! Ok.... if you don't believe that, let me try again. Here's a good strategy to improve the reliability of your systems: Any time you have an outage, find who caused it and fire that person. Eventually you'll have a company that only employs perfect people.
AT&T Survey
I got a survey from AT&T Wireless that asked a lot of questions comparing my experiences between WiFi and 3G on my AT&T mobile phone. If I were to reverse-engineer what they were getting at, either (a) they want to figure out why I dislike WiFi so they can fix those problems and encourage people to move traffic off their over-stressed 3G network, or (b) they need data to back up their coming campaign to bad-mouth WiFi and tell everyone to pay for their over-priced 3G. Based on the tone of the questions, I really think it is "b".
MacTech 2012 session listing is up!
http://www.mactech.com/conference/sessions I'll be speaking on Thursday. Don't miss this great conference, October 17-19, 2012 in Los Angeles.
How I would teach a university-level sysadmin degree
The coursework would be very focused on understanding the internals of each layer of the stack. To make a comparison to the auto industry: Your training won't result in you being a mechanic that can follow the manufacturer's manual: you will be the person that can write the manual because that's how much you understand how the car works. But the real change I'd like to see is how the labs are done. When you enter the school they give you 12 virtual machines on their VMware cluster (or Ganeti cluster). In phase one you go through a progression that ends with turning those 6 machines into 2 load balancers, 3 web servers, a replicated database, a monitoring host, etc.
4 unix commands I abuse every day
A co-worker watched me type the other day and noticed that I use certain Unix commands for purposes other than they are intended. Yes, I abuse Unix commands.
Back To School Sale: "Time Management for Sysadmins" for 50% off!
I'm proud to announce that TM4SA has been selected to be featured on this year's O'Reilly Back-to-School Special. The special runs this week only, from Sept 4th to the 11th. Save up to 50% on books, videos and courses. To receive the discount start shopping using this link http://oreil.ly/SUPaaT or use discount code "B2S2". Happy savings to all students and non-students alike!
Google reveals Goobuntu!
I'm so proud of my coworker Thomas Bushnell giving an amazing talk at LinuxCon, the Linux Foundation's annual North American technical conference. For the first time Google revealed details about how Google manages thousands of Linux desktops. We start with Ubuntu LTS, add a packages that secure it and let us manage it better, and ta-da! Read the entire article!
Android Calendar without Google calendar
Occasionally I get asked for a system that can keep a todo list and calendar and sync to a desktop directly i.e. without going through a internet-based system like Google Calendar. Pimlical has a new product that does this. I haven't tried it, but I was a big fan of their Palm OS products so I thought I'd give it a plug on my blog. The new release of Pimlical/Android and Pimlical/Desktop let you sync between your Android phone and desktop using "DirectSync" rather than syncing via Google Calendar. The press release I received says this "Removes any concern about security/safety of personal data by bypassing the cloud and Google's servers."
Matthew Sacks' new DevOps book ships!
Congrats to Matthew on the release of his new book: Pro Website Development and Operations: Streamlining DevOps for large-scale websites I look forward to reading my copy!
OpenFlow: A Radical New Idea in Networking
My ACM Queue magazine article OpenFlow: A Radical New Idea in Networking has been re-printed by CACM. Their version uses a different font and adds an awesome graphic by Jason Cook. Check it out!
Using Google 2-factor authentication on Linux or FreeBSD
HowToGeek posted a great explanation (with screenshots) of how to use Google's two-factor authentication on a Linux system. How to Secure SSH with Google Authenticator's Two-Factor Authentication If you use FreeBSD there are packages that install the same PAM module:portinstall -P security/pamgoogleauthenticator Some nice features: It is time dependent. Does not rely on Google servers. You don't have to set up a server either. There are iPhone and Android apps. Both are open source so you can independently verify their security. It is a PAM module, so it works with everything. It is also open source, thus can be independently verified.
Google hiring Ganeti software developers in Munich
As many of you know, I work at Google supporting the Ganeti open source project use within Google. I'm on the New York team that does certain functions and the Munich team is responsible for the open source project itself. Both are hiring, but this post is about Munich. The developers in Munich are taking on some new and exciting work related to the Ganeti open source project. We have a vision of where Ganeti can go and need experienced developers to make it happen. If you are interested in working in Munich on an exciting and important open source project, please check out this job advert: Google hiring Ganeti software developers in Munich Most of the Ganeti project is in Python with some Haskell.
PICC Conference becomes "LOPSA-East", May 3-4, 2013
We're 3 years old and as we have finally gotten our foothold it seems like a good time to pick a name that more accurately depicts who we are and what we do. Changing the name of the conference is a very serious matter. It is not something we take lightly. At the current growth rate this is likely to be our last opportunity to change the name (our registration numbers for the first three years were 81, 98 and 127. We hope to grow to 150-200 which is a good size for a regional conference). At the end of PICC '12 we surveyed the audience about possibly changing the name of the conference.
ANNOUNCEMENT: I'll be speaking at MacTech Conference 2012.
http://www.mactech.com/conference/sessions My talk will be titled, "Time Management tips for Mac Admins". I won't be explaining how to get Siri to schedule an appointment for you (you should be able to figure that out for yourself). I'll talk about better ways to organize your day, your time, and multiply your team's effectiveness. The Conference is October 17-19, 2012, in Los Angeles, at the Sheraton Universal hotel.
Record tonight's Mars event on NASA-TV
My cable company (Verizon FIOS) doesn't carry NASA TV channel (according to the NASA TV FAQ they can have it at no cost, which means they must not carry it because they hate space, America, freedom, and puppies). Since it is on very late in my timezone, I was hoping to Tivo it but since I don't get the channel that wasn't going to work. Luckily I found this script that lets me easily record the stream to disk using mplayer. On my mac I installed mplayer using MacPorts ("sudo port install mplayer") and the script worked on my first try.
Link of the week: The carpets are so clean, we don't need janitors!
Mark's Stories: "The carpets are so clean, we don't need janitors!" At one company I worked at, one of the problems it didn't have was IT. When someone was hired, by the time they got to their new desk, there was a computer on it with the correct image on it, their desk phone worked, their email worked... Follow the link for the full story. This is exactly why we felt it was so important that The Practice of System and Network Administration have an entire chapter about how to have great visibility when things are going well.
"IPv6 is now a reality in terms of adoption"
"There is a myth that IPv6 is only for those in Asia, but that's not true. According to new data discussed this week at an IETF conference, there are more IPv6 users in the U.S than anywhere else in the world -- coming in at 3 million. From the article: 'George Michaelson, senior R&D scientist at APNIC (Asia Pacific Network Information Centre) has a reasonable idea of what the current levels are globally for IPv6 adoption, thanks to some statistical research he has been doing. In his view, IPv6 is now a reality in terms of adoption. " I think you're used to us standing up and saying 'woe is me, woe is me, v6 isn't happening," George Michaelson, senior R&D scientist at APNIC (Asia Pacific Network Information Centre) said. "
Dear Mom And Dad
Dear Mom And Dad, Many times I've tried to explain to you what I do for a living. " Computer system administrator" or "sysadmin" is a career that is difficult to explain and I'm sure my attempts have left you even more confused. I have good news. Oxford University Press has just published a book by 4 scientists who video taped sysadmins doing their job, analysed what they do, and explains it to the non-computer person. They do it by telling compelling stories of sysadmins at work plus they give interesting analysis with great insight. Why did they do this? Because businesses depend on technology more and more and that means they depend on sysadmins more and more.
Sysadmin Appreciate Day
Happy Sysadmin Appreciation Day. Whether you keep desktops running, LOLcats broadcasting, payrolling systems paying, or blogs blogging, or any of the myriad things sysadmins do, keep doing it with integrity, grace, patience, and love. The more the world depends on computers for everything from the water we drink to the food we eat, system administration (or system engineering, or devops, or networking engineering, or storage engineering, or whatever you call yourself) is one of the most important jobs in the world. And the greatest. Appreciate that!
PCAST: 1,000Mhz for public use
If you suffered through my long rant about a totally different way to allocate wireless spectrum which would benefit everyone then you'll be happy to read this Arstechnica article about Obama's PCAST initiative moving forward. The old way to allocate frequencies was to give different industries their own "block". Each radio station, TV channel, WiFi protocol, etc. gets a block. This is inefficient. The new way is to have intelligent hardware that can share the spectrum: detect if someone is already broadcasting and back off seamlessly. The difference is that the first system is the best 1930s technology could provide. The latter is what you can do with modern systems where a computer can be programmed to actively monitor and control what is going on.
Arnie, the Doughnut! (NYC)
To all my NYC friends: Come see my sister's musical! 6 shows only! See our video preview here: http://bit.ly/Mjrfll Read the Arnie feature in Time Out New York: http://bit.ly/NKscNk The New York Musical Theatre Festival (NYMF) Presents ARNIE THE DOUGHNUT Six shows only beginning tonight - July 13th TICKETS ARE EXTREMELY LIMITED - BUY NOW! HOW TO GET YOUR TICKETS: ONLINE: Click HERE! BY PHONE: CALL 212- 352- 3101 IN PERSON: at the NYMF Hub, 330 West 42nd Street Arnie will be at the PTC Performance Space, 555 West 42nd Street: Friday, July 13th, 2012 at 7:00 pm Saturday, July 14th, 2012 at 1:00 pm Saturday July 14th, 2012 at 4:00 pm Wednesday July 18th, 2012 at 1:00 pm Saturday, July 21st, 2012 at 11:00 am Saturday July 21st, 2012 at 2:00 pm Arnie, a lovable chocolate-frosted doughnut with rainbow sprinkles, is the happiest pastry in the bakery when he's ...
Pre-order "Taming Information Technology" because it is awesome
"Taming Information Technology: Lessons from Studies of System Administrators" by Eser Kandogan, Paul Maglio, Eben Haber and John Bailey Scientists video tape sysadmins at work then analyse the footage, making interesting observations about what we do, how we do it, and why. For every CEO that thinks sysadmins just lay about all day, this book shows what risky, dangerous work we do. For the parent that doesn't quite understand what their son or daughter the system administrator does, this book spells it out in plain language stories of what we do. For the person that thinks sysadmins just sit around fixing computers with a screw driver and CD-ROM, this book shows real situations where outages cost millions and teams of technical people battle clueless (and not so clueless managers).
Final Four Days for HOPE Tickets!
A public service message to people in the NYC-area: From: Hackers On Planet Earth <[email protected]> Date: Thu, Jul 5, 2012 at 3:44 PM Subject: [nine-announce] Final Four Days for HOPE Tickets! To: [email protected] As we're closing in on HOPE Number Nine, we need to inform you of a very important deadline: advance ticket sales will closing on Sunday, July 10th. Advance tickets help us to pay for a lot of the expenses involved in putting on an event like HOPE. Renting three floors of a hotel in midtown Manhattan can be a bit pricey, so every little bit helps. Not to mention that it saves attendees from paying the more expensive price at the door.
OpenFlow: A Radical New Idea in Networking
Queue Magazine (part of ACM) has published my description of OpenFlow. It's basically "the rant I give at parties when someone asks me to explain OpenFlow and why it is important". I hope that people actually involved in OpenFlow standardization and development forgive me for my simplifications and possibly sloppy use of terminology but I think the article does a good job of explaining OF to people that aren't involved in networking: http://queue.acm.org/detail.cfm?id=2305856 I hope that OpenFlow is adopted widely. It has some cool things in it. Enjoy. Tom
Half-baked thought of the day: job titles
Is the person that hand-crafts a bed out of wood he personally chopped from the forest, designed, and built doing the same job as someone that builds a bed factory that makes 100 beds a day? I don't think so. So why do we use the same job title for a person at a 10-person company that maintains 1-2 custom-built, servers, and spends 70% of his or her day answer user questions as the person that maintains a massive 1,000-CPU cluster using Cfengine/Puppet/Chef to orchestrate hundreds of web front-ends, dozens of database servers, and huge numbers of application servers all mass-produced and automated?
Discrimination means missing out on hiring the best sysadmins
Rikki Endsley posted to Google Plus this week:I saw this tweet today from a hiring manager: "Just interviewed for a sysadmin. I'm struggling since she has no social footprint. Is that wrong, or should social be key?" What are your thoughts on a 'social footprint' requirement for sysadmins? link I'm very disturbed hearing a hiring manager say this. " Social Footprint" means how visible the person is on social networks like Facebook, G+, Twitter and so on. What does that have to do with whether or not the person is a good system administrator? It could be a bad thing if it means the person is anti-social or doesn't keep up with the latest innovations.
MicroReview: Tarsnap
I started using Tarsnap to backup my personal server "to the cloud". I found it was quick to set up, easy to learn, and works pretty well. And, yes, I've already made a wiki page that documents how my monthly restore tests will be done. The data is encrypted, which means if you lose your crypto key you can't get your data back so my restore test is done from a different machine to force me to have a copy of the key stored safely off-line. If you are looking to do backups over the internet, check this out.
When vendors don't follow through
Someone recently asked me how I should handle a vendor that wasn't being responsive: "Twice now I've sent the support team requests and received an automated response and little else. The first ticket took a month for them to answer. The second was closed with a note that they had tried to call me, but I didn't answer. Mind you, they never emailed me to say they had called." I've found that when opening a "case" or "ticket" with a vendor you have to "stay on them" or, more accurately, "manage it ruthlessly until the issue is resolved". Very few vendors are good at follow-through on tickets.
ProTip: make rsync fail more reliably
A co-worker of mine recently noticed that I tend to use rsync in a way he hadn't seen before: rsync -avP --inplace $FILE_LIST desthost:/path/to/dest/. Why the "slash dot" at the end of the destination? I do this because I want predictable behavior and the best way to achieve that is to make sure the destination is a directory that already exists. I can't be assured that /path/to/dest/ exists, but I know that if it exists then "." will exist. If the destination path doesn't exist, rsync makes a guess about what I intended, and I don't write code that relies on "guesses".
Researching looking for Sysadmins to take survey
Short version: Take this survey, you might win a $100 Amazon gift card but more importantly you'll be helping great research. Long version: Hello All, Some of you may recognize my name - and some of you may recognize my research. :) I study sysadmins and help organizations find ways to understand the work of system administration better, in part, so they can build better software. I conducted a study a few years ago that I presented at LISA, and I'm working on extending it to a journal paper. This extended publication would dramatically increase readership of the results to include top researchers and executives, so I think it's a worthy endeavor.
LOPSA Elections
The LOPSA board elections are happening. Turn-out so far is around 11%, which is pathetic. Folks, if you are a member, vote! This mailing list post has more details: https://lists.lopsa.org/pipermail/discuss/2012-June/008518.html Voting takes just a few minutes. (And if you aren't a member, join up and vote!)
Reducing latency the Google Way
Website latency is a major issue. Jeff Dean from Google has given a presentation that, for the first time, reveals some of the techniques used at Google. Seeing the presentation reminded me of the "shock and amazement" I had when RAID was invented (yes, kids, RAID used to be a "new thing"). An abstract and slides are available here http://research.google.com/people/jeff/latency.html The slides are well worth a read.
The road to intentional, formal, system administration education
System Administration is maturing and, yet, there is no accepted standard curriculum. It is ironic, and somewhat scary, that a field that society is more and more dependent on has no formal, accepted, educational path. I propose a framework that is similar to that of the electrical/electronics industry. To become a doctor there is a generally accepted educational path. Undergraduate "pre med" or biology program, medical school, internship, and so on. It gives me great comfort that the doctors that I see follow a formal path. Sysadmins, however, often "fall into" the career. I know many sysadmins whose formal education is in physics, for example, because it teaches them the rigors of mathematics, measurement, and thinking in terms of systems.
What language should a sysadmin learn?
Someone recently asked me what language a sysadmin should learn. If you are a sysadmin for Windows the answer is pretty easy: PowerShell. The answer is more complicated for Unix/Linux sysadmins because there are more choices. Rather than start a "language war", let me say this: I think every Unix/Linux sysadmin should know shell (sh or bash) plus one of Perl, Ruby, Python. It doesn't matter which. The above statement is more important to me than whether I think Perl, Python or Ruby is better, or has more job openings, or whatever criteria you use. Let me explain: It is really important to learn bash because it is so fundamental to so many parts of your job.
Re: My question for LOPSA board candidates
[Note: This is a first draft and needs a lot of editing but I know I'm not really going to come back and edit it so I might as well post it today.] LOPSA had their first "meet the candidates" a few weeks ago. I had blogged ahead of time the question I was planning to ask. The question: "I'd like to know about your experience with community-based projects. Please tell us about a project that you took responsibility for seeing through to completion. Please, only projects that are "done" or have reached a self-sustaining mode only. One or two sentences is fine.
Stipend competition to attend 2012 USENIX Women in Advanced Computing Summit
Quoting from email I received: LOPSA is pleased that USENIX shares our goal of bringing attention to the various issues facing women in our industry by hosting the Women in Advanced Computing Summit. This summit is part of their Federated Conferences week, which also includes the ATC conference and others. LOPSA would like to show our support in this area and provide something concrete toward the topic. Matt (from the LOPSA Board) came up with a great idea to provide a stipend to assist someone in attending the conference. We will award based on submission of an essay, but I'll leave those details to the posting about it.
Tom @ LOPSA PICC 2012, New Brunswick, NJ, May 11-12, 2012
I'll be there any I hope you will be there too!
Structured Speaking
I've found that a structure that gives obvious "book-ends" around each topic make it easier for the audience to follow. Most of my talks lately have been either 4-5 small case studies or a Top 10 List. Each case study is a repetition of "who are the players, what happened, what did we learn". The repetition gives the audience a clear understanding of "we're moving to the next topic now" because they see the pattern. In a Top 10 list there is the obvious "book end" of announcing the next number. I started doing this after seeing too many presentations where the presenter runs topic to topic smeared together with very little separation.
I'll be speaking at LOPSA-NJ on Thursday
The NJ Chapter of LOPSA is graciously letting me do a dress rehearsal of my Ganeti presentation that will be presented at the PICC Conference next week. http://picconf.org If you can't make it to the conference or just want to be able to attend one of the conflicting sessions, this is a great opportunity for you. Complete details are on the www.lopsanj.org website. Topic: Ganeti Virtualization Management:Improving the Utilization of Your Hardware and Your Time Date: Thursday, May 3, 2012 Time: 7:00pm (social), 7:30pm (discussion) If you are planning on coming please RSVP so we have the right amount of pizza.
Tom @ LILUG, Wed, April 10, 2012, Woodbury, Long Island, NY
I'll be giving a talk about Ganeti, the open source virtual cluster manager April 10th @ 8:00pm at the Woodbury Campus of Cold Spring Harbor Lab, in the Woodbury Auditorium. For more information visit: http://lilug.org See you there!
An Illustrated Guide to SSH Agent Forwarding
I don't think I really understood SSH "Agent Forwarding" until I read this in-depth description of what it is and how it works: http://www.unixwiz.net/techtips/ssh-agent-forwarding.html In fact, I admit I had been avoiding using this feature because it adds a security risk and it is best not to use something risky without knowing the internals of why it is risky. Now that I understand it and can use it, I find it saves me a TON of time. Highly recommended (when it is safe to use, of course!) Tom
Tom @ LOPSA-NJ, Thu May 3, 2013, Lawrenceville, NJ (near Princeton)
I'll be speaking at LOPSA-NJ's May meeting about Ganeti, the open source project I'm involved in. The title is "Ganeti Virtualization Management: Improving the Utilization of Your Hardware and Your Time". For more information check out the LOPSA NJ web site.
PICC opening keynote: Bill Cheswick
The PICC committee is excited to announce our opening keynote speaker: Bill Cheswick, Security guru and co-author of "Firewalls and Internet Security" Topic: Rethinking Passwords "We've known that passwords have been inadequate for over thirty years and they have only gotten worse. Can we escape the varying 'eye-of-newt' password rules that plague everyone's online lives? Can we get grandma safely to the other side of the authentication street? I will review some of the many research ideas that have been proposed, and offer some suggestions toward getting us out of this thicket." DINNER will be provided to all attendees on Friday at 6pm; Bill's talk will begin after dinner (8pm).
The internet needs a bill of rights
[first draft] Someone asked me about "The Internet Needs a New Pair of Pants" and I thought it would be a good chance to post some thoughts I've had. For the most part he's asking the wrong questions. Only #10 and #11 really matter. But first a quick tangent... We don't "store data" on the internet. You can 'store data' by putting it on a hard drive and then powering it off. That's easy. Anyone can do that. What you do on the internet (or "in the cloud") is you make data available (either to everyone, a restricted group, or just yourself).
Women in Advanced Computing (WiAC) Summit, June 12, 2012
Usenix is sponsoring the first Women in Advanced Computing (WiAC) Summit to run during Federated Conferences Week in Boston. WiAC will be all day June 12th, 2012. Carolyn Rowland and Nicole Forsgren Velasquez are co-chairs. Carolyn recently posted on G+ a request for ideas: What would make this a must-attend event? What topics should we cover in order to appeal to women of varying professions and backgrounds: researchers, to developers, sysadmins, IT managers, etc.? Carolyn wrote "We'd like this year to be the start of a recurring Usenix event that allows people who believe we need to support women in the computing professions to come together to share ideas, meet new people and get inspired."
IPv6 is now the default
RFC 6540: IPv6 Support Required for All IP-Capable Nodes This new RFC basically says that vendors can no longer consider IPv6 as an optional feature. If you say it supports 'IP' you better include IPv6. The RFC specifically calls out these best practices: New IP implementations must support IPv6. Updates to current IP implementations should support IPv6. IPv6 support must be equivalent or better in quality and functionality when compared to IPv4 support in a new or updated IP implementation. New and updated IP networking implementations should support IPv4 and IPv6 coexistence (dual-stack), but must not require IPv4 for proper and complete function.
Tom speaking at LILUG (Long Island) on Tuesday about Ganeti and virtualization
I'll be the guest speaker at LILUG this week. If you've never been to LILUG and live in Long Island this is a great time to check out this great Linux Users Group! I'll be giving a talk about the Ganeti open source project. Ganeti is a system that manages clusters of virtual machines. In my demo I'll build a cluster right in front of everyone and show off some of its features. If you use Xen or KVM virtual machines, Ganeti will help you do it easier, cheaper and more reliably. Tuesday, April 10th @ 8:00pm at the Woodbury Campus of Cold Spring Harbor Lab, in the Woodbury Auditorium.
Mac users: update NOW
"More than 600,000 Macs have been infected with a new version of the Flashback Trojan horse that's being installed on people's computers with the help of Java exploits, security researchers from Russian antivirus vendor Doctor Web said on Wednesday."Fast-growing Flashback Botnet Includes Over 600,000 Macs, Malware Experts Say Technical details here: New Flashback Variant Changes Tack to Infect Macs This is serious, folks. Run your "Software Update" now and reboot. Help your non-technical friends do it too.
USENIX Announces New Executive Directors: Anne and Casey!
This week the USENIX Board announced that Anne Dickison and Casey Henderson have been appointed USENIX Co-Executive Directors effective April 2, 2012. Anne and Casey have been with the Association since 2003 and 2002 respectively. Previous to their new appointments Anne was the USENIX Marketing Director and Casey was the Information Systems Director. If you've been at a USENIX conference since 2002 you've probably met them or seen them in the registration area. It was a delight to work with both of them when I co-chaired LISA '11 last year. Anne's super-power was not freaking out when I was and Casey's super-power was not being annoyed by my constant flood of technical requests.
My question for LOPSA board candidates
"I'd like to know about your experience with community-based projects. Please tell us about a project that you took responsibility for seeing through to completion (i.e. did most of the work). Please, only projects that are "done" or have reached a self-sustaining mode only. One or two sentences is fine. It doesn't have to be a project where you thought of the idea, just one where you assured it reached the finish line." I look forward to hearing their answers.
See you at CrabbyAdmins on Wednesday! (April 4)
The Baltimore/DC chapter of LOPSA is called CrabbyAdmins (a reference to the crab industry in the Chesapeake bay). I'll be speaking there on Wednesday night about the Ganeti open source project. This meeting will be in Columbia, MD, hosted at Next Century (across the street from OmniTI). It will run from 7pm-9pm. If you are interested in inexpensive virtualization or just live in the area and want to meet your local sysadmin community (or me!), please stop by! Full info and directions here: http://bit.ly/HfiVNE Tom
RFC 2410: NULL is not a joke (nor an April Fools joke)
In 2007 when Peter H. Salus and I published all the April Fools RFCs in one book we also included the poetry RFCs and the funny RFCs published outside of April Fools timeframe. Speaking of which... we included "RFC 2410: The NULL Encryption Algorithm and Its Use With IPsec" because, well, I thought it was funny. Specifying an encryption scheme for IPsec that does not encrypt the bytes is, well, funny. It turns out it wasn't published as a joke. Oops. No offense meant to the authors R. Glenn and S. Kent. Nobody pointed this out to me until years after the book was printed.
Tom @ LILUG, Tue, April 10, 2012, Woodbury, Long Island, NY
I'll be giving a talk about Ganeti, the open source virtual cluster manager April 10th @ 8:00pm at the Woodbury Campus of Cold Spring Harbor Lab, in the Woodbury Auditorium. For more information visit: http://lilug.org See you there!
PICC keynote announced: Rebecca Mercuri on "The Black Swan and Information Security"
[Note: "Early-bird" price ends in 3 days! Don't lose the discount!] The PICC committee is excited to announce our closing keynote speaker: Rebecca Mercuri on "The Black Swan and Information Security" Dr. Mercuri is the lead forensic expert at Notable Software, Inc. Her caseload has included matters from contraband, murder, viruses and malware, and election recounts (most notably Bush vs. Gore). She has testified on the federal, state, and local level as well as to the U.K. Cabinet. Talk abstract: The economic theories proposed by Nassim Nicholas Taleb in his book "The Black Swan" have strong parallels in information security. Indeed, the concepts of robustness and risk assessment mentioned in Taleb's writing are also well known to those who design software and systems intended to withstand attack.
Cascadia IT 2012: A big success!
Thanks to everyone that attended my tutorials and talk at Cascadia IT 2012. I finally got the timing right on both the Intro to Time Management for Sysadmins as well as The Limoncelli Test. I also gave a talk about the open source virtual cluster manager called Ganeti which I'm a part of via my job at Google. I'll be repeating this talk at CrabbyAdmins in Columbia, MD on Wed, April 4th. After the conference I got email from a fan that wrote "just an FYI, I've placed your book on a custom foam pedestal at my desk. Gave you the old WA state classiness."
Printable version of The Limoncelli Test
As requested, I've made a printable version of The Limoncelli Test. http://everythingsysadmin.com/the-test.pdf http://everythingsysadmin.com/the-test.html I'll be teaching a class based on this article in Fri, March 23 2012 at the Cascadia IT Conference in Seattle, WA and on Fri, May 11, 2012 at the LOPSA PICC 2012 Conference in New Brunswick, NJ. Seating is limited. Register soon! I'll also be teaching my Time Management class and giving an invited talk on the Ganeti virtual server cluster management software. See you there!
Revamping "The Limoncelli Test" for Cascadia IT 2012 (Seattle, WA)
If you are in the Pacific North West I hope you are planning on attending the Cascadia IT conference March 23-24, 2012. And if you are attending, I hope you have signed up for one or both of my tutorials. I'll be teaching "Intro to Time Management" and a new class "The Limoncelli Test: Evaluating and improving sysadmin operations". " The Limoncelli Test" is a tutorial I first did last December at LISA '11. It was kind of a half-baked idea and I got a lot of really excellent feedback. I've revamped a lot of it and I think the class it going to be much better.
Fear of Rebooting
I have two fears when I reboot a server.[1] Every time I reboot a machine I fear it won't come back up. The first cause of this fear is that some change made since the last reboot will prevent it from being able to reboot. If that last reboot was 4 months ago it could have been any change made in the last 4 months. You spend all day debugging the problem. Is it some startup script that has a typo? Is it an incompatible DLL? Sigh. This sucks. The second cause of this fear is when I've made a change to a machine (say, added new application service) and then rebooted it to make sure the service starts after reboot.
Using statements of "Undeniable Value"
I got email from someone that was having trouble convincing a boss to spend money on new PCs. The current ones are 5 years old (or older). It is a small company, owned by one man, and he runs every detail. Part of my advice to him was: Use "undeniable value" to describe requests. State things in terms of "undeniable value". The statement "we need a faster PC" doesn't do that. To you it has undeniable value: faster is better and will solve a list of problems. But to a non-technical person they can't guess all the things in your head that it will solve.
Seats still available for my classes at Cascadia IT 2012, Seattle, WA, March 23-2, 2012!
I'm teaching Intro to Time Management for Sysadmins and a new class based on The Limoncelli Test. Register before the classes fill up! http://www.casitconf.org/ My classes are both on Friday, March 23. This is a rare opportunity to catch my classes in the PNW area. The League of Professional System Administrators and the Seattle Area System Administrators Guild are proud to present the 2012 Cascadia IT Conference. Cascadia 2012 is a regional IT conference for all types of system administrators - computer, database, network, SAN, VMware, etc. It will take place on March 23 - 24th (Fri - Sat) of 2012 at Hotel DECA in Seattle's University District.
Premier 100 IT Leader profile: Ralph Loura
Congrats to Ralph Loura for being named one Computer World's Premier 100 IT Leaders. Premier 100 IT Leader profile: Ralph Loura Ralph Loura was my second manager at Bell Labs. I can't tell you which of the anonymized stories in The Practice of System and Network Administration are secretly about him, but I can say that he was the manager that encouraged me to start writing papers for conferences like Usenix LISA, which lead me to being noticed by Addison-Wesley, which got me my book deals, and the rest is history. Congrats, Ralph!
Premier 100 IT Leader profile: Ralph Loura
Congrats to Ralph Loura for being named one Computer World's Premier 100 IT Leaders. Premier 100 IT Leader profile: Ralph Loura Ralph Loura was my second manager at Bell Labs. I can't tell you which of the anonymized stories in The Practice of System and Network Administration are secretly about him, but I can say that he was the manager that encouraged me to start writing papers for conferences like Usenix LISA, which lead me to being noticed by Addison-Wesley, which got me my book deals, and the rest is history. Congrats, Ralph!
Registration is OPEN! East coast LOPSA-PICC Sysadmin Conference, May 11-12, 2012, New Brunswick, NJ
Register now and avoid the rush! http://picconf.org Space is limited! Registration is open for the 2012 LOPSA PICC conference, May 11-12, 2012 at the Hyatt Regency hotel in New Brunswick, NJ. Sysadmins and IT workers from Maine to Virginia are expected to attend the most talked about, community-driven, sysadmin conference of 2012! We're excited to announce our slate of speakers and world-class tutorials for 2012. Complete details at http://picconf.org FRIDAY is all about world-class training: This 2-day conference starts on Friday with long-format tutorials on a wide variety of topics by world-class instructors: Topics include PowerShell, Puppet, Amazon Web Services, WordPress, DNSSEC, IPv6 and much, much more!
Usenix SAGE is now Usenix LISA
Nothing is changing except the name. https://www.usenix.org/lisa I think this is a good thing. There's too much confusion over what is SAGE and what is LISA. Now people can focus on what LISA does, not figuring out which name to use when. Tom
Who to trust?
We are two people. The person that calmly makes plans and the person that executes them. The first person is calm and thoughtful and has the right amount of doubt to make sure a plan will work. The second person rushes to judgement and is full of hubris. " What was I thinking! I can do it more/better/differently." is what the second person says. The second person often forgets how much work went into the planning or the rationale for why things were set in a particular order. If an outside knows of the plan, it can confuse things if the second person "optimizes" the plan leaving those other people out of the loop.
See all the "Best Picture" Oscar nominated films in 2 days
I usually don't blog about something that has so little to do with system administration, but in this case I consider it a "time management tip". Each year AMC theaters run their "Best Picture Showcase". They show all of the "best picture" nominated films in a marathon. They show 4 films on one Saturday and the other 5 on the following Saturday. This year it is Sat, Feb 18 and Sat, Feb 25. You can buy tickets for either or both days. (Some theaters show all 9 in a row on one day.. 23 hours of movies!) We went last year and it was awesome.
What to do about SOPA/PIPA?
The headlines like, "Sen. Reid kills SOPA bill" should really read, "Sen. Reid tells people SOPA is dead so Hollywood can work on more stealthy bill." What to do about this kind of thing in general? Joel Spolsky nails it: (1) The internet seems to ignore legislation until somebody tries to take something away from us... then we carefully defend that one thing and never counter-attack. Then the other side says, "OK, compromise," and gets half of what they want. That's not the way to win... that's the way to see a steady and continuous erosion of rights online.
Time Management class at SCALE
Note: SCALE is the Southern California Linux Expo which will be held January 20-22, 2012 at the Hilton Los Angeles Airport hotel. Aleksey Tsalolikhin will be teaching a SCALE University (a joint project between SCALE and LOPSA where LOPSA instructors teach classes on topics related to system administration as part of LOPSA's mission to educate on system administration topics) based on "Time Management for System Administrators". Here is a sample success story from a student that completed the practice run of the course a week ago: I think that "The Cycle" system is a pretty comprehensive approach to time planning, but very simple concept to implement.
Book Review: 'Pro Puppet" by James Turnbull and Jeffrey McCune
I think this is the first book that actually helped me see both the "big picture" of how Puppet's components fit together as well as learn the language itself. After months of reading Puppet online documentation this book put it all together in a way that gives me the confidence to start a big Puppet project. Trying to learn Puppet from the online documentation is often like learning to drive by studying how a car is manufactured. " Pro Puppet" doesn't suffer from that problem. The hardest part of doing Configuration Management is getting started. If you join a company that already uses Puppet it is easy to hop in and add to it.
ACM Queue Programming Challenge starts soon!
ACM Queue is hosting an online programming competition on its website from January 15 through February 12, 2012. Using either Java, C++, C#, Python, or JavaScript, code an AI to compete against other participant's programs in a territory-capture game called, "Coercion". The competition is open to everyone. Details at: http://queue.acm.org/icpc/
Reminder: RSVP for LOPSA-NYC's Tuesday meeting
If you are coming to hear my presentation, please RSVP so you can get into the building. (Try to arrive early... getting in takes a while if there is a line). RSVP using the link on this page. Tue, Jan 11. Starts at 7pm. Title: "SRE@Google: Thousands of DevOps since 2004"
Reminder: RSVP for LOPSA-NYC's Wedneday meeting
If you are coming to hear my presentation, please RSVP so you can get into the building. (Try to arrive early... getting in takes a while if there is a line). RSVP using the link on this page. Date: 01/11/2012 at 7pm Title: "SRE@Google: Thousands of DevOps since 2004"
Book-signing at Arisia, Boston, Sunday
If you are going to be at the Arisia science fiction and fantasy convention please stop by the autograph signing area (Galleria - Autograph Space) on Sunday, January 15 at 11:30-1pm. I'll be there along with more reputable folks like Adrianne Brennan and Greg R. Fishbone. They'll be signing their most excellent works of fiction. I'll be signing my technical books. I have no shame and will also sign other books too, as well as napkins and scraps of paper. I'm really just excited to show that I know how to work a pen.
Tom @ LOPSA-NJ, Thu Jan 5, 2012, Lawrenceville, NJ (near Princeton)
I'll be speaking at LOPSA-New Jersey on Thursday. This will be a repeat of the keynote I did in North Carolina last November. While it says "security" in the title, it will make sense whether you work in security or not. All are invited! (no charge to attend) Topic: You Suck At Time Management (but it isn't your fault!) Date: Thursday, January 5 2012 Time: 7:00pm (social), 7:30pm (presentation) Pizza and Soda being brought to you by: INetU Managed Hosting If you are planing on coming please RSVP so we have a good count for the Pizza. Location: Lawrence Headquarters Branch of the Mercer County Library2751 US Highway 1Lawrenceville, 08648-4132 So much to do!
PICC '12 planning committee needs YOU!
Interested in helping make PICC '12 happen? The committee is the most fun group of people I've planned a conference with. If you live within 500 miles of New Brunswick, NJ we'd love for you to help out. Commitment is about an hour a week plus a short phone conference call every other Monday at 8pm. Here's some typical volunteer tasks: (we'll ask you to pick one) Forward our mailings to user groups' mailing lists (The hard part is making sure it actually went out!) Someone to maintain our Facebook/LinkedIn/Twitter presences. Invent new ways to get the word out about the conference.
NYC and NJ LOPSA meeting reminders
I'm the speaker at LOPSA's New Jersey chapter on Thursday and at LOPSA's NYC chapter the following Tuesday. Obviously I have too much free time on my hands :-) But seriously... Thu, Jan 5 near Princeton: LOPSA-NJ. Title: "You suck at Time Management... but it isn't your fault" Tue, Jan 11 in NYC: LOPSA-NYC. Title: "SRE@Google: Thousands of DevOps since 2004" New members are always welcome! Hope to see you there!Tom
Tom @ LOPSA-NYC, Tue Jan 11, 2012, NYC, NY
I'll be the speaker at LOPSA NYC's meeting in January. This will be a repeat of the "DevOps@Google" talk that I gave at LISA 2011. It was very well-received. If you missed it at LISA, this may be your last chance to see it live. Official announcement: http://www.lopsa-nyc.org/content/google-sre-when-developers-and-sysadmins-collaborate-things-get-better-faster (Please register so you can get into the building. The registration form is at the bottom of the log post) The talk starts at 7pm. Please come early so you can get through security.
Tom @ LOPSA-NJ, Thu Jan 5, 2012, Lawrenceville, NJ (near Princeton)
I'll be speaking at LOPSA-New Jersey on Thursday. This will be a repeat of the keynote I did in North Carolina last November. While it says "security" in the title, it will make sense whether you work in security or not. All are invited! (no charge to attend) Topic: You Suck At Time Management (but it isn't your fault!) Date: Thursday, January 5 2012 Time: 7:00pm (social), 7:30pm (presentation) Pizza and Soda being brought to you by: INetU Managed Hosting If you are planing on coming please RSVP so we have a good count for the Pizza. Location: Lawrence Headquarters Branch of the Mercer County Library2751 US Highway 1Lawrenceville, 08648-4132 So much to do!
The Dumbest Idea In The World: Maximizing Shareholder Value
For a long time I've had some serious issues with CEOs putting such a focus on the stock price instead of customer satisfaction. I've usually figured that I was an outsider, too ignorant of how economics or how business works to know any better. In fact, there was a time (about 5 years ago) that I was seriously considering going for an MBA so I could understand this all better. However I realized that what I really wanted to do was wait for various principles to be explained (like, "focus on shareholder value") and bring up my all my counter-examples. That's not a good reason to get an MBA.
Super Wi-Fi is better than just "super"
[This is still at 'first draft' quality, but I thought I'd post it sooner rather than later. Please ignore the typos for now.] I recently twittered my delight that the FCC approval of "super Wi-Fi" is going to be regarded as a historic moment five years from now. I mean it. Here's why: In geek terms: This gives permission to treat the airwaves like Ethernet networking, not like Teleco networking. More modern and more flexible. In non-geek terms, this decision by the FCC makes it easier to innovate. It makes it safe and easy to try new things With the possibility of experimentation comes new applications and ideas.
Stopping SOPA
The problem with companies that used to support SOPA but have turned around, is that they supported it in the first place. The problem with stopping SOPA is that the people behind it are committed to bringing it back in another form, some day, some how. The problem with SOPA is that many of the bad things in SOPA are things that the U.S. government has been doing lately either unofficially or through "cooperation" with companies. The defeat of SOPA will not be the end of the general problem.
New Years Resolutions and Time Management
One part of "The Cycle" is that you should keep a list of long-term projects and review it every few months. There are two specific times you should always review it: around budget time (this is where you record those great projects that are so big they'll require funding), and around New Years (the list usually inspires good New Years Resolutions). The list has a couple "secret" functions. First, when your main todo list is growing out of control, it can be very useful to move some of the more audacious goals onto this list. Secondly, sometimes an idea is taking up brain-space and you just need to write it somewhere.
Config Management Rosetta Stone
Yesterday on the SysAdvent calendar Aleksey Tsalolikhin has an article about configuration management. It includes a comparison of how to the same in in various languages: bash, CFEngine, chef and Puppet. Seeing how the languages differ is very interesting! SysAdvent: December 19 - Configuration Management
Two interesting Python tutorials
A great explanation about "yield" followed by a discussion of coroutines and more: Generator Tricks for Systems Programmers http://www.dabeaz.com/generators/Generators.pdf In the sequel, he goes into even more detail and the uses all the information to write an operating system in Python. A Curious Course on Coroutines and Concurrency: http://www.dabeaz.com/coroutines/Coroutines.pdf
Message from the Program Co-Chairs
USENIX Association LISA '10: 24th Large Installation System Administration Conference (This "welcome" letter appeared on the USB stick given to all attendees. Since most people probably missed it I thought I'd repost it here.) Message from the Program Co-Chairs Dear LISA '11 Attendee, There are two kinds of LISA attendees: those who read this letter at the conference and those who read it after they've returned home. To the first group, get ready for six days of brain-filling, technology-packed, geek-centric tutorials, speakers, papers, and more! To those that are reading this after the conference, we ask, "What's it like living in the future?
Impromptu talk about Google SRE tonight at 9pm.
I've booked a BoF room at 9pm to give my talk "SRE@Google: Thousands of DevOps Since 2004". Tuesday, December 6, 9:00 p.m.-10:00 p.m., Fairfax B "Tom will describe technologies and policies that Google uses to do what is (now) called DevOps. Google doesn't just empower developers and operations to work together, we have a system that empowers all groups to be their own devops team. (This is based on my opening keynote at the Pittsburgh Perl Workshop.)"
LISA11 Diary: Tue, Dec 6
(In an effort to get these out sooner rather than later I'm not spending a lot of time editing and proofreading. You've been warned.) Daytime: Today I spent the day in the Advanced Technology Workshop. What is a workshop? People need a space to spend an entire day (or half-day) to talk about a topic. There are workshops for people researching certain areas and their workshop at LISA is a once-a-year touchstone to meet in person, give presentations, share ideas, and so on. The Configuration Management workshop is in its 11th year. In fact, Puppet was inspired by a debate (argument?)
LISA11 Diary: Mon, Dec 5
(In an effort to get these out sooner rather than later I'm not spending a lot of time editing and proofreading. You've been warned.) Again woke up around 6am. Rehearse parts of the tutorial, got breakfast at the Sheraton Club on the 29th floor. Tutorial: The Limoncelli Test: My first new tutorial in years! Based on this blog post, the tutorial lists 32 "best practices" that sysadmin teams should do. I had enough time to discuss half of them. At the start of the class I had everyone take the test, and then focused on discussing the ones that had a lot of "no" answers (by show of hands).
SysAdvent 2011 has started!
25 days of sysadmin articles from all sorts of people. http://sysadvent.blogspot.com Enjoy!
LISA11 Diary: Sun, Dec 4
Sunday I woke up around 6am, had breakfast at the hotel "club" on the 29th floor (great view!) Tutorial: Time Management for System Administrators: In the morning I taught a half-day class on Time Management. This is the "personal" time management side of things: making your life more sane. I've taught this class at LISA every year since 2005-ish and this year the turn-out was HUGE (80+ people). No matter how many times I teach this I get new and interesting questions each time. After the tutorial I autographed books and answered questions. Lunch: I had the lunch that comes with the tutorial sessions.
LISA11 Diary: Sat, Dec 3
Saturday, Dec 3: Getting There: Since the conference is in Boston, I decided to take the train rather than fly. Amtrak costs about the same but is faster due to the lack of 2-hour wait for TSA and other airport things. I arrived in Boston at about 1pm, checked in at the hotel, changed, and went to the lobby to hang out. Registration: Registration wouldn't open until 5pm so I hung out, talked with people, got some status updates from the Usenix staff about registration numbers and so on. Registration opened at 5pm spot on and I was 2nd in line :-) so I got registered fast.
Review of T-L-T Tutorial at LISA2011
Sysadmin1138 attended my LISA 2011 Tutorial "The Limoncelli Test" yesterday and wrote this excellent summary. Check it out: http://sysadmin1138.net/mt/blog/2011/12/lisa-2011-the-limoncelli-test.shtml Thanks for the write-up!
Usenix LISA attendees get free Wifi in their hotel rooms!
Usenix has negotiated with the hotel to get the wifi fee waived for any attendee that stays in the hotel as part of the Usenix block. When you sign in to the WiFi go through the process and agree to the $12.99/day (I think) charge, but when you check out it will be removed from your bill. The conference hotel is the Sheraton Boston Hotel, 39 Dalton Street, Boston, MA
Tom @ Usenix LISA 2011, Boston, MA, Dec 4-9, 2011
I'll be teaching 3 tutorials and one "guru" session. Plus, as conference co-chair I'll be on stage many other times too. Watch this space: http://www.usenix.org/events/lisa11
Usenix LISA schedule online
Use the "Guidebook" app for Phone/Android/WinPhone7/BlackBerry: here View the schedule in Google Calendar: here (click "+Google Calendar" in the lower right) iCal feed: here (iCalendar, Outlook and others) As an RSS feed: here Advice about the Guidebook app: To see the all the schedules merged (training, invited talks, etc.) click the "schedule" icon. To search, swipe left (like you are turning to the page before the first page). Mark items you want to attend and the "My Schedule" feature will just show those items Search is probably the easiest way to find all my talks. Search for limoncelli (there are 4; yikes!)
OrgMode for iPhone
http://mph.puddingbowl.org/2010/02/org-mode-in-your-pocket-is-a-gnu-shaped-devil/ What will they think of next?
For my birthday I'm giving the gift of time
If you register for USENIX LISA'11 by the end of my birthday (today, Fri, Dec 2nd) your name will automatically be entered into a drawing for two 30-minute, one-on-one time management coaching classes with me. (that's by midnight California time... even though I live on the east coast.) This is a fairly exclusive offer. I normally only do time-management coaching for co-workers. Official announcement here. See you in Boston! Tom
Following me on Google Latitude? Please use [email protected]
http://www.google.com/latitude to sign up. It will be particularly fun at LISA to see everyone on the map!
Google booth during LISA, Dec 7-8, 2011
Google's tech blog posted info about the many things that Google is presenting or doing at Usenix LISA. Beer and ice cream on Thursday night. A "ask an SRE anything" booth in the vendor show. Presenting papers, talks and tutorials and much more! Check it out! http://goo.gl/XXkpK P.S. I'll be at the Google Vendor Booth Wed, noon-1pm.
Usenix LISA hotel discount extended to Nov 28!
That's right, folks! Book your room today!
Friday keynote at Usenix LISA: Michael P. Perrone
We have three keynotes this year: Wednesday morning, Thursday morning, and the closing keynote Friday afternoon. Our closing keynote speaker is Michael P. Perrone, Manager, Multicore Computing, IBM T.J. Watson Research Center I've been a sysadmin for long enough that not much impresses me any more. Watching IBM's "Watson" computer play Jeopardy! and beat the human contestants was, in a word, flabbergasting. Doug and I immediately began a quest to find someone from IBM that could talk about this amazing accomplishment at LISA. His talk, "What is Watson?" will be the final presentation of the conference. The talk will be 50 minutes long followed by 10 minutes of A&Q.
CHIMIT Workshop
I am fascinated by the fact that there are researchers that study system administrators and how to make their work easier/better/etc. The #1 thing they tell me is "we need more interaction with more sysadmins to help guide our research!" The "CHIMIT workshop" at Usenix LISA 2011 is an opportunity to interact with these researchers. Read about it here and register to attend!
Thursday keynote at Usenix LISA: Andy Palmer
We have three keynotes this year: Wednesday morning, Thursday morning, and the closing keynote Friday afternoon. Our Thursday keynote speaker is Andy Palmer, Global Head of Software and Data Engineering, Novartis Institute for Biomedical Research We invited Andy because he deals with peta-scale data warehousing, big databases and all that fun stuff. I love hearing talks about big big big data. There are always plenty of surprises when things get that big. As conference co-chair, I can't wait to meet him in person! Usenix LISA 2011 is Dec 4-9 in Boston. You can register any time, but you get a discount if you register by Nov 14.
My co-worker featured on CNN!
"Google engineer: What I learned in the war" Dan and I have worked at two employers: Google and Cibernet. He's a great guy and I'm proud to know him. In addition to Dan, I have 2 other friends that have served in the Iraq and Afganistan wars. I was relieved and thankful (and a whole bunch of other adjectives and emotions) they all made it all home alive. Tom
Usenix LISA early-bird discount ends today!
You can save big $$$ by registering for LISA on or by midnight tonight! (California time) Usenix LISA 2011 is Dec 4-9 in Boston. I look forward to seeing you there!
Usenix LISA early-bird discount ends tomorrow!
You can save big $$$ by registering for LISA on or by Nov 14th. Usenix LISA 2011 is Dec 4-9 in Boston. I look forward to seeing you there!
Happy Corduroy Appreciation Day to all my readers!
11/11 is the date that looks most like corduroy and 11/11/11 makes it especially special! Wear them 'cords with pride! Sincerely, Tom
Tools vs. Automation
Sysadmins talk a lot about "automation" but I think a more specific definition is needed. " Tool writing" is when we create a program (script, whatever) that takes a task that that we do and does it better/faster/more accurately. For example, creating a new account used to take 10 or more manual steps (creating the homedir, setting permissions, adding a line to /etc/passwd, /etc/group, etc). Good examples include: FreeBSD "pw adduser" or Linux "useradd". In short, a tool improves our ability to do a task. " Automation" is when we create a system that eliminates a task. Continuing with our example, if we "automate" account management we might build a system that polls our HR database and creates an account for any new employee and suspends accounts for anyone terminated.
Usenix LISA early-bird discount ends Nov 14! 4 days left!
You can save big $$$ by registering for LISA on or by Nov 14th. Usenix LISA 2011 is Dec 4-9 in Boston. I look forward to seeing you there!
SSH Fabric for ssh'ing to many hosts
Fabric is a new tool for ssh'ing to many hosts. It has some nice properties, such as lazy execution. You write the description of what is to be done in Python and Fabric takes care of executing it on all the machines you specify. Once you've used it a bunch of times you'll accumulate many "fab files" that you can re-use. You can use it to create large systems too. The API is simple but powerful. The tutorial gives you a good idea of how it works: http://docs.fabfile.org/en/1.2.2/tutorial.html It is written using the Paramiko module which is my favorite way to do SSH and SSH-like things from Python.
Happy Birthday to my co-chair, Doug Hughes!
Please wish him a happy birthday by posting a comment here or on the band new "G+ Page" for Usenix: https://plus.google.com/108588319090208187909/posts Doug: Working with you on the LISA 2011 conference has been a blast. I can't believe it is less than a month away! It is going to be the best LISA ever! Have a great day! -Tom
Call for Usenix LISA stories!
To help us celebrate Usenix LISA's 25th conference, we are looking for stories! Do you have a favorite LISA story? Something you learned at the conference A contact you made that really had an impact on your career A funny event that still makes you laugh? Nostalga! (remember facesaver?) Contemporary! Please send your story by Sunday, November 20 to [email protected].
Usenix LISA early-bird discount ends Nov 14! 1 week!
You can save big $$$ by registering for LISA on or by Nov 14th. Usenix LISA 2011 is Dec 4-9 in Boston. I look forward to seeing you there!
CHIMIT: REGISTRATIONS NOW OPEN!
If you are coming to Usenix LISA, why not come a day early and go to CHIMIT? CHIMIT is for people that study system administrators (how they work, how they communicate, how to make tools better for them) but they can't exist without sysadmins also attending their conference! Call For Papers: 5th ACM Symposium on Computer Human Interaction for Management of ITDecember 4-5, 2011 - Boston, MAchimit.acm.org Information Technology (IT) is central to modern life. We are surrounded by software and hardware systems that support our work and personal lives. The size and complexity of modern infrastructures is increasing rapidly; and we are now at a turning point where we need new approaches to IT system design,management, and services.
DevOps keynote at Usenix LISA: Ben Rockwood
We have three keynotes this year: Wednesday morning, Thursday morning, and the closing keynote Friday afternoon. The Wednesday keynote speaker is Ben Rockwood from Joyent who's talk is titled "The DevOps Transformation". DevOps has a lot of buzz, but Ben will separate the hype from the reality. DevOps may be a new term, but it's not a new idea. He will deconstruct DevOps into its three transformation phases, look back at the often referenced but rarely explained history that influences it, and see how it is a catalyst that is changing the craft of system administration. I'm really excited we were able to book Ben for the conference and can't wait to see the talk!
Futuristic System Administration
Looking at the Usenix LISA 2011 conference program I'm excited we could book some really powerful "what will I need to know next year" kind of talks. This is what first brought me to LISA many years ago... the fact that by attending I'd be one step ahead of my co-workers as far as what's new in system administration. Some example talks: "What Will Be Hot Next Year?" with moderator: Narayan Desai, Argonne National Lab. Panelists: Kris Buytaert, Inuits; John D'Ambrosia, Force10 Networks; Jacob Farmer, Cambridge Computer "Ethernet's Future Trajectory" with John D'Ambrosia, Force10 Networks "IPv6, DNSSEC, RPKI, etc.: What's the Holdup and How Can We Help?"
Config Management? yeah, we got that.
Looking at the Usenix LISA 2011 conference program I'm proud that we have the three major configuration management systems covered: CFEngine3, Chef and Puppet: 3-hour classes: "Puppet" with Nan Liu "Configuration Management Solutions with CFEngine 3" with Mark Burgess Invited Talks: "3 Myths and 3 Challenges to Bring System Administration out of the Dark Ages" with Mark Burgess (CFEngine Inc) "Building IronMan, Not Programming" with Luke Kanies, Founder, Puppet and Puppet Labs "Converting the Ad-Hoc Configuration of a Heterogeneous Environment to a CFM, or, How I Learned to Stop Worrying and Love the Chef" with Dimitri Aivaliotis "Choose Your Own Adventure" with Adam Jacob Experience Reports: "Getting to Elastic: Adapting a Legacy Vertical Application Environment for Scalability" with Eric Shamow (Puppet Labs) The Guru is In: (Q&A sessions) "Chef" with Aaron Peterson Refereed Papers: Fine-grained Access-control for the Puppet Configuration Language ...
LISA LISA LISA LISA
No, not this. I mean the Usenix LISA conference. It is only 5 weeks away. Have you registered yet? The first few days are half-day tutorials with industry leaders teaching topics like Puppet, CFEngine (we don't take sides... both get a half day!) , Time Management (that's me!) , IPv6 (real deployments are happening!) , and many many more topics. The last half of the conference is a mixture of invited speakers, refereed papers, and other good stuff. What I like about the refereed papers this year is that we hit the perfect balance: half are "oh, I can use that right now!"
Usenix LISA early-bird discount ends Nov 14! 2 weeks!
You can save big $$$ by registering for LISA on or by Nov 14th. Usenix LISA 2011 is Dec 4-9 in Boston. I look forward to seeing you there!
LOPSA PICC 2012 Call For Participation
http://www.picconf.org/cfp/ Call for Participation:LOPSA-NJ Professional IT Community Conference 2012 This year's Theme:"System Administration: Scaling, Security, and Saving Money" PICC '12: 3rd Annual Professional IT Community ConferenceMay 11- 12, 2012 New Brunswick, NJ Hyatt Regency New Brunswickhttp://www.picconf.org The organizers of the LOPSA-NJ Professional IT Community Conference (PICC) invite you to submit proposals for papers and talks to be presented at PICC '12. PICC12 is a gathering of professionals from the diverse IT (computer and network administration) community in New Jersey to learn, share ideas, and network. The conference includes invited speakers and keynotes, training by top-notch experts that is relevant, useful, and recession-friendly; plus an "unconference" track where attendees propose and host their own topics during the event.
Review: Apple's iPhone iOS5 "Reminder.app"
What does this Time Management guru think of Apple's new Reminders.app? If you read this blog your ears probably perked up when you heard that Apple's iOS5 adds a new application called "Reminders.app". It complements the todo-list infrastructure Apple has been adding to its calendar system on OS X. The big question is: Can it do "The Cycle". " The Cycle" is the todo list management system that I describe in Time Management for System Administrators. It is very full featured even though you can start just by using a couple of its principles and building up as you get used to it.
My new LISA Tutorial: 32 Questions
I'm teaching 3 tutorials at Usenix LISA this year. Two are on time management ("intro" and "team efficiency") but the third is brand new: The Limoncelli Test. I've identified 32 qualities of well-functioning system administration teams. You've seen them before as "The Limoncelli Test". In this tutorial, I'll be going into more detail about the important ones and leaving plenty of room for Q&A. It is a 3-hour class and I hope to keep it interesting by making it very interactive. The hardest part of adopting these practices is often your own co-workers resistance to change. Therefore, I'm adding a big section on influencing others and "selling" big changes within an organization.
Automate? Will I lose my job?
In my speaking and writing I always encourage people to automate what they can and document what you can't. If something can't be automated (or isn't worth automating) writing a bullet list of the steps to accomplish the task makes the task less error-prone and easier for others on the team to do it. I get replies like, "What if I automate myself out of a job?" or "but if I document what I do, anyone can do it and I won't be needed!" Sysadmin, please! Neither could be further from the truth. First of all, there's always more IT work to be done.
Who's coming to Usenix LISA?
Two things I like about Usenix LISA conference: (1) The speakers are (usually) the inventor. (2) They're accessible, not roped off into a VIP room. You can talk with them, hang out with them. (Boston, Dec 4-9; early registration discount ends soon!) There are a lot of big names this year: What's your interest? Linux? We got Jon "maddog" Hall himself! Solaris? (I mean... Illumos) We got Bryan Cantrill! Security? We got (Susan Landau! DevOps? we got Ben Rockwood), Avleen Vig, Adam Jacob, Kris Buytaert, Ian McFarland, Kastner and Goulah from Etsy talking about Deployinator and more! Puppet? We got the puppet master himself, Luke Kanies!
More... How to ask your boss to pay for you to go to Usenix LISA 2011
In reply to my last blog post a reader sent me email to say: "I've never gone to a conference. What do employers typically pay for?" Your company probably has a written travel policy and a travel budget. Most employers pay travel, the conference registration, hotel, and food (T&E); IMHO they should do this since you need to get there, attend, and while there sleeping and eating are kind of important too. Other companies pay part of the cost or a fixed amount. For example, they might have a $1000/year training budget per employee and that would consume the registration (most people would therefore only go to local conferences that do not require travel).
Usenix LISA early-bird discount ends Nov 14
You can save big $$$ by registering for LISA on or by Nov 14th. Usenix LISA 2011 is Dec 4-9 in Boston. I look forward to seeing you there!
How to ask your boss to pay for you to go to Usenix LISA 2011.
Ask early. Sometimes approval takes a long time. He/she may have to ask higher-ups. Warm them up. One day mention how you wish you had better tools to do something or wish you knew more about something ("time management" maybe?) . A few days later say that you found a class on the topic at LISA. (Intro to TM and Advanced TM are both being offered) Talk about end-results, not technologies. " There's a class that will teach me how to automate installations" is much more understandable than "there's a class on Puppet". Find 3 classes or talks you want to attend.
Tom @ Triangle InfoSeCon, Raleigh NC, Oct 20, 2011
I'll be keynoting the conference with a talk called, "You Suck At Time Management (but it ain't your fault!)" For information: http://raleigh.issa.org/conference.html
Tom @ Scalability (Server Fault), San Francisco, CA, October 14, 2011
Scalability, brought to you by Server Fault and High Scalability, have invited me to speak on the topic of "Ganeti Virtualization Management: Improving the Utilization of Your Hardware and Your Time". Info about Scalability 2011: http://scalability.serverfault.com
Dennis Ritchie
Dennis Ritchie has died after a long illness. He was 70; two years younger than when my own father died. When I joined Bell Labs in 1994 I was very excited that I would be on the sysadmin team that served computer scientists such as Dennis Ritchie. Many of my favorite textbooks were written by people that would now be my users. On my first day, however, I was told that I shouldn't ask Dennis, or anyone, to autograph a book: they didn't like that. This was disappointing. I had many books I had hoped to get autographed. None more than my original copy of The C Programming Language, also known as "The K&R Book".
Tom @ The Pittsburgh Perl Workshop, Pittsburgh, PA, October 8-9, 2011
I'll be a featured speaker at PPW with the topic "What Perl sysadmins wish Perl developers knew, and ($a,$b) = ($b,$a) ". Info about PPW 2011: http://pghpw.org/ppw2011 (and follow on Twitter)
LOPSA PICC '12 planning meeting kick-off (NY/NY/PA area)
Interested in helping plan the 2012 PICC conference? Whether you want to take on a big role or just help out in a small way, you are invited to the inaugural conference call. This is the meeting where the Call For Participation and Theme of the conference will be set. For info about the call send email to John BORIS [email protected] or me [email protected] Thanks! Tom
Update: I can't make it to LOPSA-NJ meeting next week.
Due to traveling, I won't be there. However, John Wagner will be giving an excellent talk called "The !# site is down! Again!?"
I'll be speaking at LOPSA-NJ in October
I'll be one of the speakers at the September LOPSA-NJ meeting Thu Oct 6, 2011. The topic will be "You Suck At Time Management (but it isn't your fault!)". This talk will have a lot of info specific to security engineers, so if you have a friend that does security for a living, please invite them. If you are in the area, I hope to see you there! Info about the event: On the LOPSA-NJ Website Info about LOPSA-NJ: http://lopsanj.org
LOPSA Columbus, Ohio Chapter Starting
Matt Simmons wrote me to let me know that the LOPSA Board has approved the creation of a Columbus, Ohio chapter! Times, places, and topics are still be worked on. It you are interested, join the mailing list at https://lists.lopsa.org/cgi-bin/mailman/listinfo/lopsa-us-oh-columbus Congrats and good luck to everyone in Columbus!
Time Management for Sysadmins 50% off Ebook
Use code B2SDEAL http://bit.ly/b2sdeal to get 50% off Time Management for Sysadmins in eBook format part of the Back To School Deal from O'Reilly. At $9.99 it is hard to pass up this deal. Offer expires September 28th, and may not be combined with other offers.
python-gflags: version 1.6 released
The Google flags parser (available for Python and C++) is very powerful. I use it for all my projects at work (of course) and since it has been open sourced, I use it for personal projects too. While I support open source 100% I rarely get to submit much code into other people's projects (I contribute to documentation more than code... go figure). So, even though it is only a few lines of new code, I do want to point out that the 1.6 release of the Python library has actual code from me. One of the neat features of this flags library is that you can specify a file to read the flags from.
Usenix LISA registration is open...
http://www.usenix.org/events/lisa11/ (as of a few minutes ago)
The 9-11 disaster investigation website
National Institute of Standards and Technology (NIST)'s division of building and fire safety performed the scientific investigation of the World Trade Center (WTC) disaster. Much of the related video, audio and photographic evidence was released under FOIA. Just in time for the 10th anniversary of the disaster the FOIA'd data was released on their website: http://wtcdata.nist.gov Since FOIA requires the raw, unaltered, data to be released, many of these videos are at very high resolution. (Lower res versions are available for easier viewing, of course). If you go to the website, you can watch all the material. If you go to Usenix LISA 2011, you can see a presentation by the sysadmins that built the site, and learn the technical and non-technical challenges that threatened the project along the way.
ServerFault Scalability Conference (and DevDays 2011) Cancelled
DevDays 2011 is Cancelled Q: What about the ServerFault Scalability Conference? A: That has been canceled, also. Sigh. The full story here. If you had registered hoping to see me speak, my apologies. Please refer to http://everythingsysadmin.com for a list of my other appearances. People in the Princeton, North Carolina and Pittsburgh area should be particularly interested in that list. Also... soon I'll be announcing 3 half-day tutorials that I'll be teaching at the Usenix LISA conference in December in Boston. Start warming your boss up to the idea of sending you to a conference right after Thanksgiving. I'm really psyched about the new material.
Tom @ LOPSA-NJ, Thu Sept 1, 2011, Lawrenceville, NJ (near Princeton)
Tom will be presenting a 1-hour talk titled Walk a kilometer in my shoes: What sysadmins wish developers knew and vice-versa at LOPSA-NJ (League of Professional System Administrators / New Jersey Chapter). If you are in the area, I hope to see you there!
This week at LOPSA-NJ: Tom Limoncelli will be presenting!
If you are in the Princeton, NJ area I hope you come out to join us! If you've never come to our meeting, this is a good event for first timers! http://www.lopsanj.org/ I'll be talking about ways for sysadmins and developers to work better together. It will be a rehearsal for a larger talk I'll be giving at PPW. Oh, and this month we have a sponsor supplying pizza and soda. So, come for the pizza, stay for the Limoncelli. :-) More info on the LOPSA-NJ website! Tom
Call for pizza toppings!
The New Jersey Chapter of LOPSA has gained a sponsor that will provide pizza and soda for the Thursday, September 1, 2011 meeting. William Bilancio posted on the LOPSANJ mailing list:September's meeting pizza and soda is generously provided by INetU Managed Hosting: http://www.inetu.net. So if you are planing to attend Tom Limoncelli's talk: "Walk a kilometer in my shoes: What sysadmins wish developers knew and vice-versa" please rsvp by going to http://www.lopsanj.org/rsvp and let us know if you are attending and what toppings you would like on the pizza." http://lists.lopsanj.org/pipermail/lopsanj/2011-August/003820.html See you there!
New sponsor: Scalability from serverfault.com
Notice a new advertisement on the right-hand side of this blog from ServerFault.com's "Scalability" 1-day conference. Use the discount code "everythingsysadmin" and get $100 off registration. I'll be speaking that day. I'm working on my slides right now!
I'll be speaking at LOPSA-NJ, Thu Sept 1, 2011, Lawrenceville, NJ (near Princeton)
At the September LOPSA-NJ meeting Thu Sept 1, 2011. The topic will be "What sysadmins wish developers knew and vice-versa". If you are in the area, I hope to see you there! Info about the event: http://lopsanj.org/node/696 Info about LOPSA-NJ: http://lopsanj.org
Pittsburgh, San Francisco, Raleigh, and Boston speaking engagements announced
I've updated http://everythingsysadmin.com's front page to list all upcoming speaking engagements. Look for me in Pittsburgh on Oct 8-9 for the Pittsburgh Perl Workshop, in San Francisco on Oct 14 for Scalability (sponsored by ServerFault), and in Raleigh, NC on Oct 20 for ISSA Triangle InfoSeCon. Of course, in December I'll be speaking at LISA '12 including 1.5 full days of tutorials. More info on the left navigation bar on http://everythingsysadmin.com
The Limoncelli Test: 32 Questions for Your Sysadmin Team
Just in time for Sysadmin Appreciation Week... People often ask me how they can improve their sysadmin team. It takes only a brief discussion to find fundamental gaps that, when filled, will improve the teams's productivity and the quality of the service being provided. To help you find these gaps, I present: The Limoncelli Test
Sysadmin Appreciation Day 2011
This coming Friday, July 29, 2011 is the 12th Annual System Administrator Appreciation Day. http://www.sysadminday.com/ LOPSA has a list of vendors giving discounts that day (including their own membership is 25% off!) Here are the celebrations that I know about: Los Angeles, CA: http://sysadminday.eventbrite.com/ NYC, NY: http://nycsysadminday.eventbrite.com/ Columbus, OH: http://614-sysadmin-day.eventbrite.com/ San Francisco, CA: http://opendns-sysadminappreciation2011.eventbrite.com/ Chicago, IL: https://www.facebook.com/event.php?eid=100672293360534 Buenos Aires, Argentina: http://eleccionroot.com/index.php
Data? We have a lot at.
Cluster management operates on a very large scale: whereas a storage system that can hold a petabyte of data is considered large by most people, our storage systems will send us an emergency page when it has only a few petabytes of free space remaining.This guy is not exaggerating. This is why I love working at Google. (Did I mention that my coworker has a sign on his desk that says, "I work here because I love this shit!") Tom
The end of MySpace
"After $35 million sale, MySpace is now an advertiser's space" I should have linked to this article when I saw it a few months ago: Workers inside MySpace tell me that this infrastructure, which they say has "hundreds of hacks to make it scale that no one wants to touch" is hamstringing their ability to really compete.I seem to recall another article claiming that MySpace had no dev-test-live system; developers pushed code directly onto the live system. I can't find a this article, so maybe I dreamed it. Either way, could you imagine the fear of pushing new features if you had to work like that?
Google+
In the future, the ultimate insult will be in the form: I know you can't see it, but I've added you to a Google+ circle called __.
Facebook Fatigue
I stopped using Facebook 8 months ago. CNN's article "Why some dissatisfied users are shunning Facebook" reminded me that I haven't written an article to explain why. There were a number of reasons. Obviously, yes, as a Google employee I was kind of sick of hearing the media yammer about Facebook, Facebook, Facebook. There was an extra large amount of hype then, especially since "The Social Network" film came out. (As an aside... I enjoyed that movie immensely and recommend it to all. I love Sorkin's writing style.) However the big reason was time management related. I had a number of big important projects on my plate, both at work and outside of work.
Happy 99th Birthday, Alan Turing!
"Today marks the 99th anniversary of the birth of Alan Turing, a noted polymath and cryptanalyst who is regarded by many as being the grandfather of modern computing." The link is to a great article. Worth the read.
screen gone; moving to tmux
I've been using "screen" since 1995ish. After reading "Is tmux the GNU Screen killer?" I gave tmux a try. I' a convert. Amazingly enough, I haven't even set up a config file. I'm using the defaults and so far I'm happy. The only changes I might make in the config file: The default command key should be CTRL-] (that's just a matter of taste) Cmd-SPACE should go to the next window Color me impressed!
Your Dial Telephone
If only my cell phone was so simple! Now You Can Dial: An Instructional Video from AT&T. 1954!
The journey of a thousand miles begins...
The journey of a thousand miles begins... with a field trial to verify assumptions. You need to do one before you do a thousand. If you are painting a house, try the paint on a part of the house people can't see. If you are upgrading systems, do a few first before you start the mass migration. The journey of a thousand miles begins... with doing it manually a few times, writing down the process, making sure the team agrees, then automating it. The only way to automate something is to make sure you know how to do it manually first.
Avoid using the term "Cloud Computing" except when being ironic
I've stopped using the term "Cloud Computing" except when referring to the general trend. I use SaaS, IaaS and PaaS so that I say exactly what I mean. As a technical person this helps me keep my conversation succinct and focused. SaaS: Software as as Service: Salesforce.com, Google Apps, etc. IaaS: Infrastructure as as Service: Amazon EC2, Eucalyptus, etc. PaaS: Platform as a Service: Google App Engine and similar systems. Or, if you want a way to remember it easier: SaaS: It's a web site! IaaS: It's a VM! PaaS: It's a framework! " Cloud" is what marketing and non-technical people use.
Tonight at LOPSA-NYC! Ganeti open source virtualization
As previously mentioned, I'll be the speaker at LOPSA-NYC. Come here me speak about the Ganeti open source project. Think virtualization clusters have to cost big bucks? Think virtualization isn't useful for a small site? Come and find out why a person that usually talks about Time Management thinks virtualization is his new favorite time management trick. Hope to see you there! (Please pre-register so you can get through security quickly.)
Reminder: I'll be speaking at LOPSA-NYC tomorrow
on the topic of Ganeti: http://everythingsysadmin.com/2011/05/lopsanyc201106.html
http://protolol.com/
This is too good not to share. This humor site is addicting. These are two of my favorites: "OSI model jokes work on so many levels" "WHO HAS ANY ARP JOKES?" The site is: http://protolol.com/
Best IPv6 Media Coverage?
PBS had, by far, the best coverage of World IPv6 Day. The piece was technically accurate, informative, picked excellent people to interview, and had no "doom" hype. The broadcast piece is here. Additional material that didn't make it into the broadcast is here. The worst coverage? A certain network magazine that covered it as some kind of Y2K Doomsday Conspiracy Theory. I won't mention their name. F---tards.
Last call for Usenix LISA papers!
The official deadline is today, Thursday: June 9, 2011, 11:59 p.m. PDT or UTC/GMT -7 hours (click to see when that is in your timezone). Submission instructions: http://www.usenix.org/events/lisa11/cfp About the conference: http://www.usenix.org/events/lisa11
What "World IPv6 Day" Means To Sysadmins
I'm going to be extremely technical here. Sysadmins should really understand what World IPv6 Day is. Does it mean the world is converting to IPv6 today? No. No, not at all. The upgrade requires technical planning and work. It can't happen without your help and without your knowing. Besides, the plan is to move to "dual stacked" IPv4+IPv6 on all hosts/networks and run that way for a good long time. Does it mean my ISP is going to enable IPv6 on my connection? No. Not at all. (Seriously, folks, when was the last time your ISP added a feature without you having to beg for it first?)
Tom @ LOPSA-NYC, 7pm, June 14, 2011 speaking about The Ganeti Project
I'll be speaking at LOPSA-NYC Tuesday, June 14, 7pm. Please pre-register to speed your way through security. Come here me speak about the Ganeti open source project. Think virtualization clusters have to cost big bucks? Think virtualization isn't useful for a small site? Come and find out why a person that usually talks about Time Management thinks virtualization is his new favorite time management trick. Here is the official announcement. Topic: Ganeti: Open source virtualization (like VMWare ESX + VMotion but open source) Speaker: Tom Limoncelli, Google, Inc When: Tuesday, June 14, 7pm - 9:30pm Description: Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies such as Xen or KVM and other Open Source software.
How to send email
http://matt.might.net/articles/how-to-email/ is a great piece on email and it can be read by geeks and non-geeks alike. I've read zillions of articles with similar titles: I promise you that this one has new ideas.
TONIGHT! Mark Burgess speaking at NYC DevOps
As mentioned previously Mark Burgess, creator of CFEngine, will be speaking at the NYC DevOps MeetUp tonight http://www.meetup.com/nycdevops/events/17211427/ When: Wednesday, May 25, 2011, 7:00 PM Topic: Mark Burgess presents DevOps and The Future of Configuration Management Where: New York... exact location revealed when you RSVP to the MeetUp
Time and financial management
My new bank makes it easy to establish sub-accounts and set up automatic transfers between them. Today I created 3 new subaccounts: "2021 House Repainting" ($100 transfered in each month) "2016 New car" ($yyy transfered in each month) "Annual unexpected household repair" ($zzz/month... every year we seem to have a $z,000 emergency; might as well plan for it) I try to make my time management advice "friction free". That is, low effort, easy to get started. I try to do the same with my finances. Any amount of difficulty (friction) can be used as an excuse to not do something.
Wednesday: NYC DevOps Presents: Mark Burgess / CFEngine
This month's NYC DevOps meetup has a special speaker: Mark Burgess, inventor of CFEngine, talking on the future of configuration management. http://www.meetup.com/nycdevops/events/17211427/ Wednesday, May 25, 2011, 7:00 PM Topic: Mark Burgess presents DevOps and The Future of Configuration Management Mark Burgess is the founder, chairman, CTO and principal author of Cfengine. He is Professor of Network and System Administration at Oslo University College and has led the way in theory and practice of automation and policy based management for 20 years. In the 1990s he underlined the importance of idempotent, autonomous desired state management ("convergence") and formalised cooperative systems in the 2000s ("promise theory").
The CDC has a Zombie Attack Plan
And provides this HTML "button" to help spread the word. When it comes to disaster recovery plans, is your IT department prepared for zombie attacks?
Mac Malware
Some people laughed when I tweeted http://goo.gl/3yyKg but now look at this http://goo.gl/XpG03 just 8 days later! This might be a good time to relink to my post called Yes, malware scanners on your servers too!
Please put the LISA Button on your web site or blog
Would you like to help Usenix LISA? Please put the LISA11 button on your website or blog. HTML code can be found here: http://www.usenix.org/events/lisa11/ I just put it up on www.TomOnTime.com and changed the position of where it is on EverythingSysadmin.com The graphic will change over the next 7 months so you don't have to do anything as we move from "collecting submissions" mode to "registration is open" mode to "thanks for attending" mode. Isn't distributed computing awesome? Tom
An internal Twitter-like service for your company
While teaching my Advanced Time Management tutorial at PICC11 I suggested that having an internal Twitter-like service at a company is a useful thing. Eric Shamow mentioned that he uses "Status.Net" which is open source. For those that didn't catch the link, it is http://status.net/open-source (they have a blog at http://status.net/blog)
TM4SA now available as a Google eBook
Via many places including Green Apple Books. I mention Green Apple because they made an awesomely silly video explaining how it works.
WIETSPTUL Part 2: Tip for writing Practice and Experience Reports
As a quick follow-up to Making it easier to submit papers to Usenix LISA, here's a tip for writing a good Practice and Experience Report: Work backwards from the "lessons learned". Step 1. Write down the 4-5 things that you wish you knew before you started the project. That is, things that readers will feel they've learned after reading your paper. Step 2. Work backwards from those 4-5 things to figure out what people need to know to understand them (for example, the story of how the project got started, the problems you faced, and how you solved them). You really don't need to include much more.
Making it easier to submit papers to Usenix LISA
[This was originally published on The Usenix Update Blog] We want YOU to submit a paper this year to the LISA conference Really. Yes, you! Whether you are in academia developing new algorithms that improve system administration, leader of an open source project that sysadmins find valuable, or a practitioner in industry that has written new software to improve productivity, we believe there's a paper inside all of you that wants to get out! (Usenix LISA is December 4-9, 2011 in Boston). LISA is also a great venue for student papers: it is a friendly audience and we have a "Best Student Paper" award that pays cash.
May the 4th Be With You!
Happy Star Wars Day! http://www.youtube.com/watch?v=OyYTATfgPZk http://starwarsblog.starwars.com/index.php/2009/05/04/may-the-4th-be-with-you/
Tom @ LOPSA PICC in NJ, April 29-30, 2011
I'll be teaching tutorials and maybe more. Watch this space: http://www.picconf.org/
Yes, malware scanners on your servers too!
I recently pointed my "6-point list of security minimums" for the enterprise. That is, 6 thinks that may have been "would be nice" in the past but are now absolutely required as far as I'm concerned. Most sites do not do all 6, and I think it is time that such sites got with the program 'cause you are making the rest of us look bad. I got a number of comments asking if I was serious about malware scanners on servers. I am. If the server is a file server then the files being stored should be scanned. Not only does it prevent this server from being the unintentional transmitter of contaminated files, but it is an interesting way to detect which users are not protecting themselves.
PICC Schedule available in "The Conventionist"
If you have an iPhone or Android, download "The Conventionist" and enter code "picc". You'll get the complete schedule for the conference (even if you aren't attending, it is a fun app to play with). Congrats to Matt, William and everyone for putting this together! I'm almost done with my slides for PICC. I can't wait to see everyone there! On-line registration is open for another 11 hours. After that, you can register on-site!
My thoughts on the LOPSA Board election
LOPSA board elections are upon us. The candidate statements are being published and before I read any of them I want to make this statement of my own:* It is my experience with volunteer organizations that people that have achieved tangible results are more likely to produce more tangible results. Ideas are a dime a dozen. Everyone has ideas. They pour in from everywhere. Don't worry about electing "idea people"; a group of LOPSA's size only needs 1-2 "vision" people but a lot of "do'ers". Elect people that have a track record of getting things done. Years ago when I was in college there was a student government election.
Not being attacked? Your network must be down.
Someone recently ask me how often an enterprise might expect to be attacked. Attacks are no longer something that happens now and then, they are constant. An hour without an attack is an hour your network connection was down. This is sometimes known as the "Advanced Persistent Threat". Shortly after APT was declassified someone gave a lecture about it at Usenix LISA. You can watch it here. (Note: I found some of what he revealed to be disturbing). I think the person meant how often an enterprise might expect a successful attack. That's an entirely different matter. Knowing about APT is one thing.
Don't lose $75
The early bird price for Confernece + training ends on Tuesday April 12th at 11:59 pm. Have you registerd to get the $75 discount? http://picconf.org Seating is limited for my new "Advanced Time Management: Team Efficiency" tutorial. Register soon! I won't be teaching this again until December!
What should I cut from "Team Time Management"?
I am rewriting my class "Advanced Time Management: Team Efficiency" class in preparation for teaching it at LOPSA PICC 2011. I need to cut about 30 minutes from it. If you attended when I taught it at Usenix LISA 10 you may recall that I had to rush at the end and didn't have a lot of time for Q&A. My notes say I need to cut 30 minutes. If you have thoughts about what to drop, please post a comment below. Thanks!
Small is good: Local sysadmin/devops conferences like PICC are a big value
[I just learned the early-bird discount deadline was changed. You now have a few more days to get the discount!] If you live in Silicon Valley it is easy to make technical connections; for the rest of us, regional conferences rule. I attend many conferences: small and large, invite-only and public, regional and national, vendor-specific and vender-neutral, even some international ones too. My next speaking gig is is LOPSA PICC in New Brunswick New Jersey, which is a small, regional, conference this April 29-30. People there will be from New Jersey, New York, Pennsylvania, Connecticut, Delaware, Massachusetts and more (last year 4 people came all the way from Virginia!)
Two April Fools reminders!
Today is April Fools day. If you see something fishy, pause and think before you react. It could be a joke. Today is an opportunity to show how good you are at taking a joke. We still have a few copies left of The Complete April Fools RFCs. One big book of all the funny Internet RFCs (as of a few years ago). http://www.rfc-humor.com for more info. Makes a great gift for the geek in your life and is the perfect conversation piece for your office.
First day of the month!
Today is the first day of the month. You, no doubt, have received a flood of reminders from mailing list servers about which mailing lists you are on. This is a good opportunity to unsubscribe from the lists you no longer find useful. Being able to manage a lot of email is good but getting less email is better.
First draft of my "Python for Perl Programmers" article
Now that I'm comfortable with Python I've written down all the things that, as a Perl programmer, I wish someone had told me early on. It seems that all the Python books are written for normal people, not those who have Perl embedded in their DNA so hard that it is difficult to think any way else. I hope this helps other people in my situation: http://everythingsysadmin.com/perl2python.html
CHIMIT Call For Papers and other contributions announced!
[If you are involved in UX, UI or human factors as applied to system administration...] CALL FOR PAPERS AND OTHER CONTRIBUTIONS ACM Symposium on Computer-Human Interaction for Management of Information Technology (CHIMIT 11) December 4, 2011 -- Boston, MA, USA Web site: chimit.acm.org TIMETABLE Paper submissions due: May 29th, 2011 Paper submission notifications: July 29, 2011 Posters and Presentations of Previously Published Work submissions due: September 8, 2011 Posters and Presentations of Previously Published Work submissions notifications: September 16, 2011 Camera-ready papers due: September 16, 2011 OVERVIEW Information Technology (IT) is central to modern life. From our homes to our largest enterprises, we are surrounded by software and hardware systems that support our work and personal lives: wireless access points, network routers, firewalls, virus scanners, databases, web servers, storage and backup systems, etc.
Sign up for my new Time Management class now!
I'll keep it brief: My new half-day time management class: $449 -- Apr 29 -- includes conference admission, all meals, and other training -- http://picconf.org (discount ends Apr 4)
Interview with me on the Mind Of Root podcast
Episode 152 (http://www.mindofroot.com/2011/03/26/mor-152-author-author/) includes an interview with me about the Time Management class I'll be teaching April 29th at LOPSA PICC in New Brunswick, NJ. (as previously mentioned)
Overheard at the office: Perl edition
Coworker evaluating a stack of resumes: "This guy blatantly lies on his resume. Nobody has 15 years of Perl experience!" Me: Um... I do. C.e.a.s.o.r: What? Me: (thinks a bit) 1991. Either July or August. Perl was at 4.032 or 4.033 if I recall. That would be (thinks for a second) 19 years! C.e.a.s.o.r: (look on his face of shock and horror) Oh. I guess he isn't lying. Me: Damn kids! Get off my lawn! (this happened last year... I delayed this post so-as to obscure any possibility of identifying information)
Taos Women In Technology event next week in San Jose, CA
[ I'm forwarding this invitation because I know a lot of readers are in San Jose. Also, Prof. Nicole Velasquez (Pepperdine University) will be speaking and she does some awesome sysadmin research! ] Taos is hosting a very special gathering for women in technology. If you know Perl is more than just a pretty necklace, then we want to meet you! Whether you're a system or network administrator or a Project Manager, we welcome all of you to a networking opportunity on Thursday, March 31, from 6-8 PM that will be held at FAZ Restaurant (1108 North Mathilda Avenue, Sunnyvale). This gathering is the first of its kind to exclusively promote and recognize the talented community of women in technology.
PICC "early bird" discount almost over (plus two interviews)
The generous early-bird discount for LOPSA PICC (http://picconf.org) goes away on April 4th. This is a good time to talk with your boss about sending you while he/she can save money. Registration information is here: http://www.picconf.org/registration This is the absolute least expensive way to get my time management training. There is a heck of a lot of other great training and talks planned. I hope to see you there! Speaking of the conference... Read this (brand new!) interview with William Bilancio. He is one of the PICC organizers. -Tom P.S. I'll be interviewed on Episode 151 of the (Mind Of Root Podcast](http://www.mindofroot.com/).
Free AA_Console for LOPSA_PICC Attendees
Great news from LOPSA! "Every person who comes to PICC11 will receive [AA Console](http://www.adminarsenal.com/admin-arsenal/main/) (which used to be named Admin Arsenal) PLUS they're also throwing in a copy of [PDQ Deploy Pro](http://www.adminarsenal.com/pdq-deploy-pro/main/). Together, this is a $500 value for absolutely nothing except showing up to a conference you wanted to attend anyway." http://www.standalone-sysadmin.com/blog/2011/03/admin-arsenal-ups-the-ante-for-picc/ Very impressive! Time to register, eh?
Thought for the day.
In one of Isaac Asimov's books, he foresees a day when telecommunications becomes so inexpensive that it becomes essentially flat-rate; for a monthly charge we can call anywhere in the world and talk all we want. As a result, people do begin to talk to others around the world. This breaks through the artificial walls to communication created by politics and we see that everyone is the same around the world. Soon, world peace is achieved. In the story, this happens around 2000. It was about 2000 at which point internet use was a big part of our lives and, usually, was flat-rate.
Thanks, NYLUG!
I had a great time speaking at NYLUG last night! If you are in the NYC area I highly recommend you check out this great Linux Users Group! http://www.nylug.org
Any CSS gurus in the house?
Help me fix the "awesome sysadmin conferences" box on http://everythingsysadmin.com so it works better when the window is narrow. (or other suggestions on how to make it look better) Please?
Tonight at NYLUG: Me!
http://nylug.org/meetings/index.shtml?20110300 If you are in the NYC area, come check it out. Wednesday, March 16, 2011, 6:30 PM - 8:00 PM IBM, 590 Madison Ave, New York, NY 10022 I will present his "Top 5" time management tips for better time management, and take Q&A about time management, system administration, and what it's like to work at Google. And I might have a surprise. NOTE: You have to pre-register.
Don't feed the trolls
"Dealing with Internet Trolls - the Cognitive Therapy Approach" is a "must read" article for anyone that deals with trolls in chat rooms, bulletin boards, and so on. The strategy comes from Feeling Good: The New Mood Therapy, which is a self-help book that I highly recommend. In fact, I recommend this book in both TPOSANA and TM4SA. Feeling Good suggests this strategy as a way for people dealing with periods of depression can deal with criticism. It works surprisingly well when dealing with trolls too. The full article is here:http://unarmed.shlomifish.org/909.html
I'll be speaking at NYLUG this Wednesday
http://nylug.org/meetings/index.shtml?20110300 If you are in the NYC area, come check it out. Wednesday, March 16, 2011, 6:30 PM - 8:00 PM IBM, 590 Madison Ave, New York, NY 10022 I will present his "Top 5" time management tips for better time management, and take Q&A about time management, system administration, and what it's like to work at Google. And I might have a surprise. NOTE: You have to pre-register.
LOPSA PICC Full Program Announced
What is LOPSA PICC? http://picconf.org Presentations, education, and fun. IT and syadmin (Linux/Unix, Windows, Networking & storage). 2 days, 1 night, conference. Low price/high value. Community-based, non-profit. April 29-30, 2011 @ Hyatt Regency New Brunswick, New Jersey. Where else can you find a regional conference with national speakers, hot topics that will help you advance your career, all meals included, and not have to travel 3,000 miles to get there? Find out more and register: http://www.picconf.org Twitter: http://twitter.com/picconf Facebook: http://picconf.org/facebook Email: http://lists.picconf.org/mailman/listinfo/picc11-announce HALF-DAY TRAINING SESSIONS: "PowerShell Fundamentals", Steven Murawski "Grokking Python", Brian K. Jones "Over the Edge System Administration, Volume 1", David N.
Cascadia IT Conference (Seattle)
LOPSA's Cascadia IT Conference starts today (the training portion started yesterday). Reviewing the program grid it looks like it is going to be a fantastic day. I wish I could be there! Congrats to the committee that put the conference together, especially Lee Damon the conference chair. I love to see community-based, volunteer-only conferences springing up. You can follow the conference on Twitter hashtag #casitconf (If you are on the east-coast and jealous of Cascadia, you'll be happy to know that LOPSA's PICC conference in NJ is just 6 weeks away! Register today!)
Review: AT&T 3G Microcell
I got an offer in the mail from AT&T for a "3G Microcell". (click for larger view) which offers "more bars in your house". It is free, as long as I keep it for 12 months. Normally a $199.95 value, I decided to check it out. What is it: A device you plug into your home network. Your cell phone sees it as a cell-phone tower and, since it is closer to you than the local cell-tower, uses it for phone calls. The phone calls go out as VoIP through your internet connection. It works with any ISP (I have FiOS, not AT&T.)
I'll be speaking at NYLUG a week from Wednesday
http://nylug.org/meetings/index.shtml?20110300 If you are in the NYC area, come check it out. Wednesday, March 16, 2011, 6:30 PM - 8:00 PM IBM, 590 Madison Ave, New York, NY 10022 I will present his "Top 5" time management tips for better time management, and take Q&A about time management, system administration, and what it's like to work at Google. NOTE: You have to pre-register.
Call For Papers: 5th ACM Symposium on Computer Human Interaction for Management of IT December 4-5, 2011 - Boston, MA
If you notice there is a new conference listed on the "Awesome Conferences" listing on EverythingSysadmin.com: CHIMIT 2011 CHMIT has published their "Call for Contributions" on their new website http://chimit.acm.org CHIMIT is a conference for people that study IT people, how we work, and how we can work better. It is a small conference and I encourage people (sysadmins and researchers) to participate. We need more things like this! I've mentioned CHIMIT a number of times including a longer explanation of why this is an important area of study and reported about a panel I was on too. This year CHIMIT is in Boston, Dec 4-5, 2011.
Seattle folks! Still time to register for Cascadia!
It is just days away. Seattle-folks, don't miss out! The training program is top notch and the talks on Saturday look very interesting! (I wish I could be there!) http://www.casitconf.org If your boss won't pay for it, tell him I said he/she should send you. (Forge email from me saying so. If he/she asks, I'll back your story.)
Awesome Sysadmin Conferences!
Regional: PICC 2011, April 29-30, 2011, New Brunswick, NJ Regional: Cascadia IT Conference, March 11-12th, 2011, Seattle, WA National: Velocity 2011, June 14 - 16, 2011, Santa Clara, CA National: Usenix LISA '11, December 4-9, Boston, MA National: ACM CHIMIT '11, December 4-5, Boston, MA Got something to say? Doing something interesting? Want to publicize your open source project? Submit papers and talk proposals to LISA and CHIMIT!
Improving the situation
You are in a car, driving 60 miles per hour, trying to get to Canada. You pass through Kansas. You are driving south. If you don't understand the problem I'm describing, look at a map. Now how do you think you should solve this problem? One might suggest that changing your speed to 30 miles per hour would be an improvement. Do you agree? I doubt it. I bet most people reading this web site would think that the situation isn't getting better until the car is driving the other direction. Why do I bring this up? You've heard that the internet is running out of IP addresses, right?
Training classes announced for PICC (NJ-area)
You can attend my new class "Advanced Time Management: Team Efficiency" there. http://www.picconf.org/training-program#f6 The more of the schedule PICC announces the better it looks. Save April 29-30 on your calendar. Better yet, register today!
Why are tax forms in PDF?
Dear universe, There are 10+ different organizations that have to give me some kind of tax form so that I can file my taxes. I'm really happy that they are now all electronic. It is much easier to download them off the organization's website than to get them in the mail. However, if these organizations are going to generate a PDF, can't they also generate a .irs file? A .irs file is an imaginary XML format that I wish existed. It would include all the data from the PDF but in a parsable format. I could take all the .irc files put them on a USB stick and hand it to my tax preparer or feed them into TurboTax.
Random technical tips, thoughts and rants
On a Mac, if you SHIFT-CLICK the green dot on a window it opens it as wide and tall as possible (instead of the application-defined behavior) Even though "ls -l" displays a files permissions as "-rw-r--r--", you can't use "-rw-r--r--" in a chmod command. This is probably one of the most obvious but overlooked UI inconsistencies in Unix that nobody has fixed after all these years. Instead we force people to learn octal and type 0644. Meanwhile every book on Unix/Linux spends pages explaining octal just for this purpose. Time would have been better spent contributing a patch to chmod. If a network problem always happens 300 seconds after an event (like a VPN coming up or a machine connecting to the network) the problem is ARP, which has to renew every 300 seconds.
1-hour documentary about IBM's Jeopardy!-playing computer Watson
It was broadcast on Feb 9 but you might still be able to catch it on your local PBS station. It is an hour long and is in-depth without requiring a CS PhD to understand. http://www.pbs.org/wgbh/nova/tech/smartest-machine-on-earth.html
Casitconf'11 early-bird registration savings extended to the 23rd
Cascadia IT Conference has moved the early-bird deadline to Wednesday, the 23rd of February. Call it a President's day sale. http://www.casitconf.org/ Some talks I'm particularly excited about: Incident Command for IT: What We Can Learn from the Fire Department, Brent Chapman Chose your own Adventure, Adam Jacob Talks on Chef, Puppet and Cfengine ...and the entire training schedule looks top notch! http://www.casitconf.org/ Register now!
Google Enables Two-Factor Authentication For All
My apologies for flogging my employer's product, but I enough people have asked me "how can I protect my gmail account" that I feel this is worth it. http://www.informationweek.com/news/security/vulnerabilities/showArticle.jhtml?articleID=229216897 Google has enabled 2-factor authentication for GMail. I highly recommend you enable this. Attacks on gmail accounts (and all accounts) are increasing in frequency.
Cascadia IT Conference - Early Bird Registration ends in 1 week (Feb 16)
The Cascadia IT Conference wants to remind you that this is the last week you'll be able to save money on registration by getting the Early Bird discount! http://bit.ly/CasITConfReg Each half day tutorial is $125, and each half day tech session is $100, but with the Early Bird discount, they're only $105 and $80, respectively. You can save even more money by purchasing a bulk pack of 2 days of tutorials for $399 (or one day of training, one day of tech sessions for only $359!) . The Early Bird discount ends on February 16th, which is this coming Wednesday. Register now so you don't forget!
Mind of Root podcast's "book club"
Keith Albright and Steve Murawski do a great sysadmin podcast called "Mind Of Root". It is mostly Windows-focused but has a lot of great Linux stuff too. Recently they've been reviewing chapters of The Practice of System and Network Administration one or two chapters per epidsode. They pre-announce which chapter will be discussed so you can read it before the next episode. They usually record the podcast live, and you can join in a chat room and be involved in the show. I'm really enjoying listening to people talk about the book. When they agree they usually have an excellent story to tell, and when they disagree they're right 99% of the time (I keep saying to myself, "D'oh!
Todays links: Soft Skills, Ganeti and ConfigMgmt
System Administration Soft Skills: How can system administrators reduce stress and conflict in the workplace? by Christina LearChristina is a co-author of The Practice of System and Network Administration. The article is a great overview of the soft skills needed by system administrators and for non-sysadmins it is an interesting peek into sysadmin life. Check it out! (If you didn't recognize her name, that's because it changed when she married this guy.) http://queue.acm.org/detail.cfm?id=1922541 Handling HDD failures with Ganeti by Lance AlbertsonLance uses the Ganeti virtual cluster manager (think: VMWare ESX with Vmotion but completely open source) and has written another great post about how it is making his life easier.
Memory tip: How to remember names
[This post is part of a series on improving your memory.] People say they are bad at remembering names so often it is trite. The truth is that everyone is bad at it so stating this fact out loud is like reminding people "I breathe air." You don't naturally remember someone's name, you have to work at it. People that are good at remembering names employ various tricks, i.e. they work at it. There is one fact you must know to improve your memory: Remembering something is a two-step process. First you must have the information. Then you have to commit it to memory.
Tom @ NYLUG, 6:30pm, March 16, 2011
Tom will talk about why sysadmins are bad at time management and why it usually isn't your fault. Tom will present his "Top 5" time management tips for better time management, and take Q&A about time management, system administration, and what its like to work at Google.
Memory tip: Two ways socks help you remember things
[For the next week or so I'll be posting the techniques I use to help me remember things. I'll be covering topics like memorizing short lists, oddball things, and names.] I prefer to write something down so that I don't forget it, but sometimes I forget the list! For example, I often have an idea right as I'm falling asleep. I keep a pad of paper by my bed just for this reason. However the next day I forget to look at the list. Therefore I need a way to remind myself to look at the list. All I need is for something to be "out of place" in the morning and that will jog my memory.
You did it! Fundraising goal met! ($4500+ raised!)
Thanks to everyone that participated in the fundraising for Lyon-Martin Health Services (LM). We exceeded the goal of raising $1000 from readers of this blog. I received email from Dr. Harbatkin calling this effort "Amazing!" Blog readers have donated $1365. I matched the first $1000. Since we made the goal before Thursday, I donated an additional $500 as promised. That's a total of $2865! Wait... there's more! Many of the people that donated work for employers with a "gift match" program, which doubles their donations. If my math is right, this will increase the total to more than $4500! Dr. Harbatkin asked me to remind people that there is a "Save Lyon-Martin" page on Facebook, fundraisers and other events on listed on http://lyon-martin.org/events.php, and to check out their web site http://lyon-martin.org for periodic announcements.
Memory tip: Remembering short lists
[For the next week or so I'll be posting the techniques I use to help me remember things. I'll be covering topics like memorizing short lists, oddball things, and names.] The human brain isn't good at remembering lists. Our brain didn't evolve to be good at that. Instead we evolved to be good at making tools and inventing things. One of the things we invented is paper, which is much better at storing lists than our brain. We also invented PDAs and cell phones. If I don't have paper, I can TXT the list to myself. However, we don't need those tools for short lists.
Goal at 94%! Your donation can push us over the top!
Would you please help me and help thousands of women in San Francisco? We are so close to the goal of raising $1,000 for Lyon-Martin! Thanks to everyone that has donated so far! The response has been very impressive. LM has made is very easy to donate. Click here. $10 or $5 will really help. I'm matching the first $1000 and if we reach $1000 I'll donate another $500. Hopefully one of my books has saved you much more than $20, why not "pay it forward"? To a syadmin $10-$20 is chump change. Donate today and help save this clinic! LM has made is very easy to donate.
67% of the goal reached! Please help me make it to 100%
I'm matching the first $1000 and if we make it there, I'll donate another $500. Please help us get there. Thanks to Nathan, Michael, Lee, Jennifer, David, Gabe and Dave so far! We've accumulated $670 so far. Maybe we should set the goal to $2000? Paypal makes it easy to donate. Best quote so far, "I figure the dude's helped me save AT LEAST $10 worth of wasted effort over the years, if he seems to dig Lyon-Martin Women's Health, then I do too." If my books have helped you, please donate. If for some reason you don't like me and/or want revent, donate a LOT so that we hit the $1,000 mark and I'm forced to donate another $500.
LM fundraising campaign day 2: 12%
Thanks to the people that have donated $120 so far, we have reached 12% of the goal of raising $1,000 to help keep Lyon-Martin Health Services open. This clinic served 2500 patients last year and will close if they can not raise $250,000 very soon. Remember that if we reach $1,000 by Feb 3th I'll chip in another $500. Wouldn't you love to know you forced Tom give up $500 more than he originally planned? As mentioned yesterday, I am matching the first $1,000 donated by my friends, readers, and fellow geeks. To be honest, I'm a bit disappointed that only $120 has been raised so far but the campaign is still young.
Please read if you live in/near San Francisco
This is going to be one of my rare non-sysadmin/non-technology posts on EverythingSysadmin.com. If you have a problem with that, please skip this post. Lyon-Martin Health Services (LM) needs your help. LM is a great institution in San Francisco that provides compassionate, respectful health care to women and transgender people at a sliding scale. They've done this for 30 years but yesterday they announced they will be closing their doors. If they can raise $250,000 soon they may be able to reorganize and stay open. I'm setting up a matching donation challenge: I'm asking readers of this blog to chip in $10-20 to help them reach their goal.
Video of the week: How to tell people to "go away"
As I edit the videos from my "time management" collection I see that some of them came out better than others. This is one of my favorites. Episode 27: How to say 'Go Away' to a user and still be polite? http://www.youtube.com/watch?v=vbv4yprHQXQ When a user interrupts us with a question and we are busy there are ways to say "go away" without sounding like a jerk. (1) make sure they feel heard. If they don't feel heard, anything you say will sound like a jerk. (2) re-enforce good behavior: teach them the right way to get help (file a ticket, etc.)
Calling All Windows Trainers
The PICC Conference is looking for Windows Trainers. If you have experience training people on PowerShell, ActiveDirectory, or Windows 7 (or know someone who does) please read Matt's blog post.
TPOSANA 2nd edition now on Kindle!
We're happy to announce The Practice of System and Network Administration is now available on Kindle! http://www.amazon.com/dp/B004JLMUJ0 Thanks to everyone at Addison-Wesley and Amazon for making this happen. Thanks to all the fans that clicked on the link on Amazon asking for a Kindle edition.
Video of the week: "Get boring!"
As I edit the videos from my "time management" collection I see that some of them came out better than others. This is one of my favorites. Episode 18: "Get Into That Old Boring Routine" http://www.youtube.com/watch?v=DG3aSFk5Lfk In this video I passionately describe the importance of routines and how they can be used to eliminate "the bad kind of excitement" and instead emphasize "the good kind of excitement" we deserve. He lists examples related to planning meetings, buying gasoline for his car, and preventing a very wet, messy, situation at work. These routines create a "domino effect" of benefits.
Testing is a waste of time
Do your developers do unit testing, system testing, or even worse the so called "continuous test"? Just tell them this redundancy will not be tolerated. It is a waste of time. If you skip those tests then the customers will find those bugs for free! Don't be foolish and turn down this free labor! (The owners of everythingsysadmin.com apologize for the above message. Tom was replaced by an evil pointy haired boss Tom. Evil Tom was wrestled to the ground and the real Tom has taken back control of the keyboard. Please disregard the above message.)
Don't make your own patch cables.
True story: My first job out of college we made our own patch cables. Usually we'd make them "on demand" as needed for a new server or workstation. My (then) boss didn't want to buy patch cables even though we knew that we weren't doing a perfect job (we were software people, eh?) . Any time we had a flaky server problem it would turn out to be the cable... usually one made by my (then) boss. When he left the company the first policy change we made was to start buying pre-made cables. That was during the days of Category 3 cables.