Crisis Management

Successfully navigating through an IT crisis requires both successful problem solving AND successful communication.

When it comes to a crisis, it is absolutely imperative to communicate effectively. People tend to fill in the details of what they do not know with their imaginations. Most often, those ad-lib details are inaccurate. Without good communication, you could experience the best, most ingenious solutions in the world, and you would not be the wiser. Seeing the ashes of a house that once stood might make you condemn a fire department. But knowing that the firemen saved everyone single living being in that house might allow you to focus on and appreciate the heroism, rather than to kindle any condemnation.  The facts do not change, but your perspective does.  Good communication yields more accurate perspectives and that is exactly what you want during a crisis.

Should the NSA have data monitoring capabilities?

Personal freedom is what is at stake here, or at least that is the concern of many. How can someone feel free to speak their mind on government matters if their anonymity is nothing but a transparent cloak in the eyes of the government? The ability for any person or organization to indiscriminately and discretely collect personal data about others is power, no matter how you look at it. As they say in the movies, “With great power comes great responsibility.”

Trust is the real issue for most people. Do we trust our government to use good judgment and sufficient discretion with a tool as powerful as the one that they have in front of them?

But this debate really goes even deeper than the previous questions. Without context, idealists on the right and the left will draw hard lines about this. But the challenges that our country faces are quite complicated. Working in the IT field, I have been presented with many real-world security challenges. Usually it boils down to the many vs. few predicament. Should we let the few rotten apples spoil the bunch? How do you save the good apples? If only a few people at a company present most of the risk to the entire company due to their abuse of policies, do you take away everyone’s rights by imposing restrictions on everyone? Or do you develop a comprehensive system for holding people accountable so that you can allow more freedoms? What if the same system of accountability that protects personal rights requires access that encroaches on personal freedoms? Now you have to choose between the lesser of two evils. When you throw in “trust”, it further complicates the decision. Maybe now, you too understand the predicament that we, as a country, face.

Entitlement and Control

It sounds like a harmless word.  Perhaps even a good one.  But the word entitlement can be quite a dirty word.   Let’s look at 2 examples of why that might be.

Scenario 1 – The Commute

You are driving along in your car on a 2 lane road cruising along at 60 Mph, the speed limit.   Out of nowhere a car pulls out in front of you and you have to slow down to avoid hitting it.   To make matters worse, they are driving at 55Mph – under the speed limit.  Remember, you are only on a 2-lane road.   The law states that you cannot pass over a double-yellow line, and there is most certainly one present.   You may feel entitled to pass the person because they cut you off and are driving under the speed limit.   The right thing to do is to stay behind them and to keep a safe distance.  But that feeling of entitlement can become overwhelming to some people and may make you want to tailgate them, break the law, or worse – break the law and then slam on your breaks in front of them to show them how it feels to have done to them what they did to you.   It starts with feeling entitled, and if you let that feeling take control, you may find yourself with points on your license, a car accident, or if the other driver is like you – road rage.    Not good.    Accept that you cannot control other people, accept that there are laws that you need to abide, and keep calm and carry on.   Clearly, this is the right thing to do.

Scenario 2 – The Poker Game

You are playing a friendly game of poker.   A friend is the dealer and you are surrounded by other friends around you.   You have been playing for a while and your luck has been pretty good.  Others around you have been less fortunate and they have had to buy back in a few times.  This is a defining hand.   You are feeling confident with your pair of pocket aces and you make a bold move.  You go all-in.   You want to end the game right here and take everyone’s money and leave.   The dealer goes to deal the last card to you, but he decides to change things up.   He reverses the order that he deals the cards.    One of your friends gets his 2nd ace after he already had a pair of kings.   He wins everything, and you lose everything.   Some would say that’s all part of the game and would walk away only feeling deeply disappointed.  But you may feel that the dealer’s decision to reverse the order of the last deal broke your winning streak and caused you to lose.   You may be right.   You may feel entitled to the winnings – and that is where bad things start to happen.  A friendly game of poker can now quickly turn into a shouting match.   The dealer gets blamed for his part, even though he played by the rules.  The winner is resented because he now has the winnings that you believe you should have been entitled to.   And friends become enemies.   Clearly, the right thing to do here is to recognize that the dealer’s choice is the dealer’s choice.  You cannot control that.  Losing is part of the game, and it is part of life.   You are not entitled to the winnings because you were never in control of that in the first place.   Recognizing that, the next right thing to do is to congratulate your friend, thank the dealer for keeping it interesting, and walk away disappointed, but not resentful or angry.

It is so easy to fall into the trap of letting our emotions control our actions.   We are all human, after all.   But when it comes to being hurt, our actions can either stunt the pain, or they can prolong it.   When we introduce feelings of entitlement, resentment, anger, and vindication into the mix, our pain becomes another person’s pain and if they harbor those same ill feelings, then their pain will circle back again and become ours once more.   We cannot directly control who hurts us, but we can control who we hurt – and indirectly, that results in a lot less pain for everyone, including ourselves.

Contributors of Change Within a Business

There are three types of contributors of change;  Instigators of Change, Implementers of Change, and Resistors of Change.   Each of these people are vital if any business is going to succeed.  Here’s why.

Instigators of Change are the people with uncontrollable passion.   They may often have ungrounded ideas, but their ideas might also be essential to the business’ success.  Many times, these are the people who are focused on the reward and oblivious of the risk.

Implementers of Change are the people with the technical skills needed to implement just about anything.   They often possess seemingly unbounded knowledge, and bring tremendously creative solutions to match equally difficult challenges.

Resistors of Change are the people who are usually the doubters.   They often fear change, and have a self-serving vested interest in averting it.  Many times, these are the people who are focused on the risk and unconvinced of the reward.

We need all three types of people to be successful.   The instigators are needed for their passion and for their ideas.   The implementers are needed for their innovative solutions.   The resistors are needed to keep the whole process grounded, to point out flaws, and to ensure that if the project does proceed, that every “i” is dotted and every “t” is crossed.   Each contributor is responsible for the success of the changes.

Just as the political system in the US has checks and balances, so do the contributors of change.  The implementers project a pragmatic perspective onto the instigator’s idealistic concepts.  The resistors harden the solution by weeding out potential failure points in the implementer’s solutions.   The instigators set the end-goal and many times, also the target deadline – ensuring an end in sight.  In the end, an innovative and successful change can take place.  Win-Win-Win.

Where does job stress come from, and what can we do about it?

Many of us have experienced stress at the job.   Perhaps we feel a muscle twitch, a migraine headache, or even our blood pressure might skyrocket.   Is there anything that we can do to be less stressed?   Let’s explore.

Empowerment is stress reducing.   Enabling others to make as many decisions as is prudent is one way of reducing their stress.  Having less on our plate reduces our stress.

The long way may take longer than taking shortcuts, but doing so will reduce stress later on.  When facing the choice, always try to avoid taking shortcuts.  Do things the right way the first time.

Be direct when communicating with others.  Avoiding confrontation is many times a precursor to passive aggressive behavior.  Whether you are the aggressor or the subject of the aggression, both parties will be subjected to stress.   It is always better for us to directly communicate with others, especially when our opinions differ.

Inform the uninformed.  Working for someone who shoots first and asks questions later can be very stressful.   Sometimes their decisions may be based on falsehoods rooted in ignorance.  Our job is not to be the victim.   Our job is to keep our bosses and peers informed.  Working for or with people who are informed will reduce conflict, which in turn will also reduce our stress.

Use your internal compass instead of an external one.  When we feel external pressure, we sometimes flip flop with our ideas.  We quickly, and many times incorrectly, doubt ourselves. We are often better experts in the subjects of our decisions than our critics are.   We need to believe in our decisions independently from criticism.   If we can be honest with ourselves and willing to swallow our pride if we make mistakes, then we can use our internal compass to tell us if we are headed in the right direction or not.  Having more confidence in our decisions will help us to be less defensive and that alone will help reduce stress.

The Path of Least Resistance

A simple-minded approach toward solving problems deals in black and white.  Just as a physics puzzle without friction or resistance is easy to solve, so too are these black and white simple-minded puzzles.  But the world is complex.  There is friction, and there is resistance.  When lightning is about to strike, it does not abort because there is no superconductive object below.  No sir.  Rather, it selects the object with the least resistance to the ground.   Lightning does not require that ideal conditions are satisfied.   It only requires that the bare minimum conditions are satisfied.  Too often, I see people make the mistake of focusing so much on ‘ideal’ conditions that they fail to succeed in their goals.  When the path with no resistance is unavailable, the path with the least resistance might just be the best one to take.

Busy vs. Bored

Is it better to be busy or bored?  

That’s the question that many of us already have an answer to but never really stop to give it much thought.  When we are busy, time flies.  Those are the days where there just is not enough time to get everything done that we wish to do.   At the end of the day, we usually feel accomplished, but exhausted.  When we are bored, those are the days that drag.  We watch the clock and think, “For goodness sake, let this day be over already!”   At the end of the day, we usually feel exhausted, and unaccomplished.  Most of us seek that feeling of accomplishment, and for those of us – it is far better to be busy.   However, being too busy can sometimes stunt a career.  Let’s consider the self-motivated email administrator.  Every day she comes to work and is inundated with account updates and email related tasks.  The day completes before her work is finished, leaving no time for her to gain experience in other areas of administration, such as VPN or general server administration.   With this in mind, which is the better alternative?  To be busy, or to be bored?   My take on this question is that for every gallon of business, we need a couple of ounces of boredom.  Those ounces of boredom are learning opportunities that , if capitalized on, will help us to graduate to new areas of expertise, keep our skills sharp, and will eventually allow us to grow our careers.

To summarize:   Stay busy, but not exceedingly so.

Facts and Perception

Perception is simply an interpretation of facts.  Knowing only a subset of facts might yield a different perception than knowing the full set of facts.  When managing a team, it is important to ask enough questions to gather a complete set of facts/details.  A manager who is overworked might try to cut corners by not asking questions and making gut decisions based on incomplete information.  Most often, these are bad decisions.  Making bad decisions not only demoralizes the team, but it also hurts the organization that you work for.   Always ask questions.  Always collect all of the facts.  Then make decisions.  Never make decisions without all of the facts.

Have I used the word “facts” enough?   I think you get my point.

The Ladder Story – How to Best Motivate a Team

The Ladder Story

This is a story about a kid who was afraid of heights.   Somehow, he found himself in a job that required him to paint rooftops, go figure.   The first couple rooftops were fairly low to the ground, and were not a big issue.  But then there was one that was about 25 feet high.  He needed to first climb onto a smaller roof, and then up a shaky aluminum ladder to get to the top roof.

The first one was easy, he had climbed that high many times before and knew he could do it.   But the 2nd one proved to be much more difficult for him.   He started to climb the ladder and got about half way up and looked down.  He could see the small cars below, could feel the nimble ladder beneath him, and could see the tiny movements that the ladder took with each step.  It was enough to send him back down the ladder.

Everyone else that he was working with had already made it to the top of the ladder and were on the top of the roof.  All of them were now looking down at this kid.  Some were taunting him.  Some were encouraging him.  But all of them were staring at him.   He started up the ladder again and like the first time, got a few steps up and turned back.   He was convinced at this point that he was not going to work on the roof that day.  Everyone else was too.  They turned their attention to the roof itself and they started working.

Something crazy happened at this point.  The kid suddenly realized that it was acceptable to fail.  All of the expectations that the others had of him had vanished.  Without them, a weight was lifted from his shoulders.  He thought to himself, “I am going to do this”, and he started to climb the ladder again.  This time, he kept climbing as nobody watched.  Before he knew it, he was at the top.  As he stepped onto the roof, everyone turned with a surprised look on their faces that he was there.

This “kid” was me.  And I did what I thought was impossible that day.  I learned a very valuable lesson, too.  Overcoming personal challenges in the face of external criticism and/or expectations is extremely difficult.   I was a better worker when all expectations of me were removed.  As a manger, I am cognizant of this experience while planning tasks.  At the highest level, I do hold expectations for my team.  However, at the lowest level, the individual team members are entrusted with assuring the success of their own contributions.

Micro-managers

When managers micromanage, they take the ownership of smaller decisions away from the people who are implementing them.  The manager then becomes the guy at the top of the ladder taunting the workers below as they climb it.  It creates an atmosphere that is ripe for failure.

All too often, a manager is so concerned with his/her own success, that in order to assure their own success they believe that they need to control every detail of what their team works on.  What they often fail to recognize is that in doing so, they not only hurt the productive capacity of their team (because they become a bottleneck for decisions), but the manager also ensures a high stress environment that is prone to failure.

Lessons Learned

An employee who is able to make their own choices will have a higher commitment to his/her own success.   Failure follows stress.  Stress follows a lack of empowerment.  A lack of empowerment follows a controlling boss.  A controlling boss follows an insecure boss, afraid of his own failure.

Conclusion:  An insecure boss is a bad boss.

 

My Cookie is Your Cookie – A tale of the stolen cookie.

The Problem

I was recently asked to help with a perplexing issue at work.   Someone accessed a web page and noticed that they were logged in as somebody else.  This is a fairly dangerous issue to have.  A customer could potentially be logged in as another customer and have full access to their information.  Obviously, this is unwanted behavior.  The only solace was that this issue was extremely rare.  It was only seen a few times out of thousands of web page hits.

Chasing Cookies

A team of people was assembled, including myself, and we all sat and thought about the issue.  We tried to wrap our heads around what was happening.   My immediate thought was that this was a caching issue.  But after speaking with our web developer, I learned that he controls which pages are cached and that the login page is definitely not cached.   All of us were stumped.  We met a few times and each time people became less and less interested in reproducing the issue.  It was said by one person, “This issue cannot be reproduced.”   Most people agreed, including myself.

One day, I was alerted that someone in our office was experiencing the issue right at that same moment in time.  I immediately started inspecting his network traffic to see if I could see anything amiss.  I was looking for cookies – information that the server stores on the client’s computer.  When computers talk to a server, if they have a cookie, they will send it back to the server to be processed.  The only cookies that I saw were being sent FROM his computer, not from the server.  It quickly became apparent that whatever happened that caused this issue, it already happened and it was too late to get any useful information now.   The team met and we discussed the new information.   The conclusions were mostly focused around disabling caching, but again everyone agreed that this issue would be nearly impossible to reproduce.

A few weeks passed by and our web developer wrote something to monitor when cookies were set from the web server itself.  Meanwhile, I wrote something to monitor when cookies are set from the network itself.   At first, I saw no data.  I thought my script was not functioning.  But then I realized that the login page is secure, and I was only monitoring non-secure communications.  I expected at this point that my network monitoring would be useless.  But then I saw some data fill up my screen.  It was strange.  How would someone get a cookie on a non-secure page if the only page that sets them is secure?   I thought about this for some time and then something else happened.  My screen lit up with traffic.  It was coming from the same network that I was on.   Another interesting bit of information was that every request had a different cookie.   I wanted to know who these requests were coming from.  So I took the cookie and applied it to my browser.  At this point, I was logged in as the other person.  I then browsed to their profile information to find out which coworker was affected.  I did this for a couple of the cookies that I saw and both pointed to the same coworker.  So I paid him a visit.

When I walked up to the coworker’s desk, everything looked normal on his screen.  No pop-ups, no toolbars, no secure browsing initiated.  Just a normal session.   I notified our web developer and he quickly left a meeting that he was in to investigate.  When he took a close look, he noticed something peculiar.  Our coworker had 2 cookies with the same name, but for different domains.  One was for the base web site, and one was for the www. subdomain.  So both were applied at the same time.  I thought about it for a few moments and was excited about this new information.  What I learned was that it was possible (still without knowing why) for the browser to confuse the web server into sending cookies on every request – even for images and javascript files, which it should never send cookies for.  If caching was in fact the root cause of the issue, now I had a way of getting a page other than the login page to give out cookies (and potentially make them cached).

Within an hour, I had met with the team again and presented this new information to them.  One member suggested again that we stop trying to find out what happened because the issue has not happened in weeks.  After sharing this opinion, he had an epiphany that he shared with us.  When a web server is configured to extend the expiration of cookies (called sliding cookies), it sends a new cookie out to replace the old one if it is older than the mid-way point between the cookie’s birth-date/time and expiration date/time.    This information brought a smile to my face.   At this point, everything clicked in my head.  It all made sense now.

How It Happened

Someone loaded up the web site and logged in.  They received a cookie from the server identifying them.   Their cookie was good for only 30 minutes.   After 15 minutes passed by, the next page that they clicked on triggered the server to send them a new cookie.  Not being a login page, the server was not told that it should not cache it.   So the server went ahead and cached the page – including that cookie.   The cache is very short-lived.  Only a few seconds.  But with high traffic to the web site, the next person that hit the same page was blindly given the first person’s cookie.

Reproduction

With a good idea of how the problem happened, now was the hardest task of all; to reproduce the problem.   I realized that I could leverage the repeated cookie issue to help trigger the stolen cookie issue.   I started by logging into the web site as normal.  I then took my cookie contents and created a new cookie for the sub domain.  I pasted the legit cookie contents into the new cookie and then deleted the original one.   To test, I loaded the web site again and it showed me as being logged in.  Perfect.   Now I had to get the server to send me a graphic or non-login page along with a cookie.  All I had to do at this point was wait 15 minutes.   After 15 minutes passed, the server would know that my cookie needed to be replaced.  It would send me a new cookie but my machine would still send the server the old one back.  This would repeat with every page and every object that I loaded.  I waited the 15 minutes and hit refresh.  BAM, just as I expected, the cookies started pouring in.   I was close now.   I just had to get the server to cache a page with a cookie on it.   So I kept hitting refresh on a single page.  I must have reloaded that same page a couple dozen times.   Then I went to another browser and cleared my cookies and my cache and then hit the same page as the first browser hit.  VIOLA – I had stolen a cookie!   I reproduced the issue!   The issue that was “too difficult to reproduce” was now no longer.   Just to be certain that this was not a fluke, I attempted the steps again but this time I had our web developer load the page from his computer.  BAM – he was logged in as me.  He had stolen my cookie too.

Now most of the rest of the effort is on our web developer’s shoulders.  He has to find out how to make certain that this never happens again.

Note:  All of my tests were completed in a test environment, not in a production environment.  The actual production environment had caching disabled, which rendered this issue inert.  With the goal of turning caching back on, it was in our best interests to try to get to the bottom of this issue first – and that is what I succeeded in doing.

Technical Synopsis

Root Cause:   IIS caching static objects with header information.

Primary Aggravating Factor:   IIS can resend a cookie with a static object if sliding is enabled AND the half-life of the cookie has passed..

Secondary Aggravating Factor:   Some users may receive 2 cookies (.www.domain.com and .domain.com).   .

www.domain.com remains static, with a diminishing expiration date.  After the half life of that cookie passes, IIS sends a new cookie with a new 14-day expiration date.  The browser accepts the new cookie for .domain.com and therefore does not replace .www.domain.com, which continues to diminish.  Since .www.domain.com was created first, the browser sends that cookie to IIS first.  IIS only cares about the first cookie it receives if multiple are provided with the same name.  For up to 7 days, it is therefore possible that IIS will send new cookies on ALL requests – including static objects.   When this behavior is combined with the root cause issue of IIS caching static objects with header information, this can cause user B to receive user A’s cookie.   Before the cookie swap happens, though, IIS must cache the static object that user A receives.   If user A is exhibiting the double-cookie issue, there is a much higher likelihood of one of the many cookie-laden objects to be cached by IIS.    As long as caching is enabled, the risk is always present.  However, without double cookies, the number of objects being served up with cookies is reduced from many to just one – thereby dramatically reducing the risk that a cookie might be cached.