Tag Archives: problems

Five laws of incidents and problems

Incidents and problems are in place to restore a service, fix an issue, work out why the issue or outage happened in the first place and then try and make sure this doesn’t happen again. All teams should be working together to make sure there is minimum downtime to the business on all incidents provided the right priorities are followed. We have all seen the analogies of incidents and problems.

eg. http://www.reddit.com/r/ITIL/comments/2d1zga/how_do_you_explain_the_difference_between/

https://itilbegood.com/2014/07/28/requests-incidents-problems-and-known-errors-in-a-nutshell/

However, where it gets a bit confusing is, where does investigating an incidents root cause and resolving the service cross over into problem root cause territory. Why should an engineer set about investigating an outage have to raise a problem if they have the incident from the customer, surely this all seems like a lot of paperwork for a few clicks?

Therefore, I wanted to put a stake in the ground, after a few years doing support, and then everyone can shout me down but at the end of the discussion / bloodbath we might have a solution. Of course it does depend upon organisations but there seems to be some confusion on incidents and problems.

At the heart of the matter is this truth,

Between incidents and problems, you should be able to restore the service quickly and root cause found with the cause of the incidents being mitigated or a work around published, so future incidents can be fixed quicker. The whole purpose is to provide fixes to the business so the business operation is minimally impacted. If there is an impact, the situation should be recovered and steps to mitigate the impact or minimise it, the next time it occurs.

Ok, so lets look at two incidents, one a customer can’t access their file shares and one customer calls in and says their Citrix sessions have hung…..and two minutes later another person calls up to say their citrix session have also hung.

The first one, the support engineer would pick up the call and after some trouble shooting realise the customers password had expired, reset and reboot, the customer is up and running. The way to mitigate it is to tell the customer to reset the password before it expires. So, this process has gone through the restore of service, finding the root cause and mitigating the issue.

Next, the engineer checks the Citrix session and finds out both customers are on the same server, the engineer can not remote onto the server, therefore the server looks like it has crashed. There is a known error entry which tells the engineer to take the server out of the load balancer and reset the customers sessions, the customers will re-connect to another server so service is restored. The engineer then reboots the server and upon reboot the server looks fine. However, would you put the server back into the live environment?

These two incidents illustrate the issue, the engineer on the first call was competent to go through all the steps and complete the incident. However, is the engineer competent to go through all the steps of trouble shooting the server? Maybe not, maybe a Citrix team needs to be involved in checking out the server before the server is put back in to the production environment. This is where a problem should be raised, the incident can be closed or linked to the problem but a problem should be raised as the server needs to be checked out why it crashed but the production environment continues to function.

Law one, raising a problem comes down to the competency of the support team. Can  they restore the service, find the root cause and mitigate it in an incident or can they only restore the service and then raise a problem for a specialist team to find the root cause and mitigate the issue.

Next, time needs to be monitored on incidents. Engineers love to trouble shoot it and fix issues, trying fix after fix to get to the bottom of the issue, however, this may take an hour. However, is this good for the business? If the engineer could put in a work around for the issue in the first 5 mins and leave the customer to get on with their day but raise a problem to investigate the issue further without needing to bother the customer, then surely this is a better way of working from the business point of view?

Law two, incidents, where a work around is present this should be implemented and a problem should be raised to find the root cause at a later date. The priority is to restore the service to the business.

When to raise a problem should be a thing of governance. ITIL explains this ITIL Service Operation page 99 (service operation process – Incidents versus problems)

The rules for invoking problem management during an incident can vary and are at the discretion of individual organisations.

Therefore when to raise a problem is up to the organisation. In the examples of the Citrix server, I would suggest a problem should be raise when the impact is to many customers, a key service or server is impacted or to group incidents together to raise to 3rd party suppliers in supplier meetings, eg the support teams notice a few hard drives are failing in the first few months. These incidents could be group togeher to raise to the 3rd party supplier.

Law three, governance should write up rules on when a problem should be raised and clearly communicated to the IT organisation.

eg A problem should be raised for all Citrix server crashes and assigned to the Citrix team

Incidents should be monitored for trends and to check if a problem could be raised to mitigate recurring incidents. Monitoring the incidents can also help check if a work around could be put in place for a long running incident and problem raised to find the root cause.

Law four, all incidents should be monitored for trend analysis and time to fix to see if a problem can be raised to mitigate the underlying issue.

Finally, once the root cause is found either through incidents and problems, one of two things should happen :

– Mitigate the issue.
– Add the issue to the known error database with a workaround / fix.

Law five, all root causes should be mitigated or the fix time shortened by writing up a known error entry with a fix or work around.

I believe by following these laws engineers have scope to troubleshoot issues as they come in whilst the business operation down time is minimised.

What does everyone think?

Thankyou for reading my post. This is my opportunity to blog about a subject I love but am still learning. These posts are my way of showing how I understand the subject, however, I would encourage you to leave comments, did you agree / disagree with the post? Did I not explain something well enough or incorrectly? Do you want me to blog about another subject within ITIL? All feedback helps me to understand more. Thankyou.

Is your service desk really a help desk in a fish costume?

Fish

Let me start with some honesty. I hate calling call centres. Every time I call I seem to loose a little bit of the will to live especially when they try and all call themselves all manner of different names, my recent favourite was ‘a customer experience agent’, when all I get is passed around, all telling me someone else should be dealing with my call or they will check and give me a call back, if I had a pound every time I was told that. A customer experience centre is still a bad call centre if I still get passed around and nobody really knows what to say or do.

I have just moved house and needed to register with British Gas (UK Gas and Electric utility company), I needed to do four things on the call:

– Give them all the details, name, address, occupation etc
– Set up a direct debit/giro so every month the right amount for the bill would be taken out of my account automatically.
– Register to add points to my nectar card, its a UK points card where I can get money off my supermarket shop
– Register for Hive, an awesome new system where I can control my  heating from my phone/tablet/computer and hive knows when I am coming home and leaving and switches on or off my heating accordingly.

Hmmm, so not too much that could go wrong. Immediately the person who picked up the call and heard what I wanted, paused, stuttered and said she needed to put me through to someone else. Already I am loosing the will, getting a little frustrated that I want to give them money but they are making it so hard, maybe a little harsh, but this has normally how it starts and only gets worse.

Then another lady picked up the phone and she nailed it, names and address…done, direct debit….done, nectar card points…done, hive…errr, never done one of these before, hold while she asked someone how to do it (this is not a problem, it is a new system so I thought I was probably a first), then bang I have an engineer coming out the following week.

Sill with me, wondering what this has to do with fish and ITIL. Well, recently I saw on a forum the title ‘How to change a help desk into a Service Desk’, the person had been tasked with changing a help desk into a service desk because that is what ITIL says and you can’t be ITIL compliant with out it. I thought, that is like calling a call centre agent a customer experience agent, a name doesn’t change a thing, it is what you do to change the perception of the customer that the team has changed.

People should ask How do I change my IT organisation into an ITIL IT organisation?’ rather then changing team names and think you are done.

I have worked in many support environments, all called many different names and some supposedly within an ITIL framework. However, I would say all were a help desk once you took away the nice names. A help desk to me, was and is, a team which takes all the calls about anything, tries to fix anything and if they can not then they have to beg, and plead, with other support teams to help them as there are no support agreements internally to get assistance. If the Help Desk can not get help then they hold onto the ticket and try over a few days to resolve it. The customer thinks the help desk is a little hit and miss, one customer even once said ‘why don’t you just call yourself, desk, instead of help desk.’

How does this differ to a Service Desk? This really is a trick question as if the organisation hasn’t changed to ITIL then the Service Desk is still just a Help Desk with a new name.

Please read my earlier blog posts to get an idea of what ITIL is about  :

https://itilbegood.com/2014/04/07/what-is-itil/

https://itilbegood.com/2014/07/19/service-management-as-a-rugby-game/

If the IT organisation wants to be an ITIL organisation, they should :

Have a catalogue showing the business what services are supported and the service desk knowing how these services are supported.

The services should be backed up with a configuration database, showing how these services are configured. the aim here is to give support engineers access to the latest configuration of the service with all the components to make troubleshooting easier so the service is resumed quickly. This is not an exercise in creating a database and ticking a box, it has to be usable and up-to-date. How the information should be presented should be after speaking to the stakeholder who will use this information. The database is a read and write database not a write database that nobody reads.

OLA’s should be written to show internal IT organisation resolution times for services, what is included and who can be involved in these fix times and how changes to these services should be implemented. If it comes in a 30 page document, ask yourself, if you were the engineer who just got a call saying nobody can access their e-mails, could you:

Would you know where then OLA is held?
How should the support teams escalate the incident?
Find out who should be on a bridge call to help fix it?
Use the configuration database to troubleshoot?
What support agreements with 3rd parties are in place?
What and who should communicate to the business the issue?
If an emergency change or a normal change needs to be implemented, how and who should do this?

All while users are screaming at you to fix it, if you feel you can’t, fix the documentation to make it easier. One suggestion would be to create a share point dashboard from the 30 page OLA document which support engineers can look to for easy reference.

After the OLA is created, SLA’s can be written which the business then knows how long an incident, request or outages should take to complete.

So, now the IT organisation knows :

What services are supported
How the services are configured
How the services are supported

Next, how does the business tell you they want something or something is broken. This is done through requests and incidents. The Service Desk should categorise the requests / incidents and add a priority to them. The priority comes from the OLA. When the service is restored to normal the incident is closed.

https://itilbegood.com/category/in-a-nutshell/

Overview of requests, incidents, problems : https://itilbegood.com/2014/07/28/requests-incidents-problems-and-known-errors-in-a-nutshell/

If an incident can’t be resolved definitively so a work around can only be used or an incident has been closed but the root cause could not be found. Open a problem, this can be worked on by an individual or a team of people to find the root cause and find the fix for the incident.

Though, ITIL is all about constant improvement there should be some sort of incident and problem management to analyse the incidents and problems to see if these can be reduced or done better through additional training or better procedures.

If you want to make a change to the service, a group of people (defined in the OLA) should assess the change in a regular meeting for proposed work, impact, back out plan, timings (is this within a change window defined in the OLA) and if the business needs to be aware, either by the business being in the same change meeting or a business communication, or both. This should minimise the impact to the business for any changes to services.

Finally, make sure there are some reports showing the business how IT is doing and the value provided. Maybe a report showing the number of changes (successful / unsuccessful), incidents (closure rate/time, categories), problems (types, closure rate, resolutions), SLA (within SLA and if not, what steps have been taken to rectify this)

Now the IT organisation knows :

How to log incidents and requests.
How to investigate incidents in more depth.
How to improve / spot trends with the incidents and problems process.
Make changes to services in a controlled way.

Finally, the IT organisation should put in place some method of improvement. Can areas of IT be improved to provide better service or value to the business?

If the IT organisation can provide these support structures to help the Service Desk, without them, the Service Desk is a Help desk still

Remember

Help Desk

Thankyou for reading my post. This is my opportunity to blog about a subject I love but am still learning. These posts are my way of showing how I understand the subject, however, I would encourage you to leave comments, did you agree / disagree with the post? Did I not explain something well enough or incorrectly? Do you want me to blog about another subject within ITIL? All feedback helps me to understand more. Thankyou.

 

 

 

Requests, Incidents, Problems and Known Errors in a nutshell

Over the past few weeks I have noticed some talk and discussion around what incidents, problems and requests are and what are the differences between them on some of the ITIL blogs. So here is my take :

Requests

These are requests made by the customer, eg please can you install x software or please can you replace the toner on the sales printer. These types of ‘can I haves’ should be logged as a request. These are separate to incidents, as they will have different SLA’s and priorities associated to them. Installing a piece of software for one member of the sales team has a different priority than someone in the sales team can’t access the network shares.

Incidents

These are for when thing breaks or isn’t working. eg My PC won’t turn on, I can’t access any network shares or none of the print outs are coming out of the printer. These are different to requests as it normally means the customer or team cannot work or a service is degraded so they can’t work as well. The person who picks up the incident will associate a priority eg a whole office who can’t access the network might be a Priority 1 incidents and a customer who can’t print might be a priority 3 call. These priorities should be documented with an SLA associated to them so the business will know roughly how log an incident of this type will take to fix. Again, it is up to you and the business to work out these priorities and SLA’s, ITIL is just a guide. The incident can be closed when the incident is fix permanently or a work around has been put in place which restores the service back to normal.

Ahh, and this is where some will wheel out the old chestnut, is a password reset and incident or a request?

Answer

1) Why is this not automated? Plenty of tools can allow the customer re set their password themselves without needing to log a incidents/request.

2) It is up to you and how you want to define it. All you are trying to do is separate incidents (priority) over a request (sometimes, not as higher priority, as an incident), be able to produce stats on the two to show trends to help with incident and request management and reporting to the business to show how great IT are.

Problems

What happens if all that the person who picks up the incident, can do is produce a work around or doesn’t know why the fix worked or multiple customers are logging the same type of incident eg reboot the PC and the problem goes away or all that can be done to resolve the incident is produce a work around, meaning the issues still exists but there is a sticky plaster to hold everything together? Now, problems come into play. Problems are something where a virtual problem team or an individual can look into the issue deeper, hopefully finding out the root cause and a permanent fix. A problem is also something that can be taken ‘off line’. The service has been restored as the incident has been closed so the danger has past but the problem can be used to investigate over a longer period to find the real issue.

Known errors

Through your diligent problem management and investigation, the root cause is found. However, like most things in life, it is not an easy fix. The fix requires a new server, cabling or the manufacturer of the component has acknowledged there is an issue but there is no driver update so all you can do is stick with the work around. ITIL has rather cleverly thought of this scenario and known errors can be used.

e.g.

An incident was logged and a workaround took two days to come up with but the manufacturer needs to update a drives before a permanent fix can be implemented. If someone logs a similar issues, the wheel doesn’t need to be created again, a known error should of been created after the first incidents work around was found so this can be used to implement a fix/work around quickly for the second incident.

A known error and the known error database greatly reduces the fix times for subsequent and similar incidents which are awaiting permanent fixes or there are other reasons why a permanent fix can’t be implemented, so a work around is as good as it is going to get.

Hopefully, requests, incidents, problems and known errors are a little clear on what they are and what the differences are.

Thankyou for reading my post. This is my opportunity to blog about a subject I love but am still learning. These posts are my way of showing how I understand the subject, however, I would encourage you to leave comments, did you agree / disagree with the post? Did I not explain something well enough or incorrectly? Do you want me to blog about another subject within ITIL? All feedback helps me to understand more. Thankyou.

Categories in a nutshell

Categories

I wanted to write this post to try and explain why we use categories in ITSM tools such as Remedy, Servicenow etc and why I think there is a need to be monitored regularly.

Lets start at the beginning, when logging and incident or request the team categorises the incident or request.

Please can I have Photoshop installed? – Category = Request – Software – Photoshop – Install

Or

I can’t connect to any network shares – Category = Incident – Network – Loss of network connectivity – Desktop

These categories choices are open to debate and discussion, that’s the beauty of ITIL, it is open to debate and a guide. You have to work out what works best for you and your company.

The next step in ITIL, remember you whole reason for doing ITIL is to provide value to the business, is to analyse these incidents in Incident Management. This should highlight trends, eg reviewing the incidents shows 30 calls per week to install photoshop, which is taking an engineer 30 mins per install to do. Maybe, this could be automated and therefore giving 15 hours back to the engineers and providing a better, quicker service to the business. More importantly, you can see trends with incidents, the incident with the network shares, you notice this happens to the same person every week at 2pm on a Tuesday. A reboot fixes this but it keeps happening and is probably a clue that a problem should be raised and looked at this in more detail.

One of the reasons to logged problems is if incidents are trending but the root cause isn’t found, then these can be looked into in more detail and hopefully finding the root cause. Once the root cause is found and resolved, then the incident shouldn’t happen again, meaning a happy customer and your engineers can work on something else. The whole point of incident management is to look at ways to reduce the number of incidents and requests.

However, how many categories do you have in your organisation? Could an incident be categorised few different ways depending on who picks it up? Are there duplications of categories / sub categories? How often do you look at your categories and check if they are still relevant and if some should be added or deleted?

These questions I think are the crux of why I think categories need to be monitored. Do you need all the categories / sub categories? When were each last used? A lot of ITSM tools has loads and loads of categories…….are they all used and would the engineers who are logging the incidents know which one to use or could they use a few different combinations? If so, are you sure your incident management trend analysis is picking up all the incidents and giving the true picture of what is happening week on week? How often are the categories reviewed? Do you still have a category for Windows Server 2000….do you have any servers still running Windows Server 2000?

A possible solution would be this; rip down the top level categories to your primary services in Service Catalogue eg Telecoms, E-mail etc. Using your engineers previous experience, reviewing any trends of incidents/requests and intuition make up some sub categories, limit the sub categories to less than 10 and add other to all these categories. This gives a better chance to trend incidents and requests in future. However, add ‘other’ to the sub categories so any incidents that don’t match the categories can be logged under the ‘other’ category. Create a workshop for engineers to explain which type of incidents/requests should be logged under which category and what the ‘other’ category is for and document this.

On a regular basis, initially, review the incidents and categories; looking at why the ‘other’ category has been used, does another category need to be added? This is fine-tuning the incident categories. Are the engineers using the right categories for the right types of incidents and requests?

On a bi yearly basis, a review of all categories should take place, are these still all relevant? Does some need to be deleted or added? Are you able to see trends and are you taking steps to reduce them?

I hope this shows how important getting categories right and making sure these are monitored to keep them in check.

Thankyou for reading my post. This is my opportunity to blog about a subject I love but am still learning. These posts are my way of showing how I understand the subject, however, I would encourage you to leave comments, did you agree / disagree with the post? Did I not explain something well enough or incorrectly? Do you want me to blog about another subject within ITIL? All feedback helps me to understand more. Thankyou.

 

Interior design, the ITIL way.

Decorating with ITIL

What is ITIL? ITIL is a collection of 5 topics covering Service Strategy, Design, Transition, Operations and Continuous Improvement which should be used to form, implement, keep it going and improve your ITIL strategy to improve your business to IT alignment….

That was boring. No, I believe ITIL to be bigger and at it heart more simplistic then an all or nothing approach to ITIL and must be implemented exactly how the manual says so. Let me explain using an analogy.

Imagine, IT, as a house. It is a shell of house, how are you going to decorate it? You are probably going to decorate it in ways that works best for you and the people who use your house. How will you know how to decorate your house, you need some ideas…look no further than the ITIL Interior Design book. In it, you will find loads of ideas on how to decorate your new house. The covers all shapes of houses and is designed to give you ideas for your home. The book gives ideas on how to design what you want to do, implement it, keep up the day-to-day maintenance on it and how make improvements to your house. However, a word of warning, its not a step by step book. The book is more there to give you ideas to research and find out how to use it best for your house.

Using the book you can tailor design items to fit your needs eg a twenty foot incident management dining room table doesn’t fit into your house, then buy a six foot incident management dining room table, which works much better in your house but follows the design principals of the twenty foot dining table. How about a change management media centre, do you need top of the range or mid range to suit your budget but gets similar results? These are two examples of incident management and change management which the essence of what these actually do stays the same but you need to mould it to what fits your business.

The metric you want are not the concrete composite used to make the driveway, you want to know how much the amenities cost per year. Much as the same way you need to tailor the reporting metric used to report ITIL to what is most useful to the business. Does reporting just how many changes are made each week mean as much as reporting how many changes were approved AND how many failed or were rolled back with possibly the report showing how many changes where service / customer impacting. This helps to show to the business how successful and possibility how competent IT is at implementing change.

All these services can be then upgraded when the budget allows or makes good business sense to upgrade through continuous improvement. In most houses do the wallpaper, carpets and doors stay the same in the house throughout the whole life of the house, no, these get upgraded and changed. Using the energy metric you can also see if you can save more money through changing suppliers or improving the heat insulation. All this is continuous service improvement, providing you with more value from your home.

For me, this is what ITIL is, it about returning the best value returned to the business and to do this you have to fit ITIL into what works best with the business which may mean leaving some ITIL out to start with to implement when it is time. Though what ITIL, I believe, is trying to get a department, which has traditionally, been a law unto itself thinking more about the business. So many times I have heard IT complain, ‘Without IT there would be no business’ well, without the business there would be no IT. After all, if the business didn’t make any money, IT wouldn’t have a budget. So using ITIL, I believe IT can repay the investment and provide the business with the best business aligned IT infrastructure it can to make the business do even better and hopefully make more money.

Thankyou for reading my post. This is my opportunity to blog about a subject I love but am still learning. These posts are my way of showing how I understand the subject, however, I would encourage you to leave comments, did you agree / disagree with the post? Did I not explain something well enough or incorrectly? Do you want me to blog about another subject within ITIL? All feedback helps me to understand more. Thankyou.