[NA] I'm from I.T. and I'm here to help....

[flyingcheesehead]

1,000 operators at, say $15/hr, operating at 90% capacity. Normally, you would get 1,000 operators * $15/hr * 100% productivity * 4 hrs = $60,000 worth of labor value. But we're operating at 90% which brings us down to $54,000 worth of labor value. So that weekly process is costing $4,000 per 1,000 operators per week in decreased productivity.

$4,000 * 52 = $208,000 worth of lost productivity for one year. That's based on 1,000 operators. Roll it up to 5,000 operators and you just cost the company $1,040,000 in lost productivity. That ain't exactly chump change...

But is IS exactly how you get IT off your back. Show the bean counters that IT practices are harming productivity, and things will change VERY quickly.

 

[tdager]

But is IS exactly how you get IT off your back. Show the bean counters that IT practices are harming productivity, and things will change VERY quickly.

Until someone from security, someone like me, shows the same bean counters this: http://www.informationweek.com/news/security/attacks/229300517

And then shows them the customer/client requirements that we have a secure environment, and if you are totally unlucky, the regulations they are under regarding system security and the possible fines.

I will not even toss in the Advanced Persistent Threat (APT) or massive bot-nets that would love to use your "we are not important" network for attacks on others.

Sorry...but the days of the system being wide open are waaaayyyy over.

 

[CJones]

Until someone from security, someone like me, shows the same bean counters this: http://www.informationweek.com/news/security/attacks/229300517

And then shows them the customer/client requirements that we have a secure environment, and if you are totally unlucky, the regulations they are under regarding system security and the possible fines.

I will not even toss in the Advanced Persistent Threat (APT) or massive bot-nets that would love to use your "we are not important" network for attacks on others.

Sorry...but the days of the system being wide open are waaaayyyy over.

I'm not saying the updates aren't important - I understand the requirements for a secure network - I'm currently developing a corporate suite of web tools and am working with IT to make sure it meets their security requirements.

My beef is with the timing of updates. I fail to see the value of scheduling system-wide updates to occur during peak business ours. Schedule it for off-hours. If it can't run at that time, let it run at first opportunity. The people that shutdown at night will have to deal with it first thing when they boot up. People that don't shutdown will be able to function without ever knowing the update was performed.

(Sidetrack: Is that your place for sale in your signature? Wish I had the job that would let me buy a place like that right now.)

 

[JeffDG]

I'm not saying the updates aren't important - I understand the requirements for a secure network - I'm currently developing a corporate suite of web tools and am working with IT to make sure it meets their security requirements.

My beef is with the timing of updates. I fail to see the value of scheduling system-wide updates to occur during peak business ours. Schedule it for off-hours. If it can't run at that time, let it run at first opportunity. The people that shutdown at night will have to deal with it first thing when they boot up. People that don't shutdown will be able to function without ever knowing the update was performed.

(Sidetrack: Is that your place for sale in your signature? Wish I had the job that would let me buy a place like that right now.)

Here's an issue you're not considering...IT is under pressure from bean counters to ensure that PCs are not left on overnight (security folks too)...they like to call it Green IT initiatives, and in a large organization, you're spending real money keeping PCs on overnight just in electricity and AC costs.

If we push a patch overnight, we get crappy compliance, but those people who ignored directives from senior management (not IT) and left their PCs on get limited interruptions, unless of course they left their PC on overnight calculating a 4D engineering model for presentation to the client the next day, and the patch reboot stopped it in its tracks.

Personally, I prefer prompting the user to install the patch for a reasonable time period (with the user option to defer), then if the user keeps deferring, force the install. I even came up with a system for "User Defined Maintenance Windows", so users could tell us in IT when they went for stuff like lunch, and we could then install patches and run scans during that time period.

Some of us IT geeks really do treat end-users as customers. Others like to worry about ID Ten Tango errors.

 

[CJones]

Some of us IT geeks really do treat end-users as customers. Others like to worry about ID Ten Tango errors.

I see the 'customer' view as a 'miss' as well. I see IT and Operations being partners. I guess I'm a little weird in my thinking, because every company I interviewed with a couple of years ago gave me a really funny look when I said "I want to be the translator between Ops and IT. Ops doesn't know how to ask IT for help, and when they do, they are upset because 'That isn't what I asked for'. IT doesn't really know how to help Ops achieve its goals because they are so removed from day-to-day operations that the only chance they get to help is through a service request that is written by an Ops guy that doesn't know what he's asking for. IT has capabilities that the average Ops guy doesn't even know that he CAN ask for and IT doesn't know that the tools they consider 'common sense' aren't even on the radar of Ops people." By that point, most of the interviewers had a glazed over look in their eyes.

IT should be used as a tool to transform the enormous amounts of data collected from day-to-day operations into information that can help alleviate pains in the operations themselves.

When IT runs a system-wide update in peak business hours, they do so 1.) Because there is 'eminent danger' and a patch is need immediately (I'm ok with that), or 2.) They don't understand the effect that they are having on operations at the floor-level. When Operations guys go around and install a dancing babies screen-saver, they don't realize that they are bring something into the system which could bring down operations even in another country. When the IT guy rips them a new one for doing so, all they see is "The propeller head is being a jerk again."

Luckily, I am at a company that is beginning to see the returns of having combination IT/Ops-minded resources available at the facility-level. Allowing those resources to have the ability to innovate on-the-spot has potential to have bigger ROI than any major ERP implementation could ever pretend to do.

Ok... I think I'm done ranting for now....

 

[denverpilot]

That is probably the biggest problem right now. "Someone must be blamed"... but instead of CxO types going back to the MANUFACTURER OF THE INSECURE PRODUCTS, they go after their staff who "Didn't properly secure the un-securable."

 

[flyingcheesehead]

Sorry...but the days of the system being wide open are waaaayyyy over.

I never said to not run the patches... They merely need to be scheduled in such a manner as to cause a minimum of disruption to the work that needs to be done on the computers.

 

[JeffDG]

I see the 'customer' view as a 'miss' as well. I see IT and Operations being partners. I guess I'm a little weird in my thinking, because every company I interviewed with a couple of years ago gave me a really funny look when I said "I want to be the translator between Ops and IT. Ops doesn't know how to ask IT for help, and when they do, they are upset because 'That isn't what I asked for'. IT doesn't really know how to help Ops achieve its goals because they are so removed from day-to-day operations that the only chance they get to help is through a service request that is written by an Ops guy that doesn't know what he's asking for. IT has capabilities that the average Ops guy doesn't even know that he CAN ask for and IT doesn't know that the tools they consider 'common sense' aren't even on the radar of Ops people." By that point, most of the interviewers had a glazed over look in their eyes.

IT should be used as a tool to transform the enormous amounts of data collected from day-to-day operations into information that can help alleviate pains in the operations themselves.

When IT runs a system-wide update in peak business hours, they do so 1.) Because there is 'eminent danger' and a patch is need immediately (I'm ok with that), or 2.) They don't understand the effect that they are having on operations at the floor-level. When Operations guys go around and install a dancing babies screen-saver, they don't realize that they are bring something into the system which could bring down operations even in another country. When the IT guy rips them a new one for doing so, all they see is "The propeller head is being a jerk again."

Luckily, I am at a company that is beginning to see the returns of having combination IT/Ops-minded resources available at the facility-level. Allowing those resources to have the ability to innovate on-the-spot has potential to have bigger ROI than any major ERP implementation could ever pretend to do.

Ok... I think I'm done ranting for now....

You and I are of very similar minds...

 

[jesse]

That is probably the biggest problem right now. "Someone must be blamed"... but instead of CxO types going back to the MANUFACTURER OF THE INSECURE PRODUCTS, they go after their staff who "Didn't properly secure the un-securable."

Perhaps thats a problem in some places, but overall that's not what I've seen. Many large corporations spend ridiculous amounts of money to buy sketchy software from corporations at insane prices instead of using better open source solutions or building it themselves because "we need someone to blame". Nobody wants to be that someone. Everyone wants to point to someone else.

In my IT career, I've had more problems with closed source expensive commercial software than I ever have with open source. Not only that I've received the worst support from those companies.

It's not really even a debate of "closed" vs "open" source software. It's about choosing the best product. Often that choice isn't made. Instead people choose one where they easily can point blame. Then they don't even want to implement it, they want to hire someone else, that way they can blame them too. It goes on and on.

 

[flyingcheesehead]

Here's an issue you're not considering...IT is under pressure from bean counters to ensure that PCs are not left on overnight (security folks too)...they like to call it Green IT initiatives, and in a large organization, you're spending real money keeping PCs on overnight just in electricity and AC costs.

So... On Macs you can put them to sleep, and you can wake them and put them back to sleep over the network. So, IT can install whatever they need to do even if the computers are "off", and can return them to that state when they're done.

That would seem to alleviate the power and temperature concerns... Can't Windows PC's do that?

 

[denverpilot]

But is IS exactly how you get IT off your back. Show the bean counters that IT practices are harming productivity, and things will change VERY quickly.

Good luck with that. Sony's security breach for a GAMING platform just cost them over $150 million.

A HIPAA or PCI violation where you leak customer medical or credit card data -- or say, ALL of your customer's credit card data... is going to be really really expensive.

I'm pretty sure the Security folks (not "IT" can justify any "downtime" the IT people must do to meet the Security department's requirements at ANY large company right now.

One possible set of real answers, of course, is... "Perhaps we shouldn't have trashcan'd our original paper processes when we moved to computers."

 

[JeffDG]

So... On Macs you can put them to sleep, and you can wake them and put them back to sleep over the network. So, IT can install whatever they need to do even if the computers are "off", and can return them to that state when they're done.

That would seem to alleviate the power and temperature concerns... Can't Windows PC's do that?

If the network is properly configured, and your security folks haven't demanded that Wake-on-LAN be disabled (and that will impact Macs too).

Wakeup has nothing to do with OS, because when the computer's off, the OS isn't running anyway.

 

[flyingcheesehead]

Good luck with that. Sony's security breach for a GAMING platform just cost them over $150 million.

A HIPAA or PCI violation where you leak customer medical or credit card data -- or say, ALL of your customer's credit card data... is going to be really really expensive.

So, I say again:

I never said to not run the patches... They merely need to be scheduled in such a manner as to cause a minimum of disruption to the work that needs to be done on the computers.

 

[denverpilot]

I never said to not run the patches... They merely need to be scheduled in such a manner as to cause a minimum of disruption to the work that needs to be done on the computers.

So okay...

IT gets a notice through various Security news sites of a 0-day exploit available for all Windows machines (replace with Mac to avoid politics if you like). All that's required to trigger the exploit is pointing the browser at a "naughty" website.

Company doesn't have a technical way to block any outbound web traffic. You can go anywhere you like. The business people have decided to utilize hiring/firing policy to enforce http traffic, not a technical solution or proxy.

Let's assume it's not really just a browser thing or that users can't get anything done without their browsers, and the browser update will take ten minutes and can be forced by IT. (A lot of assumptions here, trying to match the OP's problem.) Or perhaps the browser isn't at fault, but an API call to the underlying OS is insecure.

If even ONE machine goes to one "naughty" website, let's say this exploit has the capability to leak all of your company's customer data because it gains full Admin privs on the attacked machine.

When, in your opinion, is the appropriate time to push this update? Can IT interrupt work on everyone's machines for this?

Here's the scary part. This is pretty close to a DAILY occurence in the real-world right now for Windows machines, and Macs are catching up fast. Most IT departments are NOT forcing updates on a daily basis. Main reason: It's impossible to regression test all necessary functions of the new software fast enough to keep up with the patches from all manufacturers. It's truly that bad.

 

[flyingcheesehead]

IT gets a notice through various Security news sites of a 0-day exploit available for all Windows machines (replace with Mac to avoid politics if you like). All that's required to trigger the exploit is pointing the browser at a "naughty" website.

Company doesn't have a technical way to block any outbound web traffic. You can go anywhere you like. The business people have decided to utilize hiring/firing policy to enforce http traffic, not a technical solution or proxy.

If they've gotten to this point... IT Fail. Whether it's a proxy or deep packet filtering or whatever, if it requires going to a particular web site and you can't stop your users from going there... Fail.

Let's assume it's not really just a browser thing or that users can't get anything done without their browsers, and the browser update will take ten minutes and can be forced by IT. (A lot of assumptions here, trying to match the OP's problem.) Or perhaps the browser isn't at fault, but an API call to the underlying OS is insecure.

If even ONE machine goes to one "naughty" website, let's say this exploit has the capability to leak all of your company's customer data because it gains full Admin privs on the attacked machine.

When, in your opinion, is the appropriate time to push this update? Can IT interrupt work on everyone's machines for this?

If it really is a "doomsday" scenario, then yes, push the update now. And then, re-evaluate why you HAD TO do so.

Here's the scary part. This is pretty close to a DAILY occurence in the real-world right now for Windows machines

So... You mean to tell me that Microsoft is releasing OS updates daily now?

 

[jesse]

If they've gotten to this point... IT Fail. Whether it's a proxy or deep packet filtering or whatever, if it requires going to a particular web site and you can't stop your users from going there... Fail.

Generally it's not a particular website. It's a common attack that was just published and can be any website.

These things are common. So common that most IT departments really can't keep up with them anyways. The end risk is generally pretty small.

 

[ScottM]

My in house IT department is always on the watch and catches viruses like they were mice.

 

[denverpilot]

If they've gotten to this point... IT Fail. Whether it's a proxy or deep packet filtering or whatever, if it requires going to a particular web site and you can't stop your users from going there... Fail.

Most mid-sized and smaller companies are in this boat. More like CxO Fail, in not having any clue about how much all that stuff costs.

If it really is a "doomsday" scenario, then yes, push the update now. And then, re-evaluate why you HAD TO do so.

You missed the point. Every remote privilege escalation attack vector through a browser is --by definition -- the "doomsday scenario". New ones are reported daily. Generally all code quality really is THAT bad.

The "Why did we have to deal with this?" is a daily question from security pros and sysadmins.

My public Internet machines at the new gig had no less than 12,000 attack attempts of various sorts on them yesterday according to my IDS monitoring machines in the DMZ.

Even had an unauthorized data retrieval attempt from an IRS IP address today. (The security guys got some laughs out of that one. Guess who's got a compromised machine somewhere on their network?)

So... You mean to tell me that Microsoft is releasing OS updates daily now?

Two this week so far. Are you serious? Sign up for a SANS.org threat list or similar if you like. Eye opening.

Exploit creation is out-pacing patches by a significant number of days now. The "Security industry" wants everyone to think it's manageable. Sysadmins know better and keep backups to flatten a compromised system and reload it. That's not feasible on desktops. Business continuity money/funds go to the centralized machines first.

A lot of bored kids getting paid in other countries to sit in rooms and code exploits for getting credit card info or to run botnets.

At least three or four "legitimate" forms of Internet money created through money laundering schemes that are accepted by the underground cracker community.

It's ugly out there.

 

[flyingcheesehead]

My in house IT department is always on the watch and catches viruses like they were mice.

Your in-house IT department is also significantly shortening the life of your gadgets, unless you're taking them apart (I'd suggest quarterly) and getting all the cat hair out of them.

The two WORST things for electronics: Cats and smokers.

 

[denverpilot]

The two WORST things for electronics: Cats and smokers.

I'll take a cat hair radio or PC over a "I spilled a Coke in it a month ago and now it's not working right" Public Safety radio and day of the week.

Sugary soda is way above cats on my list of things destructive to electronics.

Lightning and the resulting power spikes are also far above cats and soda.

Cat damage is usually a clogged fan and a resulting single-source failure of the cooled component.

Smoke damage is either that or 10 years later when it starts eating into etching on the circuit board. The presence of humidity will accelerate that.

 

[denverpilot]

Dishwasher safe keyboards are becoming popular. Probably an idea that should have come along sooner.

http://store.sealshield.com/all-products-c8.aspx

 

[flyingcheesehead]

I'll take a cat hair radio or PC over a "I spilled a Coke in it a month ago and now it's not working right" Public Safety radio and day of the week.

Sugary soda is way above cats on my list of things destructive to electronics.

Lightning and the resulting power spikes are also far above cats and soda.

Okay - I should have said two worst things for electronics in the long term. Yeah, there's a multitude of things that will cause instant destruction such as those you've mentioned.

Cat damage is usually a clogged fan and a resulting single-source failure of the cooled component.

Cats love to sit on those cozy warm electronic devices, and their hair gets sucked into fans and vents and clogs them up.

Smoke damage is either that or 10 years later when it starts eating into etching on the circuit board. The presence of humidity will accelerate that.

Nope - What I see from smokers is that the tiny electrical currents in the chips and on the MB attract the smoke particles out of the air, which makes a very effective blanket over everything and it gets cooked. It'll never make it long enough to etch the board. It's also freaking disgusting.