Does and should IT exercise complete control over an embedded developer’s workstation?

Wednesday, October 27th, 2010 by Robert Cravotta

It seems to me everyone has their own personal IT horror stories. I am one of those few people that have lived on both sides of the fence when it comes to running an IT department and doing embedded development. My stint with IT occurred during the transition of combining many independent islands of department networks into a single robust company-wide network.

I enjoyed both jobs. They both had tough challenges, unpredictable and uncontrollable environments, limited budgets, and the end goal of keeping the system operating no matter the failures. I found that the IT team was frustrated at how the users seem bent on purposely destroying the operation of the network while the users were frustrated at how the IT team always tried to prevent them from doing their job. The truth is there were real problems that each side had to solve that the other side didn’t always understand. Worse, each side’s approach often sub-optimized the efforts of the other side.

One strategy that we used to preserve the robustness of the network while allowing the embedded developers the ability to do what they needed with their workstations was to allow them to work in isolated labs. This reduced the variability of hardware and software on the production network without restricting the types of tools the developers could use. However, there were always some on the IT team that did not like this approach because it represented exceptions to the “grand architecture” of the network.

Embedded development is the practice of trade-offs – but then, so is developing a good network design and IT team to keep it running in a productive fashion. Equipment fails all of the time – not because it is of poor quality, but because the equipment runs non-stop every day and the mechanical parts do fail over time. When you consider the thousands of network devices, something is breaking all of the time. To me, it was a system design issue that was similar to the embedded systems I worked on before transferring to the IT group.

Given the horror stories I hear from other embedded developers, maybe our site was not the norm in how we worked with the embedded development teams. Does the IT team find ways to work with your needs or do they force you to work around them like the horror stories seem to indicate. Or are the horror stories merely the result of people embellishing a single bad experience so long ago?

Tags:

21 Responses to “Does and should IT exercise complete control over an embedded developer’s workstation?”

  1. SdR @ LI says:

    Some places I’ve worked have been small enough that IT was informal and either could work well enough with the development team, or was a responsibility of the development team. This works well enough, though is sometimes a distraction to getting our real work done.

    Generally, I abhor locked down workstations. As a developer you’re always having to install new libraries, new development environments and so on. Also, to be efficient, you need to setup a workspace that’s very customized and different than the average office worker. Most places I’ve worked we have had full access to our machines, even when the office machines were more locked down.

    With the greater privilege, comes greater responsibility. You have a responsibility to not do things that hurt the network, you’re responsible for good security practices, and responsible for sys-admining and troubleshooting your own workstation. IT can’t be responsible for holding your hand when you’ve been the one to screw up your own environment.

    The best places I’ve worked had IT people for handing the overall network and the “regular” computer users, and have left engineering up to engineering. Sometimes on a different subnet, but at least on a different segment. We usually had our own servers different from the business servers, and usually even had our own server room that we could pop random servers up as we needed, without impacting the IT infrastructure.

    As for workstation-level stuff, we would have “root” on our own boxes. We typically have our own switch at our desk for multiple devices, and I’ve usually had my own router and subnet (provided by a normal home-style netgear or similar router) so I could work with devices that would not work politely on the overall network.

    I currently work at a company that does WiFi firmware and system level development. This place would be a nightmare for any IT department to lock down! By the nature of our work, we’re always starting new APs and adding network devices. IT would just get in the way in this engineering environment.

    In summary:
    1. Locked down environments and engineering don’t get along.
    2. Engineering needs to be responsible for some of their own IT work.
    3. Engineering needs to be responsible for good IT practices so they can have an effective environment.

    - S.

  2. I.R. @ LI says:

    At my previous position (see my profile) after we were bought by a large firm, the engineering workstations were “locked down”. This was a disaster and was eventually lifted for the R&D engineers after a lot of complaining and issues. Then the internet filtering was imposed. It was so tight that we could not even acces various forums and so on. One day I wanted access to a particular site so I called IT. Rather than simply looking at it and granting access, they forwarded me a form to fill out. It was unbelievable. A 5 page form requesting access to a blocked site (which included the time period you wanted access for) and required the multiple management signatures up to the VP level (and I am *not* exagerating). These concepts about giving the engineers the tools they needed to efficiently get their jobs done were something that company just didn’t get.

  3. R.A. @ LI says:

    Personally, I think developers should exercise complete control over IT’s workstations :-) ))

  4. W.R.B. says:

    Wow. This is a hard topic. Like Robert Cravotta I have worked both sides of the fence. Most recently as the single IT person and Firmware/Software Developer. The mantle of IT has come and gone as employees pursue other careers. However I have been able to keep the mantle of developer and Engineer.

    I have worked in copanies with as many as 3000 employees to as few as 14. The larger companies have stricter rules as it becomes hard to communicate with each individual the needs of security, profiling, and proper internet restrictions. Smaller companies have an advantage in that what the owner/CEO says becomes policy quickly. The rule I learned was to only give an employee access when they complain, and then it was reviewed by management. Further, as a manufacturing company, most managers do not want their employees having internet access as they often spend to much time in personal pursuits instead of focusing on tasks at hand.

  5. W.R.B. says:

    (continued from above)
    When working for as a Support person for a Software house, each engineer had full control over their system as to what they installed and used. This allowed them the atonomy they needed. Security become difficult, but the projects were done on time and everyone had a good time. Including network games after hours.

    With my current employer I am only responsible for Embedded and Analog Engineering systems. Senior Engineers have full control of their systems as they are constantly changing software, loading demo packages, and adding drivers as needed. HOWEVER, the network is maintained by IT and restrictions are in full force as to what data each person in the organization has access to. For instance, at one time our CEO owner had full access to the server. That is no longer the case as accidental deletion of files could result…

    The rules I have enforced depend on the culture and management of the company. Safeguarding IP is always of utmost importance.

  6. L.R. @LI says:

    Actually, I have been on both sides too, except I started my career in IT, and subsequently switched to embedded, and in no way I would ask this question, because IMO, today’s IT is typically incapable of understanding embedded development, otherwise they would have too moved to embedded engineering to earn more.

    There is also a fundamental conflict of objectives – while IT is (supposedly) focused on stability and safety in the production environment, developers need the freedom to change their environment and experiment.

    The most pragmatic solution in my experience, is to create a completely separate network for developers, isolated from the production network in accordance with IT level of paranoia, and proportional expense. In some cases this would lead to embedded developers having to use two computers – one for development and one to access the corporate tools (e.g. ERP).

  7. B.C. @LI says:

    I too have been in small environments where I could do any thing and I also have been severely locked down by big oppressive IT departments. Management is always pushing hard for us developers to get our jobs done. As long management is supportive of IT, then they can wait for my code. As soon as restrictions are lifted, I can turn my code faster.

    My advice, document the slow downs caused by IT restrictions. Let management know your issues.

  8. H.R. @LI says:

    H. R.. I guess I was lucky to have a rapport with the IT guys at places where I was doing embedded work. In general they just gave me a workstation and said “Go to ti!”. Installing Linux was not something they were comfortable with and simply left it to me. I never “stepped on their toes” without consulting them. Consequently there was no problem.

  9. P.F. @LI says:

    I my view IT provides a service to the company, we are their customers. If their B$ causes decreased output they are not providing the service. In my current position I’m the Linux guy they come to with problems BUT I still am not allowed root access to networked systems. This has caused huge battles, and currently there is a “don’t ask, don’t tell” situation.
    What would work better is a segmented network ( which is what I’ve been trying for) with the control system and embedded controllers on a separate isolated network, along with the dev systems.
    One of the sillier battles fought was over anti virus software and the choice of distros. It seems you can get Linux anti virus software from Norton for a FC. I don’t use FC for my own reasons ( kernal issues and drivers….).

  10. S.J. @LI says:

    Embedded engineers need to have full control of their environment. In one of my previous jobs, the IT department try to lock down our station whiteout success. And was decided that the IT department stop giving us (the embedded engineers) their support!

  11. J.G. @LI says:

    I’ve been lucky in that I’ve always had control over my work stations. Sometimes the IT guys were reluctant at first but once they learned that I knew what I was doing we worked well together.
    Embedded development requires tools that are often not part of the regular corporate applications menu and IT usually doesn’t understand them. It makes sense that the engineers be responsible for installing and maintaining the special tools as well as the relevant computer configurations.
    One way too keep the specialized tools from causing problems on the corporate network is to put in an additional LAN just for development purposes and the engineers have computers with 2 network connections. This was done at one company where I worked and it allowed us to do things that might have caused problems on the corporate network in isolation on our own dedicated network.

  12. R.A. @LI says:

    “One way too keep the specialized tools from causing problems on the corporate network is to put in an additional LAN just for development purposes and the engineers have computers with 2 network connections.”

    Of course, it isn’t necessary for engineers workstations to have two connections; you can have either the development network, or the corporate network tunneled. Probably makes sense to have the corporate network tunneled, as development protocols may not be routable.

  13. AIF @LI says:

    The most annoying part when development system is managed by IT are forced Windows updates and reboots in the middle of test runs.
    In my opinion the best bet is to have a slim sub-$300 system for e-mails and other corporate applications and a real workstation for development. Development LAN is separated from the company LAN. In reality development workstation only needs FTP to transfer files and access to SVN which can be done via secondary NIC on the slim workstation fully fire-walled and controlled by IT. Thus whatever you do on the development system would not affect corporate LAN.

  14. R.D. @LI says:

    In my earlier years in embedded development, our company’s IT was not that well equipped and also short of hands. So, in order to make my development path hassle free and without any waiting-time-for-IT-to-resolve-issues, I learned a lot of Windows and Linux system administration myself. This has helped me a lot over the years, especially now when I am working as independent professional. I don’t have to call IT support every time my desktop or server at home has a problem. Either I can fix it or browse the Internet to find the appropriate solution and apply it.

    As for large IT organisations, they are mostly incapable of understanding the nitty-gritties of amount of freedom required to undertake embedded development. So, freedom should be given to the developers to manage their own workstations. BUT, the catch here is that with such powers some sense of responsibility should also be ingrained in the developers. IT dept should at best do regular audits of the workstation to check what all is installed and figuring out how much of that is threatening to the organisation. They can raise flags if inappropriate software or unlicensed software is installed, and so on.

  15. E.P. @LI says:

    Ronald Reagan spoke of START treaty with “Trust but verify.”

    Embedded developers should have a a large measure of trust from management just as DBAs should for example. But checks and audits also should be accepted by the developers. Embedded developers often need special tools and applications. They are often adept at configuring the development system to match their needs. If management has decided a person is skilled enough to do this kind of work, they need to trust the developer.

    I have often worked on separate development networks. You should not need to do web browsing from an embedded system development station.

  16. R.A. @LI says:

    “You should not need to do web browsing from an embedded system development station.”

    I’d have to disagree strongly here.

    I simply can’t imagine how I could develop even a few hundred lines of code without Google as my Posix API reference search tool (before search engines, I actually had memorized much of the Posix API, but since Google, I have reclaimed all the neurons used to store the API, and put them to better use :-)

  17. M.P. @LI says:

    Hi All,

    Just read these posts and though I would share the following:

    After having a number of horror events over the years with my development PC. In my small company your development machine and office machines are the same. This is can be very annoying, when developing I install apps and the Windows PC just gets slower and slower.

    I recently had a hard drive failure which although bad news, in some ways it is also great; gives you the opportunity to get rid of clutter. I also took the opportunity to start working differently. I now do all my development under Wmware virtual machines. So far it has been a great success, my PC remains snappy as I am not installing loads of apps and all the tools I have tried (Orcad, Lauterbauch Trace, PAD’s, Compilers) under Vmware work perfectly fine.

    It seems to be a good compromise for me; I can have a different virtual machine to suit the project I am working on. So far there are no real down sides except waiting for the virtual machine to start.

    In my experience I would recommend that people give virtual machines a try, it’s like having multiple PC’s at your disposal.

  18. K.P. @LI says:

    http://www.DiligentMinds.com

    After reading all the posts, I was going to mention VMware Workstation, but Mark Perkins stole my thunder. With VMware Workstation, you can have the best of both worlds– IT can completely lock down the host O/S– and at the same time the user can be allowed to create any virtual machine they need (any O/S). All of the USB ports and other hardware can be used just fine from the virtual machine. I use this technique myself, and I recommend it highly.

    If IT has locked down the Internet to the degree that you can’t get to important support web sites, it might be better to bring in your own laptop (or netbook) with a cellular broadband dongle– this would allow complete unfettered access to the Internet, and IT would have zero control over your own private machine. This is a lot easier than fighting with IT every time you need access to a different support site, and the cost of the netbook and the cellular broadband service may even be tax deductible (but check with your tax advisor first.) Another way to do this is to set up a proxy server at your home (that has broadband Internet), and then set up an encrypted connection (proxy server) on one of your virtual machines that “tunnels” through the corporate IT network to you home network. That way, all Internet browsing is completely hidden from the corporate “nanny program”– and you would have free and unfettered access to the Internet– with the only cost being a little bit of initial work.

  19. P.B. @LI says:

    We were fortunate to have an IT department that trusted us enough to allow us to have laptops that were not under their control. The mac address for these were in the company’s mac filter table (Enet&WiFi) so the were disallowed from connecting (if we plug in the wrong Enet cable by mistake). We had a separate, dedicated T1 to the internet with fixed IPs so we could load the tools we needed, including the version of Windows, no auto logouts, and still have internet access. If we lost data we recover from our own backup, but that never happened.

  20. J.M. @LI says:

    The way IT is structured in many companies today just doesn’t work well when the needs of engineering conflict with the typical needs of the rest of the company. It seems that many companies don’t have an easy way to deal with the differences. I’ve run into problems with my PC being rebooted while trying to log data, insane delays getting software installed, inadequate memory/CPU/storage/bandwidth/latency, etc, etc. There are solutions to many of these problems(some of which have already been mentioned), but it makes the IT group work harder or spend more money, so it is difficult to force a change.

Leave a Reply to M.Y. @LI