From the pages of Industrial Networking, a supplement to Control Design magazine: http://www.controldesign.com
Issue Date: On the Web, Posted On: 10/14/2004
A delicate balance
Factory networks and enterprise networks seem like opposing forces in the industrial universe. There are ways, however, to find the Chi that reveals the harmony of a common business purpose.
Industrial Networking magazine
By Dan Hebert, Senior Technical Editor, and Joe Feeley, Editor in Chief
DOGS AND CATS, LIBERALS AND CONSERVATIVES, Red Sox fans and Yankee fans. Some groups share a common space but just won’t get along. Ever. A common view in our industrial networking space seems to be that factory and plant floor controls engineers and arch-rival IT professionals are polar opposites, too.
Until recently, these groups passed like ships in the night and the systems they captained did too. Plant and factory control systems running on hardy, hard-wired I/O networks went one way, mostly trafficking in isolated machine data and occasionally communicating machine to machine. Meanwhile office, enterprise data and telecommunications systems went another, running on fast, but delicate data networks that managed the flow of information across the organization. If the front office needed operating data from the factory, it came through the door on a clipboard via SneakerNet or was keyed into an office network workstation at the end of the day.
Then open-standards computing, Windows-based architectures, OPC , and Ethernet began to overrun the factory control network engineer’s domain. In an ever-growing number of companies, much of the data in the machine and process system networks now needs to flow into the office, often in real time.
The factory people didn’t like outsiders messing with their networks. The IT group viewed factory and plant floor networks as archaic undisciplined balls of data-carrying wire.
But the mandate to make everything work in harmony is unavoidable. This is a true work-in-progress. After all, it’s likely you’ve overheard some of the frustrations bubble up from the ranks of each group:
“The IT folks don’t appreciate that this is where our company makes money. They impose arbitrary standards that delay installation and support, then go home at 5:00 and we are left trying to solve the problem.”
“The plant staff act like cowboys and don’t appreciate that we have procedures that must be followed to protect the integrity of our computer systems.”
“The IT staff is so paranoid about hackers that they won’t make any changes that violate the configuration standards established for the office systems.”
“They only were concerned about the plant floor when they installed that system. The databases are a mess and it is almost impossible to translate it to our accounting system.”
“They just don’t understand our business.”
In this article we’ll explore what plant people view as the big issues and challenges, how that matches up with the perspectives of IT counterparts, and exactly where the differences are and, most important, where common ground can be found. This isn’t easy, but there’s reason to be optimistic.
Vive la Difference
IT departments constantly battle the ever-increasing need to provide more computing power and software applications, and the resulting demand for more bandwidth. A poorly designed enterprise systems easily can cripple heavily burdened communications channels. The IT department must be convinced that a new enterprise automation system will not adversely affect the rest of the communications traffic.
The plant floor operates in real time, often on a millisecond-level scan cycle. PLCs are always talking, and when polling is used to process this chatter—as is usually the case—too much traffic is created on the network, at least from an IT perspective. The traffic can slow the entire network, clogging it with a continuous flow of information.
When dealing with mission-critical ops, the factory floor needs much more reliability than IT, and “not only fewer outages, but faster recovery, along with more visibility of entire network,” says Gary Workmann, principal engineer of plant floor networks for the controls, conveyors, robotics and welding division of General Motors (www.gm.com). “Different system architectures are sometimes needed to satisfy IT vs. control needs.”
Control system downtime cannot be tolerated, stresses David Glanzer, director of technology development at the Fieldbus Foundation (www.fieldbus.org). “ Enterprise computing is mostly supervisory or accounting-related and can tolerate some downtime without affecting production or safety,” says Glanzer. “Because of this difference the plant floor personnel will want to control access to network equipment and software changes. IT personnel are not used to asking anyone for permission to make changes.”
It is imperative that plant control systems be protected. “From a network perspective the technical issues for the plant floor are fairly straight forward,” says Chris Rogers, manager of electrical and process controls at Boise Cascade’s (www.bc.com) Boise Paper Solutions in Boise , Idaho . “Some time is required working with the IT folks to develop a network standard that includes the plant floor network and the business network. The biggest issue is system security.”
Security is indeed the big issue, agrees Russ Ropken, vice president of RSI Co. (www.rsicompany.com). RSI provides PLC and HMI development services and its specialty is OPC -based HMI Systems. “When the plant floor has its own, private network, security is not an issue,” he argues. “Once connected to the enterprise, your system is open to viruses, hackers, etc. The last thing you want to deal with is a factory shutdown due to a virus that's running around the Internet. On the other hand, you don't want to lock down your plant machines to the point where your operators can't do their job.”
Get Set Up
Network setup issues are challenging for both sides, as PLCs use specific TCP /IP port numbers and getting those ports opened in firewalls and routers between the parts of the enterprise can be challenging. “IT folks are paid to keep things secure, every open port is a risk point,” says John Weber, president of SoftWare Toolbox (www.softwaretoolbox.com). “The controls engineer can't go force the PLCs to use a specific port all the time. Sometimes they can. Other times they cannot.”
Once a compatible physical setup is achieved, the next issue for the plant is to identify what data is important to process, what format it is in naturally, and what format it might need to be converted to for use by other processes, says Michael Gurney, one of the principals of Concept Systems, an industrial system integrator that recently installed a new PC-based process control system running at industrial packaging manufacturer SpaceKraft Inc., Salem, Ore.
“One of the challenges that Concept had to overcome in the SpaceKraft application is that they needed to do some data manipulation in the controlling PLC in order to provide it in the correct format at the PC. Specifically, they needed to generate a special ‘motor runtime’ parameter at the PLC by programming a counter.”
That thought gets plenty of seconds. “The biggest challenge we faced with connecting to the enterprise was displaying the data in a manner that made sense to everyone from the plant floor to the front office,” says Rob Barnett, maintenance manager for Nursery Supplies Inc. (www.nurserysupplies.com), Chambersburg , Pa. , a leading manufacturer of plastic containers for the wholesale nursery industry.
When operators get more involved with operating data, they need a strong understanding of what they’re doing and why. “Training is probably the biggest plant-floor challenge,” says Steve Keller, an independent system integrator who works on projects that include small automated assemblies, and embedded programming and design. “Previously information was written down and collected at the end of the shift and entered in the computer. Now, most of the information is entered via touchscreen, which updates the ERP system in real time.”
He says a lot of thought was put into designing the operator interface screens to be easy to use. Many of the operators are temporary workers who have very little experience.
“Controls engineers we work with often feel that IT people don't understand their world,” says Weber. “Both parties are very concerned about security, however, controls engineers tell me they have a hard time getting the IT people to understand that PLCs and control systems do things differently than traditional IT systems. Sure, it’s all Ethernet, but on the shop floor, time can be much more critical in terms of responsiveness of networks, than on the office side.”
Back to security for a moment. Ropken reminds us that the PCs in the automation system have a specific purpose: to provide a view into the plant floor system for the operators. “Once connected to the enterprise, those operators might have access to uses that are not intended,” he warns. “You don't want your operators to make bids on e-Bay or visit their favorite internet gambling site. These are new issues for automation engineers. They are not used to dealing with these outside challenges and it takes a real mindset change to insure these issues are dealt with.
Then there’s the often misguided idea that the plant loses control of how it presents data. “I think that the main concern on the plant floor in making the connection to the office systems is that the manager’s life is one of change,” says David Stupar, Director, Information Services for the Ocean County Utilities Authority (www.ocua.com), Bayville, N.J. “In many cases, plant floor personnel live in a world where the raw data is obscured from their bosses, so they can present the data in the best possible light. With transparent network connectivity, supervisors of plant floor personnel have complete access to the raw data and can form their own opinions on what is happening.”
And in This Corner
The IT group sometimes has a right to feel put upon. “The plant floor wants to use 2-3% of the bandwidth of the existing IT fiber Ethernet network instead of putting in a new Ethernet network,” says Workmann. “So the plant floor requires IT to upgrade its network, then tests the upgraded network for Quality of Service and other features needed for suitable performance in a control network.”
Part of the problem, says Workmann, is that IT feels that all networks must be designed, installed, and supported by IT professionals. “The plant floor concedes the need for professional design and installation,” says Workmann, “but feels that plant floor networks must be supportable by plant floor technicians and other plant floor personnel.”
For an IT person, says Weber, it is a challenge to understand the myriad of controls system hardware that is potentially on the plant floor. “The terminology varies, the brands are many, the setup tools often look nothing like what they find in the IT world,” says Weber. “The IT person needs the controls person to help assimilate and communicate what is needed, but in terms that the IT person can understand.”
To maintain consistency, IT will emphasize the need to install common router and firewall equipment at the site and will promote configuration standards established for the office environments, says says Tom Black, president of CygNet Software (www.cygnetetm.com). “This is often problematic for establishing enterprise automation systems because of the real-time nature of the data communications that the IT staff may not initially appreciate,” he says. “For instance, IT may not understand the need to open TCP /IP UDP ports for faster communications socket connections needed for data updates.”
Getting the data into the proper places in the ERP system is one of the bigger challenges, says Keller. He works on larger projects--such as for Plaspros (www.plaspros.com), a medium-sized custom plastic injection molder. They have a main plant in McHenry , Ill. , and a second plant in Batesville, Miss. The McHenry operation has 21 molding machines and Batesville has 15.
“The ERP software, Enterprise IQ by IQMS (www.iqms.com), is specifically designed for injection molders and other cycle-based manufacturing,” he says. “Enterprise IQ does have a system just for counting cycles, but we wanted to move more information to the ERP system, so we created a customized solution.”
Gurney agrees with Keller, saying the most important first IT step for merging the networks is defining the data that processing at the enterprise level wants to see. “Before any programming is begun, we develop an interface design spec that lays out all of the data that will be required by a process and how it will be accessed, for example, which tag within a PLC will be referenced, and how the data will be routed to the server.”
Connectivity Is Easy
For the IT folks, developing connectivity standards is fairly straight forward, says Rogers . “Addressing interfaces between multiple proprietary plant floor networks must be managed. But, managing domains on two sides of a firewall can be problematic. Not all plant floor networks are managed by the same group of resources, so multiple efforts to address an overall site architecture are needed. Eliminating multiple HMIs in the same operating space becomes a challenge.”
Workmann hits on a major difference. “IT optimizes its network for bandwidth, not for determinism,” he says. In addition, he says, IT feels that plant personnel are network novices not to be trusted, especially when connecting to IT networks.
Clearly, security is a big deal for the IT folks, says Ropken. “When an automation system is added to the IT department's network, they want to ensure that the system will not pose a threat to the rest of the network. The IT department typically has a procedure to follow and strict rules for how a PC will be structured on their network. Details such as operating system, virus scanners, log-in rights, and password requirements are all going to apply to the automation system.”
These items, although important, can cause problems on an automation PC. “As a quick example, virus scanners are very important for security, but these scanners can be very resource-intensive,” cautions Ropken. “You don't want the virus scanner to negatively impact the performance of your automation system.”
Then there’s the “check out the big brain on Brad” syndrome. IT resists plant floor use of Ethernet because they think they are the experts, believes Workmann. “At GM, plant floor personnel submit Ethernet hardware to IT for interoperability testing, and this must be done before the hardware can be used in plant floor Ethernet networks,” he adds. “The GM IT group found that hardware from different vendors—that is supposed to adhere to standards and be compatible—may not be compatible.”
Some of the issues are quite specific. “As far as IT challenges, the biggest challenge was justifying an additional web server to individuals not familiar with the data and its uses,” adds Barnett. A system that Barnett implemented had OPC interfaces but he needed a visualization front end for the information they could provide. A key design objective in building the system was the ability to access the system information anytime, anywhere, without having to load any software on a client PC (Figure 1).
FIGURE 1: INFO ON DEMAND
Nursery Supplies Inc., a manufacturer of plastic containers, needed a visualization front end that could access the system information anytime, anywhere, without having to load any software on a client.
For automation systems that require local database servers, the IT department often will adapt its policies to accommodate sharing server management tasks with the plant floor staff, says Black, managing the database servers remotely, or establishing a local IT presence. “Establishing on-site IT staff often leads to a problem down the road in that the IT department begins to feel that the local staff become ‘converted’ by the plant floor and become advocates of the plant in times of conflict with the IT department,” he adds.
Find That Common Ground
Rogers says coming to agreement on roles and responsibilities is tough. “Plant floor folks don't trust the critical nature of the plant floor network to IT folks, who have not always had stellar performance in the past,” he states. “IT folks don't really understand the real-time environment well. Their thinking is still batch.”
This is a much bigger issue than IT folks realize, says Rogers. Security issues are a real concern for the plant floor folks. All agree there is little value in redundant organizational roles. “Most of the issues are cultural and require a cultural solution, but neither IT nor plant floor folks are very good on people issues,” observes Rogers.
Often there's a dispute over who owns the data and the systems. “Program changes at the plant floor can affect the IT department's data, while IT's demand for data can affect how it is generated on the process floor,” says Gurney.
“It is fair to say that the two worlds are rapidly coming together, largely driven by the common standards and challenges that exist in both worlds,” says Marc Leroux, product marketing--collaborative production management, ABB Inc. (www.abb.com). “Both, for example, rely on Ethernet and TCP/IP as the backbone for all activities, and both are subject to the threat imposed by hackers and PC viruses, and to some extent, desktop applications may be similar as well.”
That said, Leroux sees significant remaining differences. “The plant floor is the world of OPC and single-value, time-series information,” he says. “The IT world lives with SQL and transactional data. Transactional data can potentially be rolled back, or undone if it fails during any of the steps it must take. On the plant floor, information is normally related to manufacturing steps, and once a step has occurred, the data must be recorded as a historical event. There is no possibility to undo it later.”
Then there are the separatist solutions. “I like the idea of two networks,” states Scott Hoffman of Warnecke Design Services, Ottoville, Ohio. “One for the office and administration and one for the machine or plant-floor level. The network on the plant floor is more likely to have troubles such as tow motors hitting or running over cable and machines being moved. When machines are being moved, a lot of times the network is not properly disconnected from the machine. Now if the entire facility was connected together, the entire network could go down.”
Keller client Plaspros is a relatively small company without a dedicated IT department. IT functions are contracted out, so “there were some disagreements with some of the implementation,” says Keller. “Some of the IT consultants pushed for VLANS, managed switches and tying the plant into the same switches the office was using. We opted for a separate system for simplicity. Because there is no dedicated IT staff we wanted to keep it as easy as possible to troubleshoot and repair.”
Stupar thinks differences arise from the fact that the two networks serve very different purposes and neither group wants to have their network relegated to a position of lesser importance than the other. Both sides fear compromise.
One of the involved trade groups recognizes that threat of having to compromise. “The great promise of open systems often has gone unfulfilled because the devices, programs and processes used at the various layers of the seven-layer Open System Interconnect (OSI) model have different options, capabilities and standards—or lack of,” believes Katherine Voss, executive director of the Open DeviceNet Vendors Assn. (ODVA, www.odva.org). “Integrating these networks requires extra resources and programming. Even then, gaps between the systems often cannot be fully and seamlessly bridged. Consequently, users compromise their investments and rarely achieve all of the productivity and quality benefits promised by open network technology.”
She argues that common application layers such as CIP (Figure 2), the Common Industrial Protocol that ODVA and member companies tout, are the key to advanced communication and true network integration, allowing complete integration of control with information, multiple CIP networks and Internet technologies.
FIGURE 2: SHARED PARTS
ODVA claims common application layers such as the Common Industrial Protocol (CIP) are the key to advanced communication and true network integration, allowing complete integration of control with information, multiple CIP networks and Internet technologies.
Success Breeds Familiarity
The manner of data reporting can go a long way toward satisfying both plant and office needs. “Our DataWorx software packages uses report-by-exception data logging, rather than the traditional polling method,” says Bill Glover, product development manager, Inteworx.Net (www.inteworx.com), a software provider to AutomationDirect. “Traditional SCADA or HMI polling software packages act as an intermediary between PLCs and the computing system. SCADA or HMI applications have a tendency to tie up a company’s network because they are continually asking for unnecessary data updates. Report-by-exception logging, however, allows direct communication between PLCs and the computing system, eliminating the need for an intermediary. The PLC is placed as a user on the network, which allows it to send information only when needed.”
A report-by-exception logging method can ease IT’s network traffic concerns, while still satisfying the plant floor’s need for optimum performance and maximum speed. “This minimizes some of the system challenges and communication barriers that traditionally have been seen in attempts to connect shop floor systems with corporate computing systems,” states Glover. “The software allows the two systems--which run on different timetables, speak in different languages and have notably different goals with respect to data collection--to work together more efficiently.”
Since Nursery Supplies installed a browser-based facilities-monitoring application built in-house using OPC application tools, Barnett says he’s seen efficiency gains of nearly 15% and the annual maintenance budget is at an all-time low. He attributes the improvements to knowing what’s going on in his critical facilities systems at anytime from anywhere. “With this system, we literally feel we can be in two places at one time,” says Barnett. “The knowledge we obtain from our application allows us to make intelligent, informed decisions about what systems need maintenance and when.”
After considering a variety of alternatives, Barnett settled on OPC Web Client from Software Toolbox. “It’s very cost effective, provided data in my browser that would update without refreshing the page, did not require any software to be loaded on client PCs and was easy to implement. I’m a PLC controls engineer and maintenance manager and I built this system myself.”
Barnett says the software allowed him to access equipment with the click of a button anywhere on site, corporate wide, or even from home. I no longer have to wait on reports or track them down, I can see what’s going on with critical equipment by clicking on our maintenance web site,” he says. “This solution has already paid for itself. Our IT people love it—all of our connectivity issues were handled at the maintenance and plant level.”
Gurney’s SpaceKraft solution united multiple network protocols. “The factory had three different incumbent networking technologies on the plant floor. We provided a PC with interfaces to DH+, DH485, and Ethernet/IP. Rockwell Software’s PlantMetrics was set up on this PC, which was located in the plant's server room.” In addition to being able to talk to all the machines on the floor using their native network interfaces, executives at SpaceKraft can tap into the PlantMetrics server to get reports at their desks.”
Keller says DataWorx allowed the Plaspros data from PLCs in the molding machines to be “pushed” up to the servers instead of the servers polling each machine (Figure 3). “This reduces the amount of network traffic greatly and makes the system more responsive and scaleable,” he says. “VB.NET is used to format the data before it is placed into the ERP database.”
Injection molders Plaspros pushed data from the machine PLCs to the servers instead of the servers polling each molding machine. This reduces the amount of network traffic greatly and makes the system more responsive and scaleable. VB.NET is used to format the data before it is placed into the ERP database.
Two Shall Become One
“In our experience, we find that it is just as likely that the plant floor will champion the installation of enterprise software for the immediate benefits to plant operations as it is for the general office to desire enterprise connectivity and uniformity,” claims CygNet’s Black. “The best results are achieved when both departments are championing the effort together.”
He says that over time, the front office develops a greater understanding and appreciation for the operation of the plant floor. “For instance, engineers who only visit the facility occasionally will be able to observe the operations of the plant in real time and provide feedback and guidance that would otherwise not be possible,” adds Black. “In turn, request for facility and procedure changes are more easily sold to front office management.”
After all, everyone really is on the same side here. “Enterprise systems, by nature, encourage uniformity throughout all of the plant floor sites,” reminds Black. “Best practices become apparent more quickly due to increased visibility and spread naturally to other sites. The result is better overall production and efficiency.”
Finally, there’s the need for essential upper management support. “I think the most import aspect of connecting the floor to the enterprise is the upper management’s dedication to the process,” concludes Keller. “We can collect every piece of data on the shop floor and present it many colorful ways, but if management does not bother to look at the data or require action when necessary then ultimately it has been a fun exercise in system integration, but the company will never benefit and the whole project will be looked at as a waste of money and time.”
Ask A Question
Copyright © 2016, BizWareDirect, Inc.
2450 Atlanta Highway, Suite 1202, Cumming, GA 30040 USA