Test Production Lines for IT/OT Convergence

IT/OT Convergence plays a pivotal role in improving manufacturing processes. David Ariens joins us to help the Exponential Industry GPT learn best practices for converging the two worlds.

Last week, Exponential Industry engaged Batist Leman, CEO of Azumuta, to discuss connected worker technology with ‘Henry Elon Rogo’ a custom ChatGPT. Batist was generous enough to give a behind-the-scenes look at his review:

This week, we welcome David Ariens, publisher of The IT/OT Insider. The IT/OT Insider discusses IT/OT Convergence with a focus on people and culture, not on the buzzwords! To support the transformation, discover which technologies (AI! Cloud! IIoT!) can enable this transition.

---

---

Entering the IT/OT Singularity

In today's competitive global market, continuous improvement in factories is not just advantageous but essential. Nowadays, most improvement projects are increasingly reliant on the convergence of Information Technology (IT) and Operational Technology (OT). IT/OT Convergence plays a pivotal role in streamlining manufacturing processes, enhancing data collection and analysis, and enabling real-time decision-making. However, as most practitioners have likely experienced, digital transformation projects once started never seem to end and with no chance to escape (the singularity).

Prompt: What is IT/OT Convergence?

IT/OT convergence refers to the integration of Information Technology (IT) systems, which manage data and communications, with Operational Technology (OT) systems, which manage industrial operations and physical processes. This convergence is a significant trend in the manufacturing and heavy industry sectors, driven by the rise of the Industrial Internet of Things (IIoT) and the increasing need for real-time data and analytics in industrial operations.

In traditional setups, IT and OT functions often operate independently. IT systems focused on data processing, storage, and business-level operations, while OT systems were concerned with direct control and monitoring of physical devices and processes in manufacturing or industrial environments. These might include machinery, industrial control systems, and plant floor operations.

The convergence of IT and OT brings several benefits:

• Improved Efficiency and Productivity: By integrating IT and OT systems, businesses can gain a more holistic view of their operations, leading to better decision-making and increased productivity.

• Enhanced Data Analysis: With IT/OT convergence, the vast amount of data generated by OT systems can be analyzed more effectively using advanced IT tools, leading to insights that can optimize operations and reduce downtime.

• Increased Agility: Converged systems can adapt more rapidly to changing market demands and technological advancements, allowing companies to remain competitive.

• Better Security: While convergence does bring new security challenges, it also offers an opportunity to build comprehensive security protocols that cover both IT and OT aspects.

• Cost Reduction: By optimizing processes and reducing the need for separate IT and OT infrastructures, companies can achieve cost savings.

However, this convergence also presents challenges, particularly in terms of integrating legacy OT systems with modern IT technologies, managing cybersecurity risks, and handling the cultural and organizational changes required to bridge the IT and OT domains.

David Ariens (DA): Traditionally, the IT and OT worlds meet little to never: they rarely have to work together, are often part of other departments, and have different (and sometimes even conflicting) objectives. While the CIO typically oversees enterprise IT, the COO, along with business unit managers or plant leaders, manages OT systems. Over time, these domains have developed disparate objectives, cultures, and technologies (architectures and protocols), all of them hindering integration.

One interesting example is how lifecycle management in OT typically goes. Applications tightly coupled to the actual physical process follow the life at the plant: they run when the plant runs and get a shiny new polish when there is a shutdown. In some industries that might be once per year, while in the heavy industries, it can easily go up to 6 years. 

One of the most important ways to increase agility in these applications is to decouple the applications dealing with the physical process (L2) as much as possible from the rest of the architecture. The IT/OT layer at L3 (MES/MOM/Others) plays a crucial role in this. 

In our “IT versus OT: who will win” article we make two bold statements: “OT loves customization, and it’s killing innovation” and “IT will become dominant and OT will remain a cash cow”. To help organizations find the right balance when integrating the two environments, you might want to review our “8 cooperation models”. Because of the enormous amount of feedback we received on this article, we are currently preparing Part 2 which will be released soon. 

Fitting IT into the Purdue Model

David mentions L2/L3 in his response, referring to levels in the Purdue Model, a reference model for data flows in computer-integrated manufacturing. Many, perhaps even a majority, of factories still adhere to the Purdue Model for the implementation of their networks. IT/OT Convergence efforts require an understanding of each of these model layers.

Prompt: How does IT fit into the Purdue model?  Review

ISA-95 and the Purdue model explained

In the Purdue Model, IT primarily fits into Level 4, known as the Business Network level. This level is focused on business operations and enterprise-level planning, encompassing functions like inventory management, supply chain optimization, and enterprise resource planning (ERP) systems. The model organizes systems based on levels of control, with each level having its specific role and function in the overall architecture of manufacturing operations:

1. Level 0 is the base, dealing with the physical process.

2. Level 1 involves sensor/actuators for real-time manipulation.

3. Level 2 includes control systems like Distributed Control Systems (DCS) and Supervisory Control and Data Acquisition (SCADA).

4. Level 3 covers manufacturing operations systems such as Manufacturing Execution Systems (MES), Process Data Historians, and Visualization tools.

5. Level 4 (Business Network), where IT is predominantly situated, is about the larger scale, longer-term aspects of the business, dealing with hours to days in terms of the time scale.

This framework demonstrates how IT systems, focusing on data processing and business-level decision-making, interface with the more hands-on, operational technologies at the lower levels of the model

DA: Not much to add to this summary! I think we all know that the “IT/OT Convergence” sweet spot is at Level 3, the GPT has summarized the article very well.

Prompt (topic links added to response): How does information get transmitted between level 3 and level 4? What protocols are used?

The transmission of information between Level 3 (manufacturing operations systems) and Level 4 (business network) in the Purdue Model typically involves a variety of communication protocols and methodologies, each designed to suit the specific requirements of industrial and business applications. Common protocols and methods include:

1. OPC UA (Open Platform Communications Unified Architecture): A widely used protocol in industrial automation, OPC UA provides a secure, reliable means to transport data from the plant floor to business systems.

2. SQL Databases and Data Warehouses: Information from Level 3 systems like MES can be stored in SQL databases or data warehouses, which are then accessed by ERP systems at Level 4.

3. RESTful APIs and Web Services: These are often used for transferring data between systems, allowing different software applications to communicate over a network.

4. MQTT (Message Queuing Telemetry Transport): A lightweight messaging protocol, MQTT is used for simple data exchange between systems.

5. File-Based Transfer: Sometimes, data is exchanged through scheduled file transfers, although this is less real-time than other methods.

6. Custom Integration Solutions: Depending on the specific requirements and existing infrastructure, custom solutions or middleware might be developed to facilitate this communication.

These protocols ensure that data from the manufacturing operations (Level 3), which could include production data, quality metrics, and process information, is accurately and efficiently transmitted to business systems (Level 4) for enterprise-level analysis, planning, and decision-making. The choice of protocol depends on factors like the scale of operations, the level of real-time data needed, existing IT/OT infrastructure, and specific industry requirements.

DA: I think we would all be happy campers when 1 to 4 would be the standard. I didn’t do market research, but I would guess custom integrations, good old file transfers, and direct database connections still make up the majority of all L3 to L4 connections. In any case, the GPT has made a correct list (which makes me wonder: on which data sets was it trained?)

David’s article makes no mention of any of these protocols and the GPT was able to bring in relevant outside context. Let’s check if the GPT can sketch a diagram of the Purdue Model since it seems to have a solid ‘understanding’.

Prompt: Can you draw me a picture of the Purdue model?

Here is a visual representation of the Purdue Model of Industrial Control Systems. This diagram illustrates the various levels, from the physical process at Level 0 to business systems at Level 4, each with its specific role and function in industrial control and operations.

DA: Amazing to see how these generative image algorithms amaze me time after time. Also, the progress made in just 1 year is mind blowing!

It seems the engineering diagram skill of the GPT still needs some work. Let’s see if the GPT can help us execute the convergence.

---

---

Converging IT and OT Upgrade Cycles

Modern software development practices found within IT emphasize continuous integration and delivery (CI/CD) which does not align well with OT upgrade cycles and objectives. IT development has increasingly adopted agile methodologies, where release cycles are often measured in days rather than years. The software approach of frequent, incremental updates, which enables rapid deployment of new features, quick responses to user feedback, and efficient resolution of issues does not fit with operational needs where simple mistakes can lead to significant amounts of waste and safety issues. If factory upgrades can be developed, simulated, and deployed faster the company can significantly outpace their competition.

Prompt: What recommendations would you provide to converge the upgrade cycles of OT systems vs IT systems?

Converging the upgrade cycles of OT (Operational Technology) and IT (Information Technology) systems is a complex but crucial task, especially in the context of the increasing integration of these systems. Here are some recommendations:

• Strategic Planning and Alignment: Align the upgrade cycles of OT and IT with the broader organizational strategy. This involves understanding the operational requirements and technological landscape of both domains.

• Cross-Functional Teams: Establish cross-functional teams that include both IT and OT personnel. These teams can ensure that upgrades consider the needs and constraints of both areas.

• Incremental Upgrades: Adopt an incremental upgrade approach. Instead of large, infrequent upgrades, smaller, more frequent updates can be easier to manage and less disruptive.

• Compatibility and Standardization: Focus on compatibility and standardization across systems. This reduces the complexities associated with integrating diverse technologies and platforms.

• Risk Assessment: Conduct thorough risk assessments before any upgrade. Understanding the potential impact on both IT and OT environments is crucial for minimizing disruptions.

• Vendor Collaboration: Work closely with vendors to ensure that the upgrades in OT and IT are compatible and can be integrated seamlessly.

• Training and Change Management: Invest in training for staff and implement effective change management strategies. Keeping the workforce updated with the latest technologies and processes is key for smooth transitions.

• Testing and Validation: Rigorously test new upgrades in a controlled environment before full deployment to detect and rectify any issues early.

• Lifecycle Management: Implement a lifecycle management approach for both OT and IT assets. This helps in planning for end-of-life scenarios and ensures that upgrades are timely and effective.

• Security Considerations: Ensure that security is a primary concern in the upgrade process. Both IT and OT systems have different security needs and vulnerabilities that must be addressed.

Remember, the key is to foster a collaborative environment where IT and OT upgrades are viewed not as separate initiatives but as part of an integrated approach to technology management.

DA: Automation upgrades remain largely a vicious cycle.

Starting from a technology point of view, most traditional automation vendors have not yet adapted new ways of working which are now common in IT (think microservices, frequent updates, etc). But also people in OT automation are not used to thinking and working with these new patterns while people in IT lack an understanding of how automation works, and why it is not uncommon for it to lag 10 years behind IT systems. 

So, one of the most important things we can do today is to decouple our applications as much as possible. When preparing upgrades, we want to make sure that we have tested it beforehand.

Problem: in 99.9% of the cases we don’t have a “test production line” at hand ;) That means that we will always start an upgrade with a lot of unknown risks. Given the impact of longer-than-planned unavailability of the production facilities, it is not illogical to postpone these upgrades as long as possible. That again makes the vicious cycle worse. 

In the more regulated industries, like food and pharmaceutical, this problem becomes even worse as we need to rigorously test and document every single change made to the systems that control production. 

Prompt: IT personnel are used to continuous integration and delivery (CI/CD) which can sometimes introduce regressions and bugs into the system. For manufacturing operations (OT) this is unacceptable and would lead to significant amounts of waste and potential safety hazards. How would you ensure incremental upgrades in a complex operational environment designed to run 24 hours, 7 days a week?

Ensuring incremental upgrades in a continuous-operation manufacturing environment requires a careful balance between innovation and stability. Here are key strategies:

• Staged Rollouts: Implement upgrades in stages, starting with non-critical systems or in a sandbox environment. This helps in identifying potential issues before they impact core operations.

• Rigorous Testing: Before deploying any upgrade, conduct extensive testing, including load and stress testing, to ensure stability and reliability.

• Real-time Monitoring: Utilize real-time monitoring tools to quickly identify and address any issues that arise post-upgrade.

• Rollback Plans: Have robust rollback plans in place to quickly revert to the previous stable version in case of significant issues.

• Maintenance Windows: Schedule upgrades during off-peak hours or planned maintenance windows to minimize impact.

• Cross-functional Collaboration: Ensure close collaboration between IT and OT teams. IT teams need to understand the criticality of continuous operations and the unique requirements of the OT environment.

• User Training and Communication: Train end-users on new features or changes. Clear communication can help in smoother transitions and quicker adoption.

• Incremental Change: Adopt a philosophy of small, incremental changes rather than large, sweeping upgrades. This reduces the risk of major disruptions.

• Vendor Support: Work with vendors who understand the operational environment and can provide support during and after the upgrade.

• Post-Upgrade Analysis: After each upgrade, conduct a thorough analysis to understand its impact and learn for future upgrades.

In essence, the key is to prioritize operational continuity and safety, integrate comprehensive testing and monitoring, and maintain clear communication between all stakeholders involved in the upgrade process.

DA: Easier said than done, I’m afraid…

• Staged Rollouts: Yes, it might be possible for large companies, however, in many cases automation systems are still big monolithic environments 

• Rigorous Testing: it would be amazing if we received a budget for those test production facilities ;)

• Real-time Monitoring: YES! True that (I practically never come across this in OT, although there is a huge potential for good monitoring, not only during updates but also in normal operations)

• Incremental Change: I would love to see automation vendors adapting this philosophy :) 

And as you stated in your prompt: there’s a very thin line between introducing unacceptable regression and benefiting from CI/CD.

---

---

Balancing Technology between Operations and the Enterprise

A few conclusions to share from this conversation:

• The GPT held its own in discussing a commonly squishy topic in IT/OT Convergence. By providing authoritative texts by David for the GPT to review, the GPT did a good job recalling and summarizing the content.

• Enabling the ability to test IT/OT changes in silico is a major reason why manufacturers are investing in metaverse technologies with NVIDIA and Siemens leading the way.

• The Exponential Industry GPT can be a great way for IT folks to learn about OT systems and vice versa. You can ask dumb questions all day long and the GPT won’t get mad (but it may get lazy).

We close out with a couple of final questions for David. Next week, we will start introducing skills into the Exponential Industry GPT.

Question: How are you leveraging generative AI and chatbots within your day-to-day work? Are you planning to use it more? If so, how?

DA: We use GenAI more or less in the same way as many others are doing it: as an extra pair of hands. As a developer aid, to rewrite some texts, to help us summarize stuff, etc. 

Question: What do you enjoy most about Exponential Industry? 

DA: I very much enjoy your summary of all the latest trends and videos in Industry 4.0, robotics etc… It eliminates the need for me to look for all that distributed information myself. Thanks!

---

Continue the Exponential Industry GPT series…