190. Idea: Industrial Networking Case Study - Lampe & Malphrus Lumber

00:00 Chris: 

Welcome to EECO Asks Why. Today we have an idea episode. I'm excited cause we're going to be talking about an industrial networking success story. And we went straight to the source on this one. So today I have with me, Mr. Ron Thompson, who is the controls and HMI development manager at Lampe & Malphrus Lumber Company in North Carolina.

And he has a really interesting story. So when working with our solution architecture group, you know, they had a lot of issues going on with their network. When I got to hearing all these little different details, I'm like we need to just bring Ron on and just have him share this story. So, Ron, welcome. And thank you for taking time with us today. I'm excited, man. I'm very excited. So maybe for the listeners out there as first of all for Lampe Malphurs. What are you guys producing? In the end what is your final product? 

00:51 Ron: 

Our final product is five quarter by six and five quarter by four decking.

00:59 Chris: 

Making a lot of decking board. I got it. So it's a, probably a pretty good market right now, right? For the last couple of years 

01:06 Ron: 

It comes and goes. Yeah. 

01:08 Chris: Yeah, no doubt. No doubt. Well, thank you. First of all, for what you do. We definitely support all of the manufacturing on EECO Asks Why. So maybe set the stage. There might be some listeners out there who may be inside of an industrial plant right now. And they're facing some industrial network issues. What were some of the things that were going on at Lampe that caught your attention? 

01:29 Ron: 

Well, when I first came on board, our network was pretty primitive. It'd just been continuously added to. You got a bunch of routers, wireless routers and dome switches, putting cameras online and stuff like that. It was just a lot of problems. We were always having to reboot stuff. 

01:49 Chris: 

So was it kind of like Frankensteined together? If you will. 

01:54 Ron: 

Pretty much. I mean, you never knew what brand router or switches you will run into. No kidding. 

02:01 Chris: 

So any standards on that stuff? So in those, well, it doesn't sound like there was any standards were there some of the stuff, I guess you get like for a home type system or did you try to go the industrial route? I mean, was it a hodgepodge?

02:13 Ron: 

It was hodgepodge. You just added to it over the years. 

02:18 Chris: 

Right, right. Yeah. Just to kind of make things work, right? 

02:21 Ron: 

Yup. That's for the most part, it was just barely working. Yeah. 

02:25 Chris: 

Yeah. Then the network center, the hub, was it all in one general location or is it really spread out throughout the mill? 

02:34 Ron: 

We have what we'll call it an IT closet that where most of the hardware was at, except for the, you know, the external switches and stuff.

02:41 Chris: 

I got you. So, I mean, when you start thinking of all those different nuances of the network. What were some of the biggest problems that were popping up? Was the line going down or anything like that due to the network? 

02:54 Ron: 

Well, we have some locations that use the wireless extensively, you know, for their data and they go offline a lot of times and we would have to figure out what the reboot, it was just, it was a headache. 

03:10 Chris: 

Any video over your wireless? 

03:13 Ron: 

The brand of camera that we used then, had to go through a PC. So you were on the network with the camera for going to the PC. And then we started adding IP cameras, you know, tied directly through the network and then that just kind of exacerbated the problem. 

03:33 Chris: 

Oh, okay. So is that when it really came to fruition? 

03:38 Ron: 

The video takes up a lot of bandwidth. 

03:43 Chris: 

I got you. So I guess it was kind of limping along until that video now, where were those video components added, where they added as part of the production? I just literally finished listening to a podcast talking about video and manufacturing. So I'm curious, how were you guys using it? 

03:59 Ron: 

The video cameras are aimed at different areas of the machinery. So the operator can see because they can't see a lot of it, you know, from where they're at. 

04:10 Chris: 

So that's operator feedback type a loop, so the operator can directly see what's going on. Okay. I got you now. So that's a real time as any of that video being saved or stored, or is it all just live?

04:25 Ron: 

We're not recording any of it. We have the capability to, but we don't. 

04:30 Chris: 

Has there ever been any talk about things about doing that recording to be able to look back? 

04:36 Ron: 

We've talked about it a little bit. There's places if you're having problems with a piece of equipment, sometimes if you start recording, you might be able to help track it down. 

04:47 Chris: 

No doubt. So, I mean, just adding that video component, I'm curious. How was that perceived from the operators? Do they like that stuff or they like, you know, get this outta here? I'm just, you know, you never know. 

04:59 Ron: 

Oh no. If they lose a view, they let me know. 

05:02 Chris: 

So they find value in the video aspects of what's going on the network. So, I mean, that really brought out a lot of issues. So then I guess at that point, was that causing the network to physically crash at times? Or were we just losing data? 

05:22 Ron: 

Sections would have just basically lock up. 

05:25 Chris: 

Oh, okay. And then you have to do a, what? Like a hard reboot?

05:29 Ron: 

We have to figure out which switch to reboot or router or whatever. 

05:33 Chris: 

How'd you go about figuring that out? 

05:35 Ron: 

Trial and error. 

05:37 Chris: 

Yeah. Right. It really is just by the fire of your pants, right?

05:41 Ron: 

Yup. Yup. Figure out which one it's going through and start from there. 

05:45 Chris: 

Was there ever any commonalities, did you ever find like maybe this couple routers or these couple switches were the primary culprit? 

05:54 Ron: 

Basically failed and those were pretty easy to track down. 

06:01 Chris: 

I got ya. So I guess at some point then you needed to move into the assessment stage to really understand what was going on with the network and to figure out, you know, how do you even move forward? So how did that look for you guys? 

06:15 Ron: 

Well, we had a company come in and do the assessment when they installed the equipment initially. I worked with the guy that was setting it up some so I can learn how to move around in it and make changes or whatever. We had already made the decision when we were putting the switches in that we were going to separate stuff out on the individual V LAN to limit the traffic, especially for the cameras, because that just loads the system down real quick. 

06:43 Chris: 

So did they help get those V lands set up? Did you see an improvement with that? 

06:50 Ron: 

Immediately.

06:51 Chris: 

Okay. And maybe just, if you don't mind, there may be some listeners out there when you said V LAN. They were like the what? I mean, how would you explain it just to someone in the mill? 

07:04 Ron: 

It's a virtual network, is what it is, where you can only manage switch. You can route your traffic, so you don't have collisions. And that way it streamlines it and takes a lot of burden off of the hardware. 

07:20 Chris: 

Right. Right. So instead of running all that data through one central location that V LAN kind of sets up a separate little area for that data to run. Kind of like a little suburb of the city, you know, we're not going into the city where we're staying at in the suburbs, moving stuff around. 

07:41 Ron: 

And with the managed switches, you know, you don't get to think which door you go through you set your port routing up or direction. I've learned this recently with the dumb switch on the managed network anything going through that dumb switch all your tags are gone. Back to square one. 

08:04 Chris: 

And when you say dumb switch, are you talking unmanaged, right. So you guys made the decision to go manage switch. Is that after your assessment? 

08:15 Ron: 

Yes.

08:20 Chris: 

Right. So before you had some unmanaged switches, it sounds like,

08:23 Ron: 

All of them. We didn't have any managed switches.

08:26 Chris: 

Wonder how they got in there. Is it just through the years? 

08:30 Ron: 

Yeah, just through the years, you know, they say well, we need this. So go out, buy a switch. From Walmart or wherever. 

08:38 Chris: 

Right. And I've heard that where people just go to Best Buy, Walmart, whoever Circuit City. You'd go to those places. Right. And just, you just need to switch and throw it in there. And that's what happens. I mean, through that assessment, were you able to get all that junk out of there now? Or do you feel like you have all managed switches or you still have some of that stuff linger? 

09:01 Ron: 

We still have some dumb switches in there, gradually getting them out. 

09:05 Chris: 

I think the good part is now at least, you know, which way you want to go, when something does go out. You know you want to go that managed route and you know how to set that switch up to, you know, maximize the efficiency and the performance and things like that.

So how long have you been doing the networking stuff? 

09:25 Ron: 

Just since I've been here. 

09:27 Chris: 

Okay. And how long was that? Five years? 

09:29 Ron: 

It will be six years in April.

09:31 Chris: 

Okay. So this is a relatively new area for you. 

09:35 Ron: 

Exactly, what I did before, it was just industrial controls, and HMIs.

09:40 Chris: 

Now do you like the networking component? 

09:45 Ron: 

I thought I wanted to learn it and be good at it, but with other stuff I got to do I don't have time. 

09:53 Chris: 

It's really hard, you know, when you spread cause you have so much other responsibility too, but I mean just hats off to you. I think just that you can even, the information you're sharing here for a industrial control type background. That's impressive. Extremely impressive. 

10:09 Ron: Well, like I said, it was a learning experience. I could make it work, but it wasn't optimal.

10:15 Chris: 

Right. Now is there an IT department there?

10:20 Ron: 

I'm trying to talk them into it. 

10:25 Chris: 

Well, Hey, here's the good part. We hear this a lot Ron. The, IT/OT those guys clashing heads, you know, the OT being a network on the plant floor and it being the network inside the business unit. And a lot of times those guys don't like each other, but if you didn't have that, IT group, it sounds like you were good to go.

So, I mean, so when it came to that dance, so that OT design, that was really all on you and your group? So you had your assessment, you know, when you started looking at where you were going to next, maybe walk us through what were the next steps of actually taking some action?

11:05 Ron: 

Well, over the years, we've continued to add more cameras so we were having to add more switches and stuff and get them set up initially. And like I said it wasn't the optimized way I was doing it because I didn't know. We're going to call for help. 

11:27 Chris: 

And that is that when you kind of engage some of the solution architecture group for support there?

11:32 Ron: 

Yes.

11:33 Chris: 

what was his approach and when K came in, and he's been on EECO Asks Why a few times, to look at your network, how did you know, how did that process work? Just curious from your experience as the end user working with a solution architect that's seasoned in networks, you know, what was that experience like for you?

11:51 Ron: 

Well, he's been a tremendous help because before I got K in. I could log on to a PLC and I might be able to stay online for five minutes or five hours. You never knew. We just dropped communication and that doesn't happen anymore. 

12:09 Chris: 

So, I mean, I'm hoping, you know, I've worked with K a lot the past and our group coming in with, you know, eyes wide open, trying to really understand what your goals are. So was that somewhat of the approach? Did you go okay. Here's what we're trying to do at Lampi and here's where we need support at, is it a lot of those conversations where they generated that way? 

12:30 Ron: 

Yes. I'd tell him what we're trying to do. The end goal, you know, he's has to figure out our network itself, you know?

12:38 Chris: 

And you have to pay him and sausage biscuits. I know K he, likes to eat, you know, so just take care of them that way. Right. 

12:48 Ron: 

He's bringing me food.

12:56 Chris: 

I'm just kidding with you because I mean, so you've made a lot of these changes. So I'm curious from the management standpoint from your boss and the plant managers and things like that, when you start looking at this, none of this stuff's free, right? So was there any pushback there on investment dollars and where we need to put money, or what, where do you get kind of given some free rein to, cause we knew we need to make these improvements 

13:21 Ron: 

Pretty much was given free rein. I mean, I would go to my boss with recommendations and right now, you know, we've got a server that's pretty hefty used for his frayed five system. Right. We also have a network management PC just by itself when I started seeing some of the dollars. I mean, after the initial setup was done, I mean, that was pretty expensive there. And then with the recommendations that K was giving us on hardware, more hardware that we needed, I thought I would get some pushback, but I didn't. My boss's comment. He's the, general manager, right? He says, it's, the network goes down, we're down. Right. So whatever we need to do. 

14:07 Chris: 

Well, that's good. It sounds like the leadership was behind it and they saw the value, you know, what you guys are trying to do, but also at the same time that they have expectations that these investments are going to keep that plant running. What were some of the enhancements? So you had this problem in network, it was going down and it was taking your lines down. You made these investments in managed switches and some routers. It sounds like setting up the V LANS. What's been the outcome? What's been the post side of these investment dollars?

14:41 Ron: 

The simple fact of setting up the V LANs made a big difference because I got all the video traffic all from the office strap. Right. And then we've got don't remember exactly what it's called. It's a Panduit edge control, monitors continuously. Shows you what's breathing and what's not. And that I don't have a full grasp of that yet. K is working on me though.

15:09 Chris: 

Is that the intraview Panduit? Yes, I got ya. I got ya. You have that there. You're still trying to, I guess, adopt that into your practice of how you evaluate your network. Okay. 

15:22 Ron: 

We also have a mesh network, which is on its own V LAN. So you have full wifi around the mill. 

15:39 Chris: 

So are you 

using a lot of wifi devices in the mill? Primarily what type of stuff? 

15:46 Ron: 

They have iPads out of the log sorter next to the server. We've actually just installed some equipment that had, they're not iPads, but they are handheld. But most of it, we have the boiler manager uses his phone a lot to monitor when he's, even when he's at home, he can call into the computer and then monitor the boilers.

16:12 Chris: 

Oh, so you guys have some, that's something that, that opens up a whole other area of remote access. So he, so is that through like a VPN connection that he's able to get into that? 

16:25 Ron: 

He comes in on team viewer and in the, what are users inside the mill is VNC. And then we've got our admin also, the way our system is set up, initially, I wasn't able to get online with most of the processes, just a certain few. I can get to all of them now except for the old legacy stuff. 

16:46 Chris: 

Have you found that to be helpful? So you're not having to necessarily drive in it for everything. You can actually do something remote? That's awesome. Now you just, you perked some ears right there. That comment. That's awesome. That's great. 

17:05 Ron: 

I know I upset some people because I don't show up, but I can fix it faster that way, right? 

17:10 Chris: 

That's right. I mean, at the end of the day, it's all about for as competitive as environment we're in. I mean, you're selling decking boards. You got to get them out. You got to get them that quickly. So I mean, that's really where it's at. So I am curious though any concerns or any areas. Of how, well, let me rephrase the question. How are you guys addressing cybersecurity with this? Because if you're doing remote access, do you have, is there anything in place to make sure that no cyber threats? Cause we see the stories all the time, you know? 

17:37 Ron: 

Right. Well, the Sonic won't handle a lot of that that company that managed them they keep our backups all the time. Servers and stuff. And K is actually steering me towards some more stuff to build, prevent. Protection. 

17:53 Chris: 

I got ya. Okay. Now wifi or just smart devices in general, you mentioned the iPads that the operators were using some different types of sounds like operator specific stuff, any equipment out there that you're using that will be qualified as smart now? Smart overloads, VFDs. Are you pulling any of that type of data back to the PLC now? Over to network? Is that been a common practice? Is that fairly new? 

18:24 Ron: 

I've been doing it for years, even before I was hired on here right now, I've added some different data links to the drivers to get more information out of them and stuff like that. 

18:36 Chris: 

What type, I'm just curious, what type of stuff, where are you pulling out to dress or do you find value in?

18:41 Ron: 

Current is probably the biggest, right. But I set up the data link so I can display it on my HMIs. Pull light output frequency and the current voltage of the drive like that. I mean, we had an issue with the drive that kept tripping on high temp. And come to find out the fan was blocked, plugged up because it was in a kind of a dirty environment.

19:12 Chris: 

Right. So at least we know the high temp alarm was working. Right. I mean something, sometimes we joke about that stuff but, it's there, you were able to take that data, diagnose it, and then go find the problem. I mean, that's, that's what it's about. 

19:28 Ron: 

Well, and that's what I tell my people. A lot of times, you know, they keep complaining about a tripping out or something. I says it's protecting itself. 

Right, right. you don't really worry about protection. You're just trying to get the motor run. I'm like, well, it's probably trying to tell you something, you know, because it's tripping, you know, Yeah, this is good stuff. Good stuff. So, I mean, if you think about the whole journey, that Lampi has been all on, it sounds like you've learned a lot through it.

19:56 Chris: 

We love to give advice, you know, out there for our listeners to say somebody sitting where you guys were several years ago with that network, that's tripping out and is falling. What advice would you offer up? How should they get started? And maybe what some lessons learned that would help them along their journey.

20:13 Ron: 

Technology has come a long way in just a few years. And in industrial networking managed switches, probably the best way to go, right. That way you can control what data goes where, right, right. It's not just broadcast. 

20:30 Chris: 

Yeah. So you really find a lot of value in the managed switches and how that has impacted your production floor. 

20:38 Ron: 

Now that I know about them. Yes. I knew about managed switches, but I'd never used them. Right. 

20:45 Chris: 

And then also the V LANs and conjunction with that. It sounded like that was a big area for you. 

20:51 Ron: 

I didn't know what V LAN was when I first started messing with this stuff. 

20:58 Chris: 

Have you centralized everything now? So it was all hodgepodge. Is it all kind of centralized in one location in the middle now for your networks and the center, if you will. 

21:07 Ron: 

Well, each machine center is going to have its own basically network. And then we plug. So it's kinda like different hubs throughout the plant. We do bring it all back to the IT closet, to the main switch. 

21:21 Chris: 

And it sounds like that clause has gotten a lot better. 

21:26 Ron: 

It was a mess when I first started. I've got everything rack mounted now. 

21:36 Chris: 

Well, you know, slow and steady wins the race. Right. You know, that's right. 

21:45 Ron: 

That's what I keep telling some of the mills I've been in, you know, they want to run everything at top speed or a lumber is a variable, a big variable. And you know, you're going to stop the start. And I said, slow it down a little bit, run all day.

21:56 Chris: 

No doubt. And am curious just out of curiosity, more than anything outside of the network, you know, so supply chain, how has that been hitting you guys in the lumber industry? 

22:06 Ron: 

The part is hitting us is equipment, the electronic equipment. It takes a lot longer to get switches and stuff like that. Mostly the electronic side of the controls and stuff. Just getting the hardware here. 

22:22 Chris: 

Very interesting. So it sounds like.... 

22:25 Ron: 

Log supplies are good. This last equipment that we put in back in November, when the numbers were crunched. We could run it slower with a better load fill and production would go up. Right. We didn't have to run it as fast. We do have the catch-up capability. If we need it, we don't have to run it that way. 

22:56 Chris: 

Right. That's very interesting. So how many shifts are you guys running? 

23:01 Ron: 

Just one. Production shift. We do have some cleanup crew and maintenance crew at night.

23:07 Chris: 

Very interesting. So, I mean, if you look back, if you could do anything different, what would you do? Only on the industrial network?

23:14 Ron: 

Probably only learn it more. I just don't have time. 

23:18 Chris: 

Right. I'm with you. 

23:19 Ron: 

That's when I reached out for help.

23:22 Chris: 

I mean, that shows wisdom. You didn't have that skill set and directly in house and that's okay. But you know you knew where to go. And it sounds like you were able to get a lot of help to get you the results that you need. 

23:34 Ron: 

Yeah. There's a lot of stuff you just can't do during production, because if you drop a link and it shuts the machine down, someone will be screaming. Yeah. Yeah. 

23:44 Chris: 

And maybe I am curious to give us some context of the time, you know, from the start to where you guys, were has this been a couple year journey? Is this a couple months journey? I how long did you really take to get there?

23:58 Ron: 

Right now as far as the level of hardware that we have right now, it's probably five years. And we're still getting new hardware. 

24:09 Chris: 

Right. So this, I guess the lesson there is, you know, be ready for the long haul or be in it for the long haul. You know, it's not an overnight fix and you take those incremental wins and you learn from them and you grow from there. Very cool. Well, Ron we call it EECO Asks Why, my friend. I always try to wrap up with the why at the end, I guess maybe give the why for our listeners out there for why having that robust, reliable network is so important.

24:39 Ron:
 
It's just like the general manager said, if the network goes down your down. You won't produce. I mean, everything. Everything nowadays is network. 

24:48 Chris: 

Yeah. No doubt. Well, you guys have been a, definitely a major success story. You know, for the people to out there that wants to learn more well in our show notes, we'll put the Lampi Malphurs website so they can check you guys out, you know, but, and we'll put some resources as well on network assessments, the way that we were able to try to really walk with Ron through this.

So, Ron, you got anything else on this? 

25:17 Ron: 

Like you said you'll be ready for the long haul. The bigger the system the longer the haul.

25:20 Chris: T

hat's right. That's right.

That's right. Well, we're just we're blessed to have you as a partner. And Ron, thank you so much again, I'm sure this has really helped a lot of people out there that are listening because you know, if you're in the battle, it sounds like, it seems like you're never going to end. So maybe you gave us some encouragement today. I hope you have a wonderful day. My friend, I appreciate it. 

Hey guys, what a wonderful conversation that was, right. I mean, we learned so much from Ron about his industrial network and that journey. It was amazing. I mean, from the beginning to the issues they were having to all the way that they did the assessments to the changes that they've made now, that is a true Testament to what can be done if you take the right steps, the right approach. And you commit to solving a problem. So wonderful story there. So we hope that brought a lot of value to you guys that are out there trying to figure out how your networks, how to take them to the next level. Now for our war stories, they're still coming in and we still need them. So send us those war stories, check out the show notes.

They'll have links there to be able to connect directly with us for those, if you're like an eco ecosystem. Please share with someone, you know, send a text message, send an email, whatever you need to do, go to eco S y.com and just, and this share that out. Whatever you need to do to get it out there, leave us a five-star rating, write a one sentence, review all those, make a difference.

So go out, have a great day. Remember, keep asking why .