P2P Security with Paul Marrapese

Joe [00:00:00] Hey, Paul! Welcome to the Guild.

Paul [00:00:02] Hey there, Joe. Thanks for having me.

Joe [00:00:04] So I don't think I've ever spoken to a security researcher. So what's the day in the life of a security researcher like?

Paul [00:00:11] Right. I don't think there's any hard and fast rule, really. What tends to happen is say I'll be playing with a product or a piece of software or something like that, and something will catch my attention, some sort of interesting behavior. And I'll want to dig into that. And sometimes that can cascade into an entire project.

Joe [00:00:30] Nice. And I like how it began with cats. That's triggered this all for you.

Paul [00:00:35] Yeah. If not for those little guys. This whole thing may have not happened.

Joe [00:00:39] Nice. So, for folks that don't know what is peer-to-peer?

Paul [00:00:42] So in the context of IoT peer-to-peer is a convenient feature that basically allows people to connect to their device from anywhere in the world. The second that they plug it in, they can go to their phone and connect right into it. No sort of setup required. Historically, this has been a challenging thing because people would need to forward ports or configure their network somehow to allow this type of access. But devices equipped with peer-to-peer handle all of that automatically. So, again, you don't have to fiddle with your router or your firewall. You can just connect to your device magically. But being a convenience feature, this means that there is going to be a security tradeoff. I don't think that it's any secret that IoT devices have a pretty notorious history when it comes to security. So you're taking these potentially insecure devices and exposing them to the whole world.

Joe [00:01:38] Now, is this something a lot of people should be concerned with? How many devices? Is it just cameras or could it be anything? I know IoT is going to be supposed to be exploding. So do all types of IoT devices use peer-to-peer?

Paul [00:01:53] So while all IoT devices don't use this, it is still a very common feature. There are hundreds of millions of devices out there that do use this. Cameras are probably one of the most common culprits, but there are also other devices like smart doorbells, baby monitors, and even alarm systems that this is equipped in.

Joe [00:02:12] So what is some risk of using P2P? Is it an inherently bad protocol or a way to communicate?

Paul [00:02:19] Well, peer-to-peer isn't inherently a bad thing. It's a pretty ubiquitous technology that's used in all sorts of different contexts. It really kind of boils down to the implementation. So in the case of these IoT devices, you have these peer-to-peer solutions that are being deployed in these millions of devices, and those solutions are actually where the flaws lie. It's these solutions that actually have these vulnerabilities that let attackers find devices or intercept sessions and capture video, that sort of thing.

Joe [00:02:56] Okay. So if you do find an issue with the device, how do you report it? And if so, do you get in trouble if you report it or if you announce it?

Paul [00:03:07] Well, reporting it is actually a pretty tricky thing. And the reason for this is because a lot of these devices that have these problems are actually white-labeled. And what that means is there are these manufacturers in China that actually sell their equipment to resellers. And it's those resellers that put their branding on the devices. I'm sure a lot of people have seen, especially on something like Amazon, the same product, but it's listed under tons and tons of different brand names. So, yeah, these are white-labeled, and finding the actual manufacturer, the party that's responsible for the software that's on there can be a pretty tricky thing. So if I want to report something, I'll usually need to dig a little bit and try to find a point of contact. From that point. I mean, I've reached out to a lot of companies and in a lot of cases. I won't even get a response back. But for those that do respond, usually, they're pretty friendly. I haven't ever been threatened or anything like that. There's really no way to get in trouble for doing this. I mean, I'm just reporting a vulnerability in their software. There's really nothing wrong about that.

Joe [00:04:20] Well, in order to, I guess, exploit a device or know more about it I guess you need to know about the different parts that make up that particular thing that you're trying to, I guess, investigate. So what's the architecture of a P2P type of device, I guess?

Paul [00:04:37] Right. Well, what's particularly interesting about these these these networks is there are actually these servers called peer-to-peer servers that kind of orchestrate everything. There are these servers that the devices will constantly be pinging back to and they sort of manage the state of the network. You can actually use these to find all of the devices that it manages. And this is where this really kind of starts to get scary because, you know, if you have all these vulnerable devices that you have this single server that is going to tell you where they all are, that could be very valuable to an attacker. But what's even really more scary is you can also leverage these servers to not just make connections to the devices, but also potentially intercept someone else's session. So if someone is trying to connect to their device to like say view their camera or the like, you can trick these servers into introducing yourself into that conversation, at which point you can watch all the video going by or you can steal the password or really anything you like at that point.

Joe [00:05:52] Wow, how did you figure that out? Is it like a sniffer or like how did you know what to look for or what to what the traffic was in order to exploit?

Paul [00:06:02] Yeah, it was a packet sniffer. Basically, I picked up a cheap camera on Amazon to watch over my cats when I first got them and I started watching the network traffic just to kind of see what this thing was up to. And I started to notice that, like, literally every minute this thing was pinging back to some servers and had a pretty constant flow of these cryptic messages that kind of indicated that it was up to something. So I started reversing the protocol and began to figure out that, yeah, like these messages are what control the ability to connect to this thing. And they're what handle all of the video going in and out of this. So it really caught my interest. And over time, I started to develop some scripts to kind of play with the protocol and see what I could and couldn't do. And I started to realize, like, you know these things can be hijacked, yeah.

Joe [00:06:57] Nice. So did you use a tool like Wireshark? In your session, you mentioned something about the P2P dissector?.

Paul [00:07:06] The dissector, yeah, yeah. I used Wireshark to do all of this and I ended up writing a dissector that will basically analyze all the peer-to-peer traffic and tell you what the message means and what all the fields within the message mean.

Joe [00:07:19] Cool. So what was the message? Was it, I believe also you mentioned something called UDP hole punching? Is that a method to use in order to get that data also?

Paul [00:07:28] Well, UDP hole punching is actually a way to establish a connection and the protocol is used to make this happen. But UDP hole punching is actually just a technique to establish a direct connection, even if there's a router at play or a firewall or the like. It basically happens when two sides want to connect to each other. If they can be like a specific sequence of packets that they send at each other, that will allow them to establish a direct connection. And the peer-to-peer protocol is used to perform this.

Joe [00:08:03] Nice. So you are a security professional. I mean, how many people would know how to do this?

Paul [00:08:10] So, well, here's the thing. This was actually one of my first major reverse engineering projects like this. It was one of the first things that I went and fully disassembled and figured out how all the different parts of it worked. I think a lot of people, given the right amount of persistence, could find similar things because there are a lot of clues that were really helpful and getting me in the right direction. So, for example, this particular library that I looked at left a lot of the debug logs in. And in a lot of cases, as it would send out a message on the wire, there would be a corresponding log statement that actually said what that message meant. So that really let me get a jump start on figuring out what the protocol meant and ultimately figuring out everything about it. I think that's a pretty common theme with a lot of security research is it really comes down to persistence and kind of developing the ability to figure out which rabbit holes to go down. Like, if something seems a little bit funny or intriguing, you know, go check it out and see how it actually works.

Joe [00:09:13] So it could also be used for…? I'm just thinking that in the news there's Tik Tok. They're saying that some countries are using Tik Tok to spy on you. Is this something similar or is this something totally different? Or could this be used by another country to spy on another country somehow?

Paul [00:09:29] Well, I really don't think that these problems are deliberate. I don't think that these vulnerabilities were put into place on purpose. Just, you know, so one country can spy on another. I really think that these are the result of really just companies being pretty naive when it comes to security. They don't think that people are going to figure out how these cameras work and they think that that obscurity is enough. But, you know, obviously, it's not. With that being said, however, there is a risk of people spying on each other. And I don't even really mean other countries. I mean, literally, any stranger on earth can potentially spy on another person. And the reason for this is that other people's cameras on this network are actually sometimes used to help other people make connections. And what that means is that those cameras are actually tunneling other people's video feeds through. I actually did a demo at DEFCON of this where basically if you capture enough packets on your camera for a long enough period of time, there is a pretty good chance that you will actually pick up someone else's session. And what you can do is you can take that session data and you can stream it straight into a video player and potentially view someone else's feed in a lot of cases as video is not encrypted. So the person that has this device that the traffic is going through can see it without any effort whatsoever.

Joe [00:10:58] So when you said that the developer could be naive, is it that they don't know how to do security testing? Or this won't even be something that they may know what to try to do because they don't have a security expert and host. Like, what do you recommend for people to avoid these types of issues in general before it gets in the field like this?

Paul [00:11:15] Yeah, it's a pretty good mixture of both. I think a lot of them have a very basic understanding of security. And so the measures that they take are very basic. For example, I have seen some devices that do use encryption. However, they use a hardcoded key. So that kind of defeats the point, you know. If someone looks into their firmware or the client and gets that key, then any traffic that they obtain, they're going to be able to decrypt it. So that kind of demonstrates a pretty severe misunderstanding of how cryptography works. So for that reason, yeah, I really kind of urgently recommend that a lot of these companies either hire someone internally or hire like a consultancy service to look over these products because an expert in these matters is going to be able to quickly see that that sort of use of cryptography is completely ineffective and point it out and fix it before it's deployed to millions of devices all over the place, you know.

Joe [00:12:18] Absolutely. And I believe also you mention what people could try is a man in the middle attack. An I guess, because these things are headless, that maybe developers think like oh, no one's going to know. There's nothing to look at. But it's the traffic going on. And you know you could put on fiddler or something just intercept that traffic. So what's the man in the middle attack you were talking about, especially how it relates to P2P?

Paul [00:12:40] Yeah. So that kind of goes back to as I mentioned, you can just buy one of these off of Amazon and then start snooping on people's traffic. It's kind of funny in one of the marketing decks for one of these products, they actually had a slide to address this concern. And they said if you're worried about security, there's no way to spy on people because there's no interface for this. But even though the software doesn't have a feature that lets you do that, that doesn't necessarily matter, because if you have physical access to this thing, you can see every single packet that's going through this device and you can conduct a man in the middle attack very easily. You don't even need any skill to do it, really. I mean, you can just buy a device, you plug it in, and start capturing traffic. And, you know, the session data is going to be right there. And this vulnerability is a huge concern because anytime anyone connects to their device, there is a chance that they may be connecting to a malicious situation like this. It's literally a risk that someone's taking every single time that they connect to their camera. There's no way to alert you to say, hey, this is going through someone else's device. There's no way to say that they may be capturing your traffic. So it's really a pretty scary thing.

Joe [00:13:55] So when someone is in your device, the P2P device, what exactly can they do, though? Can they, they can get your video? They can, are they able to get into other things on your network once they're in your P2P device?

Paul [00:14:07] Yeah. Absolutely. And this is kind of another scary thing about peer-to-peer in general. So these devices are often just running Linux under the hood. They run BusyBox Linux. And if you're able to exploit the device in a way that gives you access to the underlying operating system like that, then your potential as an attacker is really unlimited at that point. You can start scanning around the network to see what else you can hit. Or you can install other attack tools on the camera like someone could potentially install malware or something like that right on the device. I did another demo at DEFCON to demonstrate this exact problem. At this very moment, there are millions of devices on the Internet that have a vulnerability where as long as you have the camera's I.D., you can actually fully hijack that device. You don't need a username or password. It doesn't matter how strong the camera's password is, as long as you have the I.D., you can exploit this device to get full control like this. One thing that people often tell me is they don't really care if someone can, you know, hack into their video. They say it's just watching their yard or their dog or something like that. But really, viewing video is just the tip of the iceberg when it comes to the hazards of these things because when your device has a vulnerability like that, that is an entry point into your network. And if any anonymous person on the Internet can exploit it, then they can get on your network and they can really be the catalyst to a much larger scale attack.

Joe [00:15:43] Well, I guess what's also scary is you mentioned in your session that you could just shut down the vendor because that's not going to solve it, because they white label it. And it's everywhere, basically. You may not even know what device is using it, I guess.

Paul [00:15:55] Yeah. Yeah. That's been a real challenge throughout this whole project. And in even after the fact, because I have people who hear about these vulnerabilities and naturally they want to know if they're affected by them. So they might send me a brand name. But, you know, unfortunately, the brand name doesn't really tell me much because that brand you don't know what they're using under the hood. You don't know who the original device manufacturer actually was. So determining if someone is affected can actually be pretty challenging. So I'm still really trying to figure out a reliable way that people can identify these things on their networks.

Joe [00:16:34] Absolutely. And I guess another thing that was disturbing is when you said is that patching won't necessarily fix the issue either. So it's not like they can just release a patch and it's going to be resolved.

Paul [00:16:42] Yeah, there's a lot of challenges there. A lot of these issues are actually not fixable. They're really pretty deeply rooted architectural problems where if they fix these problems now, it's going to break every unpatched device on the network. So, frankly, you know, that's not going to happen. They can certainly move forward and start releasing new devices that are patched. But I mean, a lot of people, they buy a camera and then it's going to stay up for years. I've even seen some devices with firmware versions going back like, you know, five or six years. People just kind of plug it in and forget about it. Another problem comes down to the fact that these things are resold. So if the original device manufacturer actually comes out with a patch, that's still going to need to propagate down through all of the resellers. And you as the end-user, all you're gonna know how to do is go to your reseller's site. And if they don't offer the latest firmware download, then you're gonna think that you're up to date and you may, in fact, remain vulnerable.

Joe [00:17:44] So in the session, you also gave a slide on some reverse reversing tips so can share a little bit of those reversing tips?

Paul [00:17:50] Yeah. I think my favorite tip to give people is if you're doing reversing or you're analyzing something and you happen to stumble across like a magic string or something that looks a little funny, put that into GitHub. And the reason I do this is because it is absolutely mind-boggling how many people, without realizing it will push sensitive source code or documentation or anything like that up to GitHub. And of course, at that point it becomes public. So magic strings or keys or identifiers like that are a really great way to hone in on those potentially sensitive repositories. And by doing that, I have found very valuable things, such as firmware source code and documentation just completely exposed. So as a reverse engineer, that's gonna make your life a lot easier. You know, that's gonna give you potentially function names and all sorts of stuff that you no longer have to figure out manually.

Joe [00:18:48] So, Paul, I'll be honest, I have a small farm. We just got some alpacas, goats, and chickens and everyone's like, oh, you should install a camera. And I saw your session, I'm like, there's no way I'm installing a camera now. I mean, It's crazy. It's crazy what you could do. So I'm amazed at what you were you able to find. So really awesome stuff.

Paul [00:19:06] Oh, thank you. I sincerely appreciate it.

Joe [00:19:09] So Paul are there any resources people can go to to learn more about P2P?

Paul [00:19:13] Yeah. So I actually have a small website up that people can go to for some more information. If you got a hacked camera, there's an FAQ that people can look at That answers quite a few questions. There's a link to the DEFCON video. I have an interactive map that shows these devices all over the world and where they're really heavily concentrated. And I have the Wireshark dissector up there as well. So if people have one of these devices at home and they want to take a look and see what it's doing, they can use that dissector to actually analyze the traffic going on the wire.

Joe [00:19:45] Awesome. Okay Paul, before we go, is there one piece of actionable advice you can give to someone to help them with the security testing efforts? And what's the best way to find or contact you?

Paul: [00:19:54] Yeah, I think if you're developing a product and you're not really an expert on security or you just have a basic understanding, it can really go a long way to consult with someone who is an expert because they're going to be able to provide a lot of insight into potential flaws and help you resolve those, you know, early on in the development process. And obviously, it's a lot better to do that prerelease before something's out in the wild. And for those looking to get in touch with me, my contact information is right up on the website hacked.camera.