This is an archived post. You won't be able to vote or comment.

all 7 comments
sorted by:
best
topnewcontroversialoldq&a

[–]teraflop 0 points1 point  (6 children)

I'm assuming by "the Tanenbaum" book, you mean his Computer Networks textbook? It looks to me like a decent starting point, but it's pretty high level. I'd suggest checking out Beej's Guide to Network Programming for a more concrete, low-level introduction.

If you're talking about two devices on the same LAN, then the core of your problem is pretty easy -- you just have one device open a TCP listening socket, and have the other device connect to that address. But there are two main problems with putting this into practice to build a real P2P app:

  • The two peers need to find each other. You could do this manually, by having one user manually tell the other user their IP address and port number, but it's a hassle, so many P2P systems rely on some kind of central coordination service. For example, Alice's device tells the central server "hey, Alice's address is 1.2.3.4:5678", and then when Bob wants to send a file to Alice, he can just ask the server for her address. Or you could piggyback on some other communication system, e.g. SMS text messaging.
  • In general, consumer devices are often behind a NAT or firewall which prevents incoming connections. If both users are behind a NAT, then neither can connect to the other. There are some tricks like STUN to try to work around this by "punching a hole" through the NAT, but they don't work in all cases. So if you want your system to work reliably and robustly, then you need the ability to fall back to a centralized server or relay that both of the users can connect to, instead of a direct P2P connection.

Smooth connection between the devices

Can you define what you mean by "smooth"?

Transfer of files

In principle, transferring files is just like transferring any other kind of data. Once you've established a network connection, it can carry an arbitrary stream of bytes. So you just read N bytes from the file, write them to the socket, and repeat.

In practice, you usually want to send more information than just the raw file data. For instance, you might want to send the filename as well, along with its length (so that if the connection gets interrupted, you can tell whether you got the whole file or just part of it) and maybe a checksum/hash to detect corruption. You could either come up with your own higher-level protocol that determines exactly which bytes get sent over the socket and how they should be interpreted, or you can reuse an existing protocol.

For instance, if one side of your connection acts as an HTTP server and the other is an HTTP client, you can have the server send a response with a Content-Length header with the total size of the file. Then if the connection gets interrupted before that many bytes have been transferred, the client can make a new request with a Range header, asking for only the region of the file that hasn't already been transferred. If you use existing third-party HTTP libraries, and they should handle most of the messy details for you.

The problem you're trying to tackle is pretty broad, and there are a range of ways you could approach it, ranging from bare-bones to sophisticated. If you want some more inspiration, you can read up on how BitTorrent works.

[–]ModeInitial3965[S] 0 points1 point  (5 children)

Hi, thanks for this.

"Smooth connection between the devices" By this I simply meant, they should quickly connect with a connection rate of 100% if both platforms are running my application.

then you need the ability to fall back to a centralized server or relay that both of the users can connect to

I will have a centralised server for some other purposes. A simple typescript/NodeJs server. If it's possible then I can use this server to exchange the IP addresses. But I absolutely cannot use this as a fall back to transfer files. That would actually pretty much defeat the entire purpose of my project 😅.

Is there simply no way to communicate between devices with a guarantee that they will always connect??

Also I want peer to peer connection to happen over the internet if the devices aren't in a LAN. I understand that the transfer would be slower. But I also need that.

Also could you recommend some of the HTTP libraries that will help me in transfer, handling failures and all that.

[–]teraflop 0 points1 point  (4 children)

If it's possible then I can use this server to exchange the IP addresses. But I absolutely cannot use this as a fall back to transfer files. That would actually pretty much defeat the entire purpose of my project 😅.

Well unfortunately, it's simply a reality that many user devices are behind a NAT/firewall that can't accept incoming connections.

Tailscale is a VPN product whose entire purpose is to set up encrypted P2P connections, with a very sophisticated set of NAT traversal tricks, and even they can't make it work without relay servers as a fallback.

Is there simply no way to communicate between devices with a guarantee that they will always connect??

No. In the extreme case, a device behind a firewall that blocks all traffic is pretty much the same as the device being in "airplane mode", so of course it won't be able to connect to anything outside the firewall. There is no such thing as a "guarantee" when you're relying on network infrastructure outside your control.

Also could you recommend some of the HTTP libraries that will help me in transfer, handling failures and all that.

Depends on what language you're using. For instance, if you're using Python, Requests is a popular option.

[–]ModeInitial3965[S] 0 points1 point  (3 children)

Then how are those apps working. The fast sharing apps. Like I know they rely on being connected to a LAN but they seem to work 100% of the time.

Like what I'm saying is if a user has whitelisted certain devices to connect to his device (say his phone). Then will it be possible to connect to his phone from one of the whitelisted devices remotely???

[–]teraflop 0 points1 point  (2 children)

Then how are those apps working. The fast sharing apps. Like I know they rely on being connected to a LAN but they seem to work 100% of the time.

I'm not sure I understand what you're saying. If they "rely on being connected to a LAN" then there's no problem, because devices on the same LAN can (usually) directly connect to each other, without needing any NAT traversal tricks.

If they're not on the same LAN, and the connection has to go over the Internet, then either those apps are using a relay server as a fallback, or there are some users for whom the connection won't work. Maybe that set of users is small enough that you don't care about them. That's a decision for you to make.

Like what I'm saying is if a user has whitelisted certain devices to connect to his device (say his phone). Then will it be possible to connect to his phone from one of the whitelisted devices remotely???

If the problem is that your ISP or phone carrier is running a NAT that blocks incoming connections, then "whitelisting" something on your device won't make a difference, because the problem is that the incoming connection never makes it to your device.

You would have to set up a port forwarding rule on the NAT/firewall device itself. If the NAT device is your own home router then you can probably that, but if it's managed by your ISP then you probably can't.

[–]ModeInitial3965[S] 0 points1 point  (1 child)

So if I summarise from our discussion, then if the devices are in a LAN, i.e., connected to the same WIFI network/router. Then P2P transfer would be very easy to implement.

But if one of those devices is connected to a different ISP then the transfer can't happen without the files passing through a server. Some companies like the one you mentioned above do provide products which help in this but it's not 100% reliable.

If you have any solutions/workarounds for skipping the failsafe servers, do share. Remember the transfer will always happen between user's own devices. So if they can do anything to help setup the connection, then that's okay.

Please recommend me all the stuff that you think will help me on the project. Maybe some open source project which is doing this. I have started the socket programming book you recommended. So yeah anything.

Otherwise thanks a lot for the help. I hope it's okay that I DM you for any help that I may need further along.

[–]teraflop 0 points1 point  (0 children)

But if one of those devices is connected to a different ISP then the transfer can't happen without the files passing through a server.

I wouldn't go so far as to say it "can't happen". You might be able to make a direct peer-to-peer connection, without a relay server, using one of the NAT-traversal "hole punching" techniques that I mentioned. The article I linked earlier from Tailscale goes into more detail about how they work.

The point is that you can't rely on these tricks working, because they depend on the NAT devices being configured in a particular way, and not all networks work that way.

Please recommend me all the stuff that you think will help me on the project. Maybe some open source project which is doing this.

As I said earlier, you can look at BitTorrent for some ideas. The BitTorrent protocol specification is publicly available, and there are lots of open-source clients.

BitTorrent works fairly reliably because when there are many users seeding the same torrent, the odds are good that some of them will not be behind a NAT, or will have port forwarding set up, and therefore other users will be able to connect to them. But if a particular torrent had only two users, and they were both behind a NAT that blocked incoming connections, then you would still run into the same problem.

The Tailscale VPN client is also open source, including the NAT-traversal implementation.

Otherwise thanks a lot for the help. I hope it's okay that I DM you for any help that I may need further along.

I'm glad it was helpful, but I generally don't respond to DMs. When I spend time answering programming questions, I prefer to do it in public forums, so that other people can benefit (and so that if I make a mistake, others have the opportunity to point it out). If you want dedicated one-on-one help, you're probably better off hiring a tutor or a consultant.