Introduction
To solve the load balancing of multiple network cards, specify the network card to send data, and the -I option of ping can specify the network card device
setsockopt(handle, SOL_SOCKET, SO_BINDTODEVICE, (char*)&binddevice, sizeof(binddevice));
SO_BINDTODEVICE socket option
origin
The reason for this is that I am going to use two CDMA modems to expand the bandwidth of the point-to-point connection, and I want to achieve load balancing between the two modems. But unfortunately, China Unicom's access equipment does not support Multilink-PPP. So, there is no way, I have to implement load balancing myself. There are several ways to achieve load balancing. One of the methods given on the network is to use iproute2 to complete packet-level load balancing, which is implemented at the kernel level. But I don't want to leave everything to the kernel, I want to be able to control the traffic on each modem myself. So, what should I do?
solution
At first, the solution I thought of was to create two sockets, and then bind each socket to a local IP address, I thought this would cause data to be sent from the network device where the IP address was bound. go out. But practice has proved that this idea is wrong. Because every time before sending a packet, the kernel has to look up the routing table to decide which network interface to send the packet from. Once a suitable routing table entry is found, the data packet is sent out on the network interface indicated by the routing table entry. In this way, there is a problem, because the routing table is cached, so after each packet is sent, the interface that sent the packet will have a greater chance of being selected by the kernel again. In the worst case, this will result in one modem being too busy while the other is "unattended". This obviously defeats my original intention. The test results show that when one modem sends hundreds of KB of data, the other one still only sends dozens of B. It seems that this road does not work!
Following the above line of thinking, a slightly staggering way to do this is to adjust the routing table every time a packet is sent. Adjusting the routing table is easy to do. However, doing so would be a bit too much trouble, so I abandoned this idea. I haven't even tested if this works, but in theory it should work. Moreover, in the method introduced on the Internet, the load balancing at the routing level seems to be implemented in this way, but it is just like, I have not delved into it.
After several unsuccessful attempts, I took out "UNIX Network Programming" and sifted through the UDP-related parts. Only found a place that may be useful: you can set a socket option through setsockopt(): SO_DONTROUTE, but the role of this option is vague. The vagueness is vague, I actually tried it, and the result is still not working, the reason is unknown.
So, I had to go back to Linux itself and read it to the mighty man pages. When I saw socket(7), I suddenly saw a socket option that caught my eye: SO_BINDTODEVICE. Literally, this option should be what I want. Subsequent test results proved this to be the case.
The description of the socket option in socket(7) is as follows:
SO_BINDTODEVICE
Bind this socket to a particular device like "eth0", as
specified in the passed interface name. If the name is an empty
string or the option length is zero, the socket device binding
is removed. The passed option is a variable-length null terminated
interface name string with the maximum size of IFNAMSIZ.
If a socket is bound to an interface, only packets received from
that particular interface are processed by the socket.
Note that this only works for some socket types,
particularly AF_INET sockets. It is not supported for packet
sockets (use normal bind(8) there).
Here, I copied it directly. However, the last bind(8) is definitely wrong, obviously it should be bind(2). Regardless of it, it's not something I'm trying to solve right now. The central meaning of this passage is that after a socket is bound to a specified network device interface, only packets from that device will be processed by the socket. So, what if the socket is sending packets out? Is it also only sent from this network interface? Damn, it's not said here. But it doesn't matter, we'll test it out.
At first, I took it for granted that I could do something like this:
char *dev = "ppp0";
int sock1 = socket(AF_INET, SOCK_DGRAM, 0);
setsockopt(sock1, SOL_SOCKET, SO_BINDTODEVICE, dev, sizeof(dev));说明 转者注：When the author wrote this place, the string length calculation was wrong. There is no problem with this method. It should be as follows. Of course, it is also possible to use the following struct ifreq binding.
char *dev = "ppp0";
int sock1 = socket(AF_INET, SOCK_DGRAM, 0);
setsockopt(sock1, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);However, practice once again proved me wrong. But what can I do, the description in socket(7) is written so obscurely, I can't see the clue and it's forgivable. However, with Google in hand, why should I be afraid of this little problem? So Google quickly found the crux of the problem: under Linux, references to network devices are done through struct ifreq. The description of the structure in netdevice(7) is as follows:
struct ifreq
{
char ifr_name[IFNAMSIZ]; /* Interface name */
union {
struct sockaddrifr_addr;
struct sockaddrifr_dstaddr;
struct sockaddrifr_broadaddr;
struct sockaddrifr_netmask;
struct sockaddrifr_hwaddr;
short ifr_flags;
int ifr_ifindex;
int ifr_metric;
int ifr_mtu;
struct ifmapifr_map;
char ifr_slave[IFNAMSIZ];
char ifr_newname[IFNAMSIZ];
char *ifr_data;
};
};Here, I only need ifr_name this member field is enough. The code is modified as follows:
struct ifreq if_ppp0;
struct ifreq if_ppp1;
strncpy(if_ppp0.ifr_name, "ppp0", IFNAMSIZ);
strncpy(if_ppp1.ifr_name, "ppp1", IFNAMSIZ);
sock1 = socket(AF_INET, SOCK_DGRAM, 0);
sock2 = socket(AF_INET, SOCK_DGRAM, 0);
if (setsockopt(sock1, SOL_SOCKET, SO_BINDTODEVICE,
(char *)&if_ppp0, sizeof(if_ppp0)) < 0)
{
/*error handling*/
}
if (setsockopt(sock2, SOL_SOCKET, SO_BINDTODEVICE,
(char *)&if_ppp1, sizeof(if_ppp1)) < 0)
{
/*error handling*/
}Then, in the main part of the program, every time a data packet is sent on sock1, a data packet is also sent on sock2, and there is no action to receive data in the program. Since all packets are equal in size. Therefore, it can be expected that the amount of data sent on the two network interfaces should be similar. The test results strongly support this conjecture: after running the program for a period of time, the amount of data sent on interface ppp0 is 702KB, while the amount of data sent on interface ppp1 is 895KB. Although there is still a difference of nearly 200KB, in any case, it has improved a lot compared to the original situation. As for why there is such a 200KB gap, the author is also looking for the reason.
//set socket to bind a device
static jint osNetworkSystem_bindDeviceIntfImpl(JNIEnv* env, jclass, jobject fileDescriptor, jstring ifname)
{
// LOGD("ENTER bindDeviceIntfImpl");
const char* intf;
struct ifreq binddevice;
//LOGD("*** into osNetworkSystem_bindDeviceIntfImpl obj=0x%x, 0x%x\n", &fileDescriptor, fileDescriptor);
if(fileDescriptor == NULL)
{
jniThrowNullPointerException(env, NULL);
return -1;
}
int handle = jniGetFDFromFileDescriptor(env, fileDescriptor);
int result;
//LOGD("*** osNetworkSystem_bindDeviceIntfImpl FD=%d\n", handle);
if(handle == 0 || handle == -1) {
jniThrowSocketException(env, NULL);
return -1;
}
intf = env->GetStringUTFChars(ifname, NULL);
if(intf == NULL)
{
/*maybe we should un-bind the interface...*/
/*jniThrowException(env, "java/lang/NullPointerException", NULL); */
jniThrowNullPointerException(env, NULL);
return -1;
}
strncpy(binddevice.ifr_name, intf, strlen(intf)+1);
result = setsockopt(handle, SOL_SOCKET, SO_BINDTODEVICE,
(char*)&binddevice, sizeof(binddevice));
if(result != 0)
{
LOGE("*** bindif (%s) failed! errno is %d, %s\n",
intf, errno, strerror(errno));
}else
{
LOGE("*** bindif (%s) OK!\n", intf);
}
env->ReleaseStringUTFChars(ifname, intf);
return result;
}more conclusions
Regarding the SO_BINDTODEVICE socket option, the author came to the following conclusions after fully reading the man manual:
(1) For TCP sockets, UDP sockets, and RAW sockets, the sockets can be bound to the specified network interface through the SO_BINDTODEVICE socket option. After binding, all data packets sent and received on the socket interface only pass through the specified network interface;
(2) For the PACKET type socket, it cannot be bound to the specified network interface through SO_BINDTODEVICE, but must be bound to a specific network interface through bind(2). The socket address structure used is struct sockaddr_ll. The interface address structure is the address structure of the link layer and is independent of the specific network device. For example, the address structure can be used to represent both PPP devices and ethernet devices.
(3) SO_BINDTODEVICE socket option is only applicable to Linux system. If you want to write a program running on a multi-operating system platform, you cannot rely on SO_BINDTODEVICE to complete the binding of the socket to a specific device.
However, the author did not test TCP sockets and RAW sockets. For the PACKET socket (2 layer), the above conclusion is credible, because I read the source code of dhcpd and found that the PACKET socket is indeed bound to the specified network interface through bind(2).
quote
https://beej.us/guide/bgnet/html/multi/setsockoptman.html
https://stackoverflow.com/questions/1207746/problems-with-so-bindtodevice-linux-socket-option/6857922#6857922
https://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/iputils/trusty/view/head:/ping.c