《零入门kubernetes网络实战》视频专栏地址
https://www.ixigua.com/7193641905282875942
本篇文章视频地址(稍后上传)
本篇文章主要是分享一下tap虚拟网络设备。
创建tap网络设备的方式跟tun完全一样，只需要将类型改为tap即可。
这里不再占用篇幅介绍了。
| 1、本篇文章的核心点 |
- tap虚拟网络设备一般用在什么场景下
- tap虚拟网络设备使用什么文件描述符
- 简单验证/dev/net/tun文件描述符是否支持并发读写操作？即验证在同一个宿主机上实现多个kvm类型的虚拟机网络通信的关键点
| 2、tap虚拟网络设备一般用在什么场景下 |
- openvpn
- kvm
| 3、tap虚拟网络设备使用什么文件描述符 |
tap跟tun一样，都是将此文件
/dev/net/tun
作为文件描述符的。
通过此文件跟虚拟网络设备tap、tun进行数据交互。
在centos7.5系统下，其他其他并为测试。
| 4、简单验证/dev/net/tun文件描述符是否支持并发读写操作？即验证在同一个宿主机上实现多个kvm类型的虚拟机网络通信的关键点 |
在kvm里使用tap设备进行的网络构建，那么，
本小节主要是想测试一下，文件描述符/dev/net/tun是否支持并发的读写操作。
如果支持并发读写操作的话，那么，下面的网络构建是成立的，
即在同一个宿主机上可以创建多个kvm类型的虚拟机。
在上图中：
- 在一个宿主机内部，通过物理网卡eth0跟外部网络进行数据通信
- 在宿主机内部，创建一个虚拟网桥br0
- 可以创建多个tap虚拟网络设备，
- 这些tap设备挂载到虚拟网桥上br0
- 并且这些tap设备都跟宿主机上的/dev/net/tun文件进行绑定，所有tap设备通过此文件进行读写
- 创建好的kvm类型的虚拟机里可以将数据包写入到宿主机上的/dev/net/tun文件里，
- 数据一旦到达/dev/net/tunW文件里，就相当于到达了虚拟网络设备tap里了，即到达了虚拟网桥br0
这样的话，kvm类型的虚拟机就可以跟外界进行通信了。
因此，主要是测试/dev/net/tun是否支持并发读写，其他相关操作不再测试了，如将tap设备挂载到虚拟网桥上。
| 4.1、测试/dev/net/tun支持并发读写的方案 |
我们创建两个测试用例，两个测试用例里都绑定了同一个宿主机上的/dev/net/tun文件，
运行这两个测试用例，观察是否出现问题，没有问题说明支持并发读写操作：
- 测试用例1，如何实现？
- 在前面的文章中，我们使用了tun设备来实现点对点的VPN，在此基础上，我们对其代码进行改造，将tun设备改成tap设备来实现。
- 测试用例2，如何实现？
- 将前面文章<<tun设备编程ICMP协议>>里的代码直接拿过来使用即可。
| 4.2、测试环境 |
两台虚拟机
- centos7.5 10.211.55.122
- centos7.5 10.211.55.123
| 4.3、测试用例1：基于tap虚拟网络设备实现点对点的VPN |
| 4.3.1、代码结构 |
并没有严格意义上的客户端，服务器端
请求由哪个服务发起的，被称为客户端，
被请求服务，称为服务器端。
| 4.3.2、客户端代码 |
| 4.3.2.1、main.go文件 |
package main
import (
"bytes"
"fmt"
"github.com/vishvananda/netlink"
"log"
"net"
"os"
"strings"
"syscall"
"time"
"unsafe"
)
const (
tapName = "tap99"
tapDevice = "/dev/net/tun"
ifnameSize = 16
localAddr = "10.211.55.122:8285"
remoteAddr = "10.211.55.123:8285"
tapIP = "10.244.2.2"
)
type ifreqFlags struct {
IfrnName [ifnameSize]byte
IfruFlags uint16
}
func ioctl(fd int, request, argp uintptr) error {
_, _, errno := syscall.Syscall(syscall.SYS_IOCTL, uintptr(fd), request, argp)
if errno != 0 {
fmt.Errorf("ioctl failed with '%s'\n", errno)
return fmt.Errorf("ioctl failed with '%s'", errno)
}
return nil
}
func fromZeroTerm(s []byte) string {
return string(bytes.TrimRight(s, "\000"))
}
func OpenTap(name string) (*os.File, string, error) {
tap, err := os.OpenFile(tapDevice, os.O_RDWR|syscall.O_NONBLOCK, 0)
if err != nil {
fmt.Printf("OpenTap Failed! err:%v", err.Error())
return nil, "", err
}
var ifr ifreqFlags
copy(ifr.IfrnName[:len(ifr.IfrnName)-1], []byte(name+"\000"))
ifr.IfruFlags = syscall.IFF_TAP | syscall.IFF_NO_PI
err = ioctl(int(tap.Fd()), syscall.TUNSETIFF, uintptr(unsafe.Pointer(&ifr)))
if err != nil {
fmt.Printf("OpenTap Failed! err:%v\n", err.Error())
return nil, "", err
}
ifName := fromZeroTerm(ifr.IfrnName[:ifnameSize])
return tap, ifName, nil
}
func main() {
fmt.Printf("======>Now----Client----Tap---VPN---UDP<======\n")
tapFile, err := createTap()
if err != nil {
fmt.Printf("ICMP Listen Packet Failed! err:%v\n", err.Error())
return
}
defer tapFile.Close()
udpConn, err := createUDP()
if err != nil {
fmt.Printf("UDP conn Failed! err:%v\n", err.Error())
return
}
defer udpConn.Close()
go tapToUDP(udpConn, tapFile)
go udpToTap(udpConn, tapFile)
time.Sleep(time.Hour)
}
func createUDP() (*net.UDPConn, error) {
localAddr, err := net.ResolveUDPAddr("udp", localAddr)
if err != nil {
log.Fatalln("failed to get udp socket:", err)
return nil, err
}
conn, err := net.ListenUDP("udp", localAddr)
if err != nil {
log.Fatalln("failed to listen on udp socket:", err)
return nil, err
}
return conn, nil
}
func tapToUDP(udpConn *net.UDPConn, tapFile *os.File) {
packet := make([]byte, 1024*64)
size := 0
var err error
for {
if size, err = tapFile.Read(packet); err != nil {
return
}
te := MACType(packet[:size])
printMACHeader(packet[:size])
if strings.EqualFold(fmt.Sprintf("%x", te), "0800") {
b := packet[14:size]
printIPv4Header(b)
if b[9] == 1 {
icmpPacket := b[20:]
printICMPHeader(icmpPacket)
}
if b[9] == 6 {
tcpPacket := b[20:]
printTCPHeader(tcpPacket)
}
if b[9] == 17 {
udpPacket := b[20:]
printUDPHeader(udpPacket)
}
}
if strings.EqualFold(fmt.Sprintf("%x", te), "0806") {
b := packet[14:size]
printARPHeader(b)
}
rAddr, err := net.ResolveUDPAddr("udp", remoteAddr)
if err != nil {
log.Fatalln("failed to get udp socket:", err)
return
}
if size, err = udpConn.WriteTo(packet[:size], rAddr); err != nil {
fmt.Println(err.Error())
return
}
fmt.Printf("tapToUDP--->Write Msg To UDP Conn OK! size:%d\n", size)
}
}
func udpToTap(udpConn *net.UDPConn, tapFile *os.File) {
var packet = make([]byte, 1024*64)
var size int
var err error
var addr net.Addr
for {
if size, addr, err = udpConn.ReadFrom(packet); err != nil {
continue
}
size, err = tapFile.Write(packet[:size])
if err != nil {
continue
}
fmt.Printf("udpToTap--->Write Msg To /dev/net/tun OK! size:%d\tsrcIP:%v\n", size, addr)
}
}
func createTap() (*os.File, error) {
err := addTap()
if err != nil {
return nil, err
}
err = configTap()
if err != nil {
return nil, err
}
tapFile, _, err := OpenTap(tapName)
if err != nil {
return nil, err
}
return tapFile, nil
}
func addTap() error {
la := netlink.LinkAttrs{
Name: tapName,
Index: 8,
MTU: 1500,
}
tap := netlink.Tuntap{
LinkAttrs: la,
Mode: netlink.TUNTAP_MODE_TAP,
}
l, err := netlink.LinkByName(tapName)
if err == nil {
netlink.LinkSetDown(l)
netlink.LinkDel(l)
}
err = netlink.LinkAdd(&tap)
if err != nil {
return err
}
return nil
}
func configTap() error {
l, err := netlink.LinkByName(tapName)
if err != nil {
return err
}
ip, err := netlink.ParseIPNet(fmt.Sprintf("%s/%d", tapIP, 24))
if err != nil {
return err
}
addr := &netlink.Addr{IPNet: ip, Label: ""}
if err = netlink.AddrAdd(l, addr); err != nil {
return err
}
err = netlink.LinkSetUp(l)
if err != nil {
return err
}
return nil
}
具体下一小节在介绍。
目前，先把/dev/net/tun文件的并发性介绍完。
| 4.3.2.2、etherheader.go文件 |
package main
import (
"fmt"
"net"
)
func printMACHeader(packet []byte) {
fmt.Println()
fmt.Println()
fmt.Println()
fmt.Println("----->MAC Header<----")
printSrcMACEther(packet)
printDstMACEther(packet)
printTypeEther(packet)
fmt.Println("----->MAC Header<--end--")
}
func printDstMACEther(packet []byte) {
fmt.Printf("Ether Header--->DstMac:%x:%x:%x:%x:%x:%x\n", packet[0], packet[1], packet[2], packet[3], packet[4], packet[5])
}
func printSrcMACEther(packet []byte) {
fmt.Printf("Ether Header--->SrcMac:%x:%x:%x:%x:%x:%x\n", packet[6], packet[7], packet[8], packet[9], packet[10], packet[11])
}
func printTypeEther(packet []byte) {
fmt.Printf("Ether Header--->Type:%04x\n", uint16(packet[12])<<8|uint16(packet[13]))
}
func MACDestination(macFrame []byte) net.HardwareAddr {
return net.HardwareAddr(macFrame[:6])
}
func MACSource(macFrame []byte) net.HardwareAddr {
return net.HardwareAddr(macFrame[6:12])
}
func MACType(macFrame []byte) []byte {
return macFrame[12:14]
}
func MACTypeARP(macFrame []byte) []byte {
return macFrame[2:4]
}
| 4.3.2.3、arpheader.go文件 |
package main
import (
"fmt"
"net"
)
func printARPHeader(packet []byte) {
fmt.Println()
fmt.Println()
fmt.Println()
fmt.Println("----->ARP Header<----")
printARPTYPE(packet)
printARPProTYPE(packet)
printARPHardwareLen(packet)
printARPPLen(packet)
printARPOp(packet)
printARPSrcHardwareMAC(packet)
printARPSrcIP(packet)
printARPDstHardwareMAC(packet)
printARPDstIP(packet)
fmt.Println("----->ARP Header<---END---")
}
func printARPTYPE(packet []byte) {
fmt.Printf("ARP Header--->Type:%d\n", uint16(packet[0])<<8|uint16(packet[1]))
}
func printARPProTYPE(packet []byte) {
fmt.Printf("ARP Header--->ProtocolType:%04x\n", uint16(packet[2])<<8|uint16(packet[3]))
}
func printARPHardwareLen(packet []byte) {
fmt.Printf("ARP Header--->HardwareLen:%d\n", packet[4])
}
func printARPPLen(packet []byte) {
fmt.Printf("ARP Header--->ProtocolLen:%d\n", packet[5])
}
func printARPOp(packet []byte) {
fmt.Printf("ARP Header--->op:%d\n", uint16(packet[6])<<8|uint16(packet[7]))
}
func printARPSrcHardwareMAC(packet []byte) {
fmt.Printf("ARP Header--->SrcHardwarMAC:%x:%x:%x:%x:%x:%x\n", packet[8], packet[9], packet[10], packet[11], packet[12], packet[13])
}
func printARPSrcIP(packet []byte) {
fmt.Printf("ARP Header--->SrcIP:%d.%d.%d.%d\n", packet[14], packet[15], packet[16], packet[17])
}
func printARPDstHardwareMAC(packet []byte) {
fmt.Printf("ARP Header--->DstHardwareMAC:%x:%x:%x:%x:%x:%x\n", packet[18], packet[19], packet[20], packet[21], packet[22], packet[23])
}
func printARPDstIP(packet []byte) {
fmt.Printf("ARP Header--->DstIP:%d.%d.%d.%d\n", packet[24], packet[25], packet[26], packet[27])
}
func GetIPv4SrcARP(packet []byte) net.IP {
return net.IPv4(packet[14], packet[15], packet[16], packet[17])
}
func GetIPv4DstARP(packet []byte) net.IP {
return net.IPv4(packet[24], packet[25], packet[26], packet[27])
}
| 4.3.2.4、icmpheader.go文件 |
package main
import "fmt"
func printICMPHeader(packet []byte) {
fmt.Println()
fmt.Println()
fmt.Println()
fmt.Println("----->ICMP Header<----")
printICMPType(packet)
printICMPCode(packet)
printICMPCheckSum(packet)
printICMPIdentification(packet)
printICMPSeqNum(packet)
printICMPTimestamp(packet)
printICMPData(packet)
fmt.Println("----->ICMP Header<----End----")
}
func printICMPType(packet []byte) {
fmt.Printf("ICMP Header--->Type:%d\n", packet[0])
}
func printICMPCode(packet []byte) {
fmt.Printf("ICMP Header--->Code:%d\n", packet[1])
}
func printICMPCheckSum(packet []byte) {
fmt.Printf("ICMP Header--->Checksum:%04x\n", uint16(packet[2])<<8|uint16(packet[3]))
}
func printICMPIdentification(packet []byte) {
fmt.Printf("TICMP Header--->Identification(process):%d\n", uint16(packet[4])<<8|uint16(packet[5]))
}
func printICMPSeqNum(packet []byte) {
fmt.Printf("ICMP Header--->SeqNum:%d\n", uint16(packet[6])<<8|uint16(packet[7]))
}
func printICMPTimestamp(packet []byte) {
fmt.Printf("ICMP Header --->timestamp:%02x %02x %02x %02x %02x %02x %02x %02x\n", packet[8], packet[9], packet[10], packet[11], packet[12], packet[13], packet[14], packet[15])
}
func printICMPData(packet []byte) {
fmt.Printf("ICMP Header--->data:%v\n", string(packet[20:]))
}
| 4.3.2.5、tcpheader.go文件 |
package main
import (
"fmt"
)
func printTCPHeader(packet []byte) {
fmt.Println()
fmt.Println()
fmt.Println()
fmt.Printf("----->TCP Header<----len:%d\n", len(packet[:]))
printTCPsrcPort(packet)
printTCPdstPort(packet)
printTCPSequenceNumber(packet)
printTCPACKNumber(packet)
printTCPHeaderLen(packet)
printTCPFlagCWR(packet)
printTCPFlagECE(packet)
printTCPFlagUrgent(packet)
printTCPFlagACK(packet)
printTCPFlagPSH(packet)
printTCPFlagRST(packet)
printTCPFlagSYN(packet)
printTCPFlagFIN(packet)
printTCPWindowSize(packet)
printTCPCheckSum(packet)
printTCPUrgentPointer(packet)
printTCPData(packet)
fmt.Println("----->TCP Header<---END---")
}
func printTCPsrcPort(packet []byte) {
fmt.Printf("TCP Header--->SrcPort:%d\n", int64(uint16(packet[0])<<8|uint16(packet[1])))
}
func printTCPdstPort(packet []byte) {
fmt.Printf("TCP Header--->DstPort:%d\n", int64(uint16(packet[2])<<8|uint16(packet[3])))
}
func printTCPSequenceNumber(packet []byte) {
fmt.Printf("TCP Header--->SequenceNumber:%d\n", uint32(packet[4])<<24|uint32(packet[5])<<16|uint32(packet[6])<<8|uint32(packet[7]))
}
func printTCPACKNumber(packet []byte) {
fmt.Printf("TCP Header--->ACKNum:%d\n", uint32(packet[8])<<24|uint32(packet[9])<<16|uint32(packet[10])<<8|uint32(packet[11]))
}
func printTCPHeaderLen(packet []byte) {
fmt.Printf("TCP Header--->HeaderLen:%d\n", packet[12]>>4*4)
}
func printTCPFlagCWR(packet []byte) {
fmt.Printf("TCP Header--->FlagCWR:%d\n", packet[13]&0x80>>7)
}
func printTCPFlagECE(packet []byte) {
fmt.Printf("TCP Header--->FlagEcho:%d\n", packet[13]&0x40>>6)
}
func printTCPFlagUrgent(packet []byte) {
fmt.Printf("TCP Header--->FlagUrgent:%d\n", packet[13]&0x20>>5)
}
func printTCPFlagACK(packet []byte) {
fmt.Printf("TCP Header--->FlagACK:%d\n", packet[13]>>4&0b0001)
fmt.Printf("TCP Header--->FlagACK:%d\n", packet[13]>>4&0b1)
//fmt.Printf("TCP Header--->FlagACK:%d\tpacket:%v\n", packet[13]>>4, packet[13])
fmt.Printf("TCP Header--->FlagACK:%d\n", packet[13]&0x10>>4)
}
func printTCPFlagPSH(packet []byte) {
fmt.Printf("TCP Header--->FlagPSH:%d\n", packet[13]&0x08>>3)
}
func printTCPFlagRST(packet []byte) {
fmt.Printf("TCP Header--->FlagRST:%d\n", packet[13]&0x04>>2)
}
func printTCPFlagSYN(packet []byte) {
fmt.Printf("TCP Header--->FlagSYN:%d\n", packet[13]&0x02>>1)
}
func printTCPFlagFIN(packet []byte) {
fmt.Printf("TCP Header--->FlagFIN:%d\n", packet[13]&0x01)
}
func printTCPWindowSize(packet []byte) {
fmt.Printf("TCP Header--->WindowSize:%d\n", uint16(packet[14])<<8|uint16(packet[15]))
}
func printTCPCheckSum(packet []byte) {
fmt.Printf("TCP Header--->Checksum:%04x\n", uint16(packet[16])<<8|uint16(packet[17]))
}
func printTCPUrgentPointer(packet []byte) {
fmt.Printf("TCP Header--->UrgentPointer:%d\n", uint16(packet[18])<<8|uint16(packet[19]))
}
func printTCPData(packet []byte) {
headerLen := packet[12] >> 4 * 4
p := packet[headerLen:]
dataLen := len(p)
if dataLen > 1 {
fmt.Printf("TCP Header--->Data--->:%v\theaderLen:%v\tdataLen:%d\n", string(p[:dataLen-1]), headerLen, dataLen)
}
}
| 4.3.2.6、udpheader.go文件 |
package main
import "fmt"
func printUDPHeader(packet []byte) {
fmt.Println()
fmt.Println()
fmt.Println()
fmt.Println("----->UDP Header<----")
printUDPSrcPort(packet)
printUDPDstPort(packet)
printUDPAllLen(packet)
printUDPChecksum(packet)
printUDPData(packet)
fmt.Println("----->UDP Header<---End--")
}
func printUDPSrcPort(packet []byte) {
fmt.Printf("UDP Header--->SrcPort:%d\n", uint16(packet[0])<<8|uint16(packet[1]))
}
func printUDPDstPort(packet []byte) {
fmt.Printf("UDP Header--->DstPort:%d\n", uint16(packet[2])<<8|uint16(packet[3]))
}
func printUDPAllLen(packet []byte) {
fmt.Printf("UDP Header--->AllLen:%d\n", uint16(packet[4])<<8|uint16(packet[5]))
}
func printUDPChecksum(packet []byte) {
fmt.Printf("UDP Header--->Checksum:%d\n", uint16(packet[6])<<8|uint16(packet[7]))
}
func printUDPData(packet []byte) {
fmt.Printf("UDP Header--->data:%v\n", string(packet[8:]))
}
| 4.3.2.7、ipv4header.go文件 |
package main
import (
"fmt"
"net"
)
// 打印ip报文头的详情
func printIPv4Header(packet []byte) {
fmt.Println()
fmt.Println()
fmt.Println()
fmt.Println("----->IP Header<----")
printVersionIPv4(packet)
printHeaderLenIPv4(packet)
printServiceTypeIPv4(packet)
printAllLenIPv4(packet)
printIdentificationIPv4(packet)
printFlagsIPv4(packet)
printFragmentOffsetIPv4(packet)
printTTLIPv4(packet)
printProtocolIPv4(packet)
printChecksumIPv4(packet)
printSrcIPv4(packet)
printDstIPv4(packet)
fmt.Println("----->IP Header<---End---")
}
func printVersionIPv4(packet []byte) {
header := packet[0]
fmt.Printf("IPv4 Header--->Version:%d\n", header>>4)
}
func printHeaderLenIPv4(packet []byte) {
header := packet[0]
fmt.Printf("IPv4 Header--->HeaderLen:%d byte\n", header&0x0f*4)
}
func printServiceTypeIPv4(packet []byte) {
st := packet[1]
fmt.Printf("IPv4 Header--->ServiceType:%v\n", st)
}
func printAllLenIPv4(packet []byte) {
fmt.Printf("IPv4 Header--->AllLen:%d\n", uint16(packet[2])<<8|uint16(packet[3]))
}
func printIdentificationIPv4(packet []byte) {
id := uint16(packet[4])<<8 | uint16(packet[5])
fmt.Printf("IPv4 Header--->Identification:%x\t%d\n", id, id)
}
func printFlagsIPv4(packet []byte) {
// 向右移动5位，相当于去掉后5位。数值降低
fmt.Printf("IPv4 Header--->Flags:%03b\n", packet[6]>>5)
}
func printFragmentOffsetIPv4(packet []byte) {
// 向左移动3位，相当于将前3位去掉。数值可能增加
fmt.Printf("IPv4 Header--->FragmentOffset:%013b\n", uint16(packet[6])<<3|uint16(packet[7]))
}
func printTTLIPv4(packet []byte) {
fmt.Printf("IPv4 Header--->TTL:%d\n", packet[8])
}
func printProtocolIPv4(packet []byte) {
fmt.Printf("IPv4 Header--->ProtocolType:%d\n", packet[9])
}
func printChecksumIPv4(packet []byte) {
fmt.Printf("IPv4 Header--->Checksum:%d\n", uint16(packet[10])<<8|uint16(packet[11]))
}
func printSrcIPv4(packet []byte) net.IP {
fmt.Printf("IPv4 Header--->SrcIP:%d.%d.%d.%d\n", packet[12], packet[13], packet[14], packet[15])
return net.IPv4(packet[12], packet[13], packet[14], packet[15])
}
func printDstIPv4(packet []byte) net.IP {
fmt.Printf("IPv4 Header--->DstIP:%d.%d.%d.%d\n", packet[16], packet[17], packet[18], packet[19])
return net.IPv4(packet[16], packet[17], packet[18], packet[19])
}
| 4.3.3、服务端代码 |
| 4.3.3.1、server.go代码 |
package main
import (
"bytes"
"fmt"
"github.com/vishvananda/netlink"
"log"
"net"
"os"
"syscall"
"time"
"unsafe"
)
const (
tapName = "tap99"
tapDevice = "/dev/net/tun"
ifnameSize = 16
localAddr = "10.211.55.123:8285"
remoteAddr = "10.211.55.122:8285"
tapIP = "10.244.3.3"
)
type ifreqFlags struct {
IfrnName [ifnameSize]byte
IfruFlags uint16
}
func ioctl(fd int, request, argp uintptr) error {
_, _, errno := syscall.Syscall(syscall.SYS_IOCTL, uintptr(fd), request, argp)
if errno != 0 {
fmt.Errorf("ioctl failed with '%s'\n", errno)
return fmt.Errorf("ioctl failed with '%s'", errno)
}
return nil
}
func fromZeroTerm(s []byte) string {
return string(bytes.TrimRight(s, "\000"))
}
func OpenTap(name string) (*os.File, string, error) {
tap, err := os.OpenFile(tapDevice, os.O_RDWR|syscall.O_NONBLOCK, 0)
if err != nil {
fmt.Printf("OpenTap Failed! err:%v", err.Error())
return nil, "", err
}
var ifr ifreqFlags
copy(ifr.IfrnName[:len(ifr.IfrnName)-1], []byte(name+"\000"))
ifr.IfruFlags = syscall.IFF_TAP | syscall.IFF_NO_PI
err = ioctl(int(tap.Fd()), syscall.TUNSETIFF, uintptr(unsafe.Pointer(&ifr)))
if err != nil {
fmt.Printf("OpenTap Failed! err:%v\n", err.Error())
return nil, "", err
}
ifName := fromZeroTerm(ifr.IfrnName[:ifnameSize])
return tap, ifName, nil
}
func main() {
fmt.Printf("======>Now----Server----Tap---VPN---UDP<======\n")
tapFile, err := createTap()
if err != nil {
fmt.Printf("ICMP Listen Packet Failed! err:%v\n", err.Error())
return
}
defer tapFile.Close()
udpConn, err := createUDP()
if err != nil {
fmt.Printf("UDP conn Failed! err:%v\n", err.Error())
return
}
defer udpConn.Close()
go tapToUDP(udpConn, tapFile)
go udpToTap(udpConn, tapFile)
time.Sleep(time.Hour)
}
func createUDP() (*net.UDPConn, error) {
localAddr, err := net.ResolveUDPAddr("udp", localAddr)
if err != nil {
log.Fatalln("failed to get udp socket:", err)
return nil, err
}
conn, err := net.ListenUDP("udp", localAddr)
if err != nil {
log.Fatalln("failed to listen on udp socket:", err)
return nil, err
}
return conn, nil
}
func tapToUDP(udpConn *net.UDPConn, tapFile *os.File) {
packet := make([]byte, 1024*64)
size := 0
var err error
for {
if size, err = tapFile.Read(packet); err != nil {
return
}
srcMAC := MACSource(packet[:size])
dstMAC := MACDestination(packet[:size])
fmt.Printf("Server==>TapToUDP----srcMAC:%v\tdstMAC:%v\n", srcMAC.String(), dstMAC.String())
rAddr, err := net.ResolveUDPAddr("udp", remoteAddr)
if err != nil {
log.Fatalln("failed to get udp socket:", err)
return
}
if size, err = udpConn.WriteTo(packet[:size], rAddr); err != nil {
fmt.Println(err.Error())
return
}
fmt.Printf("tapToUDP--->Write Msg To UDP Conn OK! size:%d\n", size)
}
}
func udpToTap(udpConn *net.UDPConn, tapFile *os.File) {
var packet = make([]byte, 1024*64)
var size int
var err error
var addr net.Addr
for {
if size, addr, err = udpConn.ReadFrom(packet); err != nil {
continue
}
srcMAC := MACSource(packet[:size])
dstMAC := MACDestination(packet[:size])
fmt.Printf("Server==>UDPToTap----srcMAC:%v\tdstMAC:%v\n", srcMAC.String(), dstMAC.String())
size, err = tapFile.Write(packet[:size])
if err != nil {
continue
}
fmt.Printf("udpToTap--->Write Msg To /dev/net/tun OK! size:%d\tsrcIP:%v\n", size, addr)
}
}
func createTap() (*os.File, error) {
err := addTap()
if err != nil {
return nil, err
}
err = configTap()
if err != nil {
return nil, err
}
tapFile, _, err := OpenTap(tapName)
if err != nil {
return nil, err
}
return tapFile, nil
}
func addTap() error {
la := netlink.LinkAttrs{
Name: tapName,
Index: 8,
MTU: 1500,
}
tap := netlink.Tuntap{
LinkAttrs: la,
Mode: netlink.TUNTAP_MODE_TAP,
}
l, err := netlink.LinkByName(tapName)
if err == nil {
netlink.LinkSetDown(l)
netlink.LinkDel(l)
}
err = netlink.LinkAdd(&tap)
if err != nil {
return err
}
return nil
}
func configTap() error {
l, err := netlink.LinkByName(tapName)
if err != nil {
return err
}
ip, err := netlink.ParseIPNet(fmt.Sprintf("%s/%d", tapIP, 24))
if err != nil {
return err
}
addr := &netlink.Addr{IPNet: ip, Label: ""}
if err = netlink.AddrAdd(l, addr); err != nil {
return err
}
err = netlink.LinkSetUp(l)
if err != nil {
return err
}
return nil
}
func MACDestination(macFrame []byte) net.HardwareAddr {
return net.HardwareAddr(macFrame[:6])
}
func MACSource(macFrame []byte) net.HardwareAddr {
return net.HardwareAddr(macFrame[6:12])
}
| 4.3.4、go.mod参考 |
module tun-test/icmp/v2
go 1.17
require github.com/vishvananda/netlink v1.1.1-0.20210330154013-f5de75959ad5
require (
github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae // indirect
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 // indirect
)
记得修改module名称，修改自己实际的。
同样，客户端，服务器端代码同样修改一下即可。
| 4.3.5、本地编译，上传到10.211.55.122以及10.211.55.123节点上去 |
Makefile
build:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o tap-client ./client/*.go
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o tap-server ./server/main.go
scp:
scp tap-client root@10.211.55.122:/root
scp tap-server root@10.211.55.122:/root
all:
make build && make scp
在本地执行
make all
| 4.4、测试用例2：使用tun设备实现ICMP协议 |
| 4.4.1、main.go文件内容，如下 |
package main
import (
"bytes"
"encoding/binary"
"fmt"
"github.com/vishvananda/netlink"
"golang.org/x/net/icmp"
"golang.org/x/net/ipv4"
"net"
"os"
"syscall"
"time"
"unsafe"
)
const (
tunName = "tun19"
tunDevice = "/dev/net/tun"
ifnameSize = 16
eth0 = "10.211.55.122"
tunIP = "10.244.2.9"
)
type ifreqFlags struct {
IfrnName [ifnameSize]byte
IfruFlags uint16
}
func ioctl(fd int, request, argp uintptr) error {
_, _, errno := syscall.Syscall(syscall.SYS_IOCTL, uintptr(fd), request, argp)
if errno != 0 {
fmt.Errorf("ioctl failed with '%s'\n", errno)
return fmt.Errorf("ioctl failed with '%s'", errno)
}
return nil
}
func fromZeroTerm(s []byte) string {
return string(bytes.TrimRight(s, "\000"))
}
func OpenTun(name string) (*os.File, string, error) {
tun, err := os.OpenFile(tunDevice, os.O_RDWR|syscall.O_NONBLOCK, 0)
if err != nil {
fmt.Printf("OpenTun Failed! err:%v", err.Error())
return nil, "", err
}
var ifr ifreqFlags
copy(ifr.IfrnName[:len(ifr.IfrnName)-1], []byte(name+"\000"))
ifr.IfruFlags = syscall.IFF_TUN | syscall.IFF_NO_PI
err = ioctl(int(tun.Fd()), syscall.TUNSETIFF, uintptr(unsafe.Pointer(&ifr)))
if err != nil {
fmt.Printf("OpenTun Failed! err:%v\n", err.Error())
return nil, "", err
}
ifName := fromZeroTerm(ifr.IfrnName[:ifnameSize])
return tun, ifName, nil
}
func checkSum(data []byte) uint16 {
var (
sum uint32
length int = len(data)
index int
)
for length > 1 {
sum += uint32(data[index])<<8 + uint32(data[index+1])
index += 2
length -= 2
}
if length > 0 {
sum += uint32(data[index])
}
sum += sum >> 16
return uint16(^sum)
}
func main() {
fmt.Printf("======>Now----Test----Tun<======\n")
tunFile, err := createTun()
if err != nil {
fmt.Printf("ICMP Listen Packet Failed! err:%v\n", err.Error())
return
}
defer tunFile.Close()
icmpConn, _ := icmp.ListenPacket("ip4:icmp", eth0)
defer icmpConn.Close()
go tunToIcmp(icmpConn, tunFile)
go icmpToTun(icmpConn, tunFile)
time.Sleep(time.Hour)
}
func tunToIcmp(icmpconn *icmp.PacketConn, tunFile *os.File) {
var srcIP string
packet := make([]byte, 1024*64)
size := 0
var err error
for {
if size, err = tunFile.Read(packet); err != nil {
return
}
fmt.Printf("Msg Length: %d\n", binary.BigEndian.Uint16(packet[2:4]))
fmt.Printf("Msg Protocol: %d (1=ICMP, 6=TCP, 17=UDP)\tsize:%d\n", packet[9], size)
b := packet[:size]
srcIP = GetSrcIP(b)
dstIP := GetDstIP(b)
fmt.Printf("Msg srcIP: %s\tdstIP:%v\n", srcIP, dstIP)
var raddr = net.IPAddr{IP: net.ParseIP(dstIP)}
b = b[20:size]
if size, err = icmpconn.WriteTo(b, &raddr); err != nil {
fmt.Println(err.Error())
return
}
fmt.Printf("Write Msg To Icmp Conn OK! size:%d\n", size)
}
}
func icmpToTun(icmpconn *icmp.PacketConn, tunFile *os.File) {
var sb = make([]byte, 1024*64)
var addr net.Addr
var size int
var err error
for {
if size, addr, err = icmpconn.ReadFrom(sb); err != nil {
continue
}
ipHeader := createIPv4Header(net.ParseIP(addr.String()), net.ParseIP(tunIP), os.Getpid())
iphb, err := ipHeader.Marshal()
if err != nil {
continue
}
fmt.Printf("Reply MSG Length: %d\n", binary.BigEndian.Uint16(iphb[2:4]))
fmt.Printf("Reply MSG Protocol: %d (1=ICMP, 6=TCP, 17=UDP)\n", iphb[9])
dstIP := GetDstIP(iphb)
fmt.Printf("Reply src IP: %s\tdstIP:%v\n", addr, dstIP)
var rep = make([]byte, 84)
rep = append(iphb, sb[:size]...)
size, err = tunFile.Write(rep)
if err != nil {
continue
}
fmt.Printf("Write Msg To /dev/net/tun OK! size:%d\ttime:%v\n", size, time.Now())
}
}
func createIPv4Header(src, dst net.IP, id int) *ipv4.Header {
iph := &ipv4.Header{
Version: ipv4.Version,
Len: ipv4.HeaderLen,
TOS: 0x00,
TotalLen: ipv4.HeaderLen + 64,
ID: id,
Flags: ipv4.DontFragment,
FragOff: 0,
TTL: 64,
Protocol: 1,
Checksum: 0,
Src: src,
Dst: dst,
}
h, _ := iph.Marshal()
iph.Checksum = int(checkSum(h))
return iph
}
func IsIPv4(packet []byte) bool {
flag := packet[0] >> 4
return flag == 4
}
func GetIPv4Src(packet []byte) net.IP {
return net.IPv4(packet[12], packet[13], packet[14], packet[15])
}
func GetIPv4Dst(packet []byte) net.IP {
return net.IPv4(packet[16], packet[17], packet[18], packet[19])
}
func GetSrcIP(packet []byte) string {
key := ""
if IsIPv4(packet) && len(packet) >= 20 {
key = GetIPv4Src(packet).To4().String()
}
return key
}
func GetDstIP(packet []byte) string {
key := ""
if IsIPv4(packet) && len(packet) >= 20 {
key = GetIPv4Dst(packet).To4().String()
}
return key
}
func createTun() (*os.File, error) {
err := addTun()
if err != nil {
return nil, err
}
err = configTun()
if err != nil {
return nil, err
}
tunFile, _, err := OpenTun(tunName)
if err != nil {
return nil, err
}
return tunFile, nil
}
func addTun() error {
la := netlink.LinkAttrs{
Name: tunName,
Index: 8,
MTU: 1500,
}
tun := netlink.Tuntap{
LinkAttrs: la,
Mode: netlink.TUNTAP_MODE_TUN,
}
l, err := netlink.LinkByName(tunName)
if err == nil {
netlink.LinkSetDown(l)
netlink.LinkDel(l)
}
err = netlink.LinkAdd(&tun)
if err != nil {
return err
}
return nil
}
func configTun() error {
l, err := netlink.LinkByName(tunName)
if err != nil {
return err
}
ip, err := netlink.ParseIPNet(fmt.Sprintf("%s/%d", tunIP, 24))
if err != nil {
return err
}
addr := &netlink.Addr{IPNet: ip, Label: ""}
if err = netlink.AddrAdd(l, addr); err != nil {
return err
}
err = netlink.LinkSetUp(l)
if err != nil {
return err
}
return nil
}
| 4.4.2、编译、上传到10.211.55.122上 |
Makefile
build:
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o tun-driver main.go
scp:
scp tun-driver root@10.211.55.122:/root
all:
make build && make scp
本地执行
make all
即可
| 4.5、注意事项 |
修改代码时，要注意下
- 第一个测试用例中，创建是tap类型的虚拟网络设备tap99
- 第二个测试用例中，创建的tun类型的虚拟网络设备tun19
我们测试的关键点是，不同的虚拟网络设备绑定在同一个文件描述符/dev/net/tun上
只要这两个设备能够同时读写操作，即可。
实话实话，主要是不想再修改第二个测试用例的代码，将其改成tap类型的。
在tun、tap上已经用了很多时间了，太累了。
| 4.6、启动服务 |
| 4.6.1、登录到10.211.55.123节点上 |
| 4.6.1.1、启动第一个测试用例的服务器端 |
./tap-server
| 4.6.1.2、设置路由表，针对的是数据包去往10.211.55.122 |
route -n
route add -net 10.244.2.0/24 dev tap99
route -n
| 4.6.2、登录到10.211.55.122节点上 |
| 4.6.2.1、启动tap-client服务 |
| 4.6.2.1.1、启动tap-client服务 |
./tap-client
| 4.6.2.1.2、设置通往10.211.55.123节点的路由 |
route -n
route add -net 10.244.3.0/24 dev tap99
route -n
| 4.6.2.2、启动第2个测试用例 |
./tun-driver
设置路由反向检测属性
sysctl net.ipv4.conf.all.rp_filter
sysctl -w net.ipv4.conf.all.rp_filter=2
sysctl net.ipv4.conf.all.rp_filter
| 4.7、通过ping命令来测试/dev/net/tun是否支持并发读写 |
登录到
10.211.55.122 节点上去
| 4.7.1、在10.211.55.122 节点上，发起ping命令去ping10.211.55.123节点上的tap99设备 |
ping 10.244.3.3 -I tap99
抓包命令如下：
tcpdump -nn -i tap99
| 4.7.2、在10.211.55.122 节点上，发起ping命令去ping10.211.55.123节点上的对外物理网卡eth0 |
ping 10.211.55.123 -I tun19
抓包命令如下：
tcpdump -nn -i tun19
| 5、总结 |
在10.211.55.122节点上，
通过ping命令同时使用tun19虚拟设备以及tap99虚拟设备发起ping请求
整个请求中，数据包有去有回，无丢包现象，
而tap19,tun99同时绑定了/dev/net/tun文件描述符，
说明/dev/net/tun文件描述符支持并发读写操作。
这样的话，本次测试的目的之一就到达了。
<<零入门kubernetes网络实战>>技术专栏之文章目录
