diff options
Diffstat (limited to 'test/packetimpact/README.md')
| -rw-r--r-- | test/packetimpact/README.md | 531 |
1 files changed, 0 insertions, 531 deletions
diff --git a/test/packetimpact/README.md b/test/packetimpact/README.md deleted file mode 100644 index ece4dedc6..000000000 --- a/test/packetimpact/README.md +++ /dev/null @@ -1,531 +0,0 @@ -# Packetimpact - -## What is packetimpact? - -Packetimpact is a tool for platform-independent network testing. It is heavily -inspired by [packetdrill](https://github.com/google/packetdrill). It creates two -docker containers connected by a network. One is for the test bench, which -operates the test. The other is for the device-under-test (DUT), which is the -software being tested. The test bench communicates over the network with the DUT -to check correctness of the network. - -### Goals - -Packetimpact aims to provide: - -* A **multi-platform** solution that can test both Linux and gVisor. -* **Conciseness** on par with packetdrill scripts. -* **Control-flow** like for loops, conditionals, and variables. -* **Flexibilty** to specify every byte in a packet or use multiple sockets. - -## When to use packetimpact? - -There are a few ways to write networking tests for gVisor currently: - -* [Go unit tests](https://github.com/google/gvisor/tree/master/pkg/tcpip) -* [syscall tests](https://github.com/google/gvisor/tree/master/test/syscalls/linux) -* [packetdrill tests](https://github.com/google/gvisor/tree/master/test/packetdrill) -* packetimpact tests - -The right choice depends on the needs of the test. - -Feature | Go unit test | syscall test | packetdrill | packetimpact -------------- | ------------ | ------------ | ----------- | ------------ -Multiplatform | no | **YES** | **YES** | **YES** -Concise | no | somewhat | somewhat | **VERY** -Control-flow | **YES** | **YES** | no | **YES** -Flexible | **VERY** | no | somewhat | **VERY** - -### Go unit tests - -If the test depends on the internals of gVisor and doesn't need to run on Linux -or other platforms for comparison purposes, a Go unit test can be appropriate. -They can observe internals of gVisor networking. The downside is that they are -**not concise** and **not multiplatform**. If you require insight on gVisor -internals, this is the right choice. - -### Syscall tests - -Syscall tests are **multiplatform** but cannot examine the internals of gVisor -networking. They are **concise**. They can use **control-flow** structures like -conditionals, for loops, and variables. However, they are limited to only what -the POSIX interface provides so they are **not flexible**. For example, you -would have difficulty writing a syscall test that intentionally sends a bad IP -checksum. Or if you did write that test with raw sockets, it would be very -**verbose** to write a test that intentionally send wrong checksums, wrong -protocols, wrong sequence numbers, etc. - -### Packetdrill tests - -Packetdrill tests are **multiplatform** and can run against both Linux and -gVisor. They are **concise** and use a special packetdrill scripting language. -They are **more flexible** than a syscall test in that they can send packets -that a syscall test would have difficulty sending, like a packet with a -calcuated ACK number. But they are also somewhat limimted in flexibiilty in that -they can't do tests with multiple sockets. They have **no control-flow** ability -like variables or conditionals. For example, it isn't possible to send a packet -that depends on the window size of a previous packet because the packetdrill -language can't express that. Nor could you branch based on whether or not the -other side supports window scaling, for example. - -### Packetimpact tests - -Packetimpact tests are similar to Packetdrill tests except that they are written -in Go instead of the packetdrill scripting language. That gives them all the -**control-flow** abilities of Go (loops, functions, variables, etc). They are -**multiplatform** in the same way as packetdrill tests but even more -**flexible** because Go is more expressive than the scripting language of -packetdrill. However, Go is **not as concise** as the packetdrill language. Many -design decisions below are made to mitigate that. - -## How it works - -``` - +--------------+ +--------------+ - | | TEST NET | | - | | <===========> | Device | - | Test | | Under | - | Bench | | Test | - | | <===========> | (DUT) | - | | CONTROL NET | | - +--------------+ +--------------+ -``` - -Two docker containers are created by a script, one for the test bench and the -other for the device under test (DUT). The script connects the two containers -with a control network and test network. It also does some other tasks like -waiting until the DUT is ready before starting the test and disabling Linux -networking that would interfere with the test bench. - -### DUT - -The DUT container runs a program called the "posix_server". The posix_server is -written in c++ for maximum portability. It is compiled on the host. The script -that starts the containers copies it into the DUT's container and runs it. It's -job is to receive directions from the test bench on what actions to take. For -this, the posix_server does three steps in a loop: - -1. Listen for a request from the test bench. -2. Execute a command. -3. Send the response back to the test bench. - -The requests and responses are -[protobufs](https://developers.google.com/protocol-buffers) and the -communication is done with [gRPC](https://grpc.io/). The commands run are -[POSIX socket commands](https://en.wikipedia.org/wiki/Berkeley_sockets#Socket_API_functions), -with the inputs and outputs converted into protobuf requests and responses. All -communication is on the control network, so that the test network is unaffected -by extra packets. - -For example, this is the request and response pair to call -[`socket()`](http://man7.org/linux/man-pages/man2/socket.2.html): - -```protocol-buffer -message SocketRequest { - int32 domain = 1; - int32 type = 2; - int32 protocol = 3; -} - -message SocketResponse { - int32 fd = 1; - int32 errno_ = 2; -} -``` - -##### Alternatives considered - -* We could have use JSON for communication instead. It would have been a - lighter-touch than protobuf but protobuf handles all the data type and has - strict typing to prevent a class of errors. The test bench could be written - in other languages, too. -* Instead of mimicking the POSIX interfaces, arguments could have had a more - natural form, like the `bind()` getting a string IP address instead of bytes - in a `sockaddr_t`. However, conforming to the existing structures keeps more - of the complexity in Go and keeps the posix_server simpler and thus more - likely to compile everywhere. - -### Test Bench - -The test bench does most of the work in a test. It is a Go program that compiles -on the host and is copied by the script into test bench's container. It is a -regular [go unit test](https://golang.org/pkg/testing/) that imports the test -bench framework. The test bench framwork is based on three basic utilities: - -* Commanding the DUT to run POSIX commands and return responses. -* Sending raw packets to the DUT on the test network. -* Listening for raw packets from the DUT on the test network. - -#### DUT commands - -To keep the interface to the DUT consistent and easy-to-use, each POSIX command -supported by the posix_server is wrapped in functions with signatures similar to -the ones in the [Go unix package](https://godoc.org/golang.org/x/sys/unix). This -way all the details of endianess and (un)marshalling of go structs such as -[unix.Timeval](https://godoc.org/golang.org/x/sys/unix#Timeval) is handled in -one place. This also makes it straight-forward to convert tests that use `unix.` -or `syscall.` calls to `dut.` calls. - -For example, creating a connection to the DUT and commanding it to make a socket -looks like this: - -```go -dut := testbench.NewDut(t) -fd, err := dut.SocketWithErrno(unix.AF_INET, unix.SOCK_STREAM, unix.IPPROTO_IP) -if fd < 0 { - t.Fatalf(...) -} -``` - -Because the usual case is to fail the test when the DUT fails to create a -socket, there is a concise version of each of the `...WithErrno` functions that -does that: - -```go -dut := testbench.NewDut(t) -fd := dut.Socket(unix.AF_INET, unix.SOCK_STREAM, unix.IPPROTO_IP) -``` - -The DUT and other structs in the code store a `*testing.T` so that they can -provide versions of functions that call `t.Fatalf(...)`. This helps keep tests -concise. - -##### Alternatives considered - -* Instead of mimicking the `unix.` go interface, we could have invented a more - natural one, like using `float64` instead of `Timeval`. However, using the - same function signatures that `unix.` has makes it easier to convert code to - `dut.`. Also, using an existing interface ensures that we don't invent an - interface that isn't extensible. For example, if we invented a function for - `bind()` that didn't support IPv6 and later we had to add a second `bind6()` - function. - -#### Sending/Receiving Raw Packets - -The framework wraps POSIX sockets for sending and receiving raw frames. Both -send and receive are synchronous commands. -[SO_RCVTIMEO](http://man7.org/linux/man-pages/man7/socket.7.html) is used to set -a timeout on the receive commands. For ease of use, these are wrapped in an -`Injector` and a `Sniffer`. They have functions: - -```go -func (s *Sniffer) Recv(timeout time.Duration) []byte {...} -func (i *Injector) Send(b []byte) {...} -``` - -##### Alternatives considered - -* [gopacket](https://github.com/google/gopacket) pcap has raw socket support - but requires cgo. cgo is not guaranteed to be portable from the host to the - container and in practice, the container doesn't recognize binaries built on - the host if they use cgo. -* Both gVisor and gopacket have the ability to read and write pcap files - without cgo but that is insufficient here. -* The sniffer and injector can't share a socket because they need to be bound - differently. -* Sniffing could have been done asynchronously with channels, obviating the - need for `SO_RCVTIMEO`. But that would introduce asynchronous complication. - `SO_RCVTIMEO` is well supported on the test bench. - -#### `Layer` struct - -A large part of packetimpact tests is creating packets to send and comparing -received packets against expectations. To keep tests concise, it is useful to be -able to specify just the important parts of packets that need to be set. For -example, sending a packet with default values except for TCP Flags. And for -packets received, it's useful to be able to compare just the necessary parts of -received packets and ignore the rest. - -To aid in both of those, Go structs with optional fields are created for each -encapsulation type, such as IPv4, TCP, and Ethernet. This is inspired by -[scapy](https://scapy.readthedocs.io/en/latest/). For example, here is the -struct for Ethernet: - -```go -type Ether struct { - LayerBase - SrcAddr *tcpip.LinkAddress - DstAddr *tcpip.LinkAddress - Type *tcpip.NetworkProtocolNumber -} -``` - -Each struct has the same fields as those in the -[gVisor headers](https://github.com/google/gvisor/tree/master/pkg/tcpip/header) -but with a pointer for each field that may be `nil`. - -##### Alternatives considered - -* Just use []byte like gVisor headers do. The drawback is that it makes the - tests more verbose. - * For example, there would be no way to call `Send(myBytes)` concisely and - indicate if the checksum should be calculated automatically versus - overridden. The only way would be to add lines to the test to calculate - it before each Send, which is wordy. Or make multiple versions of Send: - one that checksums IP, one that doesn't, one that checksums TCP, one - that does both, etc. That would be many combinations. - * Filtering inputs would become verbose. Either: - * large conditionals that need to be repeated many places: - `h[FlagOffset] == SYN && h[LengthOffset:LengthOffset+2] == ...` or - * Many functions, one per field, like: `filterByFlag(myBytes, SYN)`, - `filterByLength(myBytes, 20)`, `filterByNextProto(myBytes, 0x8000)`, - etc. - * Using pointers allows us to combine `Layer`s with a one-line call to - `mergo.Merge(...)`. So the default `Layers` can be overridden by a - `Layers` with just the TCP conection's src/dst which can be overridden - by one with just a test specific TCP window size. Each override is - specified as just one call to `mergo.Merge`. - * It's a proven way to separate the details of a packet from the byte - format as shown by scapy's success. -* Use packetgo. It's more general than parsing packets with gVisor. However: - * packetgo doesn't have optional fields so many of the above problems - still apply. - * It would be yet another dependency. - * It's not as well known to engineers that are already writing gVisor - code. - * It might be a good candidate for replacing the parsing of packets into - `Layer`s if all that parsing turns out to be more work than parsing by - packetgo and converting *that* to `Layer`. packetgo has easier to use - getters for the layers. This could be done later in a way that doesn't - break tests. - -#### `Layer` methods - -The `Layer` structs provide a way to partially specify an encapsulation. They -also need methods for using those partially specified encapsulation, for example -to marshal them to bytes or compare them. For those, each encapsulation -implements the `Layer` interface: - -```go -// Layer is the interface that all encapsulations must implement. -// -// A Layer is an encapsulation in a packet, such as TCP, IPv4, IPv6, etc. A -// Layer contains all the fields of the encapsulation. Each field is a pointer -// and may be nil. -type Layer interface { - // toBytes converts the Layer into bytes. In places where the Layer's field - // isn't nil, the value that is pointed to is used. When the field is nil, a - // reasonable default for the Layer is used. For example, "64" for IPv4 TTL - // and a calculated checksum for TCP or IP. Some layers require information - // from the previous or next layers in order to compute a default, such as - // TCP's checksum or Ethernet's type, so each Layer has a doubly-linked list - // to the layer's neighbors. - toBytes() ([]byte, error) - - // match checks if the current Layer matches the provided Layer. If either - // Layer has a nil in a given field, that field is considered matching. - // Otherwise, the values pointed to by the fields must match. - match(Layer) bool - - // length in bytes of the current encapsulation - length() int - - // next gets a pointer to the encapsulated Layer. - next() Layer - - // prev gets a pointer to the Layer encapsulating this one. - prev() Layer - - // setNext sets the pointer to the encapsulated Layer. - setNext(Layer) - - // setPrev sets the pointer to the Layer encapsulating this one. - setPrev(Layer) -} -``` - -For each `Layer` there is also a parsing function. For example, this one is for -Ethernet: - -``` -func ParseEther(b []byte) (Layers, error) -``` - -The parsing function converts bytes received on the wire into a `Layer` -(actually `Layers`, see below) which has no `nil`s in it. By using -`match(Layer)` to compare against another `Layer` that *does* have `nil`s in it, -the received bytes can be partially compared. The `nil`s behave as -"don't-cares". - -##### Alternatives considered - -* Matching against `[]byte` instead of converting to `Layer` first. - * The downside is that it precludes the use of a `cmp.Equal` one-liner to - do comparisons. - * It creates confusion in the code to deal with both representations at - different times. For example, is the checksum calculated on `[]byte` or - `Layer` when sending? What about when checking received packets? - -#### `Layers` - -``` -type Layers []Layer - -func (ls *Layers) match(other Layers) bool {...} -func (ls *Layers) toBytes() ([]byte, error) {...} -``` - -`Layers` is an array of `Layer`. It represents a stack of encapsulations, such -as `Layers{Ether{},IPv4{},TCP{},Payload{}}`. It also has `toBytes()` and -`match(Layers)`, like `Layer`. The parse functions above actually return -`Layers` and not `Layer` because they know about the headers below and -sequentially call each parser on the remaining, encapsulated bytes. - -All this leads to the ability to write concise packet processing. For example: - -```go -etherType := 0x8000 -flags = uint8(header.TCPFlagSyn|header.TCPFlagAck) -toMatch := Layers{Ether{Type: ðerType}, IPv4{}, TCP{Flags: &flags}} -for { - recvBytes := sniffer.Recv(time.Second) - if recvBytes == nil { - println("Got no packet for 1 second") - } - gotPacket, err := ParseEther(recvBytes) - if err == nil && toMatch.match(gotPacket) { - println("Got a TCP/IPv4/Eth packet with SYNACK") - } -} -``` - -##### Alternatives considered - -* Don't use previous and next pointers. - * Each layer may need to be able to interrogate the layers aroung it, like - for computing the next protocol number or total length. So *some* - mechanism is needed for a `Layer` to see neighboring layers. - * We could pass the entire array `Layers` to the `toBytes()` function. - Passing an array to a method that includes in the array the function - receiver itself seems wrong. - -#### Connections - -Using `Layers` above, we can create connection structures to maintain state -about connections. For example, here is the `TCPIPv4` struct: - -``` -type TCPIPv4 struct { - outgoing Layers - incoming Layers - localSeqNum uint32 - remoteSeqNum uint32 - sniffer Sniffer - injector Injector - t *testing.T -} -``` - -`TCPIPv4` contains an `outgoing Layers` which holds the defaults for the -connection, such as the source and destination MACs, IPs, and ports. When -`outgoing.toBytes()` is called a valid packet for this TCPIPv4 flow is built. - -It also contains `incoming Layers` which holds filter for incoming packets that -belong to this flow. `incoming.match(Layers)` is used on received bytes to check -if they are part of the flow. - -The `sniffer` and `injector` are for receiving and sending raw packet bytes. The -`localSeqNum` and `remoteSeqNum` are updated by `Send` and `Recv` so that -outgoing packets will have, by default, the correct sequence number and ack -number. - -TCPIPv4 provides some functions: - -``` -func (conn *TCPIPv4) Send(tcp TCP) {...} -func (conn *TCPIPv4) Recv(timeout time.Duration) *TCP {...} -``` - -`Send(tcp TCP)` uses [mergo](https://github.com/imdario/mergo) to merge the -provided `TCP` (a `Layer`) into `outgoing`. This way the user can specify -concisely just which fields of `outgoing` to modify. The packet is sent using -the `injector`. - -`Recv(timeout time.Duration)` reads packets from the sniffer until either the -timeout has elapsed or a packet that matches `incoming` arrives. - -Using those, we can perform a TCP 3-way handshake without too much code: - -```go -func (conn *TCPIPv4) Handshake() { - syn := uint8(header.TCPFlagSyn) - synack := uint8(header.TCPFlagSyn) - ack := uint8(header.TCPFlagAck) - conn.Send(TCP{Flags: &syn}) // Send a packet with all defaults but set TCP-SYN. - - // Wait for the SYN-ACK response. - for { - newTCP := conn.Recv(time.Second) // This already filters by MAC, IP, and ports. - if TCP{Flags: &synack}.match(newTCP) { - break // Only if it's a SYN-ACK proceed. - } - } - - conn.Send(TCP{Flags: &ack}) // Send an ACK. The seq and ack numbers are set correctly. -} -``` - -The handshake code is part of the testbench utilities so tests can share this -common sequence, making tests even more concise. - -##### Alternatives considered - -* Instead of storing `outgoing` and `incoming`, store values. - * There would be many more things to store instead, like `localMac`, - `remoteMac`, `localIP`, `remoteIP`, `localPort`, and `remotePort`. - * Construction of a packet would be many lines to copy each of these - values into a `[]byte`. And there would be slight variations needed for - each encapsulation stack, like TCPIPv6 and ARP. - * Filtering incoming packets would be a long sequence: - * Compare the MACs, then - * Parse the next header, then - * Compare the IPs, then - * Parse the next header, then - * Compare the TCP ports. Instead it's all just one call to - `cmp.Equal(...)`, for all sequences. - * A TCPIPv6 connection could share most of the code. Only the type of the - IP addresses are different. The types of `outgoing` and `incoming` would - be remain `Layers`. - * An ARP connection could share all the Ethernet parts. The IP `Layer` - could be factored out of `outgoing`. After that, the IPv4 and IPv6 - connections could implement one interface and a single TCP struct could - have either network protocol through composition. - -## Putting it all together - -Here's what te start of a packetimpact unit test looks like. This test creates a -TCP connection with the DUT. There are added comments for explanation in this -document but a real test might not include them in order to stay even more -concise. - -```go -func TestMyTcpTest(t *testing.T) { - // Prepare a DUT for communication. - dut := testbench.NewDUT(t) - - // This does: - // dut.Socket() - // dut.Bind() - // dut.Getsockname() to learn the new port number - // dut.Listen() - listenFD, remotePort := dut.CreateListener(unix.SOCK_STREAM, unix.IPPROTO_TCP, 1) - defer dut.Close(listenFD) // Tell the DUT to close the socket at the end of the test. - - // Monitor a new TCP connection with sniffer, injector, sequence number tracking, - // and reasonable outgoing and incoming packet field default IPs, MACs, and port numbers. - conn := testbench.NewTCPIPv4(t, dut, remotePort) - - // Perform a 3-way handshake: send SYN, expect SYNACK, send ACK. - conn.Handshake() - - // Tell the DUT to accept the new connection. - acceptFD := dut.Accept(acceptFd) -} -``` - -## Other notes - -* The time between receiving a SYN-ACK and replying with an ACK in `Handshake` - is about 3ms. This is much slower than the native unix response, which is - about 0.3ms. Packetdrill gets closer to 0.3ms. For tests where timing is - crucial, packetdrill is faster and more precise. |