1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
# Benchmark tools
These scripts are tools for collecting performance data for Docker-based tests.
## Setup
The scripts assume the following:
* You have a local machine with bazel installed.
* You have some machine(s) with docker installed. These machines will be
refered to as the "Environment".
* Environment machines have the runtime(s) under test installed, such that you
can run docker with a command like: `docker run --runtime=$RUNTIME
your/image`.
* You are able to login to machines in the environment with the local machine
via ssh and the user for ssh can run docker commands without using `sudo`.
* The docker daemon on each of your environment machines is listening on
`unix:///var/run/docker.sock` (docker's default).
For configuring the environment manually, consult the
[dockerd documentation][dockerd].
## Environment
All benchmarks require a user defined yaml file describe the environment. These
files are of the form:
```yaml
machine1: local
machine2:
hostname: 100.100.100.100
username: username
key_path: ~/private_keyfile
key_password: passphrase
machine3:
hostname: 100.100.100.101
username: username
key_path: ~/private_keyfile
key_password: passphrase
```
The yaml file defines an environment with three machines named `machine1`,
`machine2` and `machine3`. `machine1` is the local machine, `machine2` and
`machine3` are remote machines. Both `machine2` and `machine3` should be
reachable by `ssh`. For example, the command `ssh -i ~/private_keyfile
username@100.100.100.100` (using the passphrase `passphrase`) should connect to
`machine2`.
The above is an example only. Machines should be uniform, since they are treated
as such by the tests. Machines must also be accessible to each other via their
default routes. Furthermore, some benchmarks will meaningless if running on the
local machine, such as density.
For remote machines, `hostname`, `key_path`, and `username` are required and
others are optional. In addition key files must be generated
[using the instrcutions below](#generating-ssh-keys).
The above yaml file can be checked for correctness with the `validate` command
in the top level perf.py script:
`bazel run :benchmarks -- validate $PWD/examples/localhost.yaml`
## Running benchmarks
To list available benchmarks, use the `list` commmand:
```bash
bazel run :benchmarks -- list
...
Benchmark: sysbench.cpu
Metrics: events_per_second
Run sysbench CPU test. Additional arguments can be provided for sysbench.
:param max_prime: The maximum prime number to search.
```
To run benchmarks, use the `run` command. For example, to run the sysbench
benchmark above:
```bash
bazel run :benchmarks -- run --env $PWD/examples/localhost.yaml sysbench.cpu
```
You can run parameterized benchmarks, for example to run with different
runtimes:
```bash
bazel run :benchmarks -- run --env $PWD/examples/localhost.yaml --runtime=runc --runtime=runsc sysbench.cpu
```
Or with different parameters:
```bash
bazel run :benchmarks -- run --env $PWD/examples/localhost.yaml --max_prime=10 --max_prime=100 sysbench.cpu
```
## Writing benchmarks
To write new benchmarks, you should familiarize yourself with the structure of
the repository. There are three key components.
## Harness
The harness makes use of the [docker py SDK][docker-py]. It is advisable that
you familiarize yourself with that API when making changes, specifically:
* clients
* containers
* images
In general, benchmarks need only interact with the `Machine` objects provided to
the benchmark function, which are the machines defined in the environment. These
objects allow the benchmark to define the relationships between different
containers, and parse the output.
## Workloads
The harness requires workloads to run. These are all available in the
`workloads` directory.
In general, a workload consists of a Dockerfile to build it (while these are not
hermetic, in general they should be as fixed and isolated as possible), some
parses for output if required, parser tests and sample data. Provided the test
is named after the workload package and contains a function named `sample`, this
variable will be used to automatically mock workload output when the `--mock`
flag is provided to the main tool.
## Writing benchmarks
Benchmarks define the tests themselves. All benchmarks have the following
function signature:
```python
def my_func(output) -> float:
return float(output)
@benchmark(metrics = my_func, machines = 1)
def my_benchmark(machine: machine.Machine, arg: str):
return "3.4432"
```
Each benchmark takes a variable amount of position arguments as
`harness.Machine` objects and some set of keyword arguments. It is recommended
that you accept arbitrary keyword arguments and pass them through when
constructing the container under test.
To write a new benchmark, open a module in the `suites` directory and use the
above signature. You should add a descriptive doc string to describe what your
benchmark is and any test centric arguments.
## Generating SSH Keys
The scripts only support RSA Keys, and ssh library used in paramiko. Paramiko
only supports RSA keys that look like the following (PEM format):
```bash
$ cat /path/to/ssh/key
-----BEGIN RSA PRIVATE KEY-----
...private key text...
-----END RSA PRIVATE KEY-----
```
To generate ssh keys in PEM format, use the [`-t rsa -m PEM -b 4096`][RSA-keys].
option.
[dockerd]: https://docs.docker.com/engine/reference/commandline/dockerd/
[docker-py]: https://docker-py.readthedocs.io/en/stable/
[paramiko]: http://docs.paramiko.org/en/2.4/api/client.html
[RSA-keys]: https://serverfault.com/questions/939909/ssh-keygen-does-not-create-rsa-private-key
|
