This is the third evaluation in our series of basket2 evaluation updates (previous at 1 and 2).
We use four datasets:
- 100x100+10,000 gathered week 45 with a local bridge (download features).
- 100x100+10,000 gathered week 46 with a local bridge (download features).
- 100x100+10,000 gathered week 47 with a remote bridge in Germany (download features).
- 100x100+10,000 gathered week 48 with a remote bridge in Candada (download features).
The local bridge was running on the same 1-Gbit LAN as the clients, and both the bridge and the gathering clients (see gathering tools) were running in Docker containers on CoreOS, where CoreOS was virtualised by VMware. Hurray for abstractions.
Hetzner bridge (Germany)
The remote bridge in Germany was hosted at Hetzner on a vServer CX30. From one of the CoreOS machines used by our clients:
--- X.X.X.X ping statistics ---
1001 packets transmitted, 1000 received, 0% packet loss, time 1001110ms
rtt min/avg/max/mdev = 27.964/28.174/33.653/0.385 msA fast and stable connection. The way there:
$ tracepath X.X.X.X
1?: [LOCALHOST] pmtu 1500
1: 192.168.60.1 0.218ms
1: 192.168.60.1 0.266ms
2: 130.243.27.193 0.568ms
3: kau1-kaugw1.kau.se 0.639ms
4: kau-r1.sunet.se 0.714ms
5: orebro-lba-r1.sunet.se 1.980ms
6: vasteras-fsn2-r1.sunet.se 3.126ms
7: stockholm-fre-r1.sunet.se 5.086ms
8: se-fre.nordu.net 5.080ms
9: xe-2-0-7-0.sth10.core-backbone.com 5.470ms
10: ae10-2021.fra20.core-backbone.com 23.700ms asymm 15
11: core-backbone-100g-fra.hetzner.de 23.569ms asymm 16
12: core4.hetzner.de 23.529ms asymm 17
13: core24.hetzner.de 28.660ms asymm 19
14: ex9k2.rz17.hetzner.de 28.899ms asymm 20
15: static.85-10-250-246.clients.your-server.de 40.470ms asymm 20
16: no reply
17: cnode11.21.flk1.your-cloud.host 28.085ms asymm 22
18: static.162.86.47.78.clients.your-server.de 28.754ms asymm 24
19: static.162.86.47.78.clients.your-server.de 28.875ms reached
Resume: pmtu 1500 hops 19 back 40The traffic first goes through Sweden (from Karlstad to Stockholm). Next, the jump from Stockholm to Frankfurt (9->10) through Core-Backbone is where we spend most of our time. Finally, the traffic hits Hetzner’s infrastructure.
OVH bridge (Canada)
The remote bridge in Canada was hosted at OVH on VPS SSD 3. From one of the CoreOS machines used by our clients:
--- X.X.X.X ping statistics ---
1138 packets transmitted, 1138 received, 0% packet loss, time 1138677ms
rtt min/avg/max/mdev = 127.510/127.733/128.296/0.244 ms
Significantly higher latency then the German bridge, but more consistent latency.
$ tracepath X.X.X.X
1?: [LOCALHOST] pmtu 1500
1: 192.168.60.1 0.280ms
1: 192.168.60.1 0.264ms
2: 130.243.27.193 49.808ms
3: kau1-kaugw1.kau.se 0.799ms
4: kau-r1.sunet.se 0.867ms
5: orebro-lba-r1.sunet.se 2.093ms
6: vasteras-fsn2-r1.sunet.se 3.215ms
7: stockholm-fre-r1.sunet.se 5.073ms
8: se-fre.nordu.net 5.096ms
9: dk-ore.nordu.net 11.595ms asymm 10
10: dk-ore.nordu.net 10.890ms asymm 11
11: nl-sar.nordu.net 21.380ms asymm 12
12: no reply
13: be10-1184.rbx-g1-a9.fr.eu 27.284ms asymm 14
14: be100-1187.ldn-1-a9.uk.eu 30.935ms asymm 15
15: be100-1295.nwk-1-a9.nj.us 128.099ms asymm 18
16: be10-1037.bhs-g1-a9.qc.ca 128.525ms asymm 17
17: vl20.bhs-g1-a75.qc.ca 127.818ms asymm 18
18: po5.bhs-z2g1-a75.qc.ca 127.848ms
19: po5.bhs-z2b1-a70.qc.ca 127.873ms asymm 20
20: 158.69.44.32 127.872ms asymm 21
21: 158.69.61.20 128.086ms asymm 22
22: X.ip-X-X-X.net 127.851ms reached
Resume: pmtu 1500 hops 22 back 43
Once again, the traffic makes its way through Sweden from Karlstad to Stockholm, then to Denmark, the Netherlands, France, UK, US and then Canada. Most of the time is spent in OVH’s infrastructure, already in step 13.
Recall and precision
Below you find the precision and recall for the six different basket2 methods for the four datasets described above.
Week 45 (local bridge)
Week 46 (local bridge)
Week 47 (remote bridge, Hetzner, Germany)
Week 47 (remote bridge, OVH, Canada)
Top ten useful features
We look closer at weights of different features using the weights tool. Note that the feature number matches that of Wang’s PhD thesis and that the relative weight (rw) only shows the average relative weight for one run of go-kNN: you cannot compare the weights between datasets or runs. The weights are normalized such that the least useful feature has weight 1.0.
Week 45 (local bridge)
Null: top 10 features
1. std of interpacket time, feat 1225, rw 444.24
2. total transmission time, feat 4, rw 389.44
3. mean interpacket time, feat 1224, rw 201.49
4. number of outgoing packets, feat 2, rw 170.84
5. position of outgoing packet 3, feat 7, rw 125.81
6. number of bursts, feat 1107, rw 115.60
7. position of outgoing packet 6, feat 10, rw 95.14
8. position of outgoing packet 7, feat 11, rw 86.69
9. number of bursts longer than 2, feat 1108, rw 76.93
10. position of outgoing packet 8, feat 12, rw 76.27
Obfs4Burst: top 10 features
1. total transmission time, feat 4, rw 404.52
2. std of interpacket time, feat 1225, rw 304.40
3. mean interpacket time, feat 1224, rw 248.30
4. number of outgoing packets, feat 2, rw 241.31
5. number of bursts, feat 1107, rw 132.46
6. position of outgoing packet 3, feat 7, rw 100.54
7. position of outgoing packet 4, feat 8, rw 100.35
8. position of outgoing packet 12, feat 16, rw 90.53
9. number of bursts longer than 2, feat 1108, rw 90.41
10. position of outgoing packet 7, feat 11, rw 89.71
Obfs4BurstIAT: top 10 features
1. std of interpacket time, feat 1225, rw 597.22
2. total transmission time, feat 4, rw 366.82
3. mean interpacket time, feat 1224, rw 321.77
4. number of outgoing packets, feat 2, rw 249.34
5. position of outgoing packet 3, feat 7, rw 177.02
6. number of bursts, feat 1107, rw 126.17
7. position of outgoing packet 4, feat 8, rw 94.57
8. number of bursts longer than 2, feat 1108, rw 93.81
9. position of outgoing packet 9, feat 13, rw 89.23
10. position of outgoing packet 7, feat 11, rw 85.56
Obfs4PacketIAT: top 10 features
1. std of interpacket time, feat 1225, rw 343.13
2. number of outgoing packets, feat 2, rw 340.04
3. total transmission time, feat 4, rw 328.30
4. mean interpacket time, feat 1224, rw 214.65
5. number of bursts, feat 1107, rw 171.94
6. number of bursts longer than 2, feat 1108, rw 153.33
7. number of packets, feat 1, rw 82.19
8. number of incoming packets, feat 3, rw 78.60
9. number of bursts longer than 5, feat 1109, rw 73.21
10. position of outgoing packet 17, feat 21, rw 70.91
Tamaraw: top 10 features
1. total transmission time, feat 4, rw 741.38
2. std of interpacket time, feat 1225, rw 734.88
3. mean interpacket time, feat 1224, rw 615.72
4. number of outgoing packets, feat 2, rw 67.53
5. longest burst, feat 1105, rw 62.22
6. number of bursts longer than 15, feat 1111, rw 53.92
7. number of bursts longer than 5, feat 1109, rw 52.07
8. number of bursts, feat 1107, rw 51.69
9. number of bursts longer than 2, feat 1108, rw 47.26
10. number of bursts longer than 10, feat 1110, rw 45.49
TamarawBulk: top 10 features
1. mean interpacket time, feat 1224, rw 748.88
2. total transmission time, feat 4, rw 716.57
3. std of interpacket time, feat 1225, rw 582.12
4. number of bursts longer than 5, feat 1109, rw 134.62
5. number of bursts longer than 20, feat 1112, rw 128.42
6. number of bursts longer than 15, feat 1111, rw 102.88
7. longest burst, feat 1105, rw 94.49
8. number of bursts longer than 10, feat 1110, rw 92.34
9. number of bursts longer than 2, feat 1108, rw 77.03
10. number of bursts, feat 1107, rw 57.32Week 46 (local bridge)
Null: top 10 features
1. total transmission time, feat 4, rw 292.28
2. number of outgoing packets, feat 2, rw 223.72
3. std of interpacket time, feat 1225, rw 186.43
4. mean interpacket time, feat 1224, rw 156.82
5. number of bursts, feat 1107, rw 128.37
6. length of burst 3, feat 1116, rw 96.57
7. number of bursts longer than 2, feat 1108, rw 95.54
8. position of outgoing packet 8, feat 12, rw 94.16
9. position of outgoing packet 7, feat 11, rw 91.66
10. position of outgoing packet 11, feat 15, rw 89.65
Obfs4Burst: top 10 features
1. total transmission time, feat 4, rw 345.85
2. number of outgoing packets, feat 2, rw 197.16
3. number of bursts longer than 2, feat 1108, rw 149.73
4. number of bursts, feat 1107, rw 145.68
5. mean interpacket time, feat 1224, rw 138.22
6. std of interpacket time, feat 1225, rw 134.71
7. length of burst 4, feat 1117, rw 115.21
8. length of burst 3, feat 1116, rw 111.77
9. longest burst, feat 1105, rw 107.16
10. length of burst 7, feat 1120, rw 104.87
Obfs4BurstIAT: top 10 features
1. total transmission time, feat 4, rw 380.03
2. number of outgoing packets, feat 2, rw 341.69
3. std of interpacket time, feat 1225, rw 268.80
4. number of bursts, feat 1107, rw 230.95
5. number of bursts longer than 2, feat 1108, rw 215.33
6. length of burst 4, feat 1117, rw 187.98
7. mean interpacket time, feat 1224, rw 182.60
8. length of burst 3, feat 1116, rw 161.51
9. length of burst 7, feat 1120, rw 153.35
10. length of burst 6, feat 1119, rw 153.27
Obfs4PacketIAT: top 10 features
1. number of outgoing packets, feat 2, rw 433.86
2. total transmission time, feat 4, rw 377.18
3. number of bursts, feat 1107, rw 208.74
4. number of bursts longer than 2, feat 1108, rw 155.19
5. std of interpacket time, feat 1225, rw 136.16
6. number of packets, feat 1, rw 134.11
7. mean interpacket time, feat 1224, rw 122.64
8. position of outgoing packet 22, feat 26, rw 117.38
9. position of outgoing packet 5, feat 9, rw 116.05
10. position of outgoing packet 11, feat 15, rw 115.22
Tamaraw: top 10 features
1. total transmission time, feat 4, rw 1016.39
2. std of interpacket time, feat 1225, rw 621.32
3. mean interpacket time, feat 1224, rw 574.04
4. number of outgoing packets, feat 2, rw 90.25
5. longest burst, feat 1105, rw 83.50
6. number of bursts longer than 15, feat 1111, rw 81.06
7. number of bursts longer than 20, feat 1112, rw 70.51
8. number of bursts longer than 5, feat 1109, rw 65.07
9. number of bursts longer than 2, feat 1108, rw 63.80
10. number of bursts longer than 10, feat 1110, rw 50.47
TamarawBulk: top 10 features
1. mean interpacket time, feat 1224, rw 555.89
2. total transmission time, feat 4, rw 426.16
3. std of interpacket time, feat 1225, rw 314.91
4. number of bursts longer than 10, feat 1110, rw 87.15
5. number of bursts longer than 20, feat 1112, rw 79.53
6. longest burst, feat 1105, rw 78.43
7. number of bursts longer than 5, feat 1109, rw 75.66
8. number of bursts longer than 15, feat 1111, rw 67.13
9. number of bursts longer than 2, feat 1108, rw 42.41
10. number of bursts, feat 1107, rw 40.51Week 47 (remote bridge, Hetzner, Germany)
Null: top 10 features
1. std of interpacket time, feat 1225, rw 448.63
2. total transmission time, feat 4, rw 399.09
3. mean interpacket time, feat 1224, rw 284.97
4. number of outgoing packets, feat 2, rw 196.13
5. length of burst 3, feat 1116, rw 110.56
6. length of burst 4, feat 1117, rw 105.11
7. position of outgoing packet 3, feat 7, rw 101.55
8. position of outgoing packet 10, feat 14, rw 100.33
9. number of bursts, feat 1107, rw 94.11
10. position of outgoing packet 7, feat 11, rw 93.05
Obfs4Burst: top 10 features
1. std of interpacket time, feat 1225, rw 337.16
2. total transmission time, feat 4, rw 336.03
3. number of outgoing packets, feat 2, rw 238.99
4. mean interpacket time, feat 1224, rw 236.38
5. number of bursts, feat 1107, rw 145.35
6. number of bursts longer than 2, feat 1108, rw 103.84
7. length of burst 3, feat 1116, rw 77.42
8. length of burst 4, feat 1117, rw 76.52
9. position of outgoing packet 7, feat 11, rw 73.14
10. position of outgoing packet 11, feat 15, rw 72.96
Obfs4BurstIAT: top 10 features
1. total transmission time, feat 4, rw 481.58
2. number of outgoing packets, feat 2, rw 290.21
3. std of interpacket time, feat 1225, rw 251.36
4. mean interpacket time, feat 1224, rw 191.55
5. number of bursts, feat 1107, rw 160.81
6. number of bursts longer than 2, feat 1108, rw 126.68
7. length of burst 4, feat 1117, rw 122.59
8. position of outgoing packet 10, feat 14, rw 110.52
9. length of burst 5, feat 1118, rw 104.21
10. number of packets, feat 1, rw 99.94
Obfs4PacketIAT: top 10 features
1. number of outgoing packets, feat 2, rw 414.75
2. total transmission time, feat 4, rw 280.50
3. std of interpacket time, feat 1225, rw 208.87
4. mean interpacket time, feat 1224, rw 183.27
5. number of bursts longer than 2, feat 1108, rw 176.84
6. number of bursts, feat 1107, rw 168.19
7. number of packets, feat 1, rw 115.92
8. number of bursts longer than 5, feat 1109, rw 92.89
9. number of incoming packets, feat 3, rw 90.95
10. position of outgoing packet 17, feat 21, rw 84.27
Tamaraw: top 10 features
1. total transmission time, feat 4, rw 822.44
2. mean interpacket time, feat 1224, rw 485.82
3. std of interpacket time, feat 1225, rw 340.30
4. number of bursts longer than 5, feat 1109, rw 135.04
5. number of bursts longer than 10, feat 1110, rw 82.34
6. number of bursts longer than 15, feat 1111, rw 75.89
7. number of outgoing packets, feat 2, rw 66.43
8. longest burst, feat 1105, rw 63.73
9. number of bursts, feat 1107, rw 59.44
10. number of bursts longer than 2, feat 1108, rw 55.97
TamarawBulk: top 10 features
1. total transmission time, feat 4, rw 1085.11
2. mean interpacket time, feat 1224, rw 846.70
3. std of interpacket time, feat 1225, rw 475.49
4. number of bursts longer than 20, feat 1112, rw 183.01
5. number of bursts longer than 15, feat 1111, rw 175.70
6. number of bursts longer than 10, feat 1110, rw 139.24
7. longest burst, feat 1105, rw 129.73
8. number of bursts longer than 2, feat 1108, rw 97.75
9. number of bursts longer than 5, feat 1109, rw 91.48
10. number of bursts, feat 1107, rw 74.32Week 48 (remote bridge, OVH, Canada)
Null: top 10 features
1. total transmission time, feat 4, rw 291.38
2. std of interpacket time, feat 1225, rw 272.16
3. mean interpacket time, feat 1224, rw 184.28
4. number of outgoing packets, feat 2, rw 146.40
5. number of bursts, feat 1107, rw 86.10
6. position of outgoing packet 7, feat 11, rw 74.62
7. position of outgoing packet 5, feat 9, rw 71.90
8. position of outgoing packet 10, feat 14, rw 67.14
9. position of outgoing packet 6, feat 10, rw 64.64
10. position of outgoing packet 9, feat 13, rw 63.01
Obfs4Burst: top 10 features
1. std of interpacket time, feat 1225, rw 363.89
2. total transmission time, feat 4, rw 332.78
3. mean interpacket time, feat 1224, rw 237.25
4. number of outgoing packets, feat 2, rw 153.48
5. number of bursts, feat 1107, rw 74.85
6. number of packets, feat 1, rw 65.72
7. position of outgoing packet 6, feat 10, rw 51.14
8. position of outgoing packet 11, feat 15, rw 50.02
9. number of bursts longer than 2, feat 1108, rw 49.56
10. number of incoming packets, feat 3, rw 48.56
Obfs4BurstIAT: top 10 features
1. std of interpacket time, feat 1225, rw 407.42
2. total transmission time, feat 4, rw 353.02
3. mean interpacket time, feat 1224, rw 250.95
4. number of outgoing packets, feat 2, rw 195.01
5. position of outgoing packet 4, feat 8, rw 90.95
6. number of bursts, feat 1107, rw 81.30
7. number of packets, feat 1, rw 72.61
8. position of outgoing packet 5, feat 9, rw 69.83
9. position of outgoing packet 9, feat 13, rw 66.28
10. position of outgoing packet 14, feat 18, rw 66.22
Obfs4PacketIAT: top 10 features
1. number of outgoing packets, feat 2, rw 423.21
2. total transmission time, feat 4, rw 378.60
3. mean interpacket time, feat 1224, rw 274.48
4. std of interpacket time, feat 1225, rw 259.29
5. number of bursts longer than 2, feat 1108, rw 245.34
6. number of bursts, feat 1107, rw 194.38
7. number of bursts longer than 5, feat 1109, rw 157.43
8. longest burst, feat 1105, rw 144.12
9. number of bursts longer than 10, feat 1110, rw 98.74
10. number of packets, feat 1, rw 93.61
Tamaraw: top 10 features
1. total transmission time, feat 4, rw 700.58
2. mean interpacket time, feat 1224, rw 314.20
3. std of interpacket time, feat 1225, rw 240.64
4. number of bursts longer than 5, feat 1109, rw 103.48
5. number of bursts longer than 2, feat 1108, rw 64.86
6. number of bursts longer than 10, feat 1110, rw 54.92
7. number of bursts, feat 1107, rw 52.37
8. number of bursts longer than 20, feat 1112, rw 45.02
9. number of bursts longer than 15, feat 1111, rw 44.23
10. longest burst, feat 1105, rw 41.45
TamarawBulk: top 10 features
1. total transmission time, feat 4, rw 447.56
2. mean interpacket time, feat 1224, rw 377.98
3. std of interpacket time, feat 1225, rw 247.17
4. number of bursts longer than 2, feat 1108, rw 129.75
5. number of bursts longer than 15, feat 1111, rw 119.54
6. longest burst, feat 1105, rw 116.58
7. number of bursts longer than 20, feat 1112, rw 111.25
8. number of bursts longer than 10, feat 1110, rw 105.02
9. number of bursts longer than 5, feat 1109, rw 82.78
10. number of bursts, feat 1107, rw 60.96Most volatile features
Removing Tamaraw and TamarawBulk, we look at the most volatile (geometric mean) features.
Week 45 (local bridge)
1. std of interpacket time, feat 1225, volatility 408.0
2. total transmission time, feat 4, volatility 371.1
3. number of outgoing packets, feat 2, volatility 243.1
4. mean interpacket time, feat 1224, volatility 242.5
5. number of bursts, feat 1107, volatility 135.0Week 46 (local bridge)
1. total transmission time, feat 4, volatility 346.9
2. number of outgoing packets, feat 2, volatility 284.4
3. std of interpacket time, feat 1225, volatility 174.1
4. number of bursts, feat 1107, volatility 173.3
5. mean interpacket time, feat 1224, volatility 148.4Week 47 (remote bridge, Hetzner, Germany)
1. total transmission time, feat 4, volatility 366.9
2. std of interpacket time, feat 1225, volatility 298.5
3. number of outgoing packets, feat 2, volatility 274.1
4. mean interpacket time, feat 1224, volatility 220.5
5. number of bursts, feat 1107, volatility 138.7Week 48 (remote bridge, OVH, Canada)
1. total transmission time, feat 4, volatility 337.4
2. std of interpacket time, feat 1225, volatility 319.8
3. mean interpacket time, feat 1224, volatility 234.3
4. number of outgoing packets, feat 2, volatility 207.5
5. number of bursts, feat 1107, volatility 100.5Discussion
Regarding recall, we see that for the local bridge (both datasets) that Obfs4PacketIAT gives highest recall of all methods and all others being more-or-less equal. As we go to the remote bridge in Germany, Obfs4PacketIAT provides even higher recall while—surprisingly—Obfs4Burst also sees increased recall over Null. Obfs4BurstIAT has slightly lower recall than Null. Moving to the dataset for the Canadian bridge, there’s no difference in recall between Null and Obfs4Burst. As for the remote bridge in Germany, Obfs4BurstIAT has lower recall than Null, and, finally, we now see Obfs4PacketIAT provide lower recall than Null.
We conclude that the latency between a bridge and its connecting client matters for what kind of basket2 method provides the best protection against WF attacks. On low latency connections, Obfs4PacketIAT actually improves performance of WF attacks, paying off as a WF defense first on high latency connections. Always obfuscating inter-arrival time appears like a good idea
on a per-burst level as in Obfs4BurstIAT with only small gains for attackers
worst-case.
In terms of features, we see the same set regardless of method. Notably, features related to time are always really useful even in the face of obfuscating inter-arrival times. The decision by Wang et al. to exclude them by default in their implementations might have been a mistake.