-
Notifications
You must be signed in to change notification settings - Fork 300
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
next: Changes queued for the v2016.05 release #888
Conversation
Link from the top-level README.md to make the current state of platform support more apparent to users. Inspired by #701.
Message: master branch supports Linux/x86-64. Your branch can support any platform you like.
Previously a warning but promoted on suggestion from @wingo.
This was written for the @alexangergall's ALX appliance but I believe he has replaced it with other code.
This enables assembly language code to access freelists in a fairly straightforward manner.
braindump on performance parameters
added info on mlock=on
This new document explains the three core Snabb data structure: packet, link, and app. For now this is a stand-alone markdown file but it could be incorporated into the introduction in the manual. This is imported from lukego/blog#11.
Perftune - performance tuning notes
Removed a stray hyphen before the filename extension.
This is a simple and permissive trademark license taken directly from the PostgreSQL one: https://wiki.postgresql.org/wiki/Trademark_Policy
This was meant to be removed in commit 519311b.
Add /lib/luajit/usr to .gitignore
Seems like the performance regression is real. Source: #896 |
Great work, SnabbBot !!! I will investigate now. |
@eugeneia I am having trouble running this benchmark in the Docker environment on Interlaken:
Any hints what might be wrong? I compiled Snabb in dock.sh. This is on branch next. |
oh hm seem to be two issues. First I used the wrong PCI address. Second seems that I need to fully qualify the PCI address for some reason. This now does work on interlaken:
Groovy. I have a test env for debugging the performance issue now. |
@eugeneia Good catch on s/2015/2016/ in the title of the PR, but to me it feels like the PR title is owned by the submitter and it seems intrusive for somebody else to change it directly rather than point out the problem. |
@lukego You are right, sorry. |
@eugeneia I think our tests on |
... or maybe I am mistakenly thinking they are yours because I see |
Sorry about the wait. Currently I am concerned about the lower results we are seeing on the I see two related issues really:
These issues interact: the variability makes it harder to compare results between branches. I am now approaching this in the "scientific testing" style by collecting more performance samples and comparing them using R scripts. I am becoming incrementally less ad-hoc in my R scripting with support from @domenkozar and http://hydra.snabb.co/. Here is what I have so far:
The immediate next step is to experiment with a few changes to Going forward I am very interested to have the CI measure variability and reduce this over time. I cannot account for this at the moment. I am keen to make the timeline viewer into the tool of choice for post-mortem analysis of runs with unexpected results e.g. the slowest of the samples here. |
So based on the idea that performance decreased because 382827d increased the NIC descriptor ring size from 512 packets to 2048 packets I tried a compromise by patching See Report for full details. Overview below. Using 1024 packets in the descriptor ring seems to partly resolve the performance regression: The average is still lower but Tukey's Test tells us that this is not statistically significant. Specifically, with the data we have the best we can say with 95% confidence is that the Seen in this table excerpt where we have the average difference (
So it is easy to believe that there is still a modest performance reduction on this benchmark but the noise from uncontrolled variation means that a lot of testing would be required to confirm this. This is the point where I start to be more interested in understanding the variation itself. I also briefly checked
So the action plan that comes to my mind is:
Though this is negotiable if other people have strong opinions. |
This value is probably not perfectly optimal. See discussion: #888 (comment)
Relatedly: SnabbBot reports low scores for the iperf benchmark mainly for three reasons:
Merging the new checksum routine from #899 may well provide a boost to all platforms including these pre-Haswell machines that lack AVX2. Could be something for |
I did a little more R hacking. I am really excited to be starting to "productionize" some of this performance analysis to become part of the release process. This time it is only the The new report has updated visualizations, wikipedia links explaining how to interpret each plot, and my own interpretation of the results. The Hydra build publishes everything needed to reproduce the report. TLDR pretty picture: |
@lukego nice. Wouldn't it be better to remove some outliers before plotting results? |
@domenkozar Good question. I would like to understand what the outliers mean first. Is it that we have uncontrolled background activities on the test server and need to improve our scheduling? Is it that we have a problem in our test scripts that fails to control something important? Is it that we have a bug in our software that causes the JIT to generate the wrong code in certain instances? It's possible that the outliers are boring but it is also possible that they are the most important points of all. The best strategy I have in mind would be if each blue dot becomes a hyperlink to a Hydra build for that one specific test case. Then we can click on data points to zoom in on the best/worst/middle results and understand what is different between the runs. Can we setup Hydra to publish each individual test run on its own page? R will be able to do the hyperlinking if we include some identifying information in the CSV file and use the SVG backend for the graph. I am imagining the CSV file we want Hydra to produce for R will be something like...
Meanwhile... I added a section to the report with some numeric statistics including median and quartiles that are not too sensitive to outliers:
|
... one interesting observation from the numeric table above is that there are only 46 values available for the branch |
No description provided.