CS-438: Tests running "forever"

Hello,

I have finished going through all the tests in gossiper_test.go, and after some work I manage to pass all of them individually. When running the entire file tough, it keeps running the test for more than 10 minutes (when pushing on gitlab, the pipeline fails because it runs for more than 10 minutes and 20 seconds). I am struggling to figure our the reason for such behaviour, since the tests have a timeout of 30s each. I have no experience with go and tests, so I am sorry if the question turns to be a silly one.

Regards!

Re: Tests running "forever"

by Kirill Nikitin - Friday, 13 November 2020, 14:59

Hi,

Hmm yes it is expected that the tests take more than 10 minutes in total to run but the pipeline time limit on Gitlab seems to be 60 minutes. Does it work if you run the entire test file on your local machine?

Kirill

Re: Tests running "forever"

by Peter Krcmar - Tuesday, 17 November 2020, 00:44

Hi,

I've encountered the same issue a couple of times (pass locally, hang on gitlab) with the tests that use the complex topo i.e:

TestGossiper_Search_Query_Complex_Topo

TestGossiper_Search_Query_No_Budget_Complex_Topo

After some debugging I managed to find out that the problem was that I wasn't receiving my "stop" message and so the gossipers never stop listening on the UDPConn.

The thing is, I'm doing nothing differently than in the other tests, I'm simply writing my stop message with a WriteToUDP. I checked that it's the correct address and there is no error when sending it, the message is simply not received.

It also isn't deterministic (I can just rerun the build and it passes half of the time) and I didn't manage to reproduce the error locally on either windows or linux.

Any help/tips are appreciated :)

Re: Tests running "forever"

by Olivér Facklam - Tuesday, 17 November 2020, 14:14

Hello,

I'm getting similar issues on GitLab. All tests work fine locally (2-3mins), but sometimes they timeout in the CI (>10mins). I don't know if this is an issue linked to my code or not? Has anyone been able to solve this?

Thanks,

Olivér

Re: Tests running "forever"

by Peter Krcmar - Tuesday, 17 November 2020, 17:38

Ok so I finally managed to make some progress:

Managed to recreate the error on ubuntu. With a lot of nodes (complex topos), some of my messages were being lost (including the stop messages). So I set my read and write buffers on my socket to a much larger value and I am consistently passing the tests now locally.

However, it doesn't seem to have much effect on gitlab...

Re: Tests running "forever"

by Kirill Nikitin - Tuesday, 17 November 2020, 17:50

It is quite difficult to exactly say what is going on but if your implementation passes all the tests locally if you run them all at once, then you should be fine. We do not look at the results of the gitlab pipeline but grade your submissions by running the tests locally.

Kirill