| opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Add Cyborg MDEV driver lifecycle scenario tests https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997352 | 06:44 |
|---|---|---|
| opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Prefer lspci over sysfs for PCI device detection https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997684 | 06:44 |
| opendevreview | Joan Gilabert proposed openstack/cyborg master: Configure pci lifecycle tempest settings via devstack plugin https://review.opendev.org/c/openstack/cyborg/+/997365 | 06:45 |
| *** jgilaber_ is now known as jgilaber | 10:02 | |
| jgilaber | sean-k-mooney Hi! I've pushed a patch to change how we check for pci devices in guest vms https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997684 to prefer using lspci | 10:03 |
| jgilaber | I've tested the change in https://review.opendev.org/c/openstack/cyborg/+/997364 and https://review.opendev.org/c/openstack/cyborg/+/997365 multiple times and seems to work well | 10:03 |
| sean-k-mooney | cool | 10:04 |
| sean-k-mooney | were you able ot replciate the failure locally? | 10:04 |
| jgilaber | I've run the tests at lest 8 times between yesterday and today | 10:04 |
| jgilaber | no I have not tested that | 10:04 |
| sean-k-mooney | well im more askign did you see it fail | 10:04 |
| jgilaber | only on ci | 10:04 |
| sean-k-mooney | i have not seen the older verion fail locally either | 10:05 |
| sean-k-mooney | ack ok well we can recheck it a few times | 10:05 |
| jgilaber | yes, we can recheck it a few times more today and check back towards eod or next week | 10:05 |
| jgilaber | I will try it locally as well | 10:05 |
| sean-k-mooney | cool the change looks good | 10:06 |
| sean-k-mooney | out of inteest did we looks at how we do this in whitebox | 10:06 |
| sean-k-mooney | https://opendev.org/openstack/whitebox-tempest-plugin/src/commit/a464d48e5e517ac8e4d04ca72b25042067abcb31/whitebox_tempest_plugin/api/compute/test_sriov.py#L862 | 10:07 |
| sean-k-mooney | so ya def _check_device_in_guest(self, linux_client, vendor_id, product_id): | 10:08 |
| sean-k-mooney | is doing cmd = "lspci -nn | grep {0}:{1} | wc -l".format( | 10:08 |
| sean-k-mooney | vendor_id, product_id) | 10:08 |
| sean-k-mooney | we may wantt to condier just porting that function as this has been sabel but its usign lspci as well | 10:08 |
| sean-k-mooney | so thats good at least | 10:09 |
| jgilaber | very similar | 10:09 |
| sean-k-mooney | so we dont technially need the fall back | 10:09 |
| sean-k-mooney | well | 10:09 |
| jgilaber | I've left the current approach as fallback but we could remove it to simplify maybe | 10:09 |
| sean-k-mooney | lspci is expected to be installled in the ci image | 10:09 |
| sean-k-mooney | for now i think we can keep it but we might simplfy this in a followup | 10:10 |
| sean-k-mooney | i kind of like that _check_device_in_guest is doing the filstering on the vm rather then copying back all the data | 10:10 |
| jgilaber | sounds good to me | 10:11 |
| jgilaber | I've been running the version of the tests in master which failed yesterday but could not reproduce the problem either | 11:38 |
| jgilaber | I've run them 6 or 7 time nows, it's quite weird that it seemed so consistent yesterday | 11:38 |
| chandankumar | https://zuul.opendev.org/t/openstack/builds?change=997365&result=FAILURE&skip=0 , one time py3 and ipv6 failed job ran on rax-ord-main . | 11:51 |
| chandankumar | yesterday | 11:52 |
| chandankumar | https://zuul.opendev.org/t/openstack/builds?job_name=cyborg-tempest-ipv6&change=997365&skip=0 - rest of the passing ipv6 ran on different provider. | 11:53 |
| chandankumar | I am not sure it is linked with provider also. | 11:54 |
| opendevreview | Takashi Kajinami proposed openstack/cyborg master: Add cors middleware defaults https://review.opendev.org/c/openstack/cyborg/+/997712 | 12:00 |
| jgilaber | today it's been running in different providers, will recheck a few more times to see | 12:00 |
| opendevreview | Takashi Kajinami proposed openstack/cyborg master: Add cors middleware defaults https://review.opendev.org/c/openstack/cyborg/+/997712 | 13:11 |
| jgilaber | chandankumar, I think you might be onto something, the ipv6 failed now and it ran on rax-ord-main https://zuul.opendev.org/t/openstack/build/87e90ed8ebbc4b7cad73c3d69d76c6a0 | 13:31 |
| sean-k-mooney | jgilaber: it could be timeing related but i think the lspci approch is going to more robust anyway | 13:43 |
| sean-k-mooney | so ya if its still stable ill approve it in an hour or two | 13:43 |
| sean-k-mooney | i was surpsied when you menteiond it sicne i had never seen it fail personally before | 13:44 |
| sean-k-mooney | but again the ci can show issue we dont see on our local system so im not that surpised either | 13:45 |
| jgilaber | yes, it did seem stable before that point | 13:45 |
| jgilaber | it's super stable in my dev env as well, could not get it to fail there | 13:45 |
| chandankumar | Do we want to add some wait condition https://github.com/openstack/cyborg-tempest-plugin/blob/master/cyborg_tempest_plugin/tests/scenario/test_pci_driver_lifecycle.py#L194 after reboot to make sure server is ip and call _assert_server_pci_ready ? | 13:47 |
| chandankumar | from multiple runs, it is failing on rax-ord-main provider only | 13:47 |
| sean-k-mooney | we shodul rewrite that test i think | 14:10 |
| sean-k-mooney | well | 14:11 |
| sean-k-mooney | we can keep it but yes we need a retry loop | 14:11 |
| sean-k-mooney | i was thinkin gif we wanted to delete that test and just rely on teh nova api reboot tests | 14:11 |
| sean-k-mooney | but for the gues one we need to wait for the vm to be back up before calling _assert_server_pci_ready | 14:12 |
| sean-k-mooney | if we really want this to be more roboust https://github.com/openstack/cyborg-tempest-plugin/blob/master/cyborg_tempest_plugin/tests/scenario/test_pci_driver_lifecycle.py#L161 shoudl be SSHABLE instead of active as well | 14:13 |
| sean-k-mooney | since we are going to try adn ssh into it waiting for SSHABLE which is a tempest concept is more accurate as ACTIVE just means nova thinks its running | 14:15 |
| sean-k-mooney | it does nto mean the guest os has fully booted | 14:15 |
| jgilaber | sean-k-mooney, sorry I had to step out for a bit | 14:52 |
| sean-k-mooney | its fine | 14:53 |
| jgilaber | I can change the state there for 'SSHABLE' but I don't think that is called in the test in question | 14:53 |
| sean-k-mooney | lets revisit that | 14:53 |
| sean-k-mooney | we can look at doign that speratly | 14:53 |
| sean-k-mooney | beasue we likely shoudl do that consitently not just in those tests | 14:53 |
| sean-k-mooney | also when waiting for the server to create | 14:54 |
| sean-k-mooney | not just reboot ectra | 14:54 |
| sean-k-mooney | jgilaber: are you happy enough to proceed with the currnt improment | 14:54 |
| sean-k-mooney | if so i can appove the test change and we can then look at appoving the job change patch | 14:55 |
| sean-k-mooney | ill be around for anouther hour or two then ill be finishing up | 14:55 |
| jgilaber | we could go ahead, I think it is slightly better, although it does not completely solve the problem | 14:58 |
| jgilaber | i.e if we enable the tests we might get gate failures which will be annoying | 14:59 |
| sean-k-mooney | yes but i wont be here next week | 15:00 |
| sean-k-mooney | so we wont be merging anything anyway and ye can tell me how it goes when i get back :P | 15:00 |
| sean-k-mooney | so that sounds like a you problem to me :) | 15:00 |
| jgilaber | yep, that's about right | 15:01 |
| sean-k-mooney | on a seious not however i think i woudl prefer having the test run for a week even if they fail | 15:01 |
| sean-k-mooney | just to actullly flesh out any opther issues with them | 15:02 |
| jgilaber | we can go ahead since it won't be blocking anything and we can collect more data from other patches | 15:02 |
| opendevreview | Merged openstack/cyborg-tempest-plugin master: Prefer lspci over sysfs for PCI device detection https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997684 | 15:47 |
| opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Add Cyborg MDEV driver lifecycle scenario tests https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997352 | 15:48 |
| opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Verify guest reboot started in PCI lifecycle test https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997751 | 15:48 |
| jgilaber | I'm not convinced this will fix the problem but I'll try during the week | 15:49 |
| jgilaber | sean-k-mooney, do you want to merge https://review.opendev.org/c/openstack/cyborg/+/997365 before you leave? | 15:50 |
| sean-k-mooney | i will yes on a call | 15:59 |
| sean-k-mooney | ok its reappoved | 16:47 |
| sean-k-mooney | assuming it passes in gate that shoudl merged | 16:48 |
| sean-k-mooney | im goign to finish up in the next 15 minutes or so | 16:48 |
| sean-k-mooney | but i can take a look ifthere is anythin else pressing | 16:49 |
| sean-k-mooney | i will have some conenctiviy while traveling next week but im not sure how much so i will check in when possible | 16:49 |
| jgilaber | ack thanks don't worry | 16:59 |
| jgilaber | I'll continue working on it | 16:59 |
| opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Verify guest reboot started in PCI lifecycle test https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997751 | 17:27 |
| opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Add Cyborg MDEV driver lifecycle scenario tests https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997352 | 17:27 |
| sean-k-mooney | jgilaber: os tempest has some conenctivty checks that we coudl reuse | 17:28 |
| sean-k-mooney | im not againt doing it our selves | 17:28 |
| sean-k-mooney | but just pointing that out | 17:28 |
| jgilaber | I'll look into that next week if the current version works | 17:29 |
| jgilaber | can you add +2 to https://review.opendev.org/c/openstack/cyborg/+/997365? It looks like the gate pipeline has not started | 17:29 |
| sean-k-mooney | oh lol | 17:29 |
| sean-k-mooney | i tough i already had | 17:30 |
| sean-k-mooney | done | 17:30 |
| jgilaber | it got deleted in one of the updates | 17:31 |
| sean-k-mooney | i left a comment on your patch but this is the helper i think you shoudl use https://github.com/openstack/tempest/blob/master/tempest/common/compute.py#L113-L153 | 17:33 |
| sean-k-mooney | or use https://github.com/openstack/tempest/blob/master/tempest/common/waiters.py#L727-L736 or https://github.com/openstack/tempest/blob/master/tempest/common/waiters.py#L678-L686 | 17:34 |
| sean-k-mooney | directly | 17:34 |
| sean-k-mooney | wait_for_server_status dose not actully supporort SSHABLE as it not actully a nova status | 17:35 |
| jgilaber | yes, I've changed that I saw it failed in CI | 17:36 |
| jgilaber | I'm leaving know I'll change the patch on Monday to use the tempest functions, thanks for looking and safe travels next week! | 17:38 |
| sean-k-mooney | no worries | 17:38 |
| opendevreview | Merged openstack/cyborg master: Configure pci lifecycle tempest settings via devstack plugin https://review.opendev.org/c/openstack/cyborg/+/997365 | 18:34 |
| opendevreview | melanie witt proposed openstack/cyborg master: Add Intel XXV710 25GbE backplane NIC to known NIC list https://review.opendev.org/c/openstack/cyborg/+/997782 | 19:14 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!