Thursday, 2021-07-15

*** ykarel|away is now known as ykarel04:32
*** amoralej|off is now known as amoralej06:10
bhagyashris|ruckarxcruz, hi06:29
ykarelbhagyashris|ruck, chkumar|rover are you aware about ci jobs config errors in https://review.rdoproject.org/zuul/config-errors?06:57
ykarelmainly the promotion-staging one or jobs defined in ci-config06:58
ykareli didn't checked if those jobs are running or being monitored by CI team06:58
chkumar|roverykarel: we donot monitor those, but good to check those07:02
ykarelchkumar|rover, yes please07:02
ykarelwe are checking non promoter ones, u can check those07:02
mariosbhagyashris|ruck: maybe we need to shift the scrum by half hour today there is a clash with all-hands fyi08:36
bhagyashris|ruckmarios, ack08:39
*** chem is now known as Guest87409:16
*** ykarel is now known as ykarel|lunch09:17
arxcruzbhagyashris|ruck: hey09:53
bhagyashris|ruckarxcruz,  i have one question- here on the dashboard http://dashboard-ci.tripleo.org/d/3pUqDadGk/tempest-skipped-tests?orgId=1 i see some of the test are not present in the skiplist https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml10:18
arxcruzbhagyashris|ruck: because it's being skipped the whole namespace10:19
arxcruzfor example TestNetworkBasicOps10:19
bhagyashris|ruckfor example this test tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details is shown on the dashboard but not in ths skiplist10:19
arxcruztempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details might not be in the skiplist, because tempest.scenario.test_network_basic_ops.TestNetworkBasicOps is being skipped 10:20
bhagyashris|rucki try to search tempest.scenario.test_network_basic_ops.TestNetworkBasicOps but that is also not in the skiplist10:21
bhagyashris|ruckand on the dashboard that test is skipped for all the branches as well10:25
bhagyashris|rucklooks like the dashboard is not showing the correct result10:25
*** ykarel|lunch is now known as ykarel11:00
*** dviroel|out is now known as dviroel11:02
arxcruzbhagyashris|ruck: https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L113111:47
arxcruzTestNetworkBasicOps is skipped11:47
arxcruzbhagyashris|ruck: nevermind, it's newton lol 11:47
arxcruzbhagyashris|ruck: https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L1755 test_subnet_details is also being skipped 11:48
bhagyashris|ruckarxcruz, yes some are there some are not11:50
*** amoralej is now known as amoralej|lunch11:50
arxcruzbhagyashris|ruck: i'll take a look 11:50
bhagyashris|ruckarxcruz, thanks11:50
*** amoralej|lunch is now known as amoralej|away11:50
bhagyashris|ruckarxcruz, plz review https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/800886 when you have time 11:54
arxcruzbhagyashris|ruck: was doing that just right now :) 11:55
bhagyashris|ruckarxcruz, ok thanks :)11:55
* frenzy_friday lunch11:55
bhagyashris|ruckarxcruz, weshay|ruck chkumar|rover please have a look when you have time https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/80091312:26
chkumar|roverdone12:27
arxcruzbhagyashris|ruck: did you check all the branches right ?12:29
bhagyashris|ruckarxcruz, yes12:30
weshay|ruckarxcruz++ bhagyashris|ruck++ chkumar|rover++12:44
zbr i need some clarifications regarding where the get_hash config is supporsed to be loaded from.12:57
zbrat this moment load_config has some kind of weird lookup logic, apparently looking for current directry config.yaml or  usr/local/etc/tripleo_get_hash/config.yaml --- as you can guess, that is not reliable.12:59
dviroelzbr: marios will be the person to answer you about that13:07
zbri have one test that confuses me, now i am trying to see if there are also others.13:08
marioszbr: lets talk on scrum i was going to ask for a sync call with you and dviroel re get hash and tripleo-repos setup.cfg moving forward13:16
marioszbr: i can explain the 'weird lookup logic too' (You're weird!)13:16
marios;)13:16
dviroelchkumar|rover: lol, i was about to ask in bhagyashris|ruck's patch, why we keep these skip tests for old releases like liberty, mitaka, newton13:17
dviroelchkumar|rover: now i saw your patch removing them :)13:17
zbrprobaly it would be better just after scrum to avoid boredom for others.13:18
marioszbr: yes i was going to suggest tomorrow morning ususally we don't have many scheduled calls... ah dviroel timezone... ok his morning then like early afternoon for us. lets setup on scrum anyway 13:23
bhagyashris|ruckakahat, arxcruz chkumar|rover frenzy_friday marios pojadhav rlandy sshnaidm zbr weshay|ruck dviroel  scrum meeting in 1 min13:29
bhagyashris|ruckReview list 13:33
bhagyashris|ruckhttps://review.opendev.org/q/topic:%22scenario010-branchful%22+(status:open%20OR%20status:merged)13:33
marioshttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/793331/14#message-8e221153fe33586675eb5e36f6f0b54fc639908f13:36
bhagyashris|ruckhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/79148613:38
rlandyhttps://review.opendev.org/q/owner:+pojadhav%2540redhat.com+topic:+wallaby-upgrade-jobs13:41
rlandyhttps://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/79323413:45
bhagyashris|ruckhttps://review.rdoproject.org/r/q/topic:%22refactor_job_names%22+(status:open%20OR%20status:merged)13:45
frenzy_fridayweshay|ruck, I have a new pastebin if you want to try https://paste.opendev.org/show/807504/ But it may not work (also the patch has been updated, you have to pull again)13:52
mariosgreat scrum... blockers and even time for some chat... 22 mins. ++13:52
weshay|ruckfrenzy_friday, rock... trying13:55
marioszbr: dviroel: sent invite for tomorrow 'usual' scrum time14:00
marioszbr: let's sync our morning tomorrow - ping me when you get in for the 'quick sync' call to unblock you?14:00
marioszbr: then we can meet with douglas for the main chat as scheduled14:00
weshay|ruckfrenzy_friday, re: INFO:paramiko:Trying ssh connection to weshayutin@review.opendev.org:2941814:00
zbrok14:01
weshay|ruckfrenzy_friday, /me assumes.. this is an error right?14:01
weshay|ruckDEBUG:paramiko.transport:Dropping user packet because connection is dead.14:01
weshay|ruckor is that normal for make build14:01
frenzy_fridayno, thats the end of make buuild14:01
frenzy_fridayYou can try make up now14:01
marioszbr: meantime can you please add this to your reviews list https://review.opendev.org/c/openstack/tripleo-quickstart/+/791486 & depends-on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/793331 as mentioned in scrum just now 14:01
weshay|ruckah cool14:01
weshay|ruckfrenzy_friday, thanks14:01
frenzy_fridayweshay|ruck, It should run with a lot of debug outputs till you get er-cron | crond: wakeup dt=6014:02
weshay|rucklooking better14:02
weshay|rucker-web exited with code 114:02
weshay|ruckAborting on container exit...14:02
weshay|ruckStopping er-bot  ... done14:02
weshay|ruckStopping er-cron ... done14:02
weshay|ruckmake: *** [Makefile:59: up] Error 114:02
mariosrlandy: weshay|ruck: please add to your reviews for when you next have time thanks https://review.opendev.org/c/openstack/tripleo-quickstart/+/791486 & depends-on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/79333114:02
zbrmarios: imho, i would prefer if we directly install the module as part of a collection and avoid the hacky installation from these reviews14:03
zbrto explain it: we would no longer need to sudo, just galaxy build/install inside the same folder.14:04
*** chkumar|rover is now known as chandankumar14:05
weshay|ruckfrenzy_friday, make up should start the containers?14:07
frenzy_fridayweshay|ruck, yes. I forgot to add one more step in the paste14:07
rlandyfrenzy_friday: is the VM updated? should I try make up again?14:10
rlandybuild worked14:10
rlandymarios: ack - pls bug me again tomorrow if I dont get to it14:11
frenzy_fridayrlandy, not yet. Right now in the original health vm it looks like the containers are running but nginx shows 40414:11
mariosrlandy: thanks14:11
rlandychandankumar: bhagyashris|ruck: have you see any ovb failures with "Property error: resources.NovaCompute.properties.image: Error validating value 'overcloud-full': No images matching {'name': 'overcloud-full'}." in deploy?15:33
rlandyshowing up in downstream15:33
rlandyjust check if you have seen the same in rdo15:33
rlandyweshay|ruck: do you know why we switched baremetal in the integration lines to fs035 and leave the component lines on fs001?15:35
weshay|ruckrlandy, better coverage15:35
chandankumarrlandy: yes15:35
rlandybarmetal and tripleo on 16.2 are way behind15:35
chandankumarrlandy: https://bugs.launchpad.net/tripleo/+bug/193628715:36
rlandychandankumar: thank yoou15:36
rlandyyou15:36
rlandyhttps://github.com/openstack/tripleo-quickstart-extras/commit/c9676e9115f1aa097b528ca7011946599141676715:42
rlandywonder if its not this15:42
rlandysshnaidm: https://github.com/openstack/tripleo-quickstart-extras/commit/9c6667d9156f7228bc91871db820823485ba94ed - should train be included in the legacy here?15:44
rlandynoting https://bugs.launchpad.net/tripleo/+bug/193628715:44
rlandytrain jobs not finding images15:45
* rlandy tried15:45
rlandytries15:45
sshnaidmhmm.. seems like tripleo operator worked: tripleo.operator.tripleo_overcloud_image_upload https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/ab8385f/job-output.txt15:47
sshnaidmrlandy, https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/ab8385f/logs/undercloud/home/zuul/overcloud_image_upload.log.txt.gz15:47
sshnaidmrlandy, if with train it works, let's use legacy, I can look later what is the difference15:49
rlandysshnaidm: going to try the adding train back to legacy15:49
rlandyit's taking out the train and 16.2 lines atm15:49
sshnaidmack15:49
rlandythanks15:49
*** marios is now known as marios|out16:25
*** ykarel is now known as ykarel|away16:32
*** sshnaidm is now known as sshnaidm|afk16:35
rlandylunch - biab17:10
weshay|ruckrlandy, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/800975  not sure if I have that right17:43
weshay|ruckrlandy, can you add something to the Examples: https://opendev.org/openstack/openstack-tempest-skiplist17:44
* dviroel will get his first dose of vaccine on saturday :)17:50
* rlandy looks17:54
rlandydviroel: :) ... don't make plans on sunday17:54
rlandyhttps://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/doc/source/listyaml/listyaml.rst17:56
dviroelrlandy: yeah =|17:56
rlandyguess that's all there is on installer17:56
rlandyhttps://opendev.org/openstack/openstack-tempest-skiplist/src/commit/85a392c55f55bd83a370c677c3174d567f288b1e/roles/list_skipped/library/list_skipped.py#L8017:57
rlandylooking at tox failure17:58
rlandyweshay|ruck: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/800975 - installer looks right but the test line needs to be spaced in 18:00
rlandyweshay|ruck: container build job ... https://logserver.rdoproject.org/53/18953/91/check/tripleo-build-containers-stream9-development/2c98310/logs/containers-built.log18:01
rlandyrunning18:01
rlandyfixing the build failure18:02
rlandystream 918:02
rlandyhttps://review.rdoproject.org/zuul/stream/46d090c2487a4e8db2bd06f7580a69d1?logfile=console.log - fix for images on train works18:03
rlandyweshay|ruck: ignore that spacing column18:12
rlandyit's the viewer I am using18:13
rlandygot that review - looking at it18:13
rlandyreview updated18:15
rlandyhttps://logserver.rdoproject.org/95/24995/58/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/46d090c/job-output.txt19:04
rlandyok - that get past image download - now introspect fails on train19:04
weshay|ruckya.. saw it was busted19:05
rlandyweshay|ruck: both19:05
rlandysame issue on train and 16.219:06
weshay|ruckrlandy, rabi has a comment on your patch19:09
rlandylooking19:09
rlandyk - editing that patch in a bit19:10
rlandyhttps://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/800980 - updated19:31
rlandyweshay|ruck: ^^19:32
weshay|ruckrock on.. thank you!19:35
rlandybaremetal is still running - so waiting until that is done19:50
rlandyto update train patch19:50
*** dviroel is now known as dviroel|brb19:53
*** dviroel|brb is now known as dviroel20:27
rlandy"success": true - yay finally one bm pass20:43
rlandyhttps://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/800980 ok to merge?21:08
weshay|ruckrlandy, aye21:14
rlandythanks21:14
*** dviroel is now known as dviroel|out21:24
rlandy2021-07-15 16:24:03.168466 | primary |21:46
rlandy2021-07-15 16:24:03.168535 | primary | TASK [os_tempest : Execute tempest tests] **************************************21:46
rlandy2021-07-15 16:24:03.168544 | primary | Thursday 15 July 2021  16:24:03 -0400 (0:00:00.036)       0:04:10.110 *********21:46
rlandy2021-07-15 21:39:16.770334 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]21:46
rlandy2021-07-15 21:39:16.791288 | POST-RUN START: [trusted : code.engineering.redhat.com/openstack/tripleo-ci-internal-config/playbooks/dlrn/post-dlrn.yaml@master]21:46
rlandyweshay|ruck: BM on tripleo component keeps timing out ^^21:46
rlandyit's blocking promotion on the 16.2 component line21:46
weshay|ruckin the post?21:46
rlandycan we reduce tempest there?21:47
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/7e1d7db9d705403dbf943f6f09cfb83f?logfile=console.log21:47
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-bm_envC-3ctlr_1comp-featureset001-tripleo-rhos-16.221:47
rlandyon good days it times out21:47
rlandyor we download images21:47
rlandywe are at the longest job time21:48
weshay|ruckrlandy, more concerned with why bm is failing21:49
weshay|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-bm_envC-3ctlr_1comp-featureset001-tripleo-rhos-16.2&pipeline=openstack-component-tripleo21:49
rlandysec - just kicking off job to test marios patch21:50
rlandyweshay|ruck: numerous failures21:51
rlandyrestarted some hardware21:51
rlandythen there was rabi's patch with networking21:51
rlandythen sandeep's patch with operators21:52
rlandybeen chasing this job for days21:52
rlandythat's why tripleo has not promoted on 16.221:52
rlandyjob is criteria21:52
rlandythe BM job from the baremetal line just passed21:53
rlandywith patch to get train off operators21:53
rlandyweshay|ruck: maybe worth skipping bm in tripleo criteria to get component to promote21:58
rlandyat least now21:58
*** rlandy is now known as rlandy|bbl22:08

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!