Monday, 2019-10-14

*** dsneddon has quit IRC00:02
*** d0ugal has quit IRC00:05
*** d0ugal has joined #oooq00:08
*** dsneddon has joined #oooq00:29
*** dsneddon has quit IRC00:39
*** dsneddon has joined #oooq00:46
*** dsneddon has quit IRC00:50
*** dsneddon has joined #oooq00:56
*** ysandeep has joined #oooq01:14
*** aakarsh|2 has quit IRC02:26
*** ykarel has joined #oooq03:55
*** udesale has joined #oooq04:07
*** soniya29 has joined #oooq04:32
soniya29Hello everyone04:41
*** sanjayu_ has joined #oooq04:53
*** surpatil has joined #oooq05:04
*** ykarel has quit IRC05:04
*** ysandeep has quit IRC05:07
*** ykarel has joined #oooq05:24
*** jbadiapa has joined #oooq05:36
*** marios has joined #oooq05:36
*** marios is now known as marios|rover05:36
*** kopecmartin|off is now known as kopecmartin05:59
*** akahat has joined #oooq06:03
*** ccamacho has joined #oooq06:10
*** holser has joined #oooq06:18
*** bogdando has joined #oooq06:42
*** akahat has quit IRC06:52
*** epoojad1 has joined #oooq06:53
*** apetrich has joined #oooq07:04
*** tesseract has joined #oooq07:04
raukadahOk I am back07:11
raukadahGood morning everyone07:11
*** raukadah is now known as chandankumar07:11
*** amoralej|off is now known as amoralej07:20
*** jfrancoa has joined #oooq07:30
*** apetrich has quit IRC07:32
*** jpena|off is now known as jpena07:46
*** ykarel is now known as ykarel|lunch07:48
chandankumarmarios|rover: arxcruz zbr panda|off https://twitter.com/NeerDoseMonster/status/1183402343481528320 pycon india team07:48
marios|rover         nice chandankumar :D07:51
chandankumarmarios|rover: :-)07:52
*** fmount has quit IRC08:14
zbrchandankumar: super!08:22
*** akahat has joined #oooq08:31
*** fmount has joined #oooq08:33
*** ykarel|lunch is now known as ykarel08:33
*** dtantsur|afk is now known as dtantsur08:34
*** derekh has joined #oooq08:38
*** akahat has quit IRC08:40
*** akahat has joined #oooq08:43
*** chem has joined #oooq08:47
*** akahat has quit IRC08:48
*** akahat has joined #oooq08:50
*** bogdando has quit IRC08:59
*** bogdando has joined #oooq09:00
chandankumarmarios|rover: Is someone looking at full tempest timeout?09:08
marios|roverchandankumar: which one https://bugs.launchpad.net/tripleo/+bug/1847585 you mean or something else09:13
openstackLaunchpad bug 1847585 in tripleo "periodic-tripleo-ci-centos-7-standalone-full-tempest-master timeout on tempest tests "Failed to establish authenticated ssh connection to cirros@192.168.24.107"" [Critical,New]09:13
*** holser has quit IRC09:13
marios|roverchandankumar: am switching to that in a bit (trying to progress https://bugs.launchpad.net/tripleo/+bug/1847385 a bit now)09:13
openstackLaunchpad bug 1847385 in tripleo "rhel 8 master fails to create cm-local-ca.pem, Error: 'openssl pkcs12 " [Critical,Triaged]09:13
*** holser has joined #oooq09:14
*** skramaja has joined #oooq09:20
chandankumarmarios|rover: yes the same bug above timedout & ssh issue09:21
marios|roverchandankumar: ack am gonna check it too in a bit but if you have time please check09:22
chandankumarmarios|rover: yes, firing up the reproducer09:22
marios|roverchandankumar: ack interested to hear how you get on there i ran against rdo cloud last week for that and hit http://paste.openstack.org/raw/782729/ - r you using libvirt?09:23
chandankumarmarios|rover: it failed at Check image are uploaded on rdo-cloud then I re-run the reproducer it vanished09:24
marios|roverchandankumar: yes i hit that every time (see fpaste there)09:28
chandankumarmarios|rover: last time i hitted twice, then I updated the whole box and installed openstackclient and checked the images, it vanished, I still donot know why it happens09:31
*** apetrich has joined #oooq09:39
chandankumarmarios|rover: http://38.145.34.6:9000/t/tripleo-ci-reproducer/stream/f7085f3900994a139085667c8c7fcf48?logfile=console.log full tempest started here09:41
marios|roverthanks chandankumar09:46
chandankumarmarios|rover: does re-running the reproducer scripts worked?09:46
marios|roverchandankumar: for the upload images thing yeah usually rerun clears that but the second one was persistent haven't been on that env since last week tho09:48
ykarelchandankumar, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-full-tempest-master&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train09:55
zbrmarios|rover: let me know if you spot any selinux failures in the following hours, I made a new release of selinux shim library.09:57
marios|roverzbr: k thx10:02
*** soniya29 has quit IRC10:15
chandankumarmarios|rover: one more thing for reproducer , we need to set registry_login_enabled: false launcher-playbook.yaml otherwise it will fail10:25
*** surpatil has quit IRC10:25
*** surpatil has joined #oooq10:25
*** akahat has quit IRC10:31
*** surpatil has quit IRC10:44
*** surpatil has joined #oooq10:46
recheck[gh-selinux] 0.2.0 → https://github.com/pycontribs/selinux/releases/tag/0.2.010:46
ykarelmarios|rover, re. that issue test 'eqalto' not found, i had hit that earlier10:47
ykareland updating ansible had fixed it for me, can u check what jinja version u have10:47
ykarelas older jinja version could cause it10:47
*** dsneddon has quit IRC10:48
*** akahat has joined #oooq11:01
marios|roverykarel: ack makes sense will check it next time... previous occasions i've done a yum update before re-run but last time i did not. wondering if a new version is pulled in and only used on the 2nd run (or sthing)11:04
ykarelack11:05
*** udesale has quit IRC11:13
*** chem has quit IRC11:15
*** dsneddon has joined #oooq11:20
*** sanjayu_ has quit IRC11:25
*** chem has joined #oooq11:29
*** jpena is now known as jpena|lunch11:31
*** epoojad1 has quit IRC11:32
weshaymarios|rover, morning..11:35
weshaymarios|rover, /var/lib/certmonger/local/creds is def.. there :)11:35
weshayand works..11:35
weshayperhaps the file perm?11:35
weshayykarel, is unmesh around?11:38
ykarelweshay, nope, he left few minutes back11:38
ykareltowards home11:38
weshayk.. thanks11:38
weshayzbr, hey brotha.. any chance you can continue your work on docker login and iptables?11:39
marios|roverweshay: o/ its 1ctrl1comp instead of 3ctrl1comp trying to find why/where11:40
zbrweshay: hi, i need to remember what was the problem.... (goldfish memory)11:40
weshayykarel, have you seen CentOS7 ceph-luminous repos fail due to11:41
marios|roverweshay: see comment #5 not sure if thats the root but that is not right anyway11:41
weshayhttp://mirror.centos.org/altarch/7/storage/x86_64/ceph-luminous/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found11:41
weshayhttp://mirror.centos.org/centos-7/7/storage/x86_64/ceph-luminous/11:41
weshayykarel, the rum repos in centos have altarch for some reason11:41
zbrweshay: but mostly yes, i am close to bring centos-8 upstream to usable.11:41
ykarelweshay, in reproducer?11:41
weshayI think it's just the node.. prior to any repos we lay down.. but can bring it back up11:42
weshayzbr, there is a bug on it that should job your memory11:42
ykarelweshay, see /etc/yum/vars/contentdir11:42
ykarelsomehow ^^ is populated with altarch11:42
zbrsure11:43
weshaymarios|rover, ah.. I see.. weird11:46
weshay[root@subnode-0 ~]# cat /etc/yum/vars/contentdir11:48
weshayaltarch11:48
weshaychandankumar, arxcruz you guys see I scheduled a sync on skip list in 10min?11:51
weshayzbr, thanks11:51
* weshay gets coffee11:51
chandankumarweshay: yes11:54
*** surpatil has quit IRC11:56
*** surpatil has joined #oooq11:57
marios|roverweshay: https://github.com/rdo-infra/rdo-jobs/blob/dc708cd177cca5c91787259a67081fa033333ee6/zuul.d/ovb-jobs.yaml#L3012:03
*** amoralej is now known as amoralej|lunch12:06
*** dsneddon has quit IRC12:08
*** rfolco has joined #oooq12:08
*** rfolco is now known as rfolco|ruck12:12
weshaymarios|rover, very weird.. because the parent of the rhel8 job is periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master12:19
weshaymaybe the node def for rhel8 is fubar'd12:19
weshaymarios|rover, the bug may be in emit releases12:22
weshayor a bug rather12:22
marios|roverweshay: yeah parent of the periodic is ok but for the non periodic (which is whats failing) its parent is tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset00112:24
marios|roverweshay: testing for now https://review.rdoproject.org/r/#/c/23124/1/zuul.d/ovb-jobs.yaml12:24
weshayah rock12:25
chandankumarweshay: https://review.opendev.org/#/c/688377/12:26
*** jpena|lunch is now known as jpena12:27
weshayrfolco|ruck, you here?12:31
rfolco|rucko/12:31
weshayah see u :)12:31
weshaythanks12:31
*** dsneddon has joined #oooq12:31
*** dsneddon has quit IRC12:37
*** soniya29 has joined #oooq12:37
weshaychandankumar, is your fs039 still up?12:38
chandankumarweshay: nope12:38
weshayk12:38
chandankumarweshay: i deleted it today to setup full tempest timedout12:38
weshaychandankumar, ya.. I was trying to bring up full tempest last night.. hit that stupid ceph repo issue :(12:39
weshaychandankumar, thanks12:39
chandankumarweshay: I am seeing how where issue, but waiting on logger screwing me12:39
chandankumar*some12:39
chandankumarweshay: marios is hitting this http://paste.openstack.org/raw/782729/ it comes sometime and vanishes12:40
marios|roverchandankumar: no the images one i hit most times and yes thats transient but the 2nd error there i didn't get passed, but that was last thursd12:41
chandankumarweshay: do we have patch for ceph issue? /me is not sure I am also hitting some issue12:42
marios|roverrfolco|ruck: weshay panda|off whats the current status of promoter is it running now12:42
weshaymarios|rover, ya.. we had several promotions over the weekend12:51
weshaymarios|rover, notes are in the etherpad .. line 3412:51
marios|roverweshay: catch up in few mins thanks (worried about master almost red now)12:53
weshayme too :)12:58
*** amoralej|lunch is now known as amoralej12:58
*** Guest31368 has joined #oooq12:58
*** Guest31368 is now known as redrobot_13:00
*** redrobot_ is now known as redrobot13:01
rfolco|ruckmarios|rover, periodic-tripleo-ci-centos-7-standalone-full-tempest-master and fs020 are failing for master still... we need to fix these to get a promotion13:04
weshayrfolco|ruck, do you need me to lead this?13:05
weshayI actually can't really have a conflict13:05
*** surpatil has quit IRC13:05
marios|roverrfolco|ruck: ack chandankumar was checking that tempest one and i was switching to it now/after scrum13:05
weshayrfolco|ruck, ?13:06
rfolco|ruckweshay, scrum?13:07
rfolco|ruckweshay, maybe chandankumar can do that? I need to ruck13:07
weshayrfolco|ruck, maybe you can arrange that in advance next time :)13:07
*** dsneddon has joined #oooq13:08
rfolco|ruckweshay, that was agreed to me13:08
*** openstackstatus has quit IRC13:15
*** sanjayu_ has joined #oooq13:17
*** d0ugal has quit IRC13:28
*** surpatil has joined #oooq13:28
*** d0ugal has joined #oooq13:29
*** sanjayu_ has quit IRC13:45
*** epoojad1 has joined #oooq13:53
chandankumarrfolco|ruck: done13:53
chandankumarrfolco|ruck: How are you ruck?13:54
rfolco|ruckchandankumar, let me know if the sprint team needs me to conduct the scrums. Otherwise I assume you guys can handle it.13:54
chandankumarrfolco|ruck: I think team can answer that. :-)13:55
*** dsneddon has quit IRC13:55
chandankumarok, I need to sleep today early13:56
chandankumarlogging out see ya tomorrow13:56
*** chandankumar is now known as raukadah13:56
*** surpatil has quit IRC14:03
*** bogdando_ has joined #oooq14:04
*** yolanda__ has joined #oooq14:06
*** bogdando has quit IRC14:06
*** bogdando_ is now known as bogdando14:06
marios|roverweshay: any ideas mid-cycle ?14:07
marios|rover;)14:08
*** yolanda has quit IRC14:09
*** epoojad1 has quit IRC14:11
weshaymarios|rover, https://code.engineering.redhat.com/gerrit/#/c/182922/14:15
weshaymarios|rover, http://rhos-release.virt.bos.redhat.com:4040/rhosp14:16
*** dtrainor has quit IRC14:17
marios|roverrfolco|ruck: do you want to sync? have time now ? me and weshay on call if you can14:20
rfolco|ruckmarios|rover, yes I can, I am fixing rhel8 container push and rm -rf14:21
rfolco|ruckmarios|rover, channel?14:21
rfolco|ruckmarios|rover, where?14:22
dtantsurheads-up: now that https://review.opendev.org/#/c/679748/ has merged, you may want to run image building jobs on ironic-python-agent-builder14:22
dtantsurykarel: ^^14:22
marios|roverhttps://meet.google.com/fag-ppbp-beo rfolco|ruck14:23
ykarelrfolco|ruck, marios|rover  ^^14:24
*** dsneddon has joined #oooq14:28
weshaymarios|rover, ^14:30
weshayrfolco|ruck, marios|rover https://review.rdoproject.org/r/#/c/22878/3/.zuul.yaml14:36
*** dtantsur is now known as dtantsur|brb14:40
*** matbu has joined #oooq14:44
*** dtrainor has joined #oooq14:45
*** surpatil has joined #oooq14:47
weshaymarios|rover, rfolco|ruck ok.. login enabled..  I would ask that any work done on the old promoter should be done w/ the centos user tmux session14:50
marios|roverweshay: ack thx14:50
*** brault has quit IRC14:50
rfolco|ruckweshay, ack thx14:50
*** surpatil has quit IRC14:51
marios|roverweshay: see pvt when you have a sec please is that the right ip14:52
marios|roverweshay: ignore in now i had wrong IP14:53
marios|roverrfolco|ruck: afaics it is the same as https://github.com/rdo-infra/ci-config/blob/a1b2d56204a1eb6e94dfe5e9ba1d42dec7377eed/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini#L16 can you confirm?14:58
rfolco|ruckmarios|rover, I don't understand the question14:59
rfolco|rucksame as what14:59
marios|roverrfolco|ruck: as what's in the promoter config (we said the current promoter has diverged in promotion criteria14:59
marios|roverrfolco|ruck: i logged in and trying to find the differences15:00
rfolco|ruckok looking15:00
marios|roverrfolco|ruck: hmm git diff shows a difference looks to be mainly stein/train/rhel8 stuffs different15:02
*** udesale has joined #oooq15:12
*** dsneddon has quit IRC15:16
*** ykarel has left #oooq15:20
*** jtomasek has quit IRC15:38
*** dsneddon has joined #oooq15:40
*** soniya29 has quit IRC15:41
*** jtomasek_ has joined #oooq15:43
*** dsneddon has quit IRC15:47
*** ccamacho has quit IRC15:47
*** dsneddon has joined #oooq15:49
marios|roverrfolco|ruck: for 'fs020' you were referring to earlier did you mean https://bugs.launchpad.net/tripleo/+bug/1843259 periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky  - see comment 24 if you get a sec please wdyt should we move that to more frequent?15:51
openstackLaunchpad bug 1843259 in tripleo "Periodic rocky fs020 job fails tempest tests tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic and tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups " [Critical,In progress] - Assigned to Luke Short (ekultails)15:51
marios|roverweshay: i think we should be running jobs that are in criteria more frequently than wed/weekend, especially if they are red (see comment 5 if you get a chance plase https://bugs.launchpad.net/tripleo/+bug/1843259 )15:52
openstackLaunchpad bug 1843259 in tripleo "Periodic rocky fs020 job fails tempest tests tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic and tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups " [Critical,In progress] - Assigned to Luke Short (ekultails)15:52
rfolco|ruckmarios|rover, sec15:52
*** dsneddon has quit IRC15:55
rfolco|ruckmarios|rover, this --> https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master15:56
rfolco|ruckmarios|rover, time out (master)... the one you pointed is rocky15:56
marios|rover rfolco|ruck: tempest timeout is master there are two15:58
marios|roverrfolco|ruck: https://bugs.launchpad.net/tripleo/+bug/1847585 tempest timeout15:58
openstackLaunchpad bug 1847585 in tripleo "periodic-tripleo-ci-centos-7-standalone-full-tempest-master timeout on tempest tests "Failed to establish authenticated ssh connection to cirros@192.168.24.107"" [Critical,New]15:58
marios|roverrfolco|ruck: and fs20 rocky https://bugs.launchpad.net/tripleo/+bug/184325915:58
openstackLaunchpad bug 1843259 in tripleo "Periodic rocky fs020 job fails tempest tests tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic and tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups " [Critical,In progress] - Assigned to Luke Short (ekultails)15:58
marios|roverrfolco|ruck: my question was about the rocky one15:58
marios|roverrfolco|ruck: /me almost out for today ... will pickup from etherpad or email me for anything urgent... gonna focus on master promotion starting with that 1847585 tomorrow16:00
*** dsneddon has joined #oooq16:00
rfolco|ruckmarios|rover, the one I referred is the fs020 master time out, which seems unrelated to the tempest ones16:00
rfolco|ruckmarios|rover, ack, will have lunch now, ping if urgent16:00
marios|roverrfolco|ruck: do we have a bug for it?16:01
marios|roverrfolco|ruck: i don't see anything at https://etherpad.openstack.org/p/ruckroversprint17 grepping fs02016:01
rfolco|ruckmarios|rover, will file a new one, but this is happening since 10-05... we should have something16:02
marios|roverrfolco|ruck: ok i hav not seen that one then yet16:02
*** openstackstatus has joined #oooq16:02
*** ChanServ sets mode: +v openstackstatus16:02
marios|roverrfolco|ruck: i wasn't aware of it when you said fs020 i thought you meant ^ the one on the ether pad for rocky16:02
rfolco|ruckmarios|rover, we got one green job at 10-07, then failed again since16:02
marios|roverrfolco|ruck: ok i will chck that too if its master criteria please add to etherpad when you have it16:03
rfolco|ruckmarios|rover, ack16:03
marios|roverrfolco|ruck: ttyl /me shutdown thanks folco have a good day16:03
rfolco|ruckmarios|rover, you 2 thx16:03
*** marios|rover has quit IRC16:07
*** dtantsur|brb is now known as dtantsur16:08
*** brault has joined #oooq16:31
*** udesale has quit IRC16:34
*** chem is now known as chem|eod16:36
*** jpena is now known as jpena|off16:55
*** jtomasek_ has quit IRC16:58
*** jtomasek has joined #oooq16:59
*** derekh has quit IRC17:00
*** tesseract has quit IRC17:25
*** dtantsur is now known as dtantsur|afk17:36
*** akahat has quit IRC17:57
*** matbu has quit IRC18:13
*** akahat has joined #oooq18:15
*** holser has quit IRC18:27
weshayrfolco|ruck, have a lint issue https://review.rdoproject.org/r/#/c/23125/18:47
rfolco|ruckugh18:48
rfolco|ruckdammn18:48
rfolco|ruckweshay, its a false positive, not sure how to workaround this...18:58
rfolco|ruckhttps://github.com/ansible/ansible-lint/issues/49018:58
rfolco|ruckzbr, around?18:58
*** kopecmartin is now known as kopecmartin|off19:00
weshayrfolco|ruck, can we get you setup on the old reproducer for the tempest fs020 timeout19:02
weshaymy tenant is full of recreates atm19:02
rfolco|ruckweshay, fs020 master?19:03
weshayaye19:03
rfolco|ruckweshay, ok will do19:03
weshayrfolco|ruck, you'll probably need a pointer or two19:03
weshaywant to tmate?19:03
*** akahat has quit IRC19:04
rfolco|ruckweshay, isn't just get the deprecated script and run it ?19:04
rfolco|ruckweshay, does -a autorun work? sorry, don't recall19:11
*** holser has joined #oooq19:20
*** brault has quit IRC19:24
*** jfrancoa has quit IRC19:30
*** holser has quit IRC21:01
*** ldumont has joined #oooq21:04
*** LaurentDumont has quit IRC21:05
*** brault has joined #oooq21:46
*** holser has joined #oooq22:13
*** holser has quit IRC22:19
*** brault has quit IRC23:36
*** brault has joined #oooq23:39

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!