Tuesday, 2018-09-25

hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci- (3 more messages)00:47
*** weshay has joined #oooq01:57
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci- (3 more messages)02:47
*** skramaja has joined #oooq02:52
*** ykarel|away has joined #oooq03:39
*** ykarel|away is now known as ykarel03:49
*** udesale has joined #oooq03:57
*** ratailor has joined #oooq04:25
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages)04:47
*** agopi_ has joined #oooq04:50
*** ykarel has quit IRC04:51
*** agopi|training has quit IRC04:53
*** sanjayu_ has quit IRC04:53
*** ykarel has joined #oooq05:07
*** jaosorior has joined #oooq05:20
*** chkumar|off is now known as chkumar|ruck05:42
*** quique|rover|off is now known as quiquell|rover05:45
*** jfrancoa has joined #oooq06:02
*** jtomasek has joined #oooq06:27
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages)06:47
*** sanjayu_ has joined #oooq07:05
*** quiquell|rover is now known as quique|rover|brb07:10
*** holser_ has joined #oooq07:10
*** ykarel is now known as ykarel|lunch07:25
*** quique|rover|brb is now known as quiquell|rover07:42
*** agopi__ has joined #oooq07:46
*** agopi_ has quit IRC07:49
*** tosky has joined #oooq07:54
*** bogdando has joined #oooq07:58
bogdandoPTAL https://review.openstack.org/#/c/595265/ https://review.openstack.org/#/c/592996/08:05
bogdandoand https://review.openstack.org/#/c/576746/08:09
chkumar|ruckbrb08:09
*** ykarel|lunch is now known as ykarel08:35
*** dtantsur|afk is now known as dtantsur08:38
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages)08:47
*** panda|off is now known as panda09:01
*** sshnaidm has joined #oooq09:07
quiquell|roversshnaidm: http://dashboard-ci.tripleo.org/d/poOr-d0mk/ansible-exploration?orgId=109:25
sshnaidmquiquell|rover, nice09:27
sshnaidmquiquell|rover, let's sync about that tomorrow09:28
sshnaidmquiquell|rover, need to think how to do it more clear and "exploratable"09:29
*** sshnaidm is now known as sshnaidm|afk09:29
quiquell|roversshnaidm: sure, btw can you take a look at this ARA issue https://bugs.launchpad.net/tripleo/+bug/1794238 ?09:29
openstackLaunchpad bug 1794238 in tripleo "overcloud ARA is the same as undercloud ARA" [High,In progress] - Assigned to Quique Llorente (quiquell)09:29
quiquell|roverDon't want to spend time there, want to go back to timeouts09:30
sshnaidm|afkquiquell|rover, yeah, will handle this09:34
quiquell|roversshnaidm|afk: I need to gather mor info, have add a "offset" parameter to telegraf python to get old jobs09:34
sshnaidm|afkquiquell|rover, but we just started to collect data for undercloud/overcloud09:35
quiquell|roversshnaidm|afk: undercloud is all09:35
quiquell|roversshnaidm|afk: undercloud was in place before August09:35
quiquell|roversshnaidm|afk: and global ara is older09:35
quiquell|roversshnaidm|afk: I am using the json, not the influxdb line there (I don't need that though)09:36
sshnaidm|afkquiquell|rover, uh, ok09:36
*** sshnaidm|afk is now known as sshnaidm|off09:37
*** sshnaidm|off has quit IRC09:42
*** jfrancoa has quit IRC09:43
*** jfrancoa has joined #oooq09:46
*** sanjayu__ has joined #oooq10:15
*** sanjayu_ has quit IRC10:18
quiquell|roverchkumar|ruck: Can you cover CIX today too ?10:21
chkumar|ruckquiquell|rover: yes10:33
chkumar|ruckquiquell|rover: but I need some help on few bugs10:34
chkumar|ruckquiquell|rover: pulling the card10:34
quiquell|roverchkumar|ruck: Let's review CIX10:34
quiquell|roverchkumar|ruck: Going to add info to cards I know of10:34
chkumar|ruckquiquell|rover: https://trello.com/c/dfAprCuv/761-cixlp1794251tripleociproa-master-undercloud-reinstall-failed-with-invalid-selinux-context-errno-95-operation-not-supported10:34
chkumar|ruckquiquell|rover: https://bugs.launchpad.net/tripleo/+bug/179425110:36
openstackLaunchpad bug 1794251 in tripleo "[master] undercloud reinstall failed with invalid selinux context: [Errno 95] Operation not supported" [Critical,Triaged]10:36
* quiquell|rover checking10:36
chkumar|ruckDoes anyone tried standalone on fs028?10:37
chkumar|ruck*fedora 2810:41
pandachkumar|ruck: there are patches to make it work10:42
pandachkumar|ruck: testing and landing thos patches is part of the next sprint10:42
*** ratailor_ has joined #oooq10:43
*** ratailor has quit IRC10:44
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages)10:47
quiquell|roverchkumar|ruck: You have stuff about this https://trello.com/c/LNHvG9LW/760-cixlp1794228tripleociproa-queenspike-no-package-uwsgi-plugin-python2-available-during-barbican-api-container-build ?10:48
chkumar|ruckquiquell|rover: yes10:48
chkumar|ruckquiquell|rover: fixes are up10:48
quiquell|roverchkumar|ruck: Going to update the review for https://trello.com/c/ocIzDwCI/750-cixlp1793293tripleociproa-pike-promotion-failing-trying-to-use-ipv6-to-get-buildlogs10:49
quiquell|roverchkumar|ruck: To remove -1 of the fixing review10:50
chkumar|ruckquiquell|rover: ack10:50
chkumar|ruckquiquell|rover: need to check fs-16 and fs016 does telemetry tests are fixed or not10:52
quiquell|roverchkumar|ruck: CIX updated, you are covered now10:58
chkumar|ruckquiquell|rover: thanks :-)10:59
quiquell|roverchkumar|ruck: about timeouts, we still have just validation, we have a disable validations  review, and also created a tool to navigate ansible between builds10:59
quiquell|roverchkumar|ruck: what's up with fs016 ?10:59
chkumar|ruckquiquell|rover: yes10:59
chkumar|ruckquiquell|rover: fs016 tempest tests failed due to ssh timed out error, but I runned again in test-project but it passed11:00
chkumar|ruckquiquell|rover: I am waiting for next periodic run if it is visible then I need to investigation in ciner or neutron logs11:00
quiquell|roverchkumar|ruck: so it fails at periodics but not manually ?11:00
chkumar|ruckquiquell|rover: yes, just one time failure11:00
quiquell|roverchkumar|ruck: But we disable tempest there11:01
quiquell|roverchkumar|ruck: It's new temptes failure ?11:01
chkumar|ruckquiquell|rover: we disable telemetry test failure, not volume boot test11:01
quiquell|roverchkumar|ruck: now volume is failing ?11:01
quiquell|roverholy sh...11:01
chkumar|ruckvolume boot test cannot be diasbled it is the minmail tempest tests running in multiple fs11:01
quiquell|roverchkumar|ruck: so you talk about disable fs016 at promotions ?11:02
chkumar|ruckquiquell|rover: no once server is boot, user tryied to ssh into the instance then ssh happened, then agin they tried to ssh into another ssh that time it was timed out after 22 attempest11:02
chkumar|ruckquiquell|rover: nope, let's wait for next run11:02
chkumar|ruckquiquell|rover: that's why I have not opened a bug11:03
quiquell|roverchkumar|ruck: ack11:13
*** udesale has quit IRC11:17
*** panda is now known as panda|afk11:19
*** dtantsur is now known as dtantsur|bbl11:25
chkumar|ruckbrb back before CIX call11:26
*** jfrancoa has quit IRC11:42
*** jfrancoa has joined #oooq11:43
weshaychkumar|ruck, quiquell|rover howdy11:45
quiquell|roverweshay: ansible explorer http://dashboard-ci.tripleo.org/d/poOr-d0mk/ansible-exploration?orgId=111:46
quiquell|roverweshay: filter by timed_out for example http://dashboard-ci.tripleo.org/d/poOr-d0mk/ansible-exploration?orgId=1&var-influxdb_filter=job_result%7C%3D%7CTIMED_OUT11:47
quiquell|roverweshay: btw, what do you mean with validations not designed for upstream ci ?11:48
quiquell|roverweshay: the disabling patch is controversial11:48
*** quiquell|rover is now known as quique|rover|lch11:49
chkumar|ruckweshay: \o/11:49
chkumar|ruckweshay: panda|afk fs021 is completely broken https://bugs.launchpad.net/tripleo/+bug/179425811:49
openstackLaunchpad bug 1794258 in tripleo "[master][Rocky] fs021 periodic job is failing with different issues" [Critical,Triaged]11:49
chkumar|ruckneed some help here11:49
weshayquique|rover|lch,++12:02
weshayquique++12:02
weshaychkumar|ruck, feel free to comment12:03
chkumar|ruckweshay: today we have rocky promotion in the morning12:06
weshayI see that now :)12:06
weshaychkumar++12:06
chkumar|ruckweshay: there are few blockers for master fixes are in progress, we will get for master also12:07
weshayquique|rover|lch, nice job diagnosing the timeouts12:08
weshayquique|rover|lch, are you available after lunch to give me an update?12:08
ssbarnea|bkpfyi fc28 public image is now available on rdo, no need to upload it anymore.12:13
ykarelweshay, i looked a little to timeouts and it doesn't look related to overcloud deployment timing out12:19
ykarelit something to do with mistral/tripleoclient12:19
ykarelthe overcloudrc create just stucks and raises exception12:19
ykarelthere me be already more findings on ^^, i just looked so shared12:20
ykarelif not already involved, good to involve someone from mistral12:21
ykarelchkumar|ruck, quique|rover|lch in case u have noticed ^^12:22
weshaychkumar|ruck, quique|rover|lch do you guys have a bug on legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch12:23
*** trown|outtypewww is now known as trown12:23
weshaychkumar|ruck, quique|rover|lch and legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master12:24
ykarelweshay, chkumar|ruck quique|rover|lch and if the timeout issue is just master, i really doubt https://review.openstack.org/#/c/603802/12:24
chkumar|ruckweshay: checking12:25
chkumar|ruckweshay: https://bugs.launchpad.net/tripleo/+bug/1793073 it might be related to that12:28
openstackLaunchpad bug 1793073 in tripleo "[queens] fs01 noop job failed with Stderr: u'/usr/bin/ironic-inspector-rootwrap: Unauthorized command: systemctl start openstack-ironic-inspector-dnsmasq.service (no filter matched)" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)12:28
*** agopi__ has quit IRC12:34
*** ratailor_ has quit IRC12:37
weshaychkumar|ruck, thanks12:38
weshaychkumar|ruck, quique|rover|lch https://review.rdoproject.org/r/1644912:41
weshayssbarnea, make any progress on fedora?12:41
ssbarneaweshay: yep, just made a minor change to https://review.openstack.org/#/c/602492/12:42
ssbarneai wanted to ask you few things, do you have some time?12:42
ssbarneacan be BJ or irc12:43
weshayssbarnea, sure man12:43
weshaylet's blue12:43
weshayquicker12:43
ssbarneazzbj12:44
weshayhttps://bluejeans.com/u/whayutin/12:44
ssbarnea|bkpweshay please send me you meeting it, i cannot use the browse bj. sound is not working in it.12:46
ssbarnea|bkpstandalone works but i need number12:46
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades- (2 more messages)12:47
*** panda|afk is now known as panda12:49
weshayssbarnea|bkp, http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/libvirt/setup/overcloud/tasks/vars/libvirt_nodepool_vars.yml12:52
*** ssbarnea has quit IRC12:56
*** agopi__ has joined #oooq13:05
*** ssbarnea has joined #oooq13:06
*** quique|rover|lch is now known as quiquell|rover13:09
quiquell|roverweshay: I am up to talk about timeouts13:11
weshayquiquell|rover, ok I am ready in 3min13:12
quiquell|roverweshay: let me know when you are free13:12
*** skramaja has quit IRC13:12
ykarelquiquell|rover, is there some place where these updates(timeouts) are tracked, i too wanted to see13:13
weshayhttps://bluejeans.com/u/whayutin/13:13
ykarelin case i looked something wrong, just correct me13:16
*** vinaykns has joined #oooq13:18
quiquell|roverykarel: we have a job's explorer http://dashboard-ci.tripleo.org/d/FEdraO0ik/jobs-exploration?orgId=113:42
quiquell|roverykarel: from there you can filter by /update/ regex and result TIMED_OUT13:43
*** sanjayu__ has quit IRC13:43
*** saneax has joined #oooq13:43
quiquell|roverykarel: for example this http://dashboard-ci.tripleo.org/d/FEdraO0ik/jobs-exploration?orgId=1&var-influxdb_filter=result%7C%3D%7CTIMED_OUT&var-influxdb_filter=job_name%7C%3D~%7C%2Fupdates%2F13:43
quiquell|roverpanda: We have to watch this https://review.openstack.org/#/c/604706/13:47
quiquell|roverpanda: It's introducing new bash13:48
weshayanybody need a 1-1 that I've had to skip recently?13:54
weshayquiquell|rover, chkumar|ruck fyi.. so for bugs that block the gate..  the current best way to handle that .. is to create a bug w/ alert && promotion blocker13:55
weshayquiquell|rover, chkumar|ruck once the card is created in trello.. add all the upstream tags... tripleo-master, tripleo-rocky, tripleo-queens13:56
weshayI'll try to add that to the ruck / rover docs so it's more clear13:56
pandaquiquell|rover: ouch13:57
quiquell|roverpanda: yep, take a look at the review, we have to help with that13:58
weshaytripleo mtg14:00
weshayrfolco, you around buddy?14:01
rfolcoweshay, o/14:01
weshayrfolco, hey.. so let's chat for a minute during this tripleo mtg14:01
rfolcobj?14:01
weshayrfolco, https://bluejeans.com/411356779814:02
*** dtantsur|bbl is now known as dtantsur14:04
weshayrfolco, https://etherpad.openstack.org/p/tripleo-meeting-items14:12
ykarelquiquell|rover, currently in meeting, will check in few minutes14:18
chkumar|ruckweshay: sure14:19
*** quiquell|rover is now known as quique|rover|off14:20
weshayrfolco, https://drive.google.com/drive/u/1/folders/1duziyA5IpQLrzs4JncjKm6FHLOlDh5_-14:23
*** fultonj has joined #oooq14:27
*** fhubik has joined #oooq14:28
ykarelquique|rover|off, ok got that, is timeout issue clear(what caused that), can discuss tomorrow, i think overcloud deployment is completed, just overcloudrc creation is stuck, actually i wanted to know what releases are having issue and when exactly it started to narrow down the problem, if it started failing on 21st then tripleoclient patch seems issue, if not then can check other parts14:29
ykareland validation parts doesn't seem to be an actual issue that's just adding extra time to job14:30
ykarelbut yes good to clear for ci14:30
*** agopi__ is now known as agopi14:33
weshaymarios, panda https://review.openstack.org/#/c/600517/3114:35
mariosweshay: ack14:40
*** jfrancoa has quit IRC14:45
*** fhubik is now known as fhubik|brb14:46
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades- (2 more messages)14:47
*** ykarel is now known as ykarel|away14:57
weshayanyone know what is going on here? https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master/093617f/job-output.txt.gz#_2018-09-25_00_51_59_31524114:58
ykarel|awayweshay, following seems actual issue15:02
ykarel|away2018-09-25 00:52:48.274 | rsync: change_dir "/home/zuul/src/*/openstack/tripleo-upgrade" failed: No such file or directory (2)15:02
ykarel|away2018-09-25 00:52:48.366 | rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]non-zero return code15:02
*** kopecmartin is now known as kopecmartin|off15:11
*** udesale has joined #oooq15:12
*** chkumar|ruck is now known as chkumar|off15:17
weshayykarel|away, thanks!15:20
*** ykarel|away has quit IRC15:22
*** fhubik|brb is now known as fhubik15:23
*** jtomasek has quit IRC15:26
chemhi I've got "Failed to parse dlrn hash" when I re-run repo-setup in the standalone-upgrade jobs, anybody familiar with this kind of error ?15:28
weshaychem, hey.. let's see the log15:30
weshaychem, probably a blip15:30
chemweshay: http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/home/zuul/repo_setup_upgrade.log.txt.gz15:30
chemweshay: hum ... anyway it's the wrong hash15:32
chemweshay: according to http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/emit_releases_file.log15:32
*** sanjayu_ has joined #oooq15:33
chemweshay: I should be using standalone_target_hash not standalone_deploy_hash at that time, so I must be missing something15:34
*** saneax has quit IRC15:35
weshayseems ritht http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/emit_releases_file.log15:35
weshaypanda, can you help me look at this w/ chem15:36
weshayI think you are more familiar w/ it15:36
chemit is but this is the repo call before the upgrade so I should be using the target hash (48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456) not the deploy hash (e89450c44e41ec2ddada7909e63f1edc1aa1afdd)15:37
pandachem: maybe you're not passing the correct release on the second playbook call15:38
chempanda: is that not enough https://review.openstack.org/#/c/604706/5/playbooks/tripleo-ci/templates/toci_quickstart.sh.j2 ?15:38
pandachem: looking15:38
weshaychem, I wonder if container-prep is messing this up http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/emit_releases_file.log15:39
weshayoops15:39
weshayhttps://review.openstack.org/#/c/604736/5/roles/standalone-upgrade/meta/main.yml15:39
weshaynope15:39
chemweshay: hum copy/pasta, but in the end It's not used I believe15:40
weshaypanda, do we have no_log set of repo setup for any particular reason?15:41
pandaweshay: not sure, maybe credentials ?15:43
pandathat's usually why we used in the other tasks15:43
chempanda: weshay so the cli is indeed correct and use the target hash -> http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/job-output.txt.gz#_2018-09-25_14_46_19_10131215:46
pandachem: the invocation looks good, let me take a look at the logs15:47
weshaychem, panda the error is from config/release/tripleo-ci/master.yml line 4015:48
weshayif [[ -z "$rdo_dlrn" || -z "$tripleo_dlrn" ]]; then15:48
weshaychem, I supsect both curl's failed?15:48
chemweshay: yes15:48
weshayhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L4015:49
weshaypanda, chem maybe we could log that call vs.. silent15:49
weshayoh wait15:50
marios /win 715:51
weshaychem, panda we should see dlrn_repo_curl_errors.log15:51
weshaymaybe ~ is not working15:52
bogdandoPTAL https://review.openstack.org/#/c/595265/ https://review.openstack.org/#/c/592996/ and https://review.openstack.org/#/c/576746/15:52
chemweshay: panda I've got to call it a day, will look again later, let me know if you get something in the review https://review.openstack.org/#/c/604706/ or somewhere :)15:52
chemweshay: panda thanks for the help15:52
bogdandothose have +2 from you, weshay so please do not bother :)15:52
bogdandothere is also https://review.openstack.org/#/c/605021/15:53
pandabogdando: I have a question pending on 59526515:55
weshaychem, panda when in doubt.. http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/8815:55
weshaylol15:55
weshayvs..15:55
weshaychem, panda when in doubt.. http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/15:55
weshaychem, you hit an infra error15:55
weshaysuprise suprsie15:55
pandaheh15:56
chemweshay: ha ... how do we solve this ?15:56
weshaychem, I enjoy weeping15:56
pandaretry until you consume the hard disk heads15:56
chemah, nice :)  will do then ...15:56
weshaychem, you could bug infra.. but they will tell you to pound sand probably15:57
weshayrfolco, ssbarnea|bkp ^15:57
weshayrfolco, ssbarnea|bkp chkumar|off FYI.. opportuity to check on the status of upstream proxies http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/15:58
weshayworks15:58
weshaybut not http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/8815:58
chemweshay: hum, nice.  need to go15:58
chemweshay: panda thanks15:58
*** ykarel|away has joined #oooq15:59
*** ykarel|away is now known as ykarel15:59
pandaweshay: why 59/88 ?15:59
pandaweshay: I see the hash is 48/a015:59
weshaypanda, that is in the log15:59
weshayreally?15:59
weshayI just closed everything16:00
* weshay looks again16:00
weshaycurl --silent http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/88/59889fc7fc8e3390d9bd4f0bf01d3a8cdaab6785_fd06359b/delorean.repo -S16:00
weshaycurl --silent http://mirror.iad.rax.openstack.org:8080/rdo/centos7/e8/94/e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c4b405a1/delorean.repo -S16:00
weshayur right16:01
weshaypanda, same story16:01
weshayhttp://mirror.iad.rax.openstack.org:8080/rdo/centos7/e8/ works16:01
pandahttp://mirror.iad.rax.openstack.org:8080/rdo/centos7/48/a0/48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456/16:01
weshayhttp://mirror.iad.rax.openstack.org:8080/rdo/centos7/e8/94/ does not16:01
weshaypanda, look at the bottom of http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/home/zuul/repo_setup_upgrade.log.txt.gz#_2018-09-25_14_46_3116:01
weshayur wrong16:02
*** sshnaidm|off has joined #oooq16:02
pandaweshay: I was looking at  http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/job-output.txt.gz#_2018-09-25_14_46_19_10131216:02
ssbarnea|bkpi have no knowledge on how the mirrors are configured (or synced).16:03
weshayssbarnea|bkp, ya.. that is an area where we need help16:03
weshaya lot16:03
weshayssbarnea|bkp, would be a +100 to find out16:03
bogdandopanda: I cannot see a case one wants to chown not being a root. That would fail miserably with a permission denied, won't it?16:03
weshayhttps://specs.openstack.org/openstack-infra/infra-specs/specs/unified_mirrors.html16:03
bogdandoand when it won't fail, there is nothing to chown16:04
bogdandocorrection to chown as a+x16:04
bogdandoand being not a non_root_user16:05
*** fhubik has quit IRC16:06
bogdandosorry, I mean if non_root_user:non_root_user wants to chown something else, it needs to be sudo16:06
ssbarnea|bkp(reading now about it, i would personally have went for a squid approach, which doesn't even need sync and also avoids the need to mention which mirrors to use, you only need to configure the proxy to use)16:06
bogdandootherwise that is a pointless16:06
bogdandopanda: does that makes any sense?16:06
*** sshnaidm|off has quit IRC16:06
*** sshnaidm has joined #oooq16:06
bogdandopanda: ok, I got it, perhaps I should fix the name as well, as I needed to run that being non root!16:07
pandaweshay: so basically the repo setup role is ignoring the dlrn_hash we're passing, and is trying to download tripleo-current anyway ?16:07
weshaypanda, sorry?16:08
weshayssbarnea|bkp, rfolco join sf-ops please16:08
weshayinternal16:08
weshaypanda, ya.. https://trunk.rdoproject.org/centos7/e8/94/16:12
weshayand https://trunk.rdoproject.org/centos7/59/8816:12
weshaydon't exist16:12
weshayssbarnea|bkp, sorry it's not a mirror issue as you saw jpena point out16:12
ssbarneaok, seen.16:13
pandaweshay: e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c4b405a1 is the hash we're using in the standalone install , 48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456/ is the one to update to. The install works, the upgrade fails IIUC. Indeed look at http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/home/zuul/repo_setup.log.txt.gz16:14
pandabut in the repo_setup_upgrade we're using the same hashes, and for some reason they now fail.16:14
weshaypanda, k.. I guess he should be installing w/ previous_current_tripleo16:14
weshayand upgrading to... current_tripleo16:14
weshaychem, ^16:14
pandaweshay: HTEY ARE FOR ROCKY16:14
pandaweshay: that's why they are not there16:15
pandaweshay: the hash you're looking for is for rocky16:15
weshayyup16:15
weshayhttps://trunk.rdoproject.org/centos7-rocky/59/88/16:15
weshaymutha fucka16:15
weshaypanda, he's using a master release file though16:15
pandabut with the wrong hashehs16:16
pandaeven if we are passing the right ones16:16
bogdandopanda: commented16:16
pandabogdando: thanks sorry I've been cauht in the middle of upgrade scandal16:17
weshaypanda, http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/job-output.txt.gz#_2018-09-25_14_46_19_09166516:17
weshaypanda, there is a bug here https://trunk.rdoproject.org/centos7-master/48/a0/16:17
weshaysome where16:17
weshayssbarnea, rfolco for future ref: https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/manifests/mirror.pp#L316:18
weshaypanda, https://review.openstack.org/#/c/604706/5/scripts/emit_releases_file/emit_releases_file.py16:20
weshayshouldn't the release dict include more than just the hash16:21
pandaweshay: how are the extra-vars we pass in command line handled in the repo_setup ?16:21
weshaypanda, I think the issue is that emit releases does not emit a release16:22
*** jfrancoa has joined #oooq16:22
weshaylol16:22
weshayhrm.. ok.. emit releases does16:22
weshaybut we're not converting the data in http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/emit_releases_file.log16:23
weshayto the release vars we need16:23
pandaweshay: http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/releases.sh16:24
pandathis is the output16:24
weshayI guess that is baked into the relase file16:24
pandaI can't find any reference of dlrn_hash in the release file. We have all sorts of dlrn_*16:24
pandatags, paths, newest path and tags16:24
weshaypanda, k .. that looks good16:24
ssbarnea|bkpneed to go, will be back in ~45mins.16:25
*** udesale has quit IRC16:27
weshaypanda, I dont see a bug :(16:29
weshay+(./toci_quickstart.sh:140): echo '--extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/master.yml -e dlrn_hash=48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456 -e get_build_command=48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456'16:29
weshay2018-09-25 14:46:19.092084 | primary | --extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/master.yml -e dlrn_hash=48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456 -e get_build_command=48a0d56e4ba547ab10a35888138370fb1ec74a97_31a6245616:29
weshaythat seems right16:29
pandayes, that part is good16:30
pandathe part that fails is that we are not using those hashes when setting up the repo16:30
pandawe are using the hashes from rocky16:31
pandarepo_setup is using e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c on master16:31
pandaand fails16:31
pandathere's nothing passing the hash we specified in command line to the setup repo16:32
weshayhttp://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/repo-setup/tasks/get-dlrn-hash-newest.yml#n3016:32
weshaypanda, maybe we couldn't override release ?16:32
pandano mention od dlrn_hash in these tasks16:33
weshay     "release": "master",16:34
weshayhttp://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz16:34
pandarelease is correct, dlrn_hash is not.16:35
weshay        "dlrn_hash": "e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c4b405a1",16:35
weshay        "dlrn_hash_newest": "59889fc7fc8e3390d9bd4f0bf01d3a8cdaab6785_fd06359b",16:35
weshay        "dlrn_hash_path": "e8/94/e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c4b405a1",16:35
weshay        "dlrn_hash_path_newest": "59/88/59889fc7fc8e3390d9bd4f0bf01d3a8cdaab6785_fd06359b",16:35
weshayhttp://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz16:35
weshayyup16:35
weshaythat's the issue16:35
weshaypanda, maybe the order of the args?16:36
*** bogdando has quit IRC16:36
weshaypanda, I think I see why16:37
pandaweshay: if you look at where we are failing https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L38 we are taking the has value from dlrn_hash_path_newest or dlrn_hash_tag_newest16:37
weshaypanda, https://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/tasks/main.yml#L316:38
weshaypanda, the fact caching kills us here16:38
weshaypanda, those always need to run when it's an upgrade16:39
weshaychem, ^16:39
*** sanjayu_ has quit IRC16:40
pandathat may be correct. I still don't see how -e dlrn_hash becomes dlrn_hash_path_newest or dlrn_hash_tag_newest16:40
weshayupgrade_type: standalone-upgrade16:41
weshaypanda, http://codesearch.openstack.org/?q=dlrn_hash_path_newest&i=nope&files=&repos=16:41
weshaypanda, http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/repo-setup/tasks/get-dlrn-hash-newest.yml#n3216:41
*** jtomasek has joined #oooq16:42
weshaypanda, there is no ansible variable indicating it's an upgrade16:43
weshayit's in the zuul inventory16:44
weshayIMHO.. the upgrade playbooks should set some fact16:44
weshayabout what it's doing16:44
weshayhttp://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz16:44
*** dtantsur is now known as dtantsur|afk16:45
pandaI still can't find any place taht takes that dlrn_hash, and transform it into something that is usable by that curl16:45
pandalike in here http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/centosci/master-current-tripleo.yml#n416:45
pandawe are clearly using "dlrn_hash"16:46
pandacan't find anything similar for the curl command16:46
pandaor any path of transformation from dlrn_has to dlrn_hash_path16:46
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages)16:47
pandabut yeah, fact caching may be an issue here anyway, depending on how we deal with thos variables16:47
*** dtantsur|afk has quit IRC16:51
*** panda is now known as panda|off16:52
weshaypanda|off, chem /me pushing a patch to the playbook16:54
weshayinstead of including the repo-setup role16:54
weshayI think it should only run the repo-setup/task/get-dlrn-hash.yml16:54
*** dtantsur has joined #oooq16:58
*** dtantsur is now known as dtantsur|afk16:58
panda|offweshay: no repo for the new releases ? so you'll just download the containers with the new tags? is that enough ?16:59
weshaypanda|off, I think https://review.openstack.org/#/c/605149/1/playbooks/multinode-standalone-upgrade.yml17:00
weshaywill do the trick17:00
weshaylines 2-1117:01
weshaypanda|off, it's the logic in main.yml that is fucking it up17:01
*** ykarel is now known as ykarel|away17:01
weshaypanda|off, chem trying it again w/ new depends-on https://review.openstack.org/#/c/604706/17:02
*** trown is now known as trown|lunch17:08
*** vinaykns has quit IRC17:27
*** panda|off has quit IRC17:41
*** holser_ has quit IRC17:43
*** jfrancoa has quit IRC17:44
*** panda has joined #oooq17:45
*** panda is now known as panda|off17:47
*** ykarel has joined #oooq17:49
*** ykarel|away has quit IRC17:49
*** trown|lunch is now known as trown18:36
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike @  (3 more messages)18:47
*** hubbot1 has quit IRC18:52
chemweshay: hey, not sure, but I think include_roles is a tasks not a playbook parameter18:52
chemweshay: meaning this new patchset should fail18:52
*** gouthamr has quit IRC18:52
*** dmellado has quit IRC18:53
chemweshay: but I silent myself and see how it goes, maybe I didn't get it right :)18:53
weshaychem, the main.yml in repo-setup is wonky18:58
chemweshay: oki, I'm pushing yet another review I nearly sure we have to use include_role at the tasks level so in the yum upgrade should do19:00
chemweshay: unless you're 100% positive this should work as is19:01
chemweshay: or I'll wait tomorrow, I'm a bit wonky myself.19:01
weshaychem, if you just take the two tasks you'll be fine19:02
weshayyou can not use main.yml19:02
*** ykarel has quit IRC19:05
*** gouthamr has joined #oooq19:12
rfolcoweshay, can you pls quick look at https://review.openstack.org/#/c/594511 ? why gate job did not trigger19:12
*** ssbarnea|bkp has quit IRC19:14
*** ssbarnea|bkp has joined #oooq19:14
chemweshay: ack19:32
*** trown is now known as trown|outtypewww20:03
*** gouthamr_ has joined #oooq20:05
*** dmellado has joined #oooq20:07
*** chem has quit IRC20:26
*** chem has joined #oooq20:26
*** gouthamr has quit IRC20:38
*** gouthamr_ is now known as gouthamr20:39
*** chem has quit IRC20:48
*** jtomasek has quit IRC21:00
*** hubbot1 has joined #oooq21:12
*** agopi has quit IRC21:52
*** agopi has joined #oooq22:32
*** agopi has quit IRC22:37
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates @ https://review.openstack.org/604298, stable/pike: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike @  (3 more messages)22:48
*** tosky has quit IRC23:11
*** agopi has joined #oooq23:22

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!