| *** rlandy is now known as rlandy|afk | 00:29 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 00:43 |
|---|---|---|
| *** rlandy|afk is now known as rlandy | 01:36 | |
| *** brault has joined #oooq | 02:34 | |
| *** brault has quit IRC | 02:34 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 02:43 |
| *** agopi has quit IRC | 02:50 | |
| *** agopi has joined #oooq | 02:50 | |
| *** agopi has quit IRC | 03:07 | |
| *** d0ugal has quit IRC | 03:17 | |
| *** d0ugal_ has joined #oooq | 03:17 | |
| *** dsneddon has quit IRC | 03:23 | |
| *** dsneddon has joined #oooq | 03:24 | |
| *** rlandy has quit IRC | 03:27 | |
| *** skramaja has joined #oooq | 03:38 | |
| *** agopi has joined #oooq | 03:42 | |
| *** ykarel_ has joined #oooq | 04:08 | |
| *** jaganathan has joined #oooq | 04:18 | |
| *** hubbot has quit IRC | 04:22 | |
| *** dmellado has quit IRC | 04:23 | |
| *** links has joined #oooq | 04:55 | |
| *** ratailor has joined #oooq | 04:56 | |
| *** ratailor has quit IRC | 05:05 | |
| *** saneax has joined #oooq | 05:26 | |
| *** tcw has quit IRC | 05:27 | |
| *** tcw has joined #oooq | 05:27 | |
| *** ykarel__ has joined #oooq | 05:50 | |
| *** ykarel_ has quit IRC | 05:53 | |
| *** pgadiya has joined #oooq | 06:02 | |
| *** pgadiya has quit IRC | 06:02 | |
| *** jfrancoa has joined #oooq | 06:06 | |
| *** agopi has quit IRC | 06:24 | |
| *** gkadam has joined #oooq | 06:38 | |
| *** jfrancoa has quit IRC | 06:38 | |
| *** holser__ has joined #oooq | 06:39 | |
| *** kopecmartin has joined #oooq | 06:42 | |
| *** ykarel_ has joined #oooq | 06:52 | |
| *** ykarel__ has quit IRC | 06:55 | |
| *** dmellado has joined #oooq | 07:04 | |
| *** brault has joined #oooq | 07:07 | |
| *** ykarel_ is now known as ykarel | 07:07 | |
| *** ratailor has joined #oooq | 07:10 | |
| *** tesseract has joined #oooq | 07:17 | |
| *** bogdando has joined #oooq | 07:20 | |
| *** tosky has joined #oooq | 07:28 | |
| *** amoralej|off is now known as amoralej | 07:32 | |
| *** florianf has joined #oooq | 07:33 | |
| *** jfrancoa has joined #oooq | 07:35 | |
| *** dmellado has quit IRC | 07:42 | |
| *** dmellado has joined #oooq | 07:42 | |
| *** ccamacho has joined #oooq | 07:51 | |
| *** jbadiapa_ is now known as jbadiapa | 07:54 | |
| *** d0ugal_ has quit IRC | 08:04 | |
| *** d0ugal has joined #oooq | 08:04 | |
| *** jfrancoa has quit IRC | 08:16 | |
| *** ykarel is now known as ykarel|lunch | 08:47 | |
| *** panda|off is now known as panda | 08:54 | |
| *** ratailor_ has joined #oooq | 09:06 | |
| *** ratailor has quit IRC | 09:10 | |
| *** jtomasek has quit IRC | 09:12 | |
| *** jtomasek_ has joined #oooq | 09:12 | |
| *** hubbot has joined #oooq | 09:14 | |
| *** quiquell|off is now known as quiquell | 09:14 | |
| sshnaidm|off | quiquell, looking at https://review.openstack.org/#/c/575242/ again - which job should we look at? | 09:18 |
| *** quiquell is now known as quiquell|rover | 09:18 | |
| quiquell|rover | sshnaidm|off: The tripleo-common change have to be in queens not master | 09:19 |
| quiquell|rover | sshnaidm|off: You can not do a n -> n + 1 with a change in master | 09:20 |
| quiquell|rover | sshnaidm|off: Sorry... let me do it in queens. | 09:20 |
| sshnaidm|off | quiquell|rover, I have one in queens too.. let's do: https://review.openstack.org/#/c/575244/ | 09:20 |
| sshnaidm|off | quiquell|rover, check please if everything is correct now: https://review.openstack.org/#/c/575244/ | 09:22 |
| quiquell|rover | sshnaidm|off: Now I think it's ok... :-) | 09:23 |
| quiquell|rover | sshnaidm|off: I am not very reliable | 09:23 |
| sshnaidm|off | quiquell|rover, ok, let's see(\ | 09:23 |
| sshnaidm|off | quiquell|rover, no worries :) it's upgrades | 09:23 |
| quiquell|rover | The new job is only configured for queens | 09:23 |
| quiquell|rover | sshnaidm|off: You don't want to know hay many [DNM] I have to test this fuc... | 09:24 |
| sshnaidm|off | quiquell|rover, believe you.. | 09:24 |
| quiquell|rover | ykarel|lunch: Damn there is no queens promotion... | 09:25 |
| *** ykarel|lunch is now known as ykarel | 09:25 | |
| ykarel | quiquell|rover, hmm :( | 09:25 |
| quiquell|rover | ykarel: Also the promoter is not working... do we have deactivate it ? | 09:25 |
| ykarel | there were some random failures | 09:25 |
| quiquell|rover | ykarel: and the queues are high... what a Friday | 09:25 |
| ykarel | quiquell|rover, hmm was about to ask about the promoter | 09:25 |
| ykarel | as master is not promoter | 09:26 |
| ykarel | as master is not promoted | 09:26 |
| quiquell|rover | ykarel: Is not running for any release | 09:26 |
| ykarel | quiquell|rover, can you check and run it for master, we need a fix for phase1 | 09:26 |
| quiquell|rover | ykarel: Running on master now | 09:32 |
| quiquell|rover | ykarel: it will do queens, then pike and finally ocata | 09:32 |
| *** zoli is now known as zoli|lunch | 09:35 | |
| quiquell|rover | ykarel: master promotion to phase1 failed at 'weirdo-master-promote-puppet-openstack-scenario002', 'weirdo-master-promote-packstack-scenario003' | 09:38 |
| quiquell|rover | ykarel: The puppet fixes are in place ? | 09:38 |
| ykarel | quiquell|rover, yes they are in place, next run should pass both packstack and poi failures | 09:42 |
| quiquell|rover | ykarel: queens failing at Skipping promotion of tripleo-ci-testing to current-tripleo, missing successful jobs: ['periodic-multinode-1ctlr-featureset016', 'periodic-multinode-1ctlr-featureset017', 'periodic-multinode-1ctlr-featureset030', 'periodic-multinode-1ctlr-featureset010', 'periodic-ovb-1ctlr_1comp-featureset002', 'periodic-multinode-1ctlr-featureset018', 'periodic-multinode-1ctlr-featureset019', | 09:43 |
| quiquell|rover | 'periodic-ovb-1ctlr_1comp-featureset020', 'periodic-ovb-3ctlr_1comp-featureset001', 'periodic-ovb-3ctlr_1comp-featureset035'] | 09:43 |
| quiquell|rover | puff | 09:43 |
| quiquell|rover | Let's go one by one | 09:43 |
| quiquell|rover | sshnaidm|off: Why is there a 24h timeshift at failed gates in the RR cockpit ? | 09:53 |
| quiquell|rover | sshnaidm|off: Do yo mean latest 24h hours instead ? | 09:53 |
| sshnaidm|off | quiquell|rover, yeah, latest | 09:53 |
| quiquell|rover | sshnaidm|off: I will change it | 09:53 |
| sshnaidm|off | quiquell|rover, thanks | 09:54 |
| quiquell|rover | sshnaidm|off: np | 09:54 |
| quiquell|rover | Any core for a +2 +1w here https://review.openstack.org/#/c/575407/ ? | 10:01 |
| quiquell|rover | rasca: Any idea on thi ? /workspace/tripleo-quickstart-gate-master-delorean-quick-basic | 10:05 |
| quiquell|rover | 17:54:46 + /home/jenkins/workspace/tripleo-quickstart-gate-master-delorean-quick-basic/bin/cico node get --arch x86_64 --release 7 --count 1 --retry-count 6 --retry-interval 60 -f csv | 10:05 |
| quiquell|rover | 17:54:47 string indices must be integers | 10:05 |
| quiquell|rover | 17:54:47 Build step 'Execute shell' marked build as failure | 10:05 |
| quiquell|rover | "string indices must be integers" | 10:05 |
| *** dtantsur|afk is now known as dtantsur | 10:06 | |
| rasca | quiquell|rover, can you give me more context on this? | 10:08 |
| quiquell|rover | rasca: https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/5982/consoleFull | 10:10 |
| quiquell|rover | arxcruz: I am testing your object-storage change | 10:10 |
| quiquell|rover | arxcruz: To check that if fixed the periodic job | 10:10 |
| arxcruz | quiquell|rover: ok | 10:11 |
| rasca | quiquell|rover, "my" object storage change? | 10:11 |
| arxcruz | rasca: no, mine :P | 10:12 |
| quiquell|rover | rasca: arxcruz change | 10:18 |
| rasca | quiquell|rover, yeah yeah, sorry, I misread IRC | 10:19 |
| ykarel | quiquell|rover, master not promoted as the random issue which we saw in queens also happened in master fs035 | 10:27 |
| ykarel | it's seen multiple time, so good to report | 10:27 |
| ykarel | and escalate | 10:27 |
| ykarel | overcloud prep image failing:- Error: IPMI call failed: power status | 10:28 |
| quiquell|rover | ykarel: Going to recheck | 10:29 |
| ykarel | quiquell|rover, so i think it would be good to stick to a hash for queens and master and promote | 10:29 |
| ykarel | weshay|ruck, ^^ | 10:30 |
| quiquell|rover | ykarel: You mean ignoring job failures and promote ? | 10:30 |
| ykarel | quiquell|rover, no | 10:30 |
| ykarel | i mean running the jobs against same hash as last one, as current issue is random one, so we get promotion, and fix the issue side by side | 10:32 |
| ykarel | quiquell|rover, i mean https://review.rdoproject.org/r/#/c/12712/ and https://review.rdoproject.org/r/#/c/10387/ | 10:34 |
| ykarel | just change hash from last promotion run ^^ | 10:34 |
| quiquell|rover | ykarel: You always have those tricks :-) | 10:36 |
| ykarel | next run is in 1.5 hour | 10:37 |
| ykarel | so good to decide before than | 10:38 |
| quiquell|rover | ykarel: Is there an RDO issue already open for this ? | 10:39 |
| ykarel | quiquell|rover, RDO issue? | 10:39 |
| quiquell|rover | ykarel: If we want to pin we need cores, for that | 10:42 |
| ykarel | quiquell|rover, you can propose, we can request core | 10:42 |
| quiquell|rover | ykarel: It can happen that we pin it and job fails again ? | 10:43 |
| ykarel | quiquell|rover, yes but we would need only fs35 to pass this time for master | 10:43 |
| ykarel | and 2 jobs for queens | 10:43 |
| ykarel | for promotion | 10:43 |
| quiquell|rover | panda: You there ? | 10:44 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 10:45 |
| *** dtantsur is now known as dtantsur|afk | 10:45 | |
| panda | quiquell|rover: more or less | 10:46 |
| quiquell|rover | panda: We need some cores to pin the master and queens hash, to rerun periodic on it | 10:47 |
| quiquell|rover | panda: We have transitory issues not releated to them | 10:47 |
| quiquell|rover | panda: Similar to this https://review.rdoproject.org/r/#/c/12712/4 | 10:47 |
| quiquell|rover | panda: and this https://review.rdoproject.org/r/#/c/10387/1 | 10:48 |
| *** zoli|lunch is now known as zoli | 10:50 | |
| *** brault has quit IRC | 10:50 | |
| quiquell|rover | ykarel: Do you have the queens failing build around ? | 10:51 |
| ykarel | quiquell|rover, also need a fix for:- No more IP addresses available on network 1f6dd7c9-1c87-4310-a1c3-bd7c0346d6ad. | 10:51 |
| ykarel | pinged #rhos-ops for this | 10:51 |
| ykarel | quiquell|rover, https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/298/consoleText | 10:53 |
| panda | quiquell: lookong | 10:56 |
| quiquell|rover | ykarel: the PROMOTE_NAME for both should be current-tripleo-rdo ? | 10:57 |
| ykarel | quiquell|rover, yes | 10:57 |
| ykarel | quiquell|rover, looks like promote name is consistent, | 10:59 |
| ykarel | checking again | 10:59 |
| ykarel | quiquell|rover, yes make it consistent | 11:00 |
| quiquell|rover | ykarel: but we are promoting to phase 1 | 11:01 |
| ykarel | quiquell|rover, we are getting hash from consistent repo in tripleo-upstream/get-hash.sh | 11:02 |
| ykarel | and run tripleo ci jobs | 11:02 |
| quiquell|rover | ykarel, panda: https://review.rdoproject.org/r/14258 | 11:02 |
| quiquell|rover | Looking comments from previous patches they say something about containers registry | 11:03 |
| *** yolanda has quit IRC | 11:03 | |
| *** yolanda has joined #oooq | 11:06 | |
| ykarel | quiquell|rover, we just need the correct hashes, i think ok to rerun the containers build | 11:11 |
| ykarel | quiquell|rover, posted some comments | 11:11 |
| quiquell|rover | ykarel: What other stuff did you say with networking | 11:12 |
| ykarel | quiquell|rover, discussion on #rhos-ops | 11:13 |
| quiquell|rover | ykarel: | 11:14 |
| quiquell|rover | ykarel: Let me check nodepool | 11:18 |
| quiquell|rover | panda: to cleanup nodepool the script is ovb-tenant-cleanup.sh ? | 11:20 |
| ykarel | quiquell|rover, ack | 11:21 |
| quiquell|rover | panda, ykarel: executing cleanup | 11:22 |
| ykarel | quiquell|rover, ack | 11:23 |
| quiquell|rover | ykarel: network problem could be releated to this too | 11:23 |
| quiquell|rover | ykarel: All those expired stacks are getting networks | 11:24 |
| ykarel | quiquell|rover, no idea, also report on #rhos-ops | 11:24 |
| quiquell|rover | ykarel: When is the next run ? | 11:25 |
| ykarel | quiquell|rover, in 45 minutes | 11:25 |
| ykarel | 12:10 UTC | 11:25 |
| quiquell|rover | ykarel: cool | 11:26 |
| *** brault has joined #oooq | 11:30 | |
| *** anande has joined #oooq | 11:38 | |
| quiquell|rover | panda: Now the promoter bash script is running under systemd | 11:40 |
| quiquell|rover | panda: This morning it was down | 11:40 |
| weshay|ruck | quiquell|rover, it was running out of the tmux | 11:45 |
| quiquell|rover | weshay|ruck: Now it's under systemd | 11:47 |
| quiquell|rover | weshay|ruck: It will survice reboots, the tmux was a fix that have survice for toooo long | 11:47 |
| ykarel | weshay|ruck, panda https://review.rdoproject.org/r/#/c/14258/ | 11:48 |
| weshay|ruck | ok, here's what I saw.. overcloud images had promoted, containers and dlrn was not promoted | 11:48 |
| quiquell|rover | ykarel, weshay|ruck: Nodepool cleanup finished | 11:49 |
| quiquell|rover | Let's see if now the periodic jobs work | 11:49 |
| quiquell|rover | weshay|ruck: We have to run nodepool cleanup at crontab or similar (rlandy agrees with that) | 11:50 |
| quiquell|rover | weshay|ruck: Or at least have an alert on it | 11:50 |
| chandankumar | arxcruz: please have a look at sprint 15 cards | 11:51 |
| chandankumar | I and kopecmartin have reviewed it | 11:51 |
| weshay|ruck | quiquell|rover, k.. thanks for taking care of it | 11:52 |
| weshay|ruck | quiquell|rover, probably need to do the same w/ telgraf | 11:52 |
| quiquell|rover | weshay|ruck: Will do | 11:52 |
| quiquell|rover | weshay|ruck: btw, upstream gate failing here http://logs.openstack.org/13/571613/18/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/9a52ece/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 11:52 |
| quiquell|rover | "ERROR running rpm query in container:" | 11:53 |
| quiquell|rover | weshay|ruck: also Queued are big again, did they activate the containerized stuff ? | 11:54 |
| weshay|ruck | quiquell|rover, it wasn't fully merged last night | 11:56 |
| * weshay|ruck looks | 11:56 | |
| weshay|ruck | quiquell|rover, I like having a tmux session there | 11:57 |
| weshay|ruck | :) | 11:57 |
| weshay|ruck | :P | 11:57 |
| quiquell|rover | weshay|ruck: I have kill them, now that also telegraf is up | 11:58 |
| weshay|ruck | k | 11:58 |
| weshay|ruck | https://review.openstack.org/#/c/571613/ | 11:58 |
| weshay|ruck | is the problem | 11:58 |
| weshay|ruck | this keeps dying in the gate | 11:58 |
| quiquell|rover | weshay|ruck: You mean upstream gates failing or enququed time ? | 12:00 |
| weshay|ruck | quiquell|rover, fix this up and we'll merge it https://review.rdoproject.org/r/#/c/13622/ | 12:04 |
| weshay|ruck | couple things to-do | 12:04 |
| quiquell|rover | weshay|ruck: Will add the telegraf config there too | 12:05 |
| quiquell|rover | weshay|ruck: +2 +1w https://review.openstack.org/#/c/575407/ | 12:07 |
| *** trown|outtypewww is now known as trown | 12:09 | |
| *** sanjay__u has quit IRC | 12:13 | |
| *** skramaja has quit IRC | 12:17 | |
| *** rlandy has joined #oooq | 12:19 | |
| quiquell|rover | weshay|ruck: updated https://review.rdoproject.org/r/13622 | 12:23 |
| *** ratailor_ has quit IRC | 12:23 | |
| quiquell|rover | weshay|ruck, trown, rlandy: Unit Test for promoter https://review.rdoproject.org/r/#/c/14084/ | 12:26 |
| rlandy | oh cool | 12:27 |
| trown | quiquell|rover: ya I need to download the update and retest... we dont run the tests on the patch :( | 12:27 |
| quiquell|rover | trown: The tox-py27 job https://review.rdoproject.org/r/#/c/14214/ | 12:27 |
| quiquell|rover | trown: But zuul 2.5 doesn't suppor running jobs not comitted :-( | 12:27 |
| *** tcw1 has joined #oooq | 12:31 | |
| *** tcw has quit IRC | 12:31 | |
| *** jaganathan has quit IRC | 12:32 | |
| quiquell|rover | rlandy: We have to put the nodepool cleaning script at crontab or something | 12:35 |
| rlandy | quiquell|rover: sure | 12:36 |
| rlandy | should be safe enough | 12:36 |
| quiquell|rover | rlandy: What tripleo tenant machine is the best place for it ? | 12:37 |
| rlandy | quiquell|rover: you'd need to have access to nodepool | 12:37 |
| rlandy | so only that tenant afaik | 12:37 |
| rlandy | we can't go from the infra tenant to nodepool | 12:38 |
| quiquell|rover | rlandy: Bad practice to copy the bashrc to whatever machine it runs on ? | 12:38 |
| rlandy | quiquell|rover: bashrc? openrc file I think you're referring to | 12:39 |
| rlandy | and yeah!! | 12:39 |
| rlandy | they will kill us | 12:39 |
| *** ykarel has quit IRC | 12:39 | |
| rlandy | the tenant is public | 12:39 |
| quiquell|rover | rlandy: Yep openrc :-) | 12:39 |
| *** ykarel has joined #oooq | 12:39 | |
| rlandy | quiquell|rover: we may have better luck on a internal server | 12:39 |
| rlandy | where we could put the file | 12:39 |
| rlandy | if we secured it | 12:39 |
| rlandy | like one of the osp hardware spreadsheet boxes | 12:40 |
| rlandy | ssh denied to anyone outside of the group | 12:40 |
| quiquell|rover | rlandy: Sounds like a good plan | 12:41 |
| rlandy | quiquell|rover; I am just trying to debug the last rhos-13 run - but I'll look for an open box and we yo can try it out | 12:41 |
| rlandy | check with panda re: security on that machine | 12:42 |
| rlandy | he's good at that stuff | 12:42 |
| quiquell|rover | rlandy: Will do | 12:42 |
| quiquell|rover | rlandy: btw myoung|off give us a downstream task https://trello.com/c/Sje1ZmNx/808-remove-internal-osp0-jobs-for-osp11-now-that-its-eol?menu=filter&filter=label:Sprint%2014%20CI | 12:42 |
| quiquell|rover | rlandy: It's just a matter or delete the project from jenkins ? | 12:43 |
| rlandy | quiquell|rover: yes | 12:43 |
| rlandy | just remove that option from jjb | 12:43 |
| rlandy | give me a sec, I'll send you the right link | 12:43 |
| quiquell|rover | rlandy: ok thanks | 12:44 |
| weshay|ruck | quiquell|rover, dashboard is down | 12:44 |
| * quiquell|rover checking | 12:45 | |
| quiquell|rover | weshay|ruck: I think RDO is down | 12:45 |
| *** rlandy_ has joined #oooq | 12:46 | |
| quiquell|rover | [ellorent@quiquell ~]$ openstack server list | 12:46 |
| quiquell|rover | An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-b17d441d-4ef0-432b-9b20-3a158af9c2f7) | 12:46 |
| quiquell|rover | weshay|ruck: Maybe we can put it in another tenant | 12:46 |
| *** dmellado_ has joined #oooq | 12:46 | |
| *** hubbot has quit IRC | 12:47 | |
| quiquell|rover | weshay|ruck: Cannot auth to my tenant... | 12:47 |
| rlandy_ | quiquell|rover I got disconnected from irc :( ... did you get that link? | 12:47 |
| quiquell|rover | rlandy_: nope | 12:47 |
| rlandy_ | rejoining | 12:48 |
| *** rlandy_ has quit IRC | 12:48 | |
| *** rlandy has quit IRC | 12:48 | |
| *** rlandy_ has joined #oooq | 12:48 | |
| rlandy_ | quiquell|rover: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/jenkins/jobs/tripleo-quickstart/promote-ospd-11-puddle.yml | 12:48 |
| *** dmellado has quit IRC | 12:49 | |
| *** rlandy_ is now known as rlandy | 12:49 | |
| *** myoung|off is now known as myoung | 12:49 | |
| quiquell|rover | weshay|ruck: Add an large instance at tripleo tenant and I will install the dashboard there | 12:49 |
| myoung | rlandy, quiquell|rover - not there's nothing urgent about removing the osp11 jobs, I just created the card so it's not lost. the job is disabled and it's extremly low priority...just housecleaning / low hanging fruit for a quiet day | 12:49 |
| myoung | (if we still have those :) ) | 12:50 |
| *** dougbtv_ has joined #oooq | 12:53 | |
| ykarel | quiquell|rover, so hash for master/queens picked up, somehow got lucky with queens as condition for queens didn't worked up | 12:53 |
| ykarel | [' queens = queens -a tripleo-ci-testing = consistent ']' | 12:53 |
| quiquell|rover | ykarel: damn I knew it :-) | 12:54 |
| weshay|ruck | quiquell|rover, please open an alert tracker bug on lp | 12:54 |
| ykarel | also there is outage in rdo-cloud again | 12:54 |
| weshay|ruck | tags = alert,promotion-blocker | 12:54 |
| quiquell|rover | weshay|ruck: With what ? | 12:57 |
| weshay|ruck | tracker-bug, rdo-cloud is down | 12:57 |
| weshay|ruck | just to inform | 12:57 |
| quiquell|rover | weshay|ruck: Ok | 12:58 |
| *** anande has quit IRC | 12:58 | |
| quiquell|rover | weshay|ruck: https://bugs.launchpad.net/tripleo/+bug/1777130 | 13:00 |
| openstack | Launchpad bug 1777130 in tripleo "RDO cloud is down" [Critical,New] - Assigned to Quique Llorente (quiquell) | 13:00 |
| quiquell|rover | arxcruz: RDO outage in the middle of testing your stuff with my reproducer :-( | 13:00 |
| myoung | chandankumar, kopecmartin, arxcruz, weshay|ruck: o/ sprint 15 planning for tempest squad start shortly | 13:00 |
| arxcruz | quiquell|rover: outch | 13:02 |
| myoung | weshay|ruck: are you joining us for planning? | 13:04 |
| weshay|ruck | I can't today sorry.. in other mtgs | 13:04 |
| marios | weshay|ruck: fyi filed this just now https://bugs.launchpad.net/tripleo/+bug/1777132 | 13:05 |
| openstack | Launchpad bug 1777132 in tripleo "queens branch tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades is broken" [High,Triaged] - Assigned to Marios Andreou (marios-b) | 13:05 |
| marios | weshay|ruck: (the queens scen0 broken upgrade job) | 13:05 |
| marios | weshay|ruck: spent a while digging but gonna context switch now (do the homework panda gave me :) ) so wanted to capture the info | 13:05 |
| marios | weshay|ruck: i did look but couldn't find an existing bug | 13:05 |
| myoung | weshay|ruck: ack | 13:06 |
| marios | quiquell|rover: fyi too :) https://bugs.launchpad.net/tripleo/+bug/1777132 | 13:06 |
| openstack | Launchpad bug 1777132 in tripleo "queens branch tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades is broken" [High,Triaged] - Assigned to Marios Andreou (marios-b) | 13:06 |
| marios | quiquell|rover: weshay|ruck (sorry i realise this bit is the ruck/rover job i mean filing the bugs but i'm newb and didn't think someone was looking at this yet) | 13:06 |
| marios | + i don't really have much better to do yet :) | 13:07 |
| quiquell|rover | marios: np, quite the opposite pretty cool you openned it. | 13:08 |
| marios | quiquell|rover: ack thanks | 13:09 |
| quiquell|rover | marios: Want me to take a look ? | 13:09 |
| marios | quiquell|rover: sure, i'm gonna do something else now and revisit so feel free. if you find something update the bug and that way hopefully we make progress :) eventually | 13:10 |
| marios | also i ping chem from upgrades as he was interested in that yesterday just told him about the bug | 13:10 |
| quiquell|rover | marios: Will try to reproduce and dig into it | 13:11 |
| quiquell|rover | myoung: Do I delete the disabled jobs ? | 13:11 |
| quiquell|rover | myoung, rlandy: https://code.engineering.redhat.com/gerrit/141687 jobs cleanup | 13:12 |
| myoung | quiquell|rover: can look after sprint, TLDR JJB push only adds/updates jobs, but does not delete | 13:14 |
| myoung | after sprint planning | 13:15 |
| quiquell|rover | myoung: ok | 13:15 |
| myoung | weshay|ruck: if you do have even a few minutes the team has a few questions around priority and could use input - we can be fast | 13:15 |
| rlandy | quiquell|rover: posted comments | 13:16 |
| rlandy | panda: trown: requesting reviews on https://review.openstack.org/#/c/570694/ | 13:19 |
| quiquell|rover | rlandy: Do I remove the rhos-10 also from the doc ? | 13:21 |
| *** amoralej is now known as amoralej|lunch | 13:22 | |
| rlandy | quiquell|rover: the 8, 9, 10 puddle files ar still there | 13:23 |
| rlandy | 10 is a long-support-release I think | 13:23 |
| rlandy | so idk - TC? myoung: what's the line on the osp-10 date? | 13:24 |
| *** links has quit IRC | 13:27 | |
| myoung | rlandy: https://access.redhat.com/support/policy/updates/openstack/platform - osp10 --> 2021 | 13:28 |
| marios | myoung: o/ btw i voted at https://review.openstack.org/#/c/574969 (pike) wrt your irc ping last night incase you didn't see it yet | 13:29 |
| marios | myoung: leaving the +A to you | 13:29 |
| rlandy | rasca: hi | 13:30 |
| rasca | hey rlandy | 13:30 |
| rlandy | rasca: I am looking again at your backport https://review.openstack.org/#/c/572155/2/tripleoclient/tests/v1/overcloud_deploy/test_overcloud_deploy.py | 13:30 |
| rlandy | it fails buils unit test ... | 13:30 |
| rlandy | build | 13:30 |
| rlandy | TypeError: test_tht_deploy_with_plan_environment_file() takes exactly 25 arguments (24 given) | 13:30 |
| rlandy | I was looking at your edits | 13:31 |
| rasca | uhm | 13:31 |
| * rlandy gets full logs | 13:31 | |
| *** ykarel is now known as ykarel|away | 13:31 | |
| rlandy | rasca: I checked the args | 13:31 |
| rlandy | counted them - I only see one reference | 13:32 |
| rlandy | rasca: can't see exactly where the objection is | 13:32 |
| ykarel|away | quiquell|rover, can you please take care for ci-config patch, fixing [' queens = queens -a tripleo-ci-testing = consistent ']', once review.rdo is up | 13:33 |
| ykarel|away | i need to leave | 13:34 |
| weshay|ruck | rfolco, panda ready | 13:34 |
| quiquell|rover | ykarel|away: Will do | 13:34 |
| ykarel|away | quiquell|rover, Thanks | 13:34 |
| rlandy | rasca: sorry - can't get full log rdocloud not accessible | 13:35 |
| rlandy | but ... | 13:35 |
| rlandy | rasca: line 362 | 13:36 |
| rlandy | there is an extra param added | 13:36 |
| rlandy | in addition to the 4 tacked on at the bottom | 13:36 |
| rlandy | mock_sleep | 13:36 |
| rlandy | rasca: any thoughts? | 13:39 |
| *** ykarel|away has quit IRC | 13:40 | |
| rasca | rlandy, I'm checking if I omitted something compared to the original patch | 13:42 |
| rasca | rlandy, but it doesn't seem so | 13:42 |
| rlandy | rasca: kind of stumped. I could delete the param I expect is the issue - but I'd be hacking | 13:43 |
| rasca | rlandy, let me finish the comparison | 13:43 |
| rlandy | sure - thanks | 13:44 |
| quiquell|rover | weshay|ruck: Going to include https://review.openstack.org/#/c/575492/2 at the release in the gating repo | 13:46 |
| *** quiquell|rover is now known as quique|rover|lch | 13:49 | |
| rlandy | quique|rover|lch: sorry did get to look at your review and now it's unreachable - will try review later | 13:50 |
| *** saneax has quit IRC | 13:52 | |
| rasca | rlandy, ok so I think we might want to ask slagle here because there is a high amount of differences between the two releases of the file | 13:55 |
| rlandy | jschlueter; hi - rhos-13 question - have package dependencies failing with "No package matching 'mock' found available" | 13:55 |
| rlandy | seen that of late^^? | 13:55 |
| rlandy | rasca: ok - do you have any queens tests passing on baremetal? | 13:56 |
| rasca | rlandy, we can give a try to the removal of mock_sleep, but the error is saying that we're passing one arg LESS and not MORE | 13:56 |
| rlandy | correct - read that the wrong way :( | 13:57 |
| rlandy | rasca: yeah - I think we need to ask if the back port is actually possible | 13:57 |
| rlandy | rasca: to check - do we have any bm passing with queens? | 13:57 |
| rlandy | I have two failing encs | 13:57 |
| rlandy | envs | 13:57 |
| rasca | rlandy, yes, https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/oooq-queens-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans/ it's very slow right now, but it is working since a very long time | 13:59 |
| rlandy | rasca: how though?? | 13:59 |
| * rlandy looks what is being deployed | 13:59 | |
| rasca | rlandy, that's what I think we need to understand | 14:00 |
| rasca | this is a full bm deployment | 14:00 |
| *** ratailor has joined #oooq | 14:00 | |
| rlandy | rasca: one sec to look at your config file | 14:00 |
| rlandy | and see if I am missing something pbvious | 14:00 |
| rlandy | then yeah, we need to get slagle | 14:01 |
| rasca | rlandy, I'll ping him inside the review. /me still wonders why I'm succeeding in my jobs | 14:09 |
| rasca | not that I'm complaining, look :) | 14:09 |
| rlandy | rasca: that is what I am comparing now | 14:09 |
| rlandy | bit our configs are quite diff | 14:09 |
| rasca | but a lot of things in this world are still a mistery to me | 14:09 |
| rlandy | I am deploying fs001 | 14:09 |
| rlandy | rasca: lol | 14:09 |
| rlandy | I have one idea but it may be wrong https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/ovb-ha.yaml#L3 | 14:12 |
| rlandy | and I've set ssl_overcloud false | 14:13 |
| rasca | rlandy, uhm, so it might be totally unrelated to this patch here | 14:14 |
| rlandy | rasca: really, who knows? | 14:15 |
| rlandy | rasca: tht is a mystery to me as well | 14:16 |
| trown | rlandy: trying to understand why standalone job is failing on that patch... it doesnt seem to be failing elsewhere | 14:16 |
| rlandy | interesting ... Looking | 14:17 |
| rlandy | I didn't switch the order this time | 14:17 |
| rlandy | ERROR: os-net-config configuration failed. | 14:21 |
| *** florianf has quit IRC | 14:21 | |
| *** amoralej|lunch is now known as amoralej | 14:23 | |
| *** quique|rover|lch is now known as quiquell|rover | 14:24 | |
| rlandy | trown: not sure - going to rebase and rerun to check if that is reproducible | 14:27 |
| rlandy | the args should be the same | 14:27 |
| jschlueter | rlandy: package is python2-mock ... | 14:28 |
| jschlueter | rlandy: I can get you link in 13 latest puddle if you need specific NVR of mock ... but was there an issue? | 14:29 |
| jschlueter | python-mock-2.0.0-1.el7ost is the package | 14:29 |
| trown | rlandy: ya I dug into the logs and couldnt figure out what could be caused by your patch... that job might have got broken sometime since yesterday | 14:30 |
| rlandy | jschlueter: afaict, the package looks to be missing - we should not need a particular version | 14:30 |
| jschlueter | rlandy: ci job? | 14:31 |
| rlandy | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-featureset001/21/consoleFull | 14:32 |
| jschlueter | http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/latest/RH7-RHOS-13.0/x86_64/os/Packages/python-mock-2.0.0-1.el7ost.noarch.rpm | 14:32 |
| rlandy | search for task Ensure DLRN package dependencies | 14:33 |
| rlandy | unfortunately logs are unavailable atm | 14:33 |
| jschlueter | rlandy: it's not mock it's python-mock | 14:33 |
| jschlueter | is the rpm package name | 14:33 |
| rlandy | ah | 14:33 |
| jschlueter | rlandy: at least I assume you are talking about python mock package and not something else? | 14:34 |
| jschlueter | rlandy: if you can point me to the RDO build you have that satifies your need I can help find it in OSP 13 | 14:34 |
| rlandy | jschlueter: just checking our code to see why we are picking this up now | 14:35 |
| jschlueter | rlandy: if you are wanting mock the rpm build root ... then that would be rhel base stuff | 14:35 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blame/master/roles/build-test-packages/tasks/main.yml | 14:35 |
| rlandy | that was added two years ago - so it's been like that for a while | 14:36 |
| rlandy | jschlueter: then it may be the rhel image we are picking up | 14:37 |
| jschlueter | rlandy: ack ... you may need to install it or it could be there depending on what guest image/rhel image you are using | 14:37 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/main.yml#L3 does try install it | 14:38 |
| weshay|ruck | quiquell|rover, ping.. join my blue | 14:44 |
| rfolco | panda, your bj pls | 14:45 |
| panda | rfolco: bj/u/gcerami | 14:45 |
| quiquell|rover | weshay|ruck: Hi, give 2 min | 14:45 |
| weshay|ruck | k | 14:45 |
| jschlueter | rlandy: when did it last run successfully? | 14:45 |
| jschlueter | and can you check rpm's installed from that run? | 14:45 |
| panda | lol on the "ok I'm gonna leave .. ooo no this is my house, you're going to leave" | 14:45 |
| *** ratailor has quit IRC | 14:46 | |
| rlandy | jschlueter: looking through - also rerunning w/o delete so we have the env to debug | 14:49 |
| myoung | chandankumar, kopecmartin, arxcruz thanks for the time/attention for sprint 15 planning. Per discussion we'll revisit some of the cards w.r.t. scoping on Monday, have a great weekend! | 14:52 |
| kopecmartin | myoung, have a nice weekend too | 14:52 |
| weshay|ruck | quiquell|rover, hrm.. 1 gate failure in zuul now | 14:52 |
| weshay|ruck | 2018-06-15 14:35:20 | time="2018-06-15T14:35:19Z" level=fatal msg="pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: no such host" | 14:52 |
| weshay|ruck | http://logs.openstack.org/47/566247/5/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/f0194f6/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz | 14:53 |
| myoung | weshay|ruck, EmilienM, will have sprint 14 status out later on tonight, and sprint 15 planning summaries out over the weekend/monday. I've been getting immuno run down all week and I'm sporting a fever. dropping offline for a bit and will be back later this afternoon. | 14:54 |
| *** myoung is now known as myoung|bbl | 14:54 | |
| EmilienM | cool | 15:00 |
| *** quiquell|rover is now known as quiquell|off | 15:09 | |
| weshay|ruck | rlandy, 1-1 | 15:18 |
| rlandy | weshay|ruck: on your bj | 15:18 |
| marios | have a good w/e folks /me leaving a bit early today bai | 15:22 |
| * marios kiddo school end of year play, it promises to be enthralling | 15:23 | |
| *** dmellado_ has quit IRC | 16:02 | |
| *** ykarel|away has joined #oooq | 16:05 | |
| *** tcw1 has quit IRC | 16:08 | |
| *** bogdando has quit IRC | 16:13 | |
| *** tcw has joined #oooq | 16:16 | |
| weshay|ruck | trown, can you join my blue for a minute | 16:19 |
| trown | sure | 16:22 |
| weshay|ruck | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_queue:%20gate%20AND%20build_name:%20*tripleo-ci*%20AND%20build_status:%20FAILURE | 16:24 |
| *** holser__ has quit IRC | 16:25 | |
| weshay|ruck | trown, https://etherpad.openstack.org/p/tripleo-gate-issues-june-2018 | 16:26 |
| *** panda is now known as panda|off | 16:30 | |
| weshay|ruck | clarkb> corvus: fungi mordred mwhahaha weshay|ruck following up with the proxy cache logging stuff from yesterday. The only stuff I see in the port 8081 is 307 responses from dockerhub. We don't cache these which is expected. I think this may imply that dockerhub is no longer redirecting to the location that we are caching (otherwise we would see requests for that location in the server log as well) | 16:32 |
| weshay|ruck | trown, ^ | 16:32 |
| *** gkadam has quit IRC | 16:43 | |
| *** ykarel|away has quit IRC | 16:43 | |
| rlandy | jschlueter: ok - so afaict, the diff is that 'mock' itself is available on fedora (probably centros) mock-1.4.6-1.fc27.noarch : Builds packages inside chroots but on rhel. python-mock is available in rhos repos | 16:47 |
| rlandy | but not on rhel | 16:47 |
| rlandy | iamge used rhel 7.5 | 16:47 |
| rlandy | image | 16:47 |
| *** hamzy_ has joined #oooq | 16:48 | |
| *** hamzy has quit IRC | 16:51 | |
| jschlueter | python-mock is different beast ... | 16:53 |
| jschlueter | rlandy: rhel/centos has mock just not certain which channel it comes from | 16:53 |
| rlandy | yum provides mick returned nothing on the undercloud install | 16:56 |
| rlandy | mock | 16:56 |
| *** zoli is now known as zoli|gone | 16:57 | |
| *** zoli|gone is now known as zoli | 16:57 | |
| *** tesseract has quit IRC | 17:01 | |
| *** amoralej is now known as amoralej|off | 17:14 | |
| *** kopecmartin has quit IRC | 17:14 | |
| rlandy | PSA ... internet in the area is undergoing maintenance ... may have a chopping connection for the next couple hours | 17:16 |
| *** ccamacho has quit IRC | 17:24 | |
| weshay|ruck | EmilienM, trown https://review.openstack.org/#/c/575535/ | 17:28 |
| *** gkadam has joined #oooq | 17:44 | |
| *** hubbot has joined #oooq | 17:45 | |
| *** chandankumar is now known as chkumar|pto | 18:01 | |
| chkumar|pto | weshay|ruck: EmilienM myoung|bbl tosky I will be on pto starting from tomorrow, if anything needed, feel free to bug me :-) | 18:02 |
| tosky | chkumar|pto: as you are on PTO, we *won't* bug you! | 18:03 |
| tosky | enjoy | 18:03 |
| *** agopi has joined #oooq | 18:11 | |
| rlandy | internet going off line | 18:27 |
| rlandy | back in about 30 minutes | 18:27 |
| *** rlandy_ has joined #oooq | 18:31 | |
| *** rlandy_ has quit IRC | 18:31 | |
| *** rlandy_ has joined #oooq | 18:31 | |
| *** rlandy has quit IRC | 18:33 | |
| *** gkadam_ has joined #oooq | 18:33 | |
| *** rlandy_ is now known as rlandy | 18:36 | |
| rlandy | weshay|ruck: ha - mystery solved ... http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-13.yml#n1 | 18:36 |
| *** gkadam has quit IRC | 18:37 | |
| rlandy | weshay|ruck: myoung|bbl: https://code.engineering.redhat.com/gerrit/141715 pls | 18:40 |
| *** Goneri has joined #oooq | 18:43 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 18:47 |
| *** matbu has quit IRC | 18:47 | |
| weshay|ruck | /back | 19:09 |
| *** dougbtv_ has quit IRC | 19:14 | |
| *** rlandy has quit IRC | 19:15 | |
| *** rlandy has joined #oooq | 19:18 | |
| weshay|ruck | rlandy, AH | 19:19 |
| weshay|ruck | doh | 19:19 |
| rlandy | weshay|ruck: just merged it to run again | 19:20 |
| weshay|ruck | ya.. I see | 19:20 |
| weshay|ruck | rlandy++ | 19:20 |
| hubbot | weshay|ruck: rlandy's karma is now 6 | 19:20 |
| weshay|ruck | thank you for paying attention :)) | 19:20 |
| rlandy | I think rdocloud returned | 19:21 |
| rlandy | I can access the images again | 19:21 |
| weshay|ruck | ya.. agree | 19:21 |
| * weshay|ruck removed the alert from the bug, updated the status on #tripleo | 19:21 | |
| rlandy | weshay|ruck: myoung|bbl: there are a bunch of cards in https://trello.com/b/U1ITy0cu/tripleo-and-rdo-ci in the failing jobs column that are complete | 19:22 |
| rlandy | can I just move somewhere? archive? | 19:23 |
| rlandy | rhel gates are on 7.5 | 19:23 |
| rlandy | pem is sorted out | 19:23 |
| rlandy | 23 hr 2 min queue - lovely :( | 19:27 |
| rlandy | weshay|ruck: clean up needed on openstack-nodepool - do you me to run that? | 19:30 |
| *** Goneri has quit IRC | 19:31 | |
| weshay|ruck | rlandy, thanks I got it | 19:34 |
| weshay|ruck | will do it now | 19:34 |
| weshay|ruck | rlandy, running | 19:36 |
| *** holser__ has joined #oooq | 19:37 | |
| *** rlandy_ has joined #oooq | 19:39 | |
| *** Goneri has joined #oooq | 19:39 | |
| *** rlandy has quit IRC | 19:41 | |
| *** holser___ has joined #oooq | 20:01 | |
| *** holser__ has quit IRC | 20:05 | |
| *** tcw has quit IRC | 20:30 | |
| *** tcw has joined #oooq | 20:33 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 20:47 |
| *** gkadam_ has quit IRC | 21:05 | |
| *** brault has quit IRC | 21:10 | |
| *** agopi has quit IRC | 21:14 | |
| *** agopi has joined #oooq | 21:15 | |
| *** rfolco has quit IRC | 21:15 | |
| *** holser___ has quit IRC | 21:21 | |
| *** jtomasek_ has quit IRC | 21:21 | |
| *** agopi has quit IRC | 22:00 | |
| rlandy_ | weshay|ruck: just fyi ... https://thirdparty.logs.rdoproject.org/jenkins-rlandy-poc-tripleo-quickstart-queens-dell_fc430_envB-single_nic_vlans-5/undercloud/home/stack/overcloud_deploy_post.log.txt.gz - queens bm is failing in post-deploy certificate ... https://thirdparty.logs.rdoproject.org/jenkins-rlandy-poc-tripleo-quickstart-queens-dell_fc430_envB-single_nic_vlans-5/undercloud/home/stack/overcloud_deploy_post.log.txt.gz#_2018-06-15_21_42_17 | 22:05 |
| rlandy_ | almost there | 22:05 |
| rlandy_ | hrybacki: hi there | 22:15 |
| *** rlandy_ is now known as rlandy | 22:15 | |
| rlandy | keystone question if you have a moment | 22:16 |
| rlandy | oh nvm | 22:28 |
| *** matbu has joined #oooq | 22:39 | |
| hubbot | All check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens. | 22:47 |
| *** Goneri has quit IRC | 22:50 | |
| *** strattao has quit IRC | 22:54 | |
| *** hamzy_ has quit IRC | 23:15 | |
| *** strattao has joined #oooq | 23:23 | |
| *** hamzy has joined #oooq | 23:25 | |
| *** tosky has quit IRC | 23:41 | |
| *** agopi has joined #oooq | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!