*** dalvarez has quit IRC | 02:42 | |
*** rlandy|afk is now known as rlandy | 02:44 | |
*** rlandy has quit IRC | 02:44 | |
*** radez has quit IRC | 02:46 | |
*** amoralej|off has quit IRC | 02:47 | |
*** ykarel has joined #oooq | 03:27 | |
*** udesale has joined #oooq | 03:36 | |
*** saneax has joined #oooq | 04:09 | |
*** skramaja has joined #oooq | 04:37 | |
*** jaganathan has joined #oooq | 04:37 | |
*** apetrich has joined #oooq | 05:06 | |
*** ratailor has joined #oooq | 05:27 | |
*** jfrancoa has joined #oooq | 06:13 | |
*** ykarel is now known as ykarel|afk | 06:30 | |
*** ykarel|afk has quit IRC | 06:34 | |
*** ratailor has quit IRC | 06:43 | |
*** ratailor has joined #oooq | 06:47 | |
*** gkadam has joined #oooq | 06:48 | |
*** ykarel|afk has joined #oooq | 06:53 | |
*** ykarel|afk is now known as ykarel | 06:54 | |
*** amoralej has joined #oooq | 06:58 | |
*** kopecmartin has joined #oooq | 07:11 | |
*** bogdando has joined #oooq | 07:26 | |
*** chem has joined #oooq | 07:49 | |
*** dalvarez has joined #oooq | 08:24 | |
*** panda|off is now known as panda | 08:56 | |
*** jaosorior has quit IRC | 09:10 | |
*** d0ugal has quit IRC | 09:14 | |
*** d0ugal has joined #oooq | 09:18 | |
panda | marios: change your nick | 09:22 |
---|---|---|
panda | ssbarnea: you there ? | 09:22 |
*** marios|rover has joined #oooq | 09:25 | |
jfrancoa | panda: Could you give me a hand please? do you know where can I find the job definition for legacy-tripleo-ci-centos-7-container-to-container-upgrades-queens? | 09:28 |
jfrancoa | panda: I'm debugging https://bugs.launchpad.net/tripleo/+bug/1785039 | 09:28 |
openstack | Launchpad bug 1785039 in tripleo "[Upgrades] legacy-tripleo-ci-centos-7-container-to-container-upgrades-master failing during deployment" [High,Triaged] - Assigned to Gabriele Cerami (gcerami) | 09:28 |
jfrancoa | panda: it used to be in the config project, but I saw that zuul/upstream.yml was removed and I can't find it now | 09:29 |
panda | jfrancoa: https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/zuul-legacy-jobs.yaml;h=3957f4e667932828d9c237be9674e56717aff9fc;hb=refs/heads/master#l1333 | 09:31 |
panda | jfrancoa: it's currently a mess up there, we are trying to fix all this in the next sprint | 09:31 |
jfrancoa | panda: thanks, I wasn't aware of this repo | 09:31 |
panda | jfrancoa: but for now, all the jobs are defined there, and they have a playbook per job, duplicated from the original | 09:32 |
jfrancoa | panda: cool, thanks for the explanation | 09:32 |
jfrancoa | panda: by the way, are you aware if I need to ask for any permissions to be able to access the rdoproject.org gerrit? I used to have rights in here, but lately I can't see any gerrit patch/project | 09:34 |
jfrancoa | panda: nah, some cookies issue with mozilla | 09:40 |
panda | jfrancoa: yep, just need github credentials | 09:50 |
*** ratailor has quit IRC | 09:57 | |
*** ratailor has joined #oooq | 09:57 | |
*** ykarel is now known as ykarel|lunch | 10:01 | |
jfrancoa | panda: problem solved https://review.rdoproject.org/r/#/c/15557/ , do you know any way to test the upgrades jobs with this patch? | 10:02 |
jfrancoa | panda: or will the rdo zuul trigger the affected jobs automatically? | 10:02 |
*** d0ugal has quit IRC | 10:07 | |
*** d0ugal has joined #oooq | 10:13 | |
*** rf0lc0 has quit IRC | 10:15 | |
*** rf0lc0 has joined #oooq | 10:16 | |
*** skramaja has quit IRC | 10:19 | |
*** skramaja has joined #oooq | 10:25 | |
chandankumar | arxcruz: myoung weshay when you are around https://trello.com/c/CcrOAWdI/903-tempest-squad-sprint-18-prepare-for-rocky-release we have added the cards for upcomign sprint | 10:26 |
*** jaosorior has joined #oooq | 10:28 | |
*** skramaja has quit IRC | 10:32 | |
*** panda is now known as panda|lunch | 11:02 | |
*** amoralej is now known as amoralej|lunch | 11:09 | |
*** ykarel|lunch is now known as ykarel | 11:09 | |
*** udesale has quit IRC | 11:30 | |
*** ratailor has quit IRC | 12:07 | |
*** amoralej|lunch is now known as amoralej | 12:15 | |
weshay | ssbarnea, howdy | 12:17 |
weshay | ssbarnea, is still pto I think | 12:17 |
*** skramaja has joined #oooq | 12:22 | |
*** agopi has quit IRC | 12:22 | |
*** jaosorior has quit IRC | 12:25 | |
amoralej | some jobs for rocky are failing with error | 12:25 |
amoralej | 2018-08-16 11:53:38.057117 | primary | emit_releases_file.py: error: argument --stable-release: invalid choice: 'rocky' (choose from 'newton', 'ocata', 'pike', 'queens', 'master') | 12:25 |
*** jaosorior has joined #oooq | 12:25 | |
amoralej | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-rocky/149ffa0/job-output.txt.gz | 12:25 |
ykarel | rocky missing here:- https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/emit_releases_file/emit_releases_file.py#L10 | 12:29 |
ykarel | marios|rover, ^^ | 12:38 |
ykarel | no ruck today? | 12:38 |
*** rlandy has joined #oooq | 12:41 | |
rlandy | jschlueter: hello - running into an undercloud install error on rhos-13 - https://thirdparty.logs.rdoproject.org/jenkins-tq-gate-rhos-13-ci-rhos-ovb-featureset001-129/undercloud/home/stack/undercloud_install.log.txt.gz#_2018-08-15_18_29_23 | 12:50 |
*** agopi has joined #oooq | 12:50 | |
rlandy | familiar to you? | 12:50 |
rlandy | we should be using passed_phase1 | 12:50 |
rlandy | weshay: ^^ fyi | 12:51 |
*** agopi_ has joined #oooq | 12:51 | |
*** agopi has quit IRC | 12:55 | |
*** agopi_ is now known as agopi | 12:56 | |
myoung | o/ sprint planning for CI Squad s18 starts shortly | 12:56 |
marios | thanks ykarel will have to check later in calls right now | 13:01 |
marios | ykarel: ruck is away weshay is filling in i am rovering and we are all in a call right now | 13:01 |
ykarel | marios, ack. btw sshnaidm proposed fix for that issue | 13:01 |
weshay | rlandy, hrm.. osp-13 | 13:02 |
weshay | rlandy, Duplicate key name 'uniq_ports0standard_attr_id' | 13:03 |
weshay | rlandy, maybe try passed phase 2? | 13:03 |
weshay | rlandy, /me looks at builds | 13:03 |
weshay | rlandy, http://paste.openstack.org/show/728194/ | 13:04 |
weshay | 2018-08-15 18:19:02 | 2018-08-15 18:19:02,408 INFO: [1;31mError: Command exceeded timeout[0m | 13:04 |
rlandy | weshay: should not be exceeding timeout though | 13:05 |
rlandy | I can increase the timeout | 13:05 |
rlandy | but I wanted to be sure it was not a legit error | 13:05 |
rlandy | weshay: admittedly it's slow on this platform | 13:05 |
weshay | rlandy, https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP13/ | 13:07 |
rlandy | weshay: so switch to passed phase 2? | 13:08 |
rlandy | and increase timeout? | 13:08 |
*** udesale has joined #oooq | 13:12 | |
*** jaosorior has quit IRC | 13:14 | |
*** skramaja has quit IRC | 13:14 | |
*** jaosorior has joined #oooq | 13:15 | |
*** jaosorior has quit IRC | 13:20 | |
rlandy | weshay: possibly we are running into the 'lower quality hardware' issue here ... https://review.openstack.org/#/c/428843/ | 13:23 |
*** jfrancoa has quit IRC | 13:23 | |
weshay | rlandy, ya.. sorry was the job ovb or virt? | 13:23 |
rlandy | weshay: ovb | 13:24 |
rlandy | trying to revive that job | 13:24 |
weshay | right k | 13:24 |
weshay | rlandy, so maybe bump up the cpu and memory as a test? | 13:24 |
*** jfrancoa has joined #oooq | 13:25 | |
rlandy | weshay: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/environments/ci-rhos.yml#n34 - xlarge - pretty extensive | 13:26 |
weshay | rlandy, that is rhos-ci cloud right? | 13:38 |
weshay | cirhos | 13:38 |
weshay | ya | 13:38 |
rlandy | weshay: yep | 13:38 |
* weshay looks at that | 13:38 | |
rlandy | weshay: I increased the volume to 80 | 13:39 |
weshay | k | 13:39 |
rlandy | if you look at the job config - we deploy with attached volumes | 13:39 |
rlandy | since we could not get bigger flavors | 13:39 |
weshay | rlandy, go big once w/ the undercloud m5.large | 13:40 |
weshay | :) | 13:40 |
rlandy | go big or go home | 13:41 |
* rlandy looks | 13:41 | |
*** matbu has joined #oooq | 13:42 | |
weshay | rlandy, :) | 13:44 |
*** radez has joined #oooq | 13:45 | |
*** saneax has quit IRC | 13:57 | |
amoralej | it seems we need a file for rocky in https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/validate-tempest/vars | 14:13 |
amoralej | jobs are failing because of that | 14:13 |
weshay | amoralej, k.. chandankumar just submitted a patch | 14:28 |
weshay | amoralej, https://review.openstack.org/#/c/592489/1 | 14:28 |
amoralej | cool | 14:29 |
ykarel | weshay, also add to the checklist so it's can be avoided for stein | 14:29 |
weshay | ykarel, aye.. thanks | 14:29 |
amoralej | chandankumar, there is no tag for tempestconf for rocky yet? | 14:29 |
chandankumar | amoralej: will come next week | 14:30 |
amoralej | ack | 14:30 |
chandankumar | amoralej: waiting for tosky to come | 14:30 |
*** jaosorior has joined #oooq | 14:32 | |
*** d0ugal has quit IRC | 14:39 | |
*** d0ugal has joined #oooq | 14:41 | |
*** marios|rover has quit IRC | 14:50 | |
*** ratailor has joined #oooq | 14:59 | |
weshay | was there a decision? | 15:04 |
weshay | I see more options | 15:04 |
*** ratailor has quit IRC | 15:05 | |
*** ratailor has joined #oooq | 15:06 | |
*** marios|rover has joined #oooq | 15:07 | |
chandankumar | weshay: we have few patches pending | 15:07 |
chandankumar | but i need some feedback from tosky so waiting till monday | 15:08 |
*** kopecmartin has quit IRC | 15:09 | |
*** ykarel is now known as ykarel|away | 15:09 | |
weshay | chandankumar, k | 15:12 |
*** jfrancoa has quit IRC | 15:32 | |
*** vinaykns has joined #oooq | 15:32 | |
*** verdurin has quit IRC | 15:36 | |
*** verdurin has joined #oooq | 15:38 | |
*** bogdando has quit IRC | 15:44 | |
*** d0ugal has quit IRC | 15:47 | |
*** ykarel|away has quit IRC | 15:51 | |
*** d0ugal has joined #oooq | 16:00 | |
chandankumar | myoung: Hello | 16:04 |
chandankumar | myoung: https://trello.com/c/CcrOAWdI/903-tempest-squad-sprint-18-prepare-for-rocky-release I have added sprint 18 cards reviewd with mkopec please let's know if you need anything else | 16:05 |
*** udesale has quit IRC | 16:05 | |
*** ratailor has quit IRC | 16:09 | |
*** amoralej is now known as amoralej|off | 16:22 | |
*** panda|lunch is now known as panda|off | 16:28 | |
* myoung walks over to the transmogrifier and looks for the "matt" position on the selector lever | 16:28 | |
*** myoung is now known as myoung|lunch | 16:28 | |
marios | weshay: myoung panda|off sshnaidm rf0lc0 ** have a great day folks bbyl | 16:28 |
marios | ttyl evel :) | 16:28 |
myoung|lunch | thanks everyone for patience, endurance, and a productive discussion and design session | 16:28 |
marios | even (k time to go now) | 16:29 |
weshay | l8r | 16:29 |
rf0lc0 | evil? | 16:29 |
rf0lc0 | :) | 16:29 |
marios | rf0lc0: yes :D it must be i mean 208 mins of meeting what else | 16:29 |
* rf0lc0 thinks we should rename this meeting to marathon planning | 16:30 | |
myoung|lunch | A NEW RECORD. | 16:31 |
myoung|lunch | everyone go outside and breathe some fresh air. let your craniums unmelt :) | 16:31 |
* myoung|lunch walks away from keyboard to do ^^ | 16:31 | |
panda|off | what planning ? | 16:31 |
panda|off | we didn't even talk about the card | 16:31 |
marios | myoung|lunch: teling ya 3 hours + is dangerous territory ... mass hysteria and we are locked in the call until someone external looks for us | 16:32 |
marios | it could be ages | 16:32 |
marios | thank heavens weshay joined to break the lock | 16:32 |
myoung|lunch | ok i was about to go...and going after this...but yeah, that might be a TC fail. my bad lol | 16:32 |
* myoung|lunch is still proud of us though...we did some rational design discussion and ship navigation | 16:33 | |
myoung|lunch | apologies for the length | 16:33 |
*** rlandy is now known as rlandy|brb | 16:33 | |
*** sshnaidm is now known as sshnaidm|bbl | 16:35 | |
*** rf0lc0 is now known as rfolco | 16:39 | |
*** gkadam has quit IRC | 17:03 | |
*** rlandy|brb is now known as rlandy | 17:11 | |
*** weshay is now known as weshay|ruck | 17:12 | |
weshay|ruck | rlandy, any luck w/ the UBER Flavor? | 17:13 |
rlandy | weshay|ruck: trying now - was caught up in the planning fun this morning | 17:13 |
weshay|ruck | rlandy, https://review.openstack.org/592591 | 17:19 |
*** gkadam has joined #oooq | 17:29 | |
rlandy | weshay|ruck: arxcruz is on PTO :( | 17:29 |
rlandy | so much for my coaching :( | 17:30 |
rlandy | weshay|ruck: question ... the comment says "mistral team is investigating" | 17:33 |
rlandy | when does the tempest team investigate vs, when does that team ask the product team to? | 17:34 |
rlandy | chandankumar: ^^? | 17:34 |
*** myoung|lunch is now known as myoung | 17:44 | |
chandankumar | rlandy: it is related to mistral tests, I donot have much knowledge, let's have an ETA, till then add it to ski list? | 17:44 |
chandankumar | *skip | 17:44 |
chandankumar | weshay|ruck: ^^ | 17:44 |
chandankumar | tempest team takes care of tempest tooling and the framework not the tests | 17:45 |
chandankumar | we are here to help team how they can enable and run tempest specific tests not on if the tests failed | 17:46 |
chandankumar | because those are component specific needs deeper knowlege of the servie how they are behaving | 17:46 |
weshay|ruck | chandankumar, certainly.. if there is a failing tempest test in the upstream job.. WE ALL ARE HERE to help debug it | 17:52 |
weshay|ruck | chandankumar, if needed we can bring in others to help | 17:52 |
weshay|ruck | chandankumar, w/ fs20 being the exception | 17:52 |
weshay|ruck | chandankumar, there is a pretty limited scope of tempest tests being used in upstream check / gate jobs | 17:52 |
weshay|ruck | that we should be able to debug those down and call in help when needed | 17:53 |
* chandankumar will look at this tomorrow | 17:53 | |
weshay|ruck | apetrich, or toure can help too | 17:54 |
weshay|ruck | rlandy, apetrich w/ https://bugs.launchpad.net/tripleo/+bug/1736950 | 17:54 |
openstack | Launchpad bug 1736950 in tripleo "CI: mistral testmistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2.test_get_list_actions_not_in_list_filter fails in gate scenario003 containers" [Critical,Triaged] | 17:54 |
rlandy | thanks | 18:03 |
apetrich | rlandy, weshay|ruck looking | 18:03 |
rlandy | apetrich: hello! | 18:03 |
weshay|ruck | chandankumar, arxcruz so rlandy is going to get familiar w/ setting up a tempest debug env | 18:04 |
apetrich | rlandy, hiya | 18:04 |
weshay|ruck | arxcruz, I've watched you do this.. chandankumar if you can help rlandy or point her at doc that would be very helpful | 18:04 |
chandankumar | weshay|ruck: time to catch d0ugal also :-) | 18:05 |
apetrich | weshay|ruck, isn't arxcruz fighting for the horde at the moment? | 18:05 |
weshay|ruck | ya.. he's pto | 18:05 |
chandankumar | he will be available tomorrow during planning | 18:05 |
apetrich | best excuse for a pto ever | 18:06 |
apetrich | oh that test | 18:06 |
apetrich | rlandy, yeah I think I know what the issue is | 18:07 |
rlandy | chandankumar: thanks - I'l catch arxcruz when he is back | 18:07 |
rlandy | apetrich: great | 18:07 |
* chandankumar knows fedora is good for application development but not good for openstack | 18:08 | |
* chandankumar find very hard when updating openstack related fedora dependencies because they donot care about us at all | 18:09 | |
rlandy | myoung: where was that tempest doc to review? | 18:15 |
rlandy | weshay|ruck: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-featureset001/130/console - running with UBER flavor on the undercloud - will see if it helps | 18:15 |
myoung | rlandy: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset053.yml#L145 | 18:15 |
myoung | rlandy: http://logs.openstack.org/42/589742/3/check/build-openstack-sphinx-docs/2a676c0/html | 18:16 |
rlandy | thank you | 18:16 |
weshay|ruck | chandankumar, oh ya.. | 18:16 |
myoung | rlandy: ack. i'm reading it later on as well | 18:16 |
weshay|ruck | fedora sucsk | 18:16 |
weshay|ruck | sucks | 18:16 |
* weshay|ruck votes for fedora lts | 18:17 | |
rlandy | I read it a while back but time to refresh | 18:17 |
weshay|ruck | chandankumar, I have a fedora guest image that's working for me up to a point | 18:17 |
weshay|ruck | chandankumar, on standalone | 18:17 |
weshay|ruck | chandankumar, we need to get tempest moving on standalone | 18:17 |
weshay|ruck | chandankumar, $next sprint | 18:17 |
weshay|ruck | should be easy | 18:18 |
weshay|ruck | I had some patches | 18:18 |
apetrich | rlandy, spinning an env to test it but I will have to close the lid soon. I can have a look at that early in the morning tomorrow | 18:18 |
rlandy | apetrich: k, thanks | 18:18 |
*** rlandy has quit IRC | 18:32 | |
*** myoung is now known as myoung|brb | 18:34 | |
*** rlandy_ has joined #oooq | 18:35 | |
*** gkadam_ has joined #oooq | 18:38 | |
*** gkadam has quit IRC | 18:42 | |
*** sshnaidm|bbl is now known as sshnaidm | 18:43 | |
chandankumar | weshay|ruck: sure sir | 18:53 |
chandankumar | weshay|ruck: https://trello.com/c/CcrOAWdI/903-tempest-squad-sprint-18-prepare-for-rocky-release feel free to take a look at sprint 18 topics since we did not get a chance to spea about that | 18:54 |
weshay|ruck | thanks chandankumar will catch you tomorrow | 18:56 |
weshay|ruck | :) | 18:56 |
*** gkadam_ has quit IRC | 19:00 | |
rlandy_ | panda|off: myoung|brb: I'll trying a POC review for the reparent task | 19:05 |
rlandy_ | rfolco: ^^ | 19:05 |
rfolco | rlandy_, ack, I'm checking a better way to include custom playbooks and tempest skiplist and custom fs vars and ... | 19:07 |
*** apetrich has quit IRC | 19:10 | |
weshay|ruck | bah.. data here is old | 19:19 |
weshay|ruck | http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=now-6h&to=now | 19:19 |
weshay|ruck | :( | 19:19 |
*** myoung|brb is now known as myoung | 19:19 | |
myoung | rlandy_, rfolco, I'm doing some thinking around what an ontology/namespace of job layers would like like...e.g....what logical tree shape makes sense, as well as putting into words/doc format some of the things we talked about today. Next week I'm in training so trying to get some thoughts/ideas out prior to monday. | 19:27 |
rfolco | ack young man myoung | 19:30 |
* myoung will be "forever Young" | 19:31 | |
* myoung groans at his own terrible pun | 19:31 | |
rfolco | ;) | 19:31 |
myoung | I think my brain has recently rebounded from this morning. | 19:31 |
myoung | lol | 19:31 |
weshay|ruck | rlandy_, that ovb job on 13 is using the fs001 config? | 19:35 |
rlandy_ | weshay|ruck: yes | 19:39 |
*** rlandy_ is now known as rlandy | 19:40 | |
weshay|ruck | rlandy, something is funky there | 19:40 |
weshay|ruck | rlandy, can we get on the undercloud? | 19:40 |
*** jrist has quit IRC | 19:40 | |
rlandy | weshay|ruck: I am on the undercloud now | 19:40 |
weshay|ruck | tmate | 19:40 |
rlandy | watching the tail on undercloud install | 19:40 |
weshay|ruck | rlandy, docker ps | 19:40 |
weshay|ruck | what's up w/ that | 19:40 |
weshay|ruck | oh wait.. | 19:40 |
weshay|ruck | this is the undercloud in queens | 19:41 |
weshay|ruck | no docker | 19:41 |
weshay|ruck | wth is taking so long | 19:41 |
weshay|ruck | anything funky on the networking? | 19:41 |
rlandy | weshay|ruck: this is the point where trouble starts | 19:49 |
weshay|ruck | ya | 19:50 |
weshay|ruck | haproxy | 19:50 |
weshay|ruck | hrm | 19:51 |
weshay|ruck | that seems like a puppet bug to me | 19:51 |
weshay|ruck | that should be something that rasca should be able to kill for us | 19:52 |
weshay|ruck | but let's see what happens now | 19:52 |
weshay|ruck | ya | 19:52 |
weshay|ruck | rlandy, weird | 19:53 |
weshay|ruck | such a odd thing to fail on for a GA product | 19:53 |
rlandy | weshay|ruck: something is killing it | 19:53 |
weshay|ruck | maybe there was a regression | 19:53 |
rlandy | failed in the same place yesterday | 19:53 |
weshay|ruck | ya | 19:53 |
rlandy | weird | 19:53 |
rlandy | that we should find it | 19:54 |
rlandy | surely tested elsewhere | 19:54 |
rlandy | poor haproxy | 19:54 |
rlandy | weshay|ruck: which is hwy I asked this morning | 19:54 |
rlandy | not passing undercloud install is pretty basic | 19:54 |
rlandy | unless it's env specific | 19:55 |
weshay|ruck | https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP13/job/phase1-13_director-rhel-7.5-virthost-1cont_1comp_1ceph-ipv4-vxlan-ceph-containers/lastBuild/console | 19:55 |
weshay|ruck | it's going to pass now | 19:55 |
rlandy | because you restarted haproxy | 19:56 |
rlandy | which should NOT be required | 19:57 |
rlandy | to state the obvious | 19:57 |
weshay|ruck | this one passed for QE | 19:58 |
weshay|ruck | http://cougar11.scl.lab.tlv.redhat.com/phase1-13_director-rhel-7.5-virthost-1cont_1comp_1ceph-ipv4-vxlan-ceph-containers/168/undercloud-0.tar.gz?undercloud-0/etc/yum.repos.d/rhos-release-13.repo | 19:58 |
weshay|ruck | rlandy, oh ya.. totally | 19:58 |
weshay|ruck | that was because I started it | 19:58 |
rlandy | weshay|ruck: can we check hapoxy status now? | 19:58 |
weshay|ruck | but puppet should have retried the start instead of just polling that status | 19:58 |
rfolco | docker hub down | 19:58 |
rfolco | nm sat 25th | 19:59 |
rlandy | weshay|ruck: ok - so it passed for QE but fails for us ... meaning??? | 20:00 |
rlandy | may still fail now | 20:00 |
weshay|ruck | rlandy, no.. I'm just finding a known working build | 20:00 |
weshay|ruck | from the recent past | 20:00 |
weshay|ruck | rlandy, I don't know.. | 20:00 |
weshay|ruck | I don't know what it means :) | 20:00 |
weshay|ruck | rlandy, maybe we need to try http://rhos-qe-mirror-tlv.usersys.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/7.5-z1/ | 20:01 |
weshay|ruck | which is the 13.1 release | 20:01 |
weshay|ruck | rlandy, all we're trying to do is to make sure tq/tqe is working with it right | 20:02 |
rlandy | weshay|ruck: we didn't pay off the correct OSP mafia boss this month? | 20:02 |
weshay|ruck | so we should find the MOST reliable build they have | 20:02 |
weshay|ruck | bah haha | 20:02 |
weshay|ruck | rlandy, it could be that there are workarounds | 20:02 |
* weshay|ruck runs | 20:02 | |
rlandy | weshay|ruck: no - come back - we are not don yet!!!! | 20:03 |
rlandy | done | 20:03 |
* weshay|ruck looks | 20:03 | |
rlandy | let's just see if this passes with restart | 20:03 |
weshay|ruck | it's setting up glance | 20:03 |
rlandy | ie: now | 20:04 |
weshay|ruck | rlandy, ya | 20:04 |
rlandy | weshay|ruck: "MOST reliable build they have" so we should be using passed phase 2 | 20:04 |
weshay|ruck | rlandy, I would say.. something we shipped | 20:06 |
weshay|ruck | http://rhos-qe-mirror-tlv.usersys.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/7.5-z1/ | 20:06 |
weshay|ruck | well.. from a boston mirror | 20:06 |
weshay|ruck | dam this slow | 20:06 |
rlandy | legit timeout? | 20:08 |
rlandy | weshay|ruck: ok - will try a boston mirror when this is officially dead | 20:09 |
rfolco | rlandy, what is the multinode fs for browbeat ? fs053 is ovb... | 20:22 |
rlandy | rfolco: there is no multinode for browbeat | 20:24 |
rlandy | it's a simulation of real hw only | 20:24 |
weshay|ruck | rlandy, https://redhat.bluejeans.com/u/whayutin/ | 20:24 |
rfolco | rlandy, oh I see thx | 20:24 |
rlandy | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-13.yml | 20:30 |
myoung | rfolco: the actual browbeat (perf ci) jobs run with large numbers of actual nodes, afaik they can go up to 24 nodes per bench without having to reconfig...but not sure what # of nodes are running in rdo2 now... agopi would know | 20:32 |
rfolco | myoung, thx ... my question has much simpler reason than browbeat itself | 20:34 |
agopi | myoung, 6 nodes | 20:34 |
agopi | 3 controllers 2 computes ^ | 20:34 |
rfolco | good to know, thank you agopi ;) | 20:34 |
agopi | np rfolco :) | 20:35 |
myoung | agopi: were it possible (via configuration) to easily up the node #, is that still a goal? that was something jkilpatr used to talk about... | 20:39 |
*** jrist has joined #oooq | 20:40 | |
agopi | it should be, and given that we've got a hardware crunch i don't see it happen anytime soon. We usually do larger scale deployment in scalelab but they're not LTA. | 20:40 |
myoung | lta? | 20:40 |
agopi | long term availability my bad | 20:41 |
*** ssbarnea has quit IRC | 20:45 | |
*** apetrich has joined #oooq | 20:46 | |
rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/147299/ | 20:53 |
rlandy | myoung: ^^ pls check if I did that correctly | 20:56 |
myoung | looking now, missing one bit | 20:56 |
myoung | commenting there | 20:56 |
weshay|ruck | what's missing? | 20:57 |
rlandy | oh sorry | 20:57 |
rlandy | I will resubmit | 20:57 |
weshay|ruck | ? | 20:58 |
rlandy | myoung: did you comment? | 20:59 |
myoung | just now...i guess review window was 5 min lol | 20:59 |
rlandy | myoung: sorry, I'm not understanding | 21:00 |
rlandy | what needs to change? | 21:00 |
rlandy | this is the test for tq tqe | 21:00 |
rlandy | is there something else that needs to change on this file? | 21:01 |
rlandy | you use this file for other jobs? | 21:01 |
rlandy | so I messed that up? | 21:01 |
rlandy | should I create a separate file? | 21:02 |
myoung | i see | 21:02 |
myoung | no it's fine...it's just that the auto triggers for normal osp13 jobs are looking at a new phase 1 passed puddle | 21:02 |
myoung | what you have for the tq/tqe jobs that use that configfile (in terms of args to rhos-release) is fine | 21:03 |
myoung | and will work | 21:03 |
rlandy | myoung: I can create a new file if thta work work better | 21:03 |
myoung | but just some concern that the non tq/tqe gate jobs (that are running osp passed_phase_1) will trigger on a new puddle still, but then the config file causes to call rhos-release with z1 puddle | 21:04 |
* myoung needs to review/audit those jobs to be sure...and wants to purge them all with fire and use zuul config downstream instead and chuckles | 21:04 | |
rlandy | myoung: ok - let me know - can change it if need be | 21:05 |
rlandy | sorry for the quick merge | 21:06 |
myoung | rlandy: sok...by the time i looked was already in heh | 21:06 |
* myoung reloads osp0 state from memory banks | 21:06 | |
rlandy | myoung: we can create a rhos-13-z1 file | 21:07 |
rlandy | if we need to | 21:07 |
myoung | rlandy: ack, i'm looking here: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/osp-director-promote-13-puddle | 21:07 |
myoung | that seems to have been red since the late may | 21:08 |
myoung | well june for https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/osp-rhos-13-promote-puddle-featureset020-1ctlr_1comp_64gb/ | 21:08 |
myoung | and rasca's job is green | 21:08 |
rlandy | but rasca's job works | 21:09 |
rlandy | idk | 21:09 |
myoung | right...so https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/3/ is the one that matters | 21:09 |
rlandy | will have to check with him when he is on line | 21:10 |
rlandy | how does that work?? | 21:10 |
myoung | so the parent job uses the trigger script to look for new puddles, and set current_build, which is passed to all jobs (https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/3/parameters) | 21:11 |
myoung | ^^ that's at passed_phase_1 | 21:11 |
myoung | so looking now to see if he uses the changed file (that now uses z1 vs. passed_phase_1) - as far as i recall his stuff is pulling the puddle from the current_build variable, so should be fine but wanted to make sure | 21:12 |
myoung | so we don't get a silent puddle mismatch | 21:12 |
myoung | rlandy: i think it's fine | 21:13 |
myoung | he does pass that yaml (00:01:33.725 + ansible-playbook -vvvv /home/rhos-ci/jenkins/workspace/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/playbooks/baremetal-undercloud-validate-ha.yml -e @/home/rhos-ci/jenkins/workspace/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/config/release/rhos-13.yml ) | 21:14 |
myoung | but | 21:14 |
myoung | current_build is defined so it just uses the specifc puddle # passed to it | 21:14 |
myoung | sorry...wanted to make sure as he's on PTO and those jobs have recently been stabilized | 21:14 |
*** agopi is now known as agopi|out | 21:20 | |
*** agopi|out has quit IRC | 21:24 | |
*** jrist has quit IRC | 21:55 | |
*** apetrich has quit IRC | 22:01 | |
*** vinaykns has quit IRC | 22:06 | |
*** sshnaidm is now known as sshnaidm|off | 22:45 | |
weshay|ruck | rlandy, :( no nodes captured in the logs https://thirdparty.logs.rdoproject.org/jenkins-tq-gate-rhos-13-ci-rhos-ovb-featureset001-130/ | 22:56 |
rlandy | weshay|ruck: the create stacks and access ahve been failing | 22:56 |
rlandy | I switched to the regular tenant | 22:56 |
rlandy | not looking good | 22:56 |
weshay|ruck | rlandy, ya. but that is the undercloud install we were watching | 22:57 |
rlandy | it's not getting there | 22:57 |
weshay|ruck | k k | 22:57 |
weshay|ruck | what evs | 22:57 |
rlandy | I am watching the regular tenant now | 22:58 |
weshay|ruck | rlandy, k | 22:58 |
weshay|ruck | rlandy, https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-featureset001/134/console | 22:58 |
weshay|ruck | that one? | 22:58 |
rlandy | for the moment | 22:58 |
weshay|ruck | k | 22:58 |
rlandy | | 02c2ed16-39fe-49d6-9c29-ee51a83d6a60 | bmc-14813 | BUILD | | bmc-base | m1.small | | 22:59 |
rlandy | | f5218bbb-f809-4f9d-af75-ffbbf6f5ccde | undercloud-14813 | BUILD | | | m1.xlarge | | 22:59 |
rlandy | | 02958638-df42-443a-9e9c-5753e763e4ff | baremetal-14813_0 | ACTIVE | overcloud_tenant-14813=172.16.0.5; public-14813=10.0.0.7; overcloud_internal-14813=172.17.0.8; overcloud_storage-14813=172.18.0.2; provision-14813=192.168.24.7; overcloud_storage_mgmt-14813=172.19.0.6 | | m1.large | | 22:59 |
rlandy | | d2e58e58-e6c9-41d1-ab9d-e84c7eeff063 | baremetal-14813_1 | ACTIVE | overcloud_tenant-14813=172.16.0.9; public-14813=10.0.0.9; overcloud_internal-14813=172.17.0.5; overcloud_storage-14813=172.18.0.1; provision-14813=192.168.24.5; overcloud_storage_mgmt-14813=172.19.0.7 | | m1.large | | 22:59 |
rlandy | ^^ sitting there | 22:59 |
weshay|ruck | rlandy, are your jobs using this? | 22:59 |
weshay|ruck | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/jenkins/jobs/tripleo-quickstart/scripts/trigger-getbuild.sh#n112 | 22:59 |
rlandy | no | 23:01 |
rlandy | triggering off changes | 23:01 |
weshay|ruck | rlandy, :) | 23:01 |
weshay|ruck | ya... looks good | 23:01 |
weshay|ruck | well done | 23:01 |
*** jrist has joined #oooq | 23:01 | |
rlandy | crazy slow or broken, I don't know which atm | 23:04 |
rlandy | crazy slow | 23:04 |
rlandy | one just completed | 23:05 |
rlandy | may deploy before I officially retire | 23:05 |
weshay|ruck | lolz | 23:06 |
weshay|ruck | ya.. maybe rdo-cloud isn't so bad | 23:06 |
weshay|ruck | lolz | 23:06 |
rlandy | oh my gosh - I LOVE rdocloud | 23:06 |
rlandy | and the admins | 23:07 |
rlandy | they respond | 23:07 |
weshay|ruck | rlandy, maybe we should send those guys a redhat reward | 23:09 |
rlandy | weshay|ruck: yeah - I have sent in the past | 23:09 |
rlandy | maybe it's time again | 23:09 |
weshay|ruck | ya.. I'll see what I have as well | 23:09 |
rlandy | weshay|ruck: I can do it - then you can approve | 23:10 |
weshay|ruck | rlandy, it's their manager, not me | 23:10 |
rlandy | hmmm... I thought it was both managers | 23:10 |
rlandy | anyways | 23:10 |
rlandy | I have points if you need | 23:10 |
weshay|ruck | I'll do it, save urs.. | 23:11 |
weshay|ruck | fyi http://eavesdrop.openstack.org/irclogs/%23oooq/ | 23:21 |
weshay|ruck | rlandy, oh btw.. I requested an upshift account today, I put us as admins | 23:21 |
weshay|ruck | the compute nodes of rdo cloud may be moved to upshift | 23:22 |
weshay|ruck | so .. | 23:22 |
weshay|ruck | there's that | 23:22 |
weshay|ruck | rlandy, we have several delete_failed in ci-rhos | 23:24 |
rlandy | weshay|ruck: nice on the upshift ... | 23:24 |
* weshay|ruck remvoing | 23:24 | |
rlandy | I know about the delete failed | 23:24 |
rlandy | they are not taking resources | 23:24 |
weshay|ruck | k | 23:25 |
rlandy | I removed all the created failed ones | 23:25 |
rlandy | we do need to get them removed but they should not be holding anything up | 23:25 |
weshay|ruck | rlandy, did I get this fixed correctly https://review.rdoproject.org/r/#/c/14482/ | 23:25 |
rlandy | weshay|ruck: looks ok - I +1'ed | 23:28 |
rlandy | will have to try it out | 23:28 |
rlandy | unless you have and can post results | 23:28 |
rlandy | weshay|ruck: this internal cloud has some how slowed down even more | 23:28 |
weshay|ruck | man.. 10.0.0.8 | 23:29 |
rlandy | I know I can put in a ticket | 23:29 |
rlandy | but d we have a direct contact? | 23:29 |
weshay|ruck | rlandy, I wish we had some production jobs here.. to light them up w/ an escalation | 23:29 |
rlandy | it's unusable now | 23:29 |
weshay|ruck | rlandy, undercloud is 10.0.0.8 ya? | 23:29 |
rlandy | ssh root@10.8.241.126 | 23:29 |
weshay|ruck | whayutin•~/OPENSTACK_API/ci-rhos ᐅ ping 10.8.241.126 thinkdoe ⌚ 19:30:11 | 23:30 |
weshay|ruck | PING 10.8.241.126 (10.8.241.126) 56(84) bytes of data. | 23:30 |
weshay|ruck | ^C | 23:30 |
weshay|ruck | --- 10.8.241.126 ping statistics --- | 23:30 |
weshay|ruck | 25 packets transmitted, 0 received, 100% packet loss, time 24602ms | 23:30 |
rlandy | there is something wrong here | 23:32 |
* rlandy checks horizon | 23:32 | |
rlandy | https://ci-rhos.centralci.eng.rdu2.redhat.com/dashboard/project/instances/f5218bbb-f809-4f9d-af75-ffbbf6f5ccde/ | 23:36 |
weshay|ruck | rlandy, https://docs.engineering.redhat.com/display/CentralCI/Central+CI+Home | 23:37 |
* weshay|ruck opening a ticket | 23:37 | |
rlandy | weshay|ruck: thank you | 23:38 |
rlandy | I think it's getting slower and slower | 23:38 |
rlandy | if that i spossible | 23:38 |
rlandy | maybe they can reboot it? | 23:39 |
weshay|ruck | rlandy, https://redhat.service-now.com/surl.do?n=PNT0332785 | 23:47 |
weshay|ruck | you should be added to the ticket | 23:47 |
rlandy | weshay|ruck: thanks - I can access it | 23:48 |
rlandy | adding to the ticket | 23:48 |
weshay|ruck | rlandy, I think we should take a virthost offline | 23:49 |
rlandy | weshay|ruck: the slave? | 23:49 |
weshay|ruck | tomorrow maybe I'll try that tomorrow | 23:49 |
rlandy | ok but why? | 23:49 |
weshay|ruck | ya.. just to see if we can get osp-13 to work | 23:49 |
rlandy | ah ok | 23:50 |
weshay|ruck | I guess rasca has done that though? | 23:50 |
rlandy | yes | 23:50 |
weshay|ruck | is it a waste? | 23:50 |
weshay|ruck | ok | 23:50 |
* rlandy gets | 23:50 | |
rlandy | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/oooq-rhos-13-puddle-bmu-ha-lab-phoenix-float_nic_with_vlans/ | 23:51 |
weshay|ruck | ah cool | 23:51 |
rlandy | weshay|ruck: it's not a waste if we want that for a reason | 23:51 |
rlandy | rasca is not deploying fs 001 | 23:51 |
weshay|ruck | ya.. only we were to run it nightly | 23:51 |
weshay|ruck | or weekly as you were suggesting | 23:51 |
rlandy | weshay|ruck: we could also use one of our bm | 23:52 |
rlandy | if ovb becomes useless | 23:52 |
rlandy | we know it works with queens | 23:52 |
* rlandy needs to run out ... will pick this up later | 23:53 | |
*** rlandy is now known as rlandy|bbl | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!