*** dmellado has quit IRC | 01:22 | |
*** rlandy has quit IRC | 02:25 | |
*** apetrich has quit IRC | 02:26 | |
*** dmellado has joined #oooq | 02:38 | |
*** sanjayu_ has joined #oooq | 03:20 | |
*** sanjayu_ has quit IRC | 05:34 | |
*** honza has joined #oooq | 05:43 | |
*** ykarel has joined #oooq | 05:54 | |
*** jfrancoa has joined #oooq | 06:18 | |
*** ccamacho has joined #oooq | 06:20 | |
*** ccamacho has quit IRC | 06:20 | |
*** ccamacho has joined #oooq | 06:20 | |
*** gkadam has joined #oooq | 06:31 | |
*** saneax has joined #oooq | 06:31 | |
*** marios|rover has joined #oooq | 06:40 | |
*** amoralej|off is now known as amoralej | 07:01 | |
*** dmellado has quit IRC | 07:04 | |
*** dmellado has joined #oooq | 07:06 | |
*** bogdando has joined #oooq | 07:10 | |
*** dmellado has quit IRC | 07:12 | |
*** tosky has joined #oooq | 07:13 | |
*** sshnaidm|afk is now known as sshnaidm | 07:17 | |
*** dmellado has joined #oooq | 07:19 | |
*** dtantsur|afk is now known as dtantsur | 07:22 | |
*** ccamacho has quit IRC | 07:33 | |
*** jtomasek has joined #oooq | 07:41 | |
*** tosky has quit IRC | 07:43 | |
*** tosky has joined #oooq | 07:43 | |
*** jaganathan has quit IRC | 07:52 | |
*** amoralej is now known as amoralej|brb | 08:16 | |
*** ccamacho has joined #oooq | 08:16 | |
ssbarnea|ruck | marios|rover: what to do about the timeouts? even our fix failed during post on one job, so it was not merged. | 08:22 |
---|---|---|
*** ccamacho has quit IRC | 08:25 | |
*** ccamacho has joined #oooq | 08:26 | |
marios | ssbarnea|ruck: which fix you mean for the updates job? | 08:26 |
*** holser_ has joined #oooq | 08:26 | |
marios | ssbarnea|ruck: i didn't check there yet. I was about to look at that nova issue again since ykarel commented it didn't fix it (or we dn't have new enough nova yet) | 08:27 |
ssbarnea|ruck | https://review.openstack.org/#/c/592577/ | 08:27 |
marios | ssbarnea|ruck: ack | 08:28 |
marios | ssbarnea|ruck: maybe comment there on the review with pointers if you have more info yet | 08:28 |
ssbarnea|ruck | without this, the newer code from delorean-current would not be used, right? | 08:28 |
marios | ssbarnea|ruck: yeah the repo wouldn't be enabled | 08:29 |
ssbarnea|ruck | i only did a recheck few minutes ago, so it would take 3h to get it passed, or ... not. | 08:29 |
marios | ssbarnea|ruck: watching the console in zuul is always fun ;) http://zuul.openstack.org/ | 08:29 |
marios | autoscroll is particularly thrilling | 08:30 |
*** amoralej|brb is now known as amoralej | 08:47 | |
*** tosky has quit IRC | 08:59 | |
*** tosky has joined #oooq | 09:00 | |
*** chem has joined #oooq | 09:19 | |
*** amoralej is now known as amoralej|brb | 09:24 | |
*** ykarel_ has joined #oooq | 09:28 | |
*** ykarel has quit IRC | 09:31 | |
*** ccamacho has quit IRC | 09:54 | |
*** ccamacho has joined #oooq | 09:55 | |
*** amoralej|brb is now known as amoralej | 09:56 | |
*** ykarel_ is now known as ykarel | 09:58 | |
ssbarnea|ruck | marios: where are our jjb files for jobs defined on jenkins? | 10:01 |
marios | ssbarnea|ruck: dono | 10:02 |
*** jaosorior_ has quit IRC | 10:05 | |
ykarel | sshnaidm, ci-centos? if yes then they are in ci-config repo | 10:06 |
sshnaidm | ssbarnea|ruck, ^^ | 10:09 |
ssbarnea|ruck | sshnaidm: thanks, i didn't know the reponame. | 10:14 |
*** ykarel is now known as ykarel|lunch | 10:29 | |
*** ykarel|lunch is now known as ykarel|away | 10:34 | |
ssbarnea|ruck | marios: can you please have a look at https://ci.centos.org/job/tripleo-quickstart-gate-ocata-delorean-quick-basic/4956/console which seems to be affected by a bug that was fixed on Aug 11: https://bugs.launchpad.net/tripleo/+bug/1786106 | 10:35 |
openstack | Launchpad bug 1786106 in tripleo "containers-prep fails w/ update_containers is undefined due to lack of default" [High,Fix released] - Assigned to wes hayutin (weshayutin) | 10:35 |
ssbarnea|ruck | clearly something is not working, should I reopen the bug and put an alert on it? | 10:36 |
ssbarnea|ruck | if it still afects us, it means fix was incomplete, right. | 10:36 |
marios | ssbarnea|ruck: ack maybe add a comment with pointer to the logs on the bug as starter? agree if it doesn't fix the problem then we re-open the bug | 10:37 |
marios | ssbarnea|ruck: i'll also have a look in a sec | 10:37 |
ssbarnea|ruck | already did | 10:37 |
ssbarnea|ruck | impression that the fix should also be added to the extras-common role, initially was added to container-prep role. | 10:39 |
marios | ack ssbarnea|ruck | 10:39 |
dalvarez | chandankumar: arxcruz o/ i have a doubt regarding tempestconf | 10:40 |
dalvarez | i see that in the gate we're doing: | 10:40 |
dalvarez | --debug \ | 10:40 |
dalvarez | --remove network-feature-enabled.api_extensions=dvr \ | 10:40 |
dalvarez | chandankumar: arxcruz how can i remove other extension only for scenario007 job? i'd like to disable dhcp_agent_scheduler for the OVN case | 10:41 |
arxcruz | dalvarez: hey, so, you need to overwrite the tempest_conf_removal | 10:47 |
arxcruz | dalvarez: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/defaults/main.yml#L71 | 10:47 |
arxcruz | on the featureset that scenario007 uses | 10:47 |
ssbarnea|ruck | marios: this is weird, the extras-common/tasts/main.yml is empty in git... do we generate it? | 10:47 |
tosky | dalvarez, arxcruz: also, please note that, if I remember correctly, that setting is used only because the dvr extension was advertised by neutron also when not enabled | 10:49 |
dalvarez | arxcruz: oh nice nice! thanks a lot | 10:49 |
tosky | dalvarez, arxcruz: it would be better if you can disable dhcp_agent_scheduler at the deployment time, and neutron does not advertise it | 10:49 |
dalvarez | tosky: good point, indeed | 10:49 |
dalvarez | let me check that, this is perhaps the right way as you point out | 10:50 |
tosky | yep, that was a workaround for a neutron bug | 10:50 |
tosky | which was later fixed, so maybe at some point that override can be removed | 10:51 |
dalvarez | yeah now i remember | 10:51 |
dalvarez | good one | 10:51 |
marios | ssbarnea|ruck: added a comment | 10:52 |
marios | ssbarnea|ruck: i think it might be we need the fix in https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/ocata.yml#L61 too? | 10:52 |
dalvarez | arxcruz: still.. the tempest validator can be adjusted per featureset? | 10:52 |
arxcruz | dalvarez: what you mean ? | 10:52 |
*** sanjayu_ has joined #oooq | 10:53 | |
*** saneax has quit IRC | 10:53 | |
dalvarez | arxcruz: i mean that the validate-tempest role that you linked is the one setting the extensions removal, can that be done per featureset/scenario? | 10:54 |
dalvarez | i thought it was just a global thing for all jobs | 10:54 |
*** sanjayu__ has joined #oooq | 10:55 | |
arxcruz | dalvarez: yeah, it can be done per featureset, i don't see why not | 10:55 |
dalvarez | arxcruz: are we doing it somewhere already so that i can take it as an example? | 10:56 |
arxcruz | dalvarez: i don't think so | 10:57 |
arxcruz | dalvarez: let me check | 10:57 |
tosky | why not? It's just a value to pass to tempestconf | 10:57 |
tosky | I mean, we have examples | 10:57 |
arxcruz | tosky: on featuresets ? | 10:57 |
tosky | oh, no, that's for overriding, not for removal | 10:58 |
*** apetrich has joined #oooq | 10:58 | |
*** sanjayu_ has quit IRC | 10:58 | |
tosky | talking about overriding options | 10:58 |
arxcruz | dalvarez: we can remove the default from validate-tempest, and override in all featuresets | 10:58 |
arxcruz | adding the dvr option for removal | 10:58 |
tosky | sshnaidm: I'm not sure I get the comment in https://review.openstack.org/#/c/509554/ , does it require any action? | 10:59 |
dalvarez | arxcruz: got it im just not familiar with this, i'll take a look | 10:59 |
dalvarez | thanks! | 10:59 |
sshnaidm | tosky, your changes to scenario008 are ignored, up to you | 10:59 |
sshnaidm | tosky, this job doesn't run tempest | 11:00 |
tosky | sshnaidm: featureset008, you mean? | 11:01 |
sshnaidm | tosky, yes | 11:01 |
tosky | so it changed | 11:01 |
tosky | I'm 100% sure it did | 11:02 |
tosky | sshnaidm: so the table is not updated? https://docs.openstack.org/tripleo-quickstart/latest/feature-configuration.html | 11:02 |
*** jaosorior has joined #oooq | 11:15 | |
sshnaidm | tosky, seems so | 11:16 |
sshnaidm | tosky, it did for branches after ocata according to code in featureset | 11:16 |
sshnaidm | tosky, but because this job doesn't run in branches after ocata... | 11:17 |
tosky | sshnaidm: the job is not run, but the featureset is a featureset that deploys manila with tempest | 11:17 |
tosky | my point is that all the featuresets that deploys manila with tempest should run those tempest tests | 11:18 |
*** ccamacho has quit IRC | 11:18 | |
tosky | if the featureset is not executed on the gates, it's not a problem on my side | 11:18 |
tosky | either it will continue to live there and be used by users from time to time, or disappear at some point | 11:19 |
tosky | but I want consistency | 11:19 |
*** ykarel|away has quit IRC | 11:19 | |
sshnaidm | tosky, I see, just fyi that it will never run tempest | 11:20 |
tosky | sshnaidm: if used, that featureset runs tempest | 11:21 |
tosky | where do you see that it does not? | 11:21 |
sshnaidm | tosky, I don't remember, did we set containers as default in pike? | 11:21 |
sshnaidm | tosky, I explained above | 11:21 |
tosky | sshnaidm: how is that relevant? | 11:21 |
tosky | then I didn't get it | 11:21 |
sshnaidm | tosky, what's not clear exactly? | 11:22 |
tosky | sshnaidm: featureset007 has run_tempest: true from pike onwards | 11:22 |
tosky | if no jobs from pike uses that featureset, it's not a problem | 11:23 |
sshnaidm | tosky, it can have whatever, it doesn't mean anything if we don't support scenario004 in containers | 11:23 |
*** ccamacho has joined #oooq | 11:25 | |
tosky | sshnaidm: again, either at some point you will run a job with containers with scenario004, or you will remove that featureset | 11:26 |
sshnaidm | tosky, I'm not against this patch, it's fine with me, I'd like all realize that it won't run tempest in CI in case you expect it to do | 11:26 |
tosky | sshnaidm: I don't expect that | 11:26 |
tosky | I already wrote it | 11:26 |
tosky | what I need to ensure is that ANY featureset with deployes manila AND tempest, IF used by anyone, will run those manila tests | 11:26 |
tosky | that's it | 11:26 |
sshnaidm | tosky, containers don't run with featureset008, they run only with featureset019 | 11:26 |
tosky | see above | 11:27 |
tosky | so at some point I guess you will remove featureset017, if it should not be used because not supported | 11:27 |
tosky | sorry, featureset008 | 11:27 |
tosky | then fine, but until that point, as long as any featureset which support manila and tempest in any way is in the repository, their definition should be the same | 11:28 |
tosky | same definitions regarding the tempest tests executed | 11:28 |
sshnaidm | tosky, after ocata is EOLed it won't be used at all, right | 11:28 |
sshnaidm | tosky, ok, completely fine with it | 11:29 |
tosky | thanks | 11:30 |
*** saneax has joined #oooq | 11:39 | |
*** sanjayu__ has quit IRC | 11:42 | |
*** saneax has quit IRC | 11:42 | |
weshay | ssbarnea|ruck, ur not on the program call | 12:04 |
weshay | or I don't see u | 12:04 |
*** amoralej is now known as amoralej|lunch | 12:06 | |
weshay | ssbarnea|ruck, ? | 12:10 |
*** ssbarnea|ruck has quit IRC | 12:13 | |
*** trown|outtypewww is now known as trown | 12:21 | |
*** agopi has quit IRC | 12:28 | |
*** rlandy has joined #oooq | 12:32 | |
*** rnoriega_ has joined #oooq | 12:38 | |
sshnaidm | weshay, marios is it known that OVB jobs don't build dependencies? | 12:39 |
weshay | sshnaidm, build-test-packages is not running? | 12:39 |
weshay | sshnaidm, on check ovb or periodic? | 12:39 |
sshnaidm | weshay, check OVB jobs, seems like it's running, but doesn't build.. | 12:40 |
sshnaidm | weshay, will look later, just noticed | 12:40 |
weshay | panda|off, ready if you are | 12:41 |
panda|off | weshay: ready | 12:44 |
weshay | panda|off, https://etherpad.openstack.org/p/tripleo-python3-tripleoclient-issues | 12:49 |
*** agopi has joined #oooq | 12:56 | |
weshay | panda|off, https://nb01.openstack.org/images/ | 12:57 |
rlandy | weshay: when you have a moment, I have the logs from the overcloud image build using the diff centos node | 12:58 |
rlandy | rf0lc0: ^^ | 13:00 |
rlandy | pls see https://review.openstack.org/#/c/594308 and test job https://review.openstack.org/#/c/594548 | 13:00 |
rlandy | with logs http://logs.openstack.org/48/594548/1/check/tripleo-buildimage-overcloud-full-centos-7/ff19db5/ | 13:00 |
*** ssbarnea has joined #oooq | 13:03 | |
rf0lc0 | looking good to me, rlandy | 13:03 |
rf0lc0 | am I missing anything? the build is good | 13:03 |
ssbarnea | i just rejoined directly, after being kicked due to the |ruck suffix | 13:04 |
rlandy | rf0lc0, I am trying to compare logs | 13:04 |
rlandy | to see if the changed image made a diff | 13:04 |
*** amoralej|lunch is now known as amoralej | 13:05 | |
*** ssbarnea is now known as ssbarnea_ | 13:07 | |
rf0lc0 | ssbarnea, need to group your nicks to the registered account I think: https://freenode.net/kb/answer/registration | 13:08 |
marios | not somethign i know of sshnaidm | 13:08 |
rlandy | panda|off: sshnaidm: pls see https://review.openstack.org/#/c/594308/6/zuul.d/build-image.yaml and let me know if you have any concerns about changing the node here to single-centos--node | 13:08 |
rlandy | marios, ^^ you alredy +1'ed - I removed my w-1 | 13:09 |
marios | rlandy: ack | 13:10 |
* rlandy doesn't really know who uses this job | 13:11 | |
marios | rlandy: well i revoted | 13:11 |
rlandy | but hopes the node change is ok | 13:11 |
marios | rlandy: i dont have objections but i might not be the best person to ask ;) | 13:11 |
marios | rlandy: i can revote when we are ready | 13:11 |
sshnaidm | rlandy, don't we need now ha-utils and browbeat as they are in requirements? | 13:11 |
rlandy | sshnaidm: where are you pointing to? | 13:13 |
*** ssbarnea_ has quit IRC | 13:13 | |
sshnaidm | rlandy, https://review.openstack.org/#/c/594308/6/zuul.d/base.yaml | 13:13 |
*** ssbarnea_ has joined #oooq | 13:13 | |
rlandy | tripleo-ci-dsvm never touch either of those | 13:13 |
rlandy | they are included above | 13:13 |
rlandy | well only browbeat | 13:14 |
rlandy | tripleo-ha-utils we have not added anything yet | 13:14 |
rlandy | as we use tripleo-ha-utils, I will include it | 13:15 |
rlandy | the job would not build if we had an issue | 13:15 |
*** ssbarnea_ has quit IRC | 13:16 | |
*** ssbarnea|ruck has joined #oooq | 13:16 | |
*** ssbarnea|ruck has quit IRC | 13:18 | |
*** ssbarnea|ruck has joined #oooq | 13:18 | |
rf0lc0 | weshay, any ideas on how do I test a periodic legacy job with an upstream patch like this https://review.openstack.org/#/c/589448 ? Want to verify if the variables did not break anything. | 13:19 |
weshay | rf0lc0, I think sshnaidm has a test job out there | 13:21 |
weshay | test patch that you can use as an example | 13:21 |
weshay | rf0lc0, maybe like https://review.rdoproject.org/r/#/c/14773/ | 13:22 |
weshay | rf0lc0, https://review.rdoproject.org/r/#/c/13943/8 | 13:23 |
rf0lc0 | weshay, hmmm, moving to check pipeline... duh | 13:24 |
* rf0lc0 makes silly face | 13:24 | |
rf0lc0 | weshay, thank you master | 13:24 |
panda|off | sorry guys, I'll be off for the rest of the week. | 13:25 |
rf0lc0 | panda|off, ack let me know if you need any help | 13:27 |
rlandy | sshnaidm, https://review.openstack.org/#/c/594308/ is what we needed to fix for the reparenting to go forward | 13:27 |
rlandy | test job here ... https://review.openstack.org/#/c/594548 | 13:28 |
rlandy | apetrich: how's the tempest error debug going? | 13:41 |
rlandy | manage to reproduce it? | 13:41 |
*** dtrainor has quit IRC | 13:44 | |
apetrich | rlandy, nope. | 13:50 |
apetrich | also rdo cloud kind of behaving badly does not help | 13:52 |
apetrich | but for sure on a not-container undercloud it does not happen or does not happen often | 13:52 |
apetrich | on a container undercloud testing is a bit of a pain | 13:53 |
rlandy | apetrich: pls post the link to your bug again | 13:54 |
apetrich | rlandy https://bugs.launchpad.net/tripleo/+bug/1736950 | 13:55 |
openstack | Launchpad bug 1736950 in tripleo "CI: mistral testmistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2.test_get_list_actions_not_in_list_filter fails in gate scenario003 containers" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 13:55 |
weshay | ssbarnea|ruck, please investigate the rocky promotion job failures and get bugs asap | 13:56 |
weshay | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky/00dd61e/job-output.txt.gz | 13:56 |
weshay | ssbarnea|ruck, see https://review.rdoproject.org/zuul/status.html | 13:56 |
ssbarnea|ruck | weshay: ok | 13:56 |
weshay | sshnaidm, holla if you want/need help | 13:56 |
weshay | sshnaidm, sorry... meant ssbarnea|ruck | 13:57 |
rlandy | arxcruz: do you have some time today? I'd like to look at ^^ apetrich's bug and see what you would do about debugging that | 13:57 |
weshay | nick collision | 13:57 |
weshay | marios|rover, ^ | 13:57 |
apetrich | arxcruz, or if you want tomorrow we can meet somewhere to look at that face to face. | 13:57 |
rlandy | apetrich: I am trying to learn how to debug tempest failures better | 13:58 |
apetrich | today I'm neck deep in meetings then I have to leave | 13:58 |
rlandy | I'd like to use that as a teaching example if arxcruz has the time | 13:58 |
apetrich | rlandy, that makes us 2 | 13:58 |
marios | weshay: ack, i looked into that nova one earlier added a comment #12 @ https://bugs.launchpad.net/tripleo/+bug/1787910 | 13:59 |
openstack | Launchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 13:59 |
marios | weshay: meant to ask you about that re the versions | 13:59 |
rlandy | apetrich: ok - cool - your morning though is my midnight :( | 13:59 |
marios | weshay: err check comment #11 instead sorry | 13:59 |
*** dmellado has quit IRC | 14:01 | |
arxcruz | i have the time | 14:03 |
arxcruz | rlandy: apetrich | 14:03 |
arxcruz | although i don't know what you're talking about | 14:03 |
arxcruz | apetrich: tomorrow they are coming here to mount the wardrobe | 14:03 |
rlandy | arxcruz: https://bugs.launchpad.net/tripleo/+bug/1736950 | 14:04 |
openstack | Launchpad bug 1736950 in tripleo "CI: mistral testmistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2.test_get_list_actions_not_in_list_filter fails in gate scenario003 containers" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich) | 14:04 |
apetrich | arxcruz, so theres a mistral undercloud tempest test that shows as a failure. we are skipping it here https://review.openstack.org/#/c/592591/1/roles/validate-tempest/vars/tempest_skip_master.yml | 14:04 |
rlandy | we want to use this example to set up a debug env | 14:04 |
rlandy | and learn how you would work through this | 14:04 |
apetrich | but I'm unable to reproduce on a non-container undercloud | 14:04 |
apetrich | and I can't find a way to run a tempest container and tempest in it | 14:05 |
rlandy | tomorrow afternoon your time (morning my time)? | 14:05 |
rf0lc0 | anyone also experiencing problems with review.rdoproject.org ? | 14:06 |
rf0lc0 | review.rdoproject.org took too long to respond | 14:06 |
*** dtrainor has joined #oooq | 14:08 | |
arxcruz | rf0lc0: 1.1.1.1 ;) | 14:09 |
arxcruz | rlandy: apetrich sure, let's do it :) | 14:10 |
rlandy | arxcruz: I'm ready but apetrich has meetings | 14:12 |
rlandy | not sure if you want to do this twice over :) | 14:12 |
rlandy | up to you | 14:12 |
arxcruz | rlandy: do you have an env already up ? | 14:13 |
rlandy | arxcruz: no - that is what I want to learn to set up | 14:14 |
rlandy | you showed me your env once | 14:14 |
arxcruz | env i mean the openstack installed just to run tempest | 14:14 |
arxcruz | my env i do with reproduce script | 14:14 |
rlandy | oh no - but I can do that | 14:15 |
rlandy | depending on the responsiveness of rdocloud | 14:15 |
rlandy | arxcruz: let me set that up - will ping you when it is done | 14:15 |
arxcruz | ok | 14:15 |
arxcruz | i have an env up, but i'm testing tempestconf | 14:16 |
rlandy | trying the reproducer from this log ... | 14:16 |
rlandy | http://logs.openstack.org/16/592216/2/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/ | 14:17 |
rlandy | hmmm ... rdocloud not responding :( | 14:21 |
rlandy | arxcruz: ^^ :( | 14:21 |
rlandy | we may have to try again another time - sorry | 14:22 |
rlandy | or I could try the libvirt reproducere | 14:22 |
rlandy | or I could try the libvirt reproducer | 14:22 |
rlandy | does it matter to you which one we use? | 14:22 |
*** ssbarnea|ruck has quit IRC | 14:23 | |
rlandy | arxcruz; trying libvirt reproducer | 14:28 |
*** jrist has joined #oooq | 14:45 | |
arxcruz | rlandy: yeah, even my env is down now | 14:57 |
*** apetrich has quit IRC | 14:58 | |
rlandy | arxcruz: I am trying the libvirt reproducer but Resize undercloud image is failing | 14:58 |
marios | weshay: i've been chasing an issue (via grafana) for a bit and getting ready to file the bug but worried it might be because of the rax down? i was looking at tripleo-ci-centos-7-scenario003-multinode-oooq-container from https://review.openstack.org/#/c/560445/ via grafana. it times out on the overcloud deploy | 14:59 |
sshnaidm | weshay, marios rdo cloud is acting up again, fyi.. | 14:59 |
marios | weshay: afaics it is issue with containers http://paste.openstack.org/show/728607/ both getting registry but the error is problem with container delete | 15:00 |
marios | sshnaidm: thanks do you think this is related to rdo cloud? ^^^ or i should file a bug for that | 15:00 |
weshay | marios|rover, we need an alert bug stating rdo-cloud is down.. w/ alert and promotion blocker please | 15:00 |
weshay | marios, /me looks at your links | 15:01 |
sshnaidm | marios, no no, just fyi, not in this context | 15:01 |
*** ssbarnea|ruck has joined #oooq | 15:02 | |
marios | weshay: https://bugs.launchpad.net/tripleo/+bug/1788426 | 15:02 |
openstack | Launchpad bug 1788426 in tripleo "RDO Cloud is down" [Undecided,In progress] - Assigned to Marios Andreou (marios-b) | 15:02 |
marios | done lord vader | 15:02 |
marios | weshay: (hope thats what you meant) | 15:02 |
weshay | marios, lolz | 15:03 |
weshay | marios, that is perfect | 15:03 |
marios | ack sshnaidm thanks | 15:04 |
weshay | marios, so chat w/ me for a sec.. are you tracking down an issue w/ rax? | 15:06 |
marios | weshay: no i just looked at the first timeout in the check jobs from grafana | 15:06 |
*** rnoriega_ is now known as rnoriega | 15:07 | |
weshay | marios, k.. hard to see anything w/ rdo cloud being down :) | 15:07 |
weshay | http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 | 15:07 |
marios | weshay: which is http://logs.openstack.org/45/560445/122/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/4c6b063/ | 15:07 |
weshay | I guess openstack health | 15:07 |
marios | weshay: ok so i'll go ahead and file that bug then and see if any folks "from containers" know anything about that one | 15:08 |
marios | weshay: i want to see how often it is but cistatus.tripleo.org is nope | 15:08 |
weshay | http://status.openstack.org/elastic-recheck/ | 15:08 |
marios | (i guess rdo cloud) | 15:08 |
marios | weshay: well assuming we have a query for it already? | 15:09 |
weshay | marios, timeouts should be tracked w/ 143 error | 15:09 |
marios | weshay: or you mean propose one | 15:09 |
weshay | ya.. there is | 15:09 |
weshay | search that page for 143 | 15:09 |
weshay | marios, er.. sorry Bug 1686542 - Generic job timeout bug | 15:09 |
openstack | bug 1686542 in OpenStack-Gate "Generic job timeout bug" [Low,Confirmed] https://launchpad.net/bugs/1686542 | 15:09 |
marios | weshay: you mean for this particular ... oh you mean there is a job timeout one | 15:09 |
marios | weshay: ack | 15:09 |
weshay | http://logstash.openstack.org/#dashboard/file/logstash.json?query=(message%3A%20%5C%22FAILED%20with%20status%3A%20137%5C%22%20OR%20message%3A%20%5C%22FAILED%20with%20status%3A%20143%5C%22)%20AND%20tags%3A%20%5C%22console%5C%22%20AND%20voting%3A1 | 15:10 |
weshay | then you can add filters like tripleo | 15:10 |
marios | weshay: ok not so frequent then | 15:10 |
marios | weshay: 1 fail in 24 hours | 15:11 |
marios | weshay: but why is our upstream ci status suffering (at 76.1 now) | 15:11 |
marios | all the things are super slow today | 15:11 |
marios | weshay: k gonna hold on the bug... i have notes. lets see if it happens more than once ;) | 15:14 |
weshay | k | 15:14 |
marios | weshay: did you check comment #11 https://bugs.launchpad.net/tripleo/+bug/1787910 about the nova versions... so how do we get that newer version | 15:17 |
openstack | Launchpad bug 1787910 in tripleo "OVB overcloud deploy fails on nova placement errors" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 15:17 |
marios | sshnaidm: rlandy if you have time ^^ | 15:17 |
rlandy | marios: is there a review to look at or you want us to work on the bug? | 15:18 |
sshnaidm | marios, if we have time..? | 15:18 |
marios | sshnaidm: heh:) rlandy no i meant, does what i write make sense. we need a particular version of nova (from 22nd after the patch merged) | 15:19 |
rlandy | anyone else have libvirt reproducer fail on virt-resize? | 15:20 |
marios | sshnaidm: weshay rlandy not clear to me how we get that. does it mean a promotion (if so we have a problem since the bug is itself promotion blocker). or we just wait? | 15:20 |
marios | (aka recheck) | 15:20 |
weshay | rlandy, run the command manually from your host | 15:21 |
weshay | and it will be more clear what the error is | 15:21 |
weshay | marios, your question is in reference to 1787910 | 15:22 |
weshay | ? | 15:22 |
marios | weshay: yes comment #11 specifically | 15:22 |
marios | weshay: ykarel said 'not fixed yet' and i agree, even though the patch merged. We aren't gettign new enough version of nova. | 15:23 |
weshay | marios, ya.. if the patch is not in the build.. then bug remains open | 15:24 |
weshay | tracking down the githash for the fix.. vs. .the dlrn package id | 15:24 |
weshay | is the right way to figure that out | 15:24 |
weshay | so you did that it looks like | 15:24 |
weshay | marios, so the next question is .. is rocky consistent? | 15:24 |
* weshay looks | 15:24 | |
weshay | http://rhos-release.virt.bos.redhat.com:3030/rhosp | 15:25 |
weshay | ah.. we're not tracking rocky | 15:25 |
weshay | just master | 15:25 |
marios | weshay: damn i missed the ci escalation status call | 15:25 |
marios | :/ | 15:25 |
marios | just realised | 15:25 |
weshay | marios, meh | 15:25 |
weshay | I was there | 15:25 |
weshay | so. this bug is on master | 15:25 |
rlandy | I hate this error ... Message: No valid host was found. There are not enough hosts available., Code: 500" | 15:25 |
weshay | rlandy, ya. .terrible | 15:26 |
rlandy | openstack just spews that when it is not sure what else to say | 15:26 |
weshay | marios, /me looks at the repos | 15:26 |
weshay | just to double check | 15:26 |
weshay | openstack-nova-18.0.0-0.20180822065816.17b6957.el7.noarch.rpm | 15:27 |
weshay | marios, that is the latest patch on master nova https://github.com/openstack/nova/commits/master | 15:27 |
marios | weshay: that one should do it | 15:27 |
marios | weshay: its what in current | 15:27 |
weshay | marios, so to get ahead of it.. "IF" we wanted to | 15:28 |
marios | [2] https://trunk.rdoproject.org/centos7-master/current/ | 15:28 |
marios | weshay: ^ | 15:28 |
weshay | meh.. nvrmind | 15:28 |
marios | has nova-18.0.0-0.20180822065816.17b6957.el7.noarch.rpm | 15:28 |
weshay | marios, ya.. we don't pick up current | 15:28 |
weshay | we pick up consistent | 15:28 |
weshay | but ya | 15:28 |
weshay | atm they are the same | 15:28 |
weshay | marios ya.. so ur right man re: the bug | 15:31 |
marios | weshay: i assure you, it wasn't intentional! | 15:32 |
* marios brb | 15:32 | |
weshay | can't do shit atm | 15:33 |
weshay | no logs | 15:33 |
marios | weshay: so my question still stands. do we need promotion to get that newer version? or we just wait? | 15:34 |
marios | weshay: if it is the former, we need promotion, then we have a problem | 15:34 |
weshay | the promotion jobs ALWAYS pull the latest consistent rpms | 15:35 |
weshay | from the whole of openstack | 15:35 |
weshay | I can't tell what yatin was looking at atm.. because rdo logs are down | 15:35 |
weshay | but if we did go to the job he pulled as an example | 15:35 |
marios | weshay: oh so we dont needa promotion to get the newer nova... it will just get pulled into the next promotion job | 15:35 |
weshay | and looked at rpm-qa.txt in the base | 15:35 |
weshay | we'd know | 15:35 |
weshay | marios, the promotion jobs always run w/ the latest openstack rpms | 15:36 |
weshay | marios, ssbarnea|ruck did you guys read through my doc? | 15:36 |
weshay | :)) | 15:36 |
marios | weshay: yeah it was very helpful | 15:36 |
weshay | k.. | 15:36 |
marios | weshay: 3 times :) | 15:36 |
weshay | lolz | 15:36 |
marios | weshay: i need another pass | 15:36 |
weshay | he he | 15:36 |
weshay | lolz | 15:36 |
weshay | marios, you sound like morazi | 15:36 |
marios | weshay: i worked for him for a long time :) | 15:37 |
ssbarnea|ruck | 2 times, i only managed 0.7 times. | 15:37 |
rlandy | hmmm ... Could not open '/tmp/reproduce-tmp.jymb5/undercloud-resized.qcow2': Permission denied | 15:40 |
rlandy | run as non-root-user | 15:40 |
rlandy | weird | 15:42 |
rlandy | has not changed in a while | 15:42 |
*** dmellado has joined #oooq | 15:45 | |
weshay | rlandy, tmate? | 15:46 |
rlandy | sec setting up | 15:48 |
*** gkadam is now known as gkadam-afk | 15:52 | |
rlandy | weshay++ | 16:04 |
rlandy | for scaring my libvirt into action | 16:04 |
weshay | heh | 16:10 |
rlandy | marios: are you all set? still want help with ovb fun? | 16:13 |
marios | rlandy: /me hometime | 16:13 |
marios | continue tomorrow | 16:14 |
rlandy | marios: ack ok | 16:14 |
marios | rlandy: thanks for checking | 16:14 |
rlandy | need to take another pass at your reviews | 16:14 |
rlandy | marios: ^^ | 16:14 |
rlandy | will do so if rdocloud returns | 16:14 |
marios | rlandy: k have a good 1 | 16:15 |
rlandy | sure | 16:15 |
*** gkadam-afk is now known as gkadam | 16:49 | |
*** ykarel has joined #oooq | 16:51 | |
*** trown is now known as trown|lunch | 16:52 | |
*** gkadam has quit IRC | 16:54 | |
ykarel | weshay, sshnaidm are we waiting for something to get https://review.openstack.org/#/c/591982/ +W? | 17:04 |
rf0lc0 | weshay, do you have the power to merge this https://review.rdoproject.org/r/#/c/15825/1/resources/missing_resources_1489486510.yaml | 17:07 |
weshay | ykarel, it's been workflowed.. just also has been rebased | 17:19 |
weshay | ykarel, taking care of it | 17:19 |
weshay | rf0lc0, sec | 17:19 |
ykarel | weshay, okk Thanks | 17:19 |
rf0lc0 | weshay, nicolas said r.r.o might not have fully recovered yet | 17:20 |
weshay | rf0lc0, I just have +1 | 17:20 |
weshay | lolz | 17:20 |
*** ykarel has quit IRC | 17:20 | |
rf0lc0 | weshay, you are not helping ! | 17:21 |
rf0lc0 | :) | 17:21 |
*** ykarel has joined #oooq | 17:21 | |
*** vinaykns has joined #oooq | 17:21 | |
*** vinaykns has left #oooq | 17:22 | |
rlandy | ovb still down :( | 17:22 |
rlandy | all my jobs are red | 17:22 |
rlandy | arxcruz: I have an env up on libvirt but it did not fail tempest | 17:23 |
*** ykarel_ has joined #oooq | 17:24 | |
*** ykarel_ has quit IRC | 17:24 | |
*** ykarel_ has joined #oooq | 17:24 | |
rlandy | weshay: sorry to bug you about this ... https://review.openstack.org/#/c/594308/ - ok to merge? https://review.openstack.org/#/c/594548 shows the buildimage test job | 17:25 |
ykarel_ | many nodes here are in deleting state: https://review.rdoproject.org/zuul/nodes.html, is someone taking care of those | 17:25 |
*** rnoriega has quit IRC | 17:25 | |
*** rf0lc0 has quit IRC | 17:25 | |
*** panda|off has quit IRC | 17:25 | |
*** marios has quit IRC | 17:25 | |
*** jschlueter has quit IRC | 17:25 | |
*** weshay has quit IRC | 17:25 | |
ykarel_ | weshay, ^^ | 17:25 |
*** ykarel has quit IRC | 17:26 | |
*** ykarel_ has quit IRC | 17:27 | |
*** holser_ has quit IRC | 17:30 | |
*** bogdando has quit IRC | 17:41 | |
*** rodrigods has joined #oooq | 17:44 | |
*** rfolco has joined #oooq | 17:50 | |
*** weshay has joined #oooq | 17:50 | |
weshay | it's IDENTIFY NOT LOGIN | 17:51 |
weshay | :)) | 17:51 |
rlandy | weshay: welcome back | 17:51 |
weshay | stupid wes | 17:51 |
rfolco | I am also struggling with identify on hexchat | 17:51 |
rfolco | connect commands: I set the identify... but this happens after I get kicked from channels | 17:52 |
rfolco | where am I noob'ing ? | 17:52 |
rlandy | what is the login method in hexchat? | 17:52 |
rfolco | custom command | 17:52 |
rlandy | hexchat -> network list -> look at login command | 17:53 |
rfolco | this ==> msg NickServ IDENTIFY rf0lc0 %p | 17:53 |
rlandy | mine is SASL username and password | 17:53 |
rlandy | ^^ when you click on freenode and Edit | 17:54 |
rfolco | let me try | 17:54 |
rlandy | add the password | 17:54 |
rlandy | to the password field | 17:54 |
rlandy | make sure use glabl user info | 17:55 |
rlandy | global | 17:55 |
rfolco | ok. quit and restart | 17:55 |
rfolco | thx rlandy | 17:56 |
rlandy | rfolco: worked? | 17:56 |
rfolco | trying | 17:56 |
*** rfolco has quit IRC | 17:56 | |
rlandy | I guess now since he did not come back | 17:56 |
*** rfolco has joined #oooq | 18:05 | |
rlandy | rfolco: I guess that didn't help :( | 18:09 |
rlandy | you disappeared for too long | 18:10 |
rfolco | rlandy, :( | 18:10 |
rfolco | I can manually identify myself | 18:10 |
rfolco | after | 18:10 |
rfolco | but hexchat doesn't do that when I connect to freenode | 18:10 |
rfolco | :( | 18:10 |
rlandy | sorry - using the SASL login method and adding my password in the dialog works for me | 18:12 |
*** ChanServ has quit IRC | 18:16 | |
*** trown|lunch is now known as trown | 18:21 | |
*** ChanServ has joined #oooq | 18:22 | |
*** barjavel.freenode.net sets mode: +o ChanServ | 18:22 | |
*** dmellado has quit IRC | 18:22 | |
*** dtantsur is now known as dtantsur|afk | 18:43 | |
*** tosky has quit IRC | 18:47 | |
*** amoralej is now known as amoralej|off | 18:51 | |
rfolco | cores can please do a final pass on https://review.openstack.org/#/c/589448 ? sshnaidm marios|rover rlandy weshay | 18:56 |
rlandy | review.rdoproject still down:( | 18:56 |
rlandy | rfolco: are the results you posted from a review that ran with the new workflow | 18:59 |
rlandy | the tests in the actual review would not trigger | 18:59 |
rfolco | yes rlandy, I used different jobs for that patch | 18:59 |
rfolco | why? | 19:00 |
rlandy | only missing var to check is dryrun[4]. Should be good to merge. | 19:00 |
rlandy | ^^ not sure about that | 19:00 |
* rlandy is so out of the loop on this change | 19:02 | |
rlandy | arxcruz: you around? | 19:02 |
arxcruz | rlandy: kind of :) | 19:02 |
rfolco | rlandy, dryrun is on demand flag... not a risk | 19:03 |
arxcruz | rlandy: can we go through tomorrow? it's 9pm here :) | 19:03 |
rlandy | arxcruz: ok - I have an env ready - tomorrow is fine | 19:03 |
*** jfrancoa has quit IRC | 19:08 | |
weshay | rfolco, so question | 19:16 |
weshay | we're leaving playook | 19:16 |
weshay | sec | 19:16 |
rlandy | what happened to our reproducer card? | 19:17 |
rlandy | weshay: rfolco: can I move this card into progress ... would like to start putting notes in there wrt reproducer and zuul-emulator mode | 19:20 |
rlandy | probably won;t complete this sprint - but we can start here | 19:20 |
rfolco | rlandy, agreed... we started to accumulate leftovers from previous sprints :( | 19:21 |
weshay | rlandy, so that is tenatively working out? | 19:25 |
rlandy | weshay: the zuul_changes thing is not so simple - I am experimenting with the build-test-packages role | 19:26 |
rlandy | per adarazs's work a while back | 19:26 |
rlandy | never the less | 19:26 |
weshay | rlandy, what about zuul cloner? | 19:27 |
rlandy | I have done some experimentation and want to log notes somewhere | 19:27 |
rlandy | well if zuul_cloner is deprecated, probably not the way to go | 19:27 |
rlandy | rfolco: left a question on https://review.openstack.org/#/c/589448 | 19:28 |
rlandy | weshay: mostly I wanted somewhere to put the poc reviews and notes for tomorrow's meeting discussion | 19:29 |
rlandy | weshay: we need a supported way to get the depends-on jobs from the gerrit change | 19:29 |
rlandy | w/o relying on jenkins or zuul | 19:29 |
rlandy | because if we want a stanalone reproducer-type script | 19:29 |
rlandy | to replace quickstart.sh, we can't rely on either | 19:30 |
rlandy | build-test-packages uses one or the other | 19:30 |
rlandy | and then there are all the centos assumptions in nodepool-setup | 19:31 |
weshay | rlandy, we should chat w/ paul | 19:34 |
rlandy | weshay: about what part? | 19:35 |
rlandy | the way to pull depends-on changes w/o zuul? | 19:36 |
weshay | ya | 19:37 |
rlandy | rfolco: will try to comment intelligently but I am way out of the loop on this vars changes | 19:37 |
weshay | rfolco, so I'm starting to get interested again in what we can tear out of these jobs | 19:45 |
rfolco | weshay, which jobs ? | 19:46 |
weshay | rfolco, /me goes to review marios's patches | 19:47 |
weshay | sec.. talking to paul | 19:47 |
weshay | in rdo | 19:47 |
weshay | rfolco, rlandy asking in #zuul | 19:48 |
rlandy | ack - following | 19:49 |
rlandy | hoping # zuul will come up with a better answer but it possible to api query a gerrit change and grep for Depends-On: | 19:51 |
rlandy | ugly though | 19:51 |
rlandy | weshay: ^^ | 19:51 |
weshay | rlandy, totally | 19:51 |
weshay | rlandy, there is an endless chain there | 19:51 |
rlandy | something like ... | 19:51 |
rlandy | ssh -p 29418 review.openstack.org gerrit query 577230 --dependencies | grep Depends-On | 19:52 |
rlandy | Depends-On: I84db703ac3c5e6da260986539e74eda3d0e6198f | 19:52 |
rlandy | but then you have to query that review for its repo | 19:52 |
rlandy | ***very ugly*** but possible | 19:52 |
rfolco | random thought... can we just run our job on sf and this job connects to the jenkins host and pass all zuul vars we set in the job config as we do upstream ? | 19:53 |
rfolco | we run a fake node job that is only a wrapper to the jenkins job | 19:53 |
rlandy | rfolco: we need a way that is independent of zuul and jenkins | 19:54 |
rfolco | why | 19:54 |
rlandy | but if we had internal sf, we could have more options | 19:54 |
rlandy | because we need to replace quickstart.sh | 19:54 |
rlandy | that runs standone | 19:54 |
rlandy | that runs standalone | 19:54 |
*** panda has joined #oooq | 19:55 | |
rfolco | I still believe the idea is doable, easier than parsing depends-on | 19:55 |
rfolco | must be missing something | 19:55 |
rlandy | rfolco: which sf would a random dev use | 19:55 |
rlandy | to test his local work? | 19:55 |
rfolco | ok, I was mixing jenkins internal jobs with reproducer case | 19:57 |
rlandy | jenkins is def a problem as well | 19:59 |
rlandy | right now build-test-packages uses the library jenkins_deps | 19:59 |
rlandy | which we could reuse ... | 19:59 |
* rlandy gets | 19:59 | |
rlandy | rfolco: https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/build-test-packages/library | 20:00 |
rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/library/jenkins_deps.py | 20:01 |
rlandy | follows the workflow decsribed above | 20:01 |
rfolco | you have a reproducer, which should run just like the one that ran in zuul. So we do have the zuul inventory available. We have ZUUL_CHANGES then..... or not? | 20:01 |
rfolco | sorry I am not on the same page here | 20:05 |
rlandy | if not zuul, not zuul_changes | 20:17 |
rfolco | like reproducer for phase2 ? | 20:25 |
rlandy | ansible localhost -m jenkins_deps -M /home/rlandy/workspace/tripleo-quickstart-extras/roles/build-test-packages/library -a 'host="review.openstack.org" change_id="577230"' | 20:27 |
rlandy | ^^ that works | 20:27 |
rlandy | we just pull refspec out of that | 20:29 |
rlandy | weshay: ^^ | 20:29 |
rlandy | worst case scenario | 20:29 |
rlandy | thank you adarazs | 20:29 |
* weshay reads | 20:32 | |
weshay | rlandy, oh that's cool | 20:33 |
weshay | rlandy, what is the result of that? | 20:33 |
rlandy | [rlandy@rlandy library]$ ansible localhost -m jenkins_deps -M /home/rlandy/workspace/tripleo-quickstart-extras/roles/build-test-packages/library -a 'host="review.openstack.org" change_id="577230"' | grep "refspec" | 20:34 |
rlandy | [WARNING]: Could not match supplied host pattern, ignoring: all | 20:34 |
rlandy | [WARNING]: provided hosts list is empty, only localhost is available | 20:34 |
rlandy | "refspec": "refs/changes/30/577230/16" | 20:34 |
rlandy | "refspec": "refs/changes/77/593077/2" | 20:34 |
rlandy | but I can clean that up more | 20:34 |
rlandy | to build the zuul_changes string | 20:34 |
rlandy | it's an idea | 20:34 |
weshay | rlandy, if you can drop the refspec output into a json file | 20:42 |
weshay | that would be pretty cool | 20:42 |
rlandy | weshay: yep - playing with some json output now | 20:42 |
*** trown is now known as trown|outtypewww | 20:54 | |
*** jtomasek has quit IRC | 20:57 | |
rlandy | weshay: where do you want to go with this? make an attempt to write a localized CLI zuul-executor ot just hack something out from a library we already have? | 21:11 |
*** holser_ has joined #oooq | 21:18 | |
weshay | rlandy, something to pitch to the team | 21:52 |
weshay | I think | 21:52 |
*** agopi has quit IRC | 21:52 | |
weshay | rlandy, if you can frame up the question and some possible solutions.. I think that would lead us in the right direction | 21:57 |
weshay | rfolco, where are nodesets defined? | 22:16 |
weshay | in project config? | 22:16 |
rfolco | weshay, for what job | 22:16 |
weshay | rfolco, doesn't exist yet.. | 22:16 |
weshay | rfolco, I need a fedora node | 22:17 |
rfolco | weshay, rdo or upstream | 22:17 |
weshay | rfolco, upstream | 22:18 |
rfolco | https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/nodesets.yaml | 22:18 |
weshay | I think it's called fedora-28 | 22:18 |
* weshay looks | 22:18 | |
weshay | rfolco, that is incorrect | 22:18 |
rfolco | new workflow parents to tripleo-ci-base which uses single or multinode centos from there | 22:19 |
weshay | rfolco, does that parent to someting? | 22:19 |
weshay | rfolco, /me looking for fedora28 nodes | 22:19 |
rfolco | weshay, openstack-zuul-jobs has more nodesets | 22:19 |
rfolco | ok must be there | 22:19 |
* weshay looks | 22:19 | |
rfolco | https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/nodesets.yaml#L14 | 22:20 |
rfolco | weshay, ^ | 22:20 |
weshay | fedora-latest | 22:20 |
weshay | :) | 22:20 |
weshay | rfolco, rock on.. thanks | 22:20 |
*** holser_ has quit IRC | 22:26 | |
*** holser_ has joined #oooq | 22:34 | |
*** ChanServ has quit IRC | 22:49 | |
*** ChanServ has joined #oooq | 23:03 | |
*** barjavel.freenode.net sets mode: +o ChanServ | 23:03 | |
*** holser_ has quit IRC | 23:10 | |
*** rlandy has quit IRC | 23:39 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!