Friday, 2018-06-08

*** tosky has quit IRC00:00
rlandy|roverand failed again00:19
rlandy|roverat least it's consistent00:19
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.00:34
*** rlandy|rover has quit IRC00:52
*** rfolco has quit IRC00:54
*** sanjay__u has joined #oooq01:08
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.02:34
*** udesale has joined #oooq03:48
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.04:34
*** pgadiya has joined #oooq04:37
*** pgadiya has quit IRC04:37
*** links has joined #oooq04:41
quiquell|offarxcruz|ruck: promoter at master is failing05:56
quiquell|offarxcruz|ruck: r&r alert is legit05:56
quiquell|offarxcruz|ruck: Cannot access http://38.145.34.55/master.log05:56
*** quiquell|off is now known as quiquell05:57
Tenguinfra seems to have restarted something with zuul05:57
Tenguthey detected something weird.05:58
quiquellarxcruz|ruck, Tengu: humm ok, weshay disabled master promotions, maybe that's the cause05:59
quiquellTengu: They have update ansible to 2.5, maybe that's the problem05:59
Tenguhmm ok.06:00
-openstackstatus- NOTICE: Zuul stopped receiving gerrit events around 04:00UTC; any changes submitted between then and now will probably require a "recheck" comment to be requeued. Thanks!06:00
*** holser__ has joined #oooq06:17
*** matbu has quit IRC06:17
*** matbu has joined #oooq06:18
*** jbadiapa has joined #oooq06:22
*** jaganathan has joined #oooq06:28
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.06:34
*** zoli has quit IRC06:41
*** gchamoul has quit IRC06:41
*** zoli has joined #oooq06:48
*** gchamoul has joined #oooq06:49
*** jaosorior has quit IRC06:58
pandaquiquell: howdy07:03
*** d0ugal has joined #oooq07:03
*** amoralej|off is now known as amoralej07:23
quiquellpanda: Welcome back !!!07:24
quiquellpanda: How you doing ?07:26
pandaquiquell: wonderfully now that I'm back to work07:29
*** tosky has joined #oooq07:30
quiquellpanda: You were waitting for it :-)07:30
pandaquiquell: absolutely!07:32
pandaquiquell: what did I miss ?07:32
*** jbadiapa has quit IRC07:32
quiquellpanda: Sprint is progressing, we need some +2 +1 workflows07:33
quiquellpanda: Not much blockers, also the promoter change for hashes we where waitting for you07:33
*** jbadiapa has joined #oooq07:33
quiquellpanda: I have add some unit testing to it, to verify07:33
quiquellpanda: For the injection, we are adding release to the gating repo07:34
pandaquiquell: how much testing is needed or sprint topics ?07:34
quiquellpanda: Is good also for debuggin purposes and we don't have to do a backup07:34
quiquellpanda: What do you mean ?07:34
Tenguwould it be possible to get some review/feedback regarding this proposal? https://review.openstack.org/#/c/570841/  that would be really nice :).07:36
quiquellpanda: +2 +1w for fs037 in queens https://review.openstack.org/#/c/570902/07:44
quiquellIt's a sprint thing07:44
quiquellpanda: https://review.openstack.org/#/c/572308/07:44
quiquellpanda: 200~https://review.openstack.org/#/c/572306/07:45
quiquellpanda: https://review.openstack.org/#/c/572297/07:45
quiquellpanda: ^ the for reviews are to finish https://trello.com/c/flI683EI/774-ci-job-create-job-37-work-on-queens-and-calls-tripleo-upgrade-updates-workflow07:45
quiquellpanda: s/for/four/07:45
quiquellpanda: Also this only needs +1w https://review.openstack.org/#/c/572096/07:47
pandawow07:47
pandaI will review for the entire day07:47
quiquellpanda: You can start with the last one, just needs the +1v07:49
quiquellpanda: Other ones are will close a sprint card07:49
*** gkadam has joined #oooq08:06
quiquellmarios: You there ?08:10
marioso/ quiquell08:21
marioshi :) I did wonder if we were in same timezone08:21
mariosquiquell: i was just checking your patches08:21
mariosand i pinged the upgrades team to check that (I know jistr would be interested but he was relocating). i think they are ok lgtm for merge08:22
quiquellmarios: I am at Madrid GMT+208:22
mariosquiquell: cool. with ccamacho and jfrancoa?08:22
quiquellmarios: My only concern is if we are missing other tripleo projects08:23
quiquell marios Yep, I also worked with jfrancoa in a previous company :-)08:23
mariosquiquell: well, client/common for sure we need (I didn't check where we have this running yet? tht/? and?)08:23
mariosquiquell: ah cool08:23
mariosquiquell: were you there last year when we came? like october08:23
marios'we' the upgrades team was there08:24
quiquellmarios: we are covering tht, client and common are we missing some ?08:24
marioslooks like matbu is checking your patches now too from ping just now08:24
mariosquiquell: no that sounds like a good base for sure.08:24
quiquellmarios: I enter RedHat March08:24
mariosquiquell: i see ok so we didn't meet last year08:25
quiquellmarios: Nope, but we will, are you moving to CI ?08:25
pandamaybe in another life08:25
marioso/ panda08:25
quiquellpanda: Go back to your corner !08:25
mariosyeah maybe in another life we shared a taxi for 5 hours over a mountain in the rain. #true story bro08:25
pandaper aspera, ad astra08:26
quiquellmarios: In this life we are sharing tripleo ci (how exciting!!!)08:26
mariosquiquell: :D08:26
marioswoo08:26
marios\o/08:26
quiquellpanda, marios: cool fs037 for queens changes are gating08:27
*** kopecmartin has joined #oooq08:31
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.08:34
quiquellmarios: We are developing this tool http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1, to summaraize problems in CI08:35
pandawe ?08:38
quiquellSagi, Wes and I08:41
mariosquiquell: thanks :) nice :)08:43
mariosquiquell: is that tracked outside the sprint planning? pointers/cards etc for it?08:58
quiquellmarios: nope, is just a poc done at 20% projects08:59
mariosquiquell: ack its pretty though :)08:59
*** d0ugal_ has joined #oooq09:00
quiquellmarios: If you have stuff you are missing we can add it09:00
*** d0ugal has quit IRC09:00
*** d0ugal_ has quit IRC09:00
*** d0ugal has joined #oooq09:01
*** dtantsur|afk is now known as dtantsur09:09
*** bogdando has joined #oooq09:11
quiquellpanda, marios: I have a yum question09:20
mariosquiquell: sup09:21
quiquellmarios: if I have a .repo file at yum.repos.d09:22
quiquellAnd disabled=009:22
quiquellI just change disabled=009:22
quiquellDoing a yum update -y09:22
quiquellWill ignore it ?09:22
quiquellAnd install new stuff ?09:22
mariosquiquell: sounds right but what do you mean ignore it?09:24
quiquellmarios: To be honest, don't really know yet :-)09:24
mariosquiquell: enable/disable the repo. do you mean it was enabled=009:25
pandaquiquell: why should you update ignore a disabled=0 ?09:25
mariosquiquell: and you made it enabled=109:25
pandaah eya, disabled does not exist09:25
quiquellpanda, marios: I want to deactivate the gating.repo in the upgrade09:26
quiquellenabled=0 and yum update -y will be enough ?09:26
mariosquiquell: so make it enabled=009:26
mariosquiquell: i think so?09:26
quiquellmarios: cool09:26
quiquellalso the opposite09:26
quiquellande new .repo and calling yum update -y without the yum enablerepo=... ?09:27
quiquells/ande/add/09:27
*** jaosorior has joined #oooq09:36
*** dtantsur is now known as dtantsur|brb09:38
chandankumararxcruz|ruck: I need to inject one script for getting refstack client tests in tempest format10:04
chandankumaron downloading the tests it comes in ascii format which is not read by tempest10:04
*** udesale_ has joined #oooq10:27
*** udesale has quit IRC10:29
*** udesale_ has quit IRC10:31
*** dtantsur|brb is now known as dtantsur10:31
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.10:34
quiquellpanda, marios: do you know what MERGER_FAILURE means at zuul ?10:48
*** zoli is now known as zoli|afk11:13
mariosquiquell: where?11:13
mariosquiquell: don't know i guess some gate failed?11:13
mariosor dependency?11:13
quiquellmarios: openstack-infra suggest recheck11:13
mariosah there was announcement about zuul not receiving events from gerrit today sec11:13
quiquellLet's wait11:13
marios09:00 -openstackstatus:#oooq- NOTICE: Zuul stopped receiving gerrit events around 04:00UTC; any changes submitted between then and now will probably require a "recheck"  comment to be requeued. Thanks!11:14
mariosquiquell: ^^ like 5 hours ago11:14
quiquellpanda: Do you have a minute ?11:44
pandaquiquell: sure11:46
quiquellpanda: Let's go to bj, to help me verify the injection of zuul changes11:47
quiquellpanda: I am at your room11:50
pandaquiquell: lol, I'm at yours11:51
pandaquiquell: let's use yours11:51
pandaquiquell: quicker for me11:51
quiquellpanda: I have move to fedora + i3, have some issues with bluejeans login11:51
quiquellpanda: let me try again11:51
pandaquiquell: you don't need to login to go into your room11:52
quiquellpanda: no ?11:52
pandaquiquell: you need only if you want your moderator powa11:52
weshayarxcruz|ruck, howdy11:52
pandaquiquell: and itt seems you enabled the moderator only meeting11:53
arxcruz|ruckweshay: hey boss11:53
pandaquiquell: I'm in my room if you prefer now11:53
quiquellpanda: ok, thanks11:54
*** rfolco has joined #oooq11:56
weshayarxcruz|ruck, what's up w/ queens and the ovb promotion jobs?11:58
weshaythey should have picked up your change11:58
weshayarxcruz|ruck, also pike should have promoted but did not11:58
arxcruz|ruckweshay: the upload patch didn't get in last run11:59
weshaythat's crazy12:00
arxcruz|ruckweshay: i'll check pike, sorry, i spend the morning checking the error on master, open two bugs, then i realize rlandy had open yesterday a bug :/12:00
weshayarxcruz|ruck, heh12:01
weshayarxcruz|ruck, it should be escalated too12:01
arxcruz|ruckweshay: in pike promotion log i found this:12:01
arxcruz|ruck2018-06-08 06:22:39,762 19151 DEBUG    promoter Remaining hashes after removing already promoted ones: []12:01
weshayarxcruz|ruck, https://trello.com/c/xvRSVaAQ/615-cixlp1775698tripleociproa-master-promotion-undercloud-install-is-failing-503-errors-starting-container-trunkregistryrdoprojector12:02
arxcruz|ruckweshay: yeah, my fault, i wake up, saw master completly red12:02
arxcruz|ruckdon't even thought about check if was a known problem already12:02
weshayarxcruz|ruck, get in the habit of looking at the lp bugs every morning.. sort by the latest12:03
weshayit's a decent news feed for tripleo12:04
arxcruz|ruckyeah, i should before everything else12:04
arxcruz|rucklessons learned12:04
*** rlandy has joined #oooq12:31
*** rlandy is now known as rlandy|rover12:31
rlandy|roverweshay: arxcruz|ruck: looks like we got an issue with the running promoter process12:34
weshayrlandy|rover, jump in my blue12:34
arxcruz|ruckrlandy|rover: which one? :)12:34
rlandy|roverk12:34
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.12:34
quiquellweshay, panda: testing injection + emit script changes with https://review.openstack.org/#/c/571435/12:45
quiquellLet's see where we are12:45
quiquellis a n -> n + 112:45
weshayrlandy|rover, arxcruz|ruck12:48
weshayhttps://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/60b4155/undercloud/var/log/extra/docker/docker_allinfo.log.txt.gz12:48
weshayrlandy|rover, arxcruz|ruck grep or check for restarting12:48
weshayrlandy|rover, think we can adjust the subject of the bug12:49
arxcruz|ruckweshay: 3 restarts nova api, nova metadata and nova api cron12:49
weshayrlandy|rover, arxcruz|ruck we probably want to think about how we can raise containers like this.. to be more visible12:49
weshayquiquell, I'll be a few more minutes12:50
weshayquiquell, sorry12:50
rlandy|roverI see the restarting on centos-binary-nova-api but not on centos-binary-ironic-conductor - exited12:51
*** zoli|afk is now known as zoli12:55
*** zoli is now known as zoli|wfh12:55
*** zoli|wfh is now known as zoli12:55
weshayrlandy|rover, arxcruz|ruck fyi https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/tips_tricks.html12:56
*** myoung|off is now known as myoung12:59
*** amoralej is now known as amoralej|lunch13:02
rlandy|roverweshay: arxcruz|ruck: not to detract from the debug session - but pls see the promoter server ... 2018-06-08T13:02:40Z E! Error in plugin [inputs.procstat]: E! Error: procstat getting process, exe: [] pidfile: [] pattern: [^python.*dlrnapi_promoter.py .*pike.*] user: [] Error running /usr/bin/pgrep: exit status 113:02
rlandy|rovercan we stop that run?13:03
weshayrlandy|rover, feel free to stop the whole thing13:03
quiquellrlandy|rover: it's just a warning, let me mute i t13:03
weshayya.. pike should have promoted13:03
rlandy|roverweshay: I'll restart it13:03
quiquellrlandy|rover: This is not the promoter, this is the reporter13:04
quiquellIf you look a tmux again you see that promoter is running13:04
quiquellrlandy|rover: Is at pike now13:04
quiquellrlandy|rover:ansible docker is failing13:05
quiquellrlandy|rover: look at /var/log/message13:05
myoungquiquell: good morning, looking at your patches for injection.  Did you and panda already meet / bj around validation?13:06
quiquellmyoung: yep, manual verification with a reproducer13:06
quiquellmyoung: Looks good the n -> n + 113:07
quiquellI have recheck all the testing reviews13:07
quiquellmyoung: also now it depends on trown's change, so we test the integration of both cards13:07
myoungquiquell: (still reconstructing the earlier day) - I was about to rebase the patches to THT I've made to cause rpm rebuild in the jobs (that depend on your changes) are still needed yes?13:07
*** d0ugal_ has joined #oooq13:08
*** d0ugal has quit IRC13:08
quiquellmyoung: No need to reviews, they depends on Change-Id it doesn't chage13:08
myoungquiquell: that was my next question, have we already reparented yours or his?13:08
quiquellmyoung: Has post a recheck on them13:08
*** d0ugal_ has quit IRC13:08
* myoung looks. 13:08
myoungo/ good morning all btw :)13:08
*** d0ugal has joined #oooq13:08
*** d0ugal has quit IRC13:08
*** d0ugal has joined #oooq13:08
quiquellmyoung: I have to patches, one at TTQE with adding release to gating repo13:09
quiquellmyoung: And another at tripleo-ci to force n -> n + 113:09
quiquellmyoung: I have replace the latest to just has trown's change as parent13:09
quiquellmyoung: https://review.openstack.org/#/c/573680/13:10
myoungquiquell: cool, trown|outtypewww patch is my next stop, when last we left our hero (yesterday) that was getting  a rework to address your UX feedback13:10
quiquellmyoung: I still think it's not only UX but, the change lgfm13:11
myoungquiquell: (ack, just recounting from convo, about to look at patches.) - reading your responses to my questions in your review13:11
quiquellmyoung: Long story short, let's wait for zuul :-)13:11
quiquellmyoung: Cool, let me know if I miss any piece of your concerns on it13:12
myoungquiquell: quick chat via bj?  might be faster than here13:13
myoungre: questions in https://review.openstack.org/#/c/57273613:13
quiquellmyoung: I think we can put in place an implementation for the toci part and parent with trown13:13
quiquellmyoung: sure13:14
myoungquiquell: can chat about that too...that's my next step anyway13:14
myoungi'm in my room13:14
quiquellOk, going there13:14
myoungquiquell: re: toci part I think basically what you POC'd last week is basicaly it...with perhaps a small tweak13:15
*** Goneri has joined #oooq13:17
myoungquiquell: https://review.openstack.org/#/c/572736/6/roles/build-test-packages/defaults/main.yml13:18
*** atoth has joined #oooq13:20
rlandy|roverquiquell: we're busy debugging another container issue - but pls enlarge the sze of your window on tmux13:22
weshayquiquell, I updated our 1-113:25
quiquellweshay: ok13:26
myoungquiquell: https://review.openstack.org/#/c/573000/13:26
myoung^^ spawned --> http://logs.openstack.org/00/573000/2/check/tripleo-ci-centos-7-undercloud-upgrades/bf915c9/13:27
myoungquiquell: http://logs.openstack.org/00/573000/2/check/tripleo-ci-centos-7-undercloud-upgrades/bf915c9/job-output.txt.gz#_2018-06-06_22_06_48_04449113:28
*** jaosorior has quit IRC13:29
rlandy|roverquiquell: I still see a promoter process running - did you try it?13:42
rlandy|roverI see the ansible error13:42
chandankumarweshay: arxcruz|ruck I am facing one problem with refstack test integration with TQE13:43
arxcruz|ruckchandankumar: hold on13:44
quiquellrlandy|rover: talking with myoung give e minute13:45
-openstackstatus- NOTICE: A misapplied distro security package update caused many jobs to fail with a MERGER_FAILURE error between ~06:30-12:30 UTC; these can be safely rechecked now that the problem has been addressed13:45
rlandy|roverquiquell: ok - ping me when you are done13:45
rlandy|roverI'd like to kill and restart13:45
*** links has quit IRC13:46
chandankumarweshay: currently the download refstack test regex is in us-ascii format, i tried to convert it in utf-8 but not happening usin iconv, from the refstackc-client code i foudn that they espcially convert in the run time or my question is will i glue a role in validat-tempest for refstack installation from venv and run the same from there then the role become messy or I can package it in RDO and add it under -all13:47
chandankumarpackage so that once can use it from package adn consume it from there? or I can write a simple script which does the tempest formatted test conversion in the playbook itself as a temp solution?13:47
quiquellrlandy|rover: Have some minutes before 1-to-113:52
rlandy|roverquiquell: 1 minute13:53
quiquellrlandy|rover: shoot13:53
*** amoralej|lunch is now known as amoralej13:54
rlandy|roverquiquell: can you join weshay's bj?13:55
rlandy|roverarxcruz|ruck and I are on there13:55
quiquellsure13:56
arxcruz|ruckrasca: around ?13:56
rlandy|roverrasca: if we want to assign the nova team to a bug - which DFG would we assign?13:56
rascarlandy|rover, DFG:COMPUTE I'd say13:58
rascaarxcruz|ruck, I'm here13:58
rlandy|roverrasca: I see virtual compute13:59
rlandy|rovernot just compute13:59
rascarlandy|rover, sorry, where are you looking at?13:59
myoungpanda, are you/we attending the upgrades meeting today?13:59
rlandy|roverhttps://trello.com/c/xvRSVaAQ/615-cixlp1775698tripleociproa-master-promotion-undercloud-install-is-failing-503-errors-starting-container-trunkregistryrdoprojector13:59
rlandy|roverrasca: ^^13:59
myoungpanda: err...(typo) did you attend?14:00
rascarlandy|rover, oh the prodchain14:00
myoungpanda: nvrmnd looking at etherpad with notes form wes14:00
myoungfrom* ws14:00
*** jaganathan has quit IRC14:00
* myoung mutters curses at his keyboard14:01
rascarlandy|rover, so I'd say Virtual Compute is good, but maybe weshay can confirm14:01
*** jaganathan has joined #oooq14:01
pandamyoung: nope, didn't know we had something to discuss.14:04
myoungpanda: weshay was there all good14:05
weshaypanda, did you see the doc about software factory from rfolco ?14:08
arxcruz|ruckbogdando: is you on the tmate ?14:09
rlandy|roverrasca: yep promotion chain14:09
rlandy|roverweshay: ^^ we're assigning the bug yto virtual computer as the DFG for now14:10
rlandy|rovernoting comments from Bogdan14:10
bogdandoarxcruz|ruck: yes14:11
arxcruz|ruckbogdando: i was looking for a command to remove the healthcheck or change it14:11
arxcruz|ruckso we can see what's going on14:11
bogdandoarxcruz|ruck: I tried with paunch14:11
bogdandoupdated to config to always pass healtcheck14:11
arxcruz|ruckbogdando: yup, i saw it14:11
bogdandobut the paunch command seem didn't work as I expected14:11
bogdandoarxcruz|ruck: I'll remove that nova_api and try again with paucnh...14:13
arxcruz|ruckk14:13
* arxcruz|ruck needs to learn docker quickly14:14
pandaweshay: not yet, just finished reviews. looking now14:15
weshayquiquell, https://trunk.rdoproject.org/centos7-master/report.html14:28
myoungarxcruz|ruck: I've found the 2 o'reilly books ("Using Docker" and "Docker Cookbook" both very good / useful.  The latter being recipe based for examples, and the former being a quick read that provides good "start to finish" ref.  Neither spends a whole lot of time on compose, but both are recommended IMHO.14:31
arxcruz|ruckmyoung: cool14:31
*** quiquell is now known as quiquell|off14:32
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.14:34
weshayrlandy|rover, arxcruz|ruck so we're on the right path w/ regards to the nova container?14:48
rlandy|roverweshay: I stopped working on that debug to look at the container uploads14:49
arxcruz|ruckweshay: bogdando is doing some black magic14:49
bogdandopaunching that nova api container in its face basically, with no results14:50
rlandy|roverweshay: assigned the bug on the prodchain board to virtual compute - pleas change if that is incorrect14:51
*** dougbtv_ has joined #oooq14:54
weshayrlandy|rover, arxcruz|ruck thanks14:56
arxcruz|ruckbogdando: is mysql also containerized ?14:57
*** myoung is now known as myoung|biaf14:57
bogdandoarxcruz|ruck: yes, but it has nothing to mysql14:57
arxcruz|ruckok14:57
bogdandojust missed the config files for that start14:57
arxcruz|ruckjust checking :)14:57
bogdandotrying to figure out the way to do it right14:58
weshaypanda, arxcruz|ruck rlandy|rover need reviews on https://review.openstack.org/#/c/572217/ and the deps please14:58
rlandy|roverrabsing that14:59
rlandy|rovercannot merge14:59
rlandy|roverthere is a whole lot of patches here15:01
weshaypanda, hope you had a nice PTO.. let's make some time on monday to chat about the next sprint and the next few sprints15:02
weshaybah.. arxcruz|ruck documentation.. details.. just details.. crap.. I'm on it15:04
pandaweshay: sure15:04
arxcruz|ruckweshay: you asked me to review :P15:04
rlandy|roverweshay: panda: ^^ when ou discuss next sprint request to keep in mind changes to promoter and reproducer15:04
rlandy|roveror next few sprints15:04
weshayrlandy|rover, do you think the openshift oc changes should be handled in a sprint15:05
weshaypanda, oh.. we need to catch up about the promoter too.. want to get your thoughts..15:05
rlandy|roverweshay: I just started looking at them today ... kind of distracted15:05
rlandy|roverwhich is why I am suggesting a sprint15:05
weshaypanda, talking about the promoter today would be a good idea15:05
rlandy|roverruck rover gets very distracted15:05
rlandy|roverweshay: panda: wrt promoter - we need to talk about venv and python - some other basic things15:06
rlandy|roverlibvirt to support?15:06
rlandy|roverjust my $0.0215:06
rlandy|roverhttps://review.openstack.org/#/c/572217/ - I don;t suggest we merge toci-gate-test changes on friday afternoon15:07
arxcruz|ruckrlandy|rover: I gree with you hehehehe15:08
rlandy|roverbesides we need some rebase work there15:08
*** dtrainor has quit IRC15:11
arxcruz|ruckshit 29 euros the kindle edition, 34 euros the paperback edition15:12
*** d0ugal has quit IRC15:14
pandaweshay: how urgent is to merge  https://review.openstack.org/#/c/572217/ ?15:17
*** ccamacho has quit IRC15:23
weshaypanda, fairly urgent.. thanks for review will fix...   what is on your mind w/ regards to the urgency15:24
* weshay wants to hear15:24
pandaweshay: nothing, this change work anyway, my comment was only to remove some unecessary things, so if it's urgent, we can merge it and modify later.15:25
weshaypanda, I can change the things you highlighted today, but need reviews.. and hopefully merge by monday15:25
weshayarxcruz|ruck, can take a pass at queens again https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/278/console15:29
arxcruz|ruckweshay: this is crazy, why now is taking v2 api instead of v3 ?15:30
*** saneax has quit IRC15:31
rlandy|roverweshay: arxcruz|ruck: fyi ... https://bugs.launchpad.net/tripleo/+bug/177587415:33
openstackLaunchpad bug 1775874 in tripleo "[promotion] fs027 is using containers from docker.io not trunk.registry.rdoproject.org/tripleomaster/" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)15:34
*** dtrainor has joined #oooq15:35
arxcruz|ruckweshay: i'm not seeing a reproducer script on this job, do you know where it is ?15:36
arxcruz|ruckweshay: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/5d2fbdb/15:36
arxcruz|ruckor quickstart.sh queens works ?15:36
*** bogdando has quit IRC15:37
*** myoung|biaf is now known as myoung15:37
myoungweshay, panda is the 30m duration meeting scheduled for monday sufficient?  That's 30 focused for just CI squad (tempest is it's own meeting)?15:42
arxcruz|ruckweshay: i'll try to recreate this on libvirt and debug15:43
pandamyoung: should be enough if we don't need to make some initial design15:47
myoungpanda: i'll bump it out to an hour and we can aim to finish in 30.  weshay does that sound good? ^^15:51
myoungpanda: (parsing your input - would rather have what we need and quit early than go into sprint 15 planning not ready).  we have a lot of PTO and such in sprint 15 so want to make sure our plans / cards are well defined.15:52
*** jaganathan has quit IRC15:52
weshaymyoung, please add rfolco to that sprint planning15:54
myoungweshay: ack15:55
myoungweshay: exciting to have SF internally.  that dream / vision has been a long time coming.15:55
myoungrfolco: ^^15:55
pandaother PTOs ?15:56
pandavacations are highly overrated15:57
myoungweshay, rfolco, panda: sprint 15 planning is monday 11-june, 4:30-5:30 UTC w/ plan to quit early if possible.15:57
myoung^^ (calendars looked open) - let me know if that timeslot needs to change15:58
myoungarxcruz|ruck, rlandy|rover: I've added you to triage monday (ruck/rover) morning, invite sent, let me know if that timeslot doesn't work.16:05
rlandy|roverty16:05
myoungarxcruz|ruck: re kindle/paperback, I think we have access to the oreilly library for free via our employer fwiw.  also (at least in US) I get a lot of books used for quite a bit less.  I'm a shameless tree killer.16:07
arxcruz|ruckmyoung: oreilly? I mean, o really?16:08
arxcruz|ruckmyoung: don't know about that, let me check16:08
* myoung wonders if he has wires crossed and looks again16:08
myoung(I like paper books anyway - can read them in the sunshine easier...and battery life of paper books is SUPERB)16:09
myoungarxcruz|ruck: (sent you  DM)16:10
rlandy|roverweshay: fs027 is the only singlenode job16:12
rlandy|roveryou may be right - checking toci-gate-test16:12
*** holser__ has quit IRC16:19
*** kopecmartin has quit IRC16:24
*** dtrainor has quit IRC16:25
*** dtrainor has joined #oooq16:26
hubbotAll check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens.16:34
*** panda is now known as panda|off16:45
*** dtrainor has quit IRC16:46
*** jtomasek has quit IRC16:52
*** zoli is now known as zoli|gne17:04
*** zoli|gne is now known as zoli|gone17:04
*** zoli|gone is now known as zoli17:04
*** gkadam has quit IRC17:11
-openstackstatus- NOTICE: The Zuul scheduler was offline briefly to clean up from debugging a nodepool issue, so changes uploaded or approved between 16:50 and 17:15 UTC may need to be rechecked or reapproved (all already queued changes are in the process of being reenqueued now)17:23
*** sanjay__u has quit IRC17:24
*** dtantsur is now known as dtantsur|afk17:30
*** dougbtv_ has quit IRC17:35
*** myoung is now known as myoung|bbl17:36
*** dougbtv_ has joined #oooq17:37
rlandy|roverweshay: have a moment to go over the fs027 differences?17:42
weshayya.. need 3min17:42
rlandy|roversure - ping me when you are ready - no rush17:42
*** amoralej is now known as amoralej|off17:44
weshayrlandy|rover, k on my blue17:46
Tenguweshay: could you find the reason of the issues ?17:46
weshayTengu, I've been in other meetings since I left17:46
Tenguweshay: oh, ok17:46
Tenguweshay: ah, Bogdan added some comments on LP17:47
*** saneax has joined #oooq17:53
*** saneax has quit IRC17:53
Tenguhm. looks like either as a dns issue (hostname resolving on the wrong IP) or a misconfiguration in haproxy…17:54
*** atoth has quit IRC17:55
*** atoth has joined #oooq17:56
*** atoth has quit IRC18:00
*** atoth has joined #oooq18:01
*** dtrainor has joined #oooq18:04
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.18:35
*** tcw1 has joined #oooq18:36
*** tcw has quit IRC18:36
rfolcodo we have a group subscription to LWN.net ?18:36
*** myoung|bbl is now known as myoung19:36
*** tcw1 has quit IRC19:42
*** tcw1 has joined #oooq19:48
rookweshay myoung alright, the fact that the browbeat jobs have been pretty much dead the past 6 months has me scrambling...19:54
weshayrook, show me19:54
rookhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/19:55
rookhttps://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/browbeat-quickstart-gerrit-rhos-12-baremetal-CI/19:55
rookdead osp12 job ^19:55
weshayrook, we need to make sure you are using the upstream fs configs19:55
rookLooking through the output, it seems something has changed, and we don't have the correct info passing19:55
weshayin addition to the env settings you need19:55
weshayrook, ya.. things change often19:56
rooklol19:56
weshayif we're not using upstream config.. things will break19:56
weshaycontainers, config download etc..etc..19:56
rookso, better question... since Browbeat running is a "bolt on"... Why can't I just ride the wave as long as my bolt on is working?19:56
rookvs having to periodically fux with things?19:57
weshayrook, we had the same deal.. bm jobs using internal only config19:57
weshaybad idea19:57
weshayrlandy|rover, and I started to fix that recently19:57
weshaycan do the same thing for you guys19:57
* rlandy|rover reads back19:57
rookThe only job that seems to be green is osp11, and ain19:58
rlandy|roverI'm in the middle of a major rewrite there19:58
rook't no one care about that.19:58
rookrlandy|rover weshay ack19:58
rookok19:58
weshayrook, ya.. because containers is default in 12,13 now19:58
rookyeah19:58
weshaysame w/ some other features19:58
rookand it is really ruining my day19:59
rlandy|roverrook - of you're ok with our changing your configs, we can do the whole lot together19:59
rookdamn containers.19:59
weshayya19:59
rlandy|roversorry - keep getting distracted by other failures19:59
weshayheh19:59
weshayrlandy|rover, can you show me where you left off w/ this stuff?19:59
rookit is friday. I will stfu.19:59
weshayheh20:00
rooknot a good time to spin up new work... we ill just forget by monday20:00
rookbut i need to get this going again20:00
rookI found a huge perf delta between 12 and 13 (neutron)20:00
weshayah k20:00
rookand I have zero historcal data that can point to where this happened.20:00
rlandy|roversec - submitting last review of fs027 jobs20:01
weshayrook, you guys also need to really consider getting a nodepool based job upstream running your workflow20:01
weshayeven if it's not measuring anything20:01
weshayjust to make sure you configs don't break like this20:01
weshayrook, I can go into details on monday with you if you want.. w/ the why/how this happened20:02
weshayrlandy|rover, I should just softlink fs001 -> config/geneneral_config/minimal.yml20:03
weshayand just make everyone move to upstream config20:03
rlandy|roversec20:03
weshayand override anything diff in env config20:04
rlandy|roverweshay: https://review.openstack.org/#/q/topic:bug/1775874+(status:open+OR+status:merged) - let me know if I missed something20:05
rlandy|roverchecking status of rhos-13 jobs20:05
weshayrlandy|rover, k.. let me go through that.. can you then make sure I have the latest info on the bm changes so we can start bringing rook's jobs back online20:06
*** dougbtv_ has quit IRC20:06
rlandy|roverdidn't add newton since we don;t run that rdocloud20:07
rlandy|roveryep - reading back as to what the last issues was with rhos-1320:07
rlandy|roverI know master works20:07
rlandy|roverhave review out on queens20:07
rlandy|roverrhos-13, I need to read back20:07
rlandy|roverenvE has the hacked up change20:07
rlandy|roverconfig should be fs00120:08
rlandy|roverwith minor overrides in env_settings20:08
rlandy|rovernothing else - other than the network_environment.yaml20:08
rlandy|roverfew minutes20:08
rlandy|rovermyoung had some notes20:08
rookweshay: ack. I need to spend more time on CI as this is becoming a trend for us. I have always pawned it off and only know it by first name like my 3rd removed cousin. I need to pull CI in a bit closer...20:09
rooklets work to get me smart on CI monday20:10
myoungrlandy|rover: rhos-13 was failing for the duplicate image ID problem (first, potentially now resolved), and by the rhos-ci.yml variable override issue (arxcruz|ruck might have current notes).  I can dig up the LPs'20:10
myoungor we can  handle monday.20:10
rookbecause Friday.. well i am full of slow buffalo and it might be dropped when I clear caches tonight...20:10
* myoung has a full brain atm as well.20:11
weshayrook, so I may have an update to /config/general_config/minimal.yml that would work for you20:11
myoungrlandy|rover: detailed notes are in the sprint 13 and 14 etherpads20:12
weshaybut ideally we get you on something upstream20:12
myoung^^ that is *the* answer to get early warning of config drift.20:12
weshayrook, we lost ci.centos for jobs > newton  the perf is too shitty20:12
weshayand nothing deploys in time20:12
weshayansible-playbook -vvvv /home/rhos-ci/jenkins/minimal-1/workspace/browbeat-quickstart-gerrit-rhos-12-baremetal-CI/playbooks/prep-internal.yml -e @/home/rhos-ci/jenkins/minimal-1/workspace/browbeat-quickstart-gerrit-rhos-12-baremetal-CI/config/release/pike.yml -e @/home/rhos-ci/jenkins/minimal-1/workspace/browbeat-quickstart-gerrit-rhos-12-baremetal-CI/config/nodes/1ctlr_1comp.yml -e @/home/rhos-ci/jenkins/minimal-1/workspace/browbeat-quickstar20:12
weshayt-gerrit-rhos-12-baremetal-CI/config/general_config/minimal.yml -e @/home/rhos-ci/jenkins/minimal-1/workspace/browbeat-quickstart-gerrit-rhos-12-baremetal-CI/config/environments/default_libvirt.yml -e local_working_dir=/home/rhos-ci/jenkins/minimal-1/workspace/browbeat-quickstart-gerrit-rhos-12-baremetal-CI -e virthost=microbrow-04.perf.lab.eng.rdu.redhat.com -t untagged,provision,environment,libvirt,undercloud-scripts,undercloud-inventory,ov20:12
weshayercloud-scripts,undercloud-setup,undercloud-install,undercloud-post-install,tripleoui-validate,teardown-nodes20:12
weshayhttps://thirdparty.logs.rdoproject.org/jenkins-browbeat-quickstart-gerrit-rhos-12-baremetal-CI-12/console.txt.gz#_2018-05-10_11_47_25_11920:12
* myoung returns to sprint stuff and will respond if pinged, otherwise nose-down20:13
weshaythat job is virt? or real baremetal20:13
rookack weshay, it should be BM20:13
weshayrook, the only bm config's I know of for you guys is in https://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=tree;f=hardware_environments;h=105db670bbe73e452ec1cf707e46db80974f59e8;hb=HEAD20:13
rlandy|roverweshay: how far out are we on getting internal zuul?20:14
weshayrlandy|rover, we're starting on thrs20:14
rookcorrect, those are still the same weshay20:14
weshayrook, ok.. I just don't seem them used by the job you pointed me at20:15
weshayrook, rlandy|rover and I will have a very good bm example we can work from to get your jobs back online shortly.. ours is working now but we want one more touch if I am remembering correctly20:15
rlandy|roverhttps://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=blob;f=hardware_environments/hp_dl360_envE/network_configs/single_nic_vlans/config_files/config.yml;h=4739f349ba39659587c169233038f2c2fb774ded;hb=HEAD20:16
rlandy|rover^^  this is basically fs 00120:16
rlandy|rovereverything env specific sits here ... https://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=blob;f=hardware_environments/hp_dl360_envE/network_configs/single_nic_vlans/env_settings.yml;h=5ce53bfa4c59bd9cd0c957900938b05a1db6d182;hb=HEAD20:17
rlandy|roverthe only other thing we need is ...20:17
rlandy|roverhttps://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=blob;f=hardware_environments/hp_dl360_envE/network_configs/single_nic_vlans/single_nic_vlans.yml;h=a5db780d02a3e713dc3b8c2dfa6a8b750f987412;hb=HEAD20:17
arxcruz|rucklove?20:17
rlandy|roverwhich we can rep;lace with just settings20:17
rlandy|roverwith that we can use upstream20:18
rlandy|roverwith just one settings file from downstream20:18
rlandy|roverdo we need rho?s-1320:19
rlandy|roverbasic question is what release do we need to run with browbeat20:19
rlandy|roverI think rhos-13 is possible nwo20:19
rlandy|roverthat is what I am confirming20:19
weshayarxcruz|ruck, you are crazy20:22
weshayrlandy|rover, fyi.. minimal.yml was fs001 at one point too20:23
weshayrlandy|rover, we need to find a way to include fs001, and override what we need to w/ env20:23
weshayshouldn't be that bad20:23
rlandy|roverweshay: that is what we are doing now20:24
myounghttps://github.com/openstack/browbeat/blob/master/ci-scripts/tripleo/microbrow-browbeat-ci.sh#L92, and can use the featureset variables defined in JJB in ooo-env.20:25
rlandy|roverall we really need is one extra settings file - that we need to make sure is passed last20:25
* myoung backs away slowly and returns to nose-down20:25
myoung(for real this time)20:25
rlandy|roverthat is what I was trying to show with the ssl20:25
weshayrlandy|rover, that is in fact an exact match to fs001 minus some recent doc gen settings20:25
rlandy|roverand ssl20:25
weshayrlandy|rover, /me thinks we should focus on changing the jjb at this point20:25
rookrlandy|rover: well, OSP13 and OSP1220:26
rookReally20:26
rlandy|roverbit not rhos20:26
rlandy|roverit's upstream releases20:26
rookerm - both would be nice... but upstream for sure.20:26
rookso pike and queens20:26
weshayrlandy|rover, let's chat for a sec.. rook you are welcome if you want to20:26
weshayrlandy|rover, there is no real diff afaict20:27
rooki need to pull the rip cord I have two little ones at my ankles20:27
weshayha20:27
rookweshay rlandy|rover lets sync up early next week20:27
rlandy|roverk joining20:27
rookthank you all!20:27
rlandy|roversure20:27
weshaynp..20:28
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.20:35
*** agopi has joined #oooq20:40
rlandy|roverweshay: https://code.engineering.redhat.com/gerrit/#/c/141114/21:28
rlandy|rovertwo notes ...21:28
rlandy|rover1. I did not delete the old file in the same review21:29
rlandy|rover(we can do that later when everything is moved over21:29
rlandy|rover2. I took a chance and added PublicVirtualFixedIPs: [{ "ip_address": "10.12.150.195" }]21:29
rlandy|roverthat was not included in the original file21:30
rlandy|roverand it may fail ( we will need to test that)21:30
rlandy|roverbut I have some idea that is part of why ssl failed21:30
weshayrlandy|rover, ok.. I'll get started on the others but ya..  run this through a test as soon as you can21:30
rlandy|roverweshay: ack - we can merge it and kick env E ad see what happens21:32
rlandy|roverlet's see what's running now21:32
*** myoung is now known as myoung|off21:34
rlandy|roverweshay: master's been pretty green ... https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans/21:34
rlandy|roverweshay: up to you if you want to merge that change and run now - or leave it until we can watch it through21:35
rlandy|roverI won;t be on line only for about another hour tonight21:36
weshayrlandy|rover, imho we should merge, run it.. and reconnect on monday?21:36
weshaywdyt?21:36
rlandy|roverok21:36
rlandy|roverhere we go21:36
weshay+2 from me21:37
rlandy|roverit may break promotions - but we can revert21:37
weshayrlandy|rover, yup and have time to do so21:37
rlandy|roveroh shoot21:39
rlandy|roverwon;t work21:39
rlandy|roverwe will have to override the file21:39
rlandy|roverotherwise it will still take that file21:39
rlandy|roverwe'll need a different job21:40
rlandy|roveranyway's let's see that we don't break anything21:40
*** ykarel has joined #oooq22:09
*** rfolco_ has joined #oooq22:11
*** rfolco has quit IRC22:12
*** ykarel_ has joined #oooq22:14
*** ykarel has quit IRC22:16
*** agopi has quit IRC22:24
*** rlandy|rover has quit IRC22:27
hubbotAll check jobs are working fine on stable/queens, stable/ocata, stable/ocata, master.22:35
*** ykarel_ has quit IRC22:49
*** ykarel_ has joined #oooq23:04
*** ykarel_ has quit IRC23:09
*** atoth has quit IRC23:20

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!