| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 00:15 |
|---|---|---|
| *** hamzy has quit IRC | 00:51 | |
| *** matbu has joined #oooq | 01:28 | |
| *** matbu has quit IRC | 01:42 | |
| *** skramaja has joined #oooq | 02:09 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 02:15 |
| *** rlandy|rover|bbl is now known as rlandy|rover | 02:26 | |
| *** strattao has quit IRC | 02:56 | |
| *** strattao has joined #oooq | 02:58 | |
| *** rlandy|rover has quit IRC | 03:27 | |
| *** skramaja has quit IRC | 03:43 | |
| *** skramaja has joined #oooq | 03:43 | |
| *** hamzy has joined #oooq | 03:54 | |
| *** hamzy has quit IRC | 03:59 | |
| *** hamzy has joined #oooq | 04:01 | |
| *** udesale has joined #oooq | 04:08 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 04:15 |
| *** d0ugal has quit IRC | 04:35 | |
| *** ykarel|afk has joined #oooq | 04:35 | |
| *** XMY5KHlukasdboer has joined #oooq | 05:00 | |
| XMY5KHlukasdboer | ARE YOU LOOKING FOR A HELP CHANNEL WHERE THE HELP DON'T KNOW SHIT BUT INSTEAD OF SAYING THEY DON'T SHIT THEY WILL SPEND 10 MINS JERKING YOUR CHAIN?? LOOK NO FURTHER THAN #UBUNTU A CHANNEL FULL OF DUMB NIGGERS THAT DON'T KNOW SHIT!! | 05:00 |
| XMY5KHlukasdboer | ARE YOU LOOKING FOR A HELP CHANNEL WHERE THE HELP DON'T KNOW SHIT BUT INSTEAD OF SAYING THEY DON'T SHIT THEY WILL SPEND 10 MINS JERKING YOUR CHAIN?? LOOK NO FURTHER THAN #UBUNTU A CHANNEL FULL OF DUMB NIGGERS THAT DON'T KNOW SHIT!! | 05:00 |
| XMY5KHlukasdboer | ykarel|afk udesale hamzy skramaja strattao jaganathan yolanda tcw dtantsur|afk _jbadiapa jaosorior sanjay__u ccamacho moguimar sshnaidm|off rfolco myoung|off rnoriega jrist rasca weshay_pto jschluet fuzzball81 bandini quiquell|off tbarron hubbot ajo sdoran rodrigods EmilienM panda|off zoli gchamoul hrybacki rook honza sai_ leifmadsen dmellado amoralej|off openstackstatus openstack dsneddon trown|outtypewww ssbarnea lucasagomes chandankumar dalvarez ar | 05:00 |
| *** XMY5KHlukasdboer has quit IRC | 05:00 | |
| *** d0ugal has joined #oooq | 05:03 | |
| *** d0ugal has quit IRC | 05:10 | |
| *** ccamacho has quit IRC | 05:10 | |
| *** ratailor has joined #oooq | 05:10 | |
| *** links has joined #oooq | 05:13 | |
| *** d0ugal_ has joined #oooq | 05:14 | |
| *** d0ugal__ has joined #oooq | 05:23 | |
| *** links has quit IRC | 05:24 | |
| *** d0ugal_ has quit IRC | 05:24 | |
| *** links has joined #oooq | 05:25 | |
| *** links has quit IRC | 05:35 | |
| *** links has joined #oooq | 05:36 | |
| *** links has quit IRC | 05:36 | |
| *** links has joined #oooq | 05:36 | |
| *** udesale_ has joined #oooq | 05:39 | |
| *** udesale has quit IRC | 05:42 | |
| *** links has quit IRC | 05:42 | |
| *** links has joined #oooq | 05:46 | |
| *** jaganathan has quit IRC | 05:50 | |
| *** jaganathan has joined #oooq | 05:50 | |
| *** marios has joined #oooq | 05:56 | |
| *** pgadiya has joined #oooq | 05:58 | |
| *** pgadiya has quit IRC | 05:58 | |
| *** saneax has joined #oooq | 05:59 | |
| *** links has quit IRC | 06:03 | |
| *** links has joined #oooq | 06:04 | |
| *** jfrancoa has joined #oooq | 06:05 | |
| *** saneax has quit IRC | 06:09 | |
| quiquell|off | arxcruz|ruck: Good morning sir | 06:15 |
| *** kopecmartin has joined #oooq | 06:15 | |
| *** quiquell|off is now known as quiquell | 06:15 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 06:15 |
| *** links has quit IRC | 06:17 | |
| *** holser__ has joined #oooq | 06:21 | |
| *** links has joined #oooq | 06:22 | |
| *** udesale__ has joined #oooq | 06:36 | |
| *** links has quit IRC | 06:38 | |
| *** links has joined #oooq | 06:38 | |
| *** udesale_ has quit IRC | 06:38 | |
| *** links has quit IRC | 06:42 | |
| *** matbu has joined #oooq | 06:47 | |
| *** links has joined #oooq | 06:52 | |
| *** _jbadiapa is now known as jbadiapa | 06:53 | |
| *** ccamacho has joined #oooq | 07:03 | |
| *** ccamacho has quit IRC | 07:07 | |
| *** ccamacho has joined #oooq | 07:08 | |
| *** bogdando has joined #oooq | 07:20 | |
| *** tesseract has joined #oooq | 07:23 | |
| *** ykarel|afk is now known as ykarel|lunch | 07:30 | |
| *** udesale__ is now known as udesale | 07:53 | |
| *** sshnaidm|off is now known as sshnaidm | 07:59 | |
| sshnaidm | quiquell, do you use telegraf in docker? | 07:59 |
| quiquell | sshnaidm: Na, I needed python | 07:59 |
| quiquell | The oficial telegraf docker image doesn't have it | 08:00 |
| quiquell | I didn't want to create new docker images | 08:00 |
| quiquell | At the beginning I start grafana and influxdb from a docker-compose | 08:00 |
| sshnaidm | quiquell, yeah, the same.. | 08:00 |
| quiquell | Install telegraf directly | 08:00 |
| quiquell | Is not a big deal | 08:00 |
| quiquell | I have also instal grafan directly to work with the config | 08:00 |
| quiquell | The docker was just to start quick | 08:00 |
| sshnaidm | quiquell, I play with exec plugin, but I don't see it uses timestamps from script.. It worries me | 08:00 |
| quiquell | influxdb is still running on docker though | 08:01 |
| quiquell | sshnaidm: if you don't put a timestamp telegraf do it for you | 08:01 |
| quiquell | It depends what you want to represent | 08:01 |
| quiquell | Do you want to go a quick bj ? | 08:01 |
| sshnaidm | quiquell, sure | 08:02 |
| sshnaidm | https://bluejeans.com/u/sshnaidm/ | 08:03 |
| quiquell | btw I have link sova to the ruck rover dashboard | 08:03 |
| quiquell | http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1 | 08:03 |
| quiquell | Also the alarms show it in the notifications | 08:03 |
| *** ykarel|lunch is now known as ykarel | 08:12 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 08:15 |
| *** gkadam has joined #oooq | 08:17 | |
| *** jaganathan has quit IRC | 08:22 | |
| *** jaganathan has joined #oooq | 08:29 | |
| arxcruz|ruck | quiquell: can you give me the credentials for this dashboard ? | 08:45 |
| *** amoralej|off is now known as amoralej | 08:55 | |
| quiquell | arxcruz|ruck: Let's try the invitation features | 09:25 |
| sshnaidm | quiquell, https://github.com/sshnaidm.keys | 09:25 |
| quiquell | sshnaidm: ack | 09:25 |
| arxcruz|ruck | quiquell: https://github.com/arxcruz.keys | 09:25 |
| arxcruz|ruck | hehehe | 09:25 |
| quiquell | sshnaidm: Code for api/builds interface https://github.com/openstack-infra/zuul/blob/master/zuul/driver/sql/sqlconnection.py | 09:27 |
| quiquell | Would be nice to have an influxdb reporter | 09:28 |
| quiquell | arxcruz|ruck: Check your email | 09:29 |
| arxcruz|ruck | quiquell: I will not! | 09:29 |
| arxcruz|ruck | just kidding, i'll | 09:29 |
| *** marios has quit IRC | 09:29 | |
| *** marios has joined #oooq | 09:31 | |
| *** zoli is now known as zoli|lunch | 09:31 | |
| quiquell | arxcruz|ruck: Have check with my email, invitation works | 09:31 |
| sshnaidm | quiquell, yeah, and then we could just use our grafana with upstream influxdb :) | 09:31 |
| quiquell | Yep | 09:31 |
| quiquell | But maybe it's too much of reporting | 09:32 |
| quiquell | But maybe we can like configure the reporter at our zuul jobs | 09:32 |
| quiquell | humm you know... maybe we can integrate that with toci :-/ | 09:32 |
| quiquell | Let's forget about it for now | 09:32 |
| sshnaidm | quiquell, upstream has graphite though.. but not too much data there: http://grafana.openstack.org/ | 09:34 |
| quiquell | sshnaidm: Already found it, no use for us | 09:35 |
| sshnaidm | quiquell, we had there ovb jobs before it moved to rdo cloud | 09:35 |
| quiquell | They only put there stuff for them to monitor zuul's infra | 09:35 |
| sshnaidm | yeah | 09:35 |
| quiquell | sshnaidm: btw I like how they did it | 09:35 |
| quiquell | they have graphite.openstack.org | 09:36 |
| quiquell | But there is not much we can use from there | 09:36 |
| sshnaidm | quiquell, not sure we can connect to their graphite | 09:37 |
| quiquell | I did | 09:37 |
| quiquell | Just add anohter datasource | 09:37 |
| quiquell | There are no credentials | 09:38 |
| sshnaidm | oh, cool | 09:38 |
| sshnaidm | quiquell, nothing interesting there? | 09:38 |
| quiquell | But... data there is very openstack-infra things | 09:38 |
| sshnaidm | I see | 09:38 |
| quiquell | But take a look | 09:38 |
| quiquell | Add a data source and play with it | 09:38 |
| quiquell | in grafana | 09:38 |
| sshnaidm | ok | 09:38 |
| quiquell | I think I have break my grafana with the users... going to restart | 09:39 |
| quiquell | it | 09:39 |
| quiquell | arxcruz|ruck: Did you receive the invitation ? | 09:40 |
| quiquell | sshnaidm: try ssh centos@38.145.34.131 | 09:41 |
| sshnaidm | quiquell, I'm in | 09:42 |
| quiquell | cool | 09:42 |
| quiquell | there is a tmux | 09:42 |
| quiquell | running telegraf | 09:42 |
| sshnaidm | ok | 09:43 |
| quiquell | there is a clone of the review | 09:43 |
| quiquell | I do a git review -d 13797 and rerun it | 09:43 |
| quiquell | to debug your stuf you can run telegraf ... --test locally | 09:44 |
| quiquell | To check the influxdb lines | 09:44 |
| quiquell | If they are correctly parse it | 09:44 |
| quiquell | Add another patchset to the review | 09:44 |
| * quiquell going back to sprint13 business | 09:47 | |
| quiquell | arxcruz|ruck, sshnaidm: the list-of-upstream-reviews trello thing, do we put there all the reviews of ci stuff ? | 09:51 |
| sshnaidm | quiquell, we don't have really a exact scope of this card.. i put there reviews that require attention and not reviewed yet | 09:53 |
| quiquell | pl | 09:53 |
| quiquell | ok | 09:53 |
| *** panda|off is now known as panda | 10:03 | |
| quiquell | panda: What do we do between sprints ? | 10:11 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 10:15 |
| panda | quiquell: shepherd to remaining reviews, or what you do on your 20% of time. maybe we'll start taking about roadmaps, when we underdstand how to handle them and if it's worth | 10:23 |
| quiquell | panda: ok cool | 10:24 |
| quiquell | arxcruz|ruck: This review fails in unrelated job https://review.openstack.org/#/c/570167/ | 10:24 |
| quiquell | It's a reproducer change, do you know if we have issues at tripleo-ci-centos-7-scenario002-multinode-oooq-container ? | 10:24 |
| *** dtantsur|afk is now known as dtantsur | 10:30 | |
| arxcruz|ruck | quiquell: we don't according sova | 10:32 |
| arxcruz|ruck | quiquell: although the gate is 7 hours late... | 10:32 |
| *** zoli|lunch is now known as zoli | 10:34 | |
| quiquell | arxcruz|ruck: ok, btw did you receive the ruck rover cockpit invitation ? | 10:34 |
| arxcruz|ruck | quiquell: yup, and i'm already looking into it | 10:35 |
| quiquell | ok | 10:35 |
| quiquell | Going to add rlandy | 10:35 |
| *** amoralej is now known as amoralej|off | 11:07 | |
| panda | of course my daughter had to get chickenpox two days before our flight ... | 11:23 |
| quiquell | panda: Fucking murphy | 11:26 |
| panda | I hope I can cancel my PTO at this point. I can't go anywhere. | 11:28 |
| *** udesale_ has joined #oooq | 11:33 | |
| *** udesale has quit IRC | 11:35 | |
| *** udesale_ has quit IRC | 11:42 | |
| *** rasca has quit IRC | 11:44 | |
| *** d0ugal has joined #oooq | 11:55 | |
| *** d0ugal__ has quit IRC | 11:57 | |
| sshnaidm | arxcruz|ruck, forgot to mention yesterday, we have this bug also: https://bugs.launchpad.net/tripleo/+bug/1770944 | 12:00 |
| openstack | Launchpad bug 1770944 in tripleo "CI: centos.ci: certmonger service fails while installing undercloud" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 12:00 |
| sshnaidm | arxcruz|ruck, please pass it to rlandy | 12:00 |
| arxcruz|ruck | sshnaidm: ack | 12:01 |
| sshnaidm | tw, folks, we got corp accounts to lynda.com (who works in RH) | 12:03 |
| *** skramaja has quit IRC | 12:07 | |
| quiquell | sshnaidm: This courses are good ? | 12:11 |
| *** quiquell is now known as quiquell|lunch | 12:12 | |
| sshnaidm | quiquell, i didn't take yet, but seems like a large choice of them | 12:13 |
| *** ratailor has quit IRC | 12:14 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 12:15 |
| *** rlandy has joined #oooq | 12:28 | |
| rlandy | arxcruz|ruck: hello | 12:28 |
| rlandy | sorry I messsed up my nick yesterday | 12:29 |
| rlandy | I thought I agreed to be ruck | 12:29 |
| rlandy | but it's fine if you want to keep ruck now - let me know | 12:29 |
| arxcruz|ruck | rlandy: up to you | 12:29 |
| arxcruz|ruck | rlandy: sshnaidm was working on https://bugs.launchpad.net/tripleo/+bug/1770944 | 12:30 |
| openstack | Launchpad bug 1770944 in tripleo "CI: centos.ci: certmonger service fails while installing undercloud" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 12:30 |
| rlandy | arxcruz|ruck: ok - we can stay as we are | 12:30 |
| rlandy | I'll be rover for now | 12:30 |
| sshnaidm | rlandy, yep, forgot to mention it yesterday | 12:30 |
| arxcruz|ruck | rlandy: status for this morning is everything seems to be working smoothly, but gate are 7 hours | 12:30 |
| *** rlandy is now known as rlandy|rover | 12:30 | |
| arxcruz|ruck | i mean, the promotion are ok, the dashboard is green except for phase 2 ocata that is yellow | 12:30 |
| rlandy|rover | arxcruz|ruck: I am working on the ocata and phase2 issues | 12:31 |
| arxcruz|ruck | rlandy|rover: okay | 12:31 |
| rlandy|rover | arxcruz|ruck: the upstream zuul queues are long | 12:31 |
| arxcruz|ruck | rlandy|rover: if you need help with something let me know | 12:31 |
| rlandy|rover | but nothing seems stuck | 12:31 |
| *** udesale has joined #oooq | 12:31 | |
| arxcruz|ruck | rlandy|rover: yup | 12:31 |
| arxcruz|ruck | exactly, just the long queue | 12:31 |
| rlandy|rover | I watched them till late last night | 12:31 |
| rlandy|rover | sshnaidm: did you see that over your time in ruck/river land? | 12:32 |
| rlandy|rover | rover | 12:32 |
| rlandy|rover | the queue is at 14 hours | 12:32 |
| rlandy|rover | but everything looks to be running ok | 12:32 |
| rlandy|rover | jobs start, run, pass etc. | 12:32 |
| sshnaidm | rlandy|rover, yeah, it could be just one patch failed a few times in gate.. | 12:33 |
| arxcruz|ruck | rlandy|rover: i think it worth to mention on tripleo | 12:33 |
| sshnaidm | rlandy|rover, better to look at failed gate jobs | 12:33 |
| sshnaidm | http://cistatus.tripleo.org/gates/ | 12:34 |
| sshnaidm | arxcruz|ruck, seems like tempest fails in 002 scenario in gates: tripleo-ci-centos-7-scenario002-multinode-oooq-container | 12:35 |
| rlandy|rover | ▼ tripleo-ci-centos-7-3nodes-multinode seems to fail as well | 12:35 |
| sshnaidm | rlandy|rover, it's not gate job though | 12:36 |
| sshnaidm | arxcruz|ruck, http://logs.openstack.org/72/550072/2/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/d194dff/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-05-24_10_54_35 | 12:36 |
| arxcruz|ruck | sshnaidm: yeah, i'm investigating | 12:36 |
| arxcruz|ruck | seems to be consistently failing on test_object_services | 12:37 |
| sshnaidm | rlandy|rover, arxcruz|ruck gate queue is raising when we have gates jobs failing, I don't know other reasons | 12:37 |
| rlandy|rover | myoung|off: we're at post-deploy on phase2 - getting there | 12:37 |
| arxcruz|ruck | my first guess is that the service isn't up | 12:38 |
| *** holser__ has quit IRC | 12:38 | |
| *** holser__ has joined #oooq | 12:39 | |
| *** saneax has joined #oooq | 12:39 | |
| *** quiquell|lunch is now known as quiquell | 12:41 | |
| *** trown|outtypewww is now known as trown | 12:44 | |
| arxcruz|ruck | rlandy|rover: sshnaidm it seems some error in the object storage service, i'll reproduce it on rdocloud to check, maybe the url is wrong, not sure if there were some change in the endpoint | 12:45 |
| sshnaidm | arxcruz|ruck, did you create a bug? | 12:46 |
| arxcruz|ruck | sshnaidm: not yet, it's check, let me grab more info before open a bug | 12:46 |
| *** rasca has joined #oooq | 12:49 | |
| rlandy|rover | arxcruz|ruck: also had tempest failure here - not sure it's related ... http://logs.openstack.org/60/567060/33/gate/tripleo-ci-centos-7-containers-multinode/3e1c634/job-output.txt.gz | 12:50 |
| rlandy|rover | need to rekick that | 12:50 |
| arxcruz|ruck | rlandy|rover: that is timeout | 12:50 |
| rlandy|rover | rechecking | 12:51 |
| arxcruz|ruck | http://logs.openstack.org/60/567060/33/gate/tripleo-ci-centos-7-containers-multinode/3e1c634/logs/undercloud/home/zuul/tempest.log.txt.gz | 12:51 |
| arxcruz|ruck | tempest started to run and was killed | 12:51 |
| rlandy|rover | panda: trown: I think this is the last patch of the set ... https://review.openstack.org/#/c/567060/ - I just rechecked it | 12:52 |
| *** tcw has quit IRC | 12:54 | |
| quiquell | rlandy|rover: I see a +2v before the recheck | 12:55 |
| quiquell | Don't know why it didn't merge | 12:55 |
| rlandy|rover | marked -2 | 12:56 |
| trown | undercloud containers job timed out | 12:56 |
| trown | in gate | 12:56 |
| trown | gate queue seems pretty big | 12:56 |
| rlandy|rover | it's running though | 12:56 |
| quiquell | ok | 12:56 |
| trown | 15hours... | 12:56 |
| rlandy|rover | there are a few jobs that have been waiting a while since yesterday | 12:57 |
| rlandy|rover | but they are running now | 12:57 |
| quiquell | rlandy|rover: Have send you an invitation to the ruck/rover dashboard | 12:57 |
| quiquell | http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover | 12:57 |
| rlandy|rover | thank you | 12:58 |
| quiquell | sshnaidm and me working on it | 12:59 |
| quiquell | Let us know if it's of any help | 12:59 |
| rlandy|rover | nothing seems stuck | 13:00 |
| rlandy|rover | took a long time to start | 13:00 |
| rlandy|rover | once these top 4 jobs clear, then the queue will be half the time | 13:00 |
| myoung|off | o/ here ye here ye...ci squad sprint planning, we welcome you all heartily! | 13:01 |
| rlandy|rover | myoung|off: do I attend as river - I always forget these things | 13:01 |
| rlandy|rover | rover | 13:01 |
| myoung|off | rlandy|rover: all are welcome... | 13:02 |
| myoung|off | rlandy|rover: yes please...join. arxcruz|ruck that's you too | 13:03 |
| myoung|off | :) | 13:03 |
| *** myoung|off is now known as myoung | 13:03 | |
| myoung | ^^ https://github.com/openstack/tripleo-specs/blob/master/specs/policy/ci-team-structure.rst#sprint-start--day-1----25-hours | 13:03 |
| arxcruz|ruck | me too? | 13:03 |
| myoung | https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 13:04 |
| myoung | arxcruz|ruck: aye...ruck+rover for planning | 13:04 |
| *** tcw has joined #oooq | 13:04 | |
| *** saneax has quit IRC | 13:05 | |
| *** ykarel is now known as ykarel|away | 13:17 | |
| *** zoli is now known as zoli|afk | 13:20 | |
| *** tcw has quit IRC | 13:21 | |
| *** links has quit IRC | 13:24 | |
| *** amoralej|off is now known as amoralej | 13:35 | |
| *** tcw has joined #oooq | 13:41 | |
| *** ykarel|away has quit IRC | 13:46 | |
| *** udesale has quit IRC | 13:51 | |
| *** udesale has joined #oooq | 13:51 | |
| *** moguimar has quit IRC | 13:54 | |
| *** tesseract-RH has joined #oooq | 13:54 | |
| *** tesseract-RH has quit IRC | 13:55 | |
| *** tesseract-RH has joined #oooq | 13:55 | |
| *** tesseract has quit IRC | 13:56 | |
| *** sai_ has quit IRC | 14:15 | |
| *** rook has quit IRC | 14:15 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 14:15 |
| *** sai_ has joined #oooq | 14:17 | |
| *** rook has joined #oooq | 14:17 | |
| *** rook is now known as Guest11079 | 14:18 | |
| sshnaidm | myoung, can you please share link to etherpad? | 14:22 |
| myoung | sshnaidm: https://etherpad.openstack.org/p/tripleo-ci-zuul-repo-insertion | 14:22 |
| myoung | ssh https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:22 |
| myoung | sshnaidm: https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 14:22 |
| rlandy|rover | http://zuul.openstack.org/ - queue down to 4hr 50 mins | 14:22 |
| myoung | rlandy|rover: rhos-12 is happy accessing the image via id (vs. name that could be a dupe) https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-12-ci-rhos-ovb-minimal-pacemaker-public-bond/922/consoleFull | 14:24 |
| *** Guest11079 has quit IRC | 14:26 | |
| myoung | rlandy|rover: however the (non-voting and I've never seen it pass ever) proto job that attempting to deploy master is failing on UC install. I haven't debugged down and it's not a priority anyway (for now...) https://thirdparty.logs.rdoproject.org/jenkins-periodic-master-rdo_trunk-ovb-minimal-pacemaker-multiple-nics-186/undercloud/home/stack/undercloud_install.log.txt.gz#_2018-05-24_01_26_07 | 14:27 |
| myoung | ^^ but TLDR - accessing by id in rhos-ci.yml works for the centos case as well | 14:27 |
| rlandy|rover | all ... fyi ... rdocloud undergoing upgrade | 14:28 |
| rlandy|rover | jobs are not starting | 14:28 |
| rlandy|rover | arxcruz|ruck: trown: myoung: panda: sshnaidm: ^^ | 14:28 |
| arxcruz|ruck | rlandy|rover: (╯°□°）╯︵ ┻━┻ | 14:28 |
| rlandy|rover | arxcruz|ruck: see rhos-ops | 14:29 |
| rlandy|rover | arxcruz|ruck: upstream zuul back to 5 hrs | 14:29 |
| rlandy|rover | better than 15hrs | 14:29 |
| *** rook_ has joined #oooq | 14:29 | |
| sshnaidm | arxcruz|ruck, less jobs to watch | 14:30 |
| rlandy|rover | myoung: k - getting to the last failures on bm | 14:30 |
| rlandy|rover | I may disable ssl_overcloud if I can't lock down the ips | 14:31 |
| arxcruz|ruck | sshnaidm: now, but then... | 14:35 |
| *** tesseract-RH has quit IRC | 14:47 | |
| *** zoli|afk is now known as zoli | 14:48 | |
| *** zoli is now known as zoli|wfh | 14:48 | |
| *** zoli|wfh is now known as zoli | 14:48 | |
| *** jtomasek has joined #oooq | 14:54 | |
| *** matbu has quit IRC | 15:01 | |
| *** tesseract has joined #oooq | 15:03 | |
| *** matbu has joined #oooq | 15:10 | |
| quiquell | sshnaidm: Added the gauges http://localhost:3000/d/oMxnrF4mk/ruck-rover?orgId=1 | 15:19 |
| *** rook_ is now known as rook | 15:20 | |
| *** kopecmartin has quit IRC | 15:26 | |
| rlandy|rover | ok - on the ocata problem now | 15:27 |
| *** bogdando has quit IRC | 15:30 | |
| *** ccamacho has quit IRC | 15:31 | |
| sshnaidm | quiquell, changed them a little | 15:32 |
| *** gkadam has quit IRC | 15:32 | |
| *** tcw has quit IRC | 15:34 | |
| *** jtomasek has quit IRC | 15:35 | |
| quiquell | sshnaidm: Cool | 15:42 |
| *** quiquell is now known as quiquell|off | 15:42 | |
| quiquell|off | sshnaidm: Did the export/import script too, saving the changes in git | 15:42 |
| sshnaidm | quiquell|off, ok | 15:42 |
| rlandy|rover | 2018-05-24 08:55:03 | 2018-05-24 08:54:59Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error in 0 output ip_address: Unknown Error (HTTP 503) | 15:43 |
| rlandy|rover | in ocata | 15:43 |
| *** matbu has quit IRC | 15:48 | |
| rlandy|rover | arxcruz|ruck: hmm ... https://ci.centos.org/job/tripleo-quickstart-promote-ocata-rdo_trunk-minimal/ has a different failure each job | 15:49 |
| rlandy|rover | wondering if it is not a timeout | 15:49 |
| rlandy|rover | rerunning | 15:49 |
| *** ykarel|away has joined #oooq | 15:51 | |
| *** gkadam has joined #oooq | 15:53 | |
| *** gkadam has quit IRC | 15:54 | |
| *** marios has quit IRC | 15:55 | |
| *** tcw has joined #oooq | 15:56 | |
| *** sshnaidm has quit IRC | 15:56 | |
| rlandy|rover | amoralej: hello - I am looking at the weirdo failures in https://ci.centos.org/job/rdo_trunk-promote-ocata-current-tripleo/ - do you/your team still support those jobs? | 15:59 |
| amoralej | ye | 16:00 |
| amoralej | s | 16:00 |
| amoralej | let me check | 16:00 |
| *** ykarel|away has quit IRC | 16:00 | |
| rlandy|rover | thanks - new with the latest build | 16:00 |
| *** tcw has quit IRC | 16:01 | |
| *** tcw has joined #oooq | 16:01 | |
| amoralej | we are hitting some issue doing pip install in ci.centos environment | 16:01 |
| amoralej | it's weird | 16:01 |
| amoralej | rlandy|rover, for the oooq job, are you also checking it? | 16:02 |
| rlandy|rover | we have a failure on the promote job as well - unrelated | 16:02 |
| rlandy|rover | I am rechecking https://ci.centos.org/job/tripleo-quickstart-promote-ocata-rdo_trunk-minimal/ | 16:02 |
| rlandy|rover | there we have a overcloud deploy failure | 16:05 |
| *** matbu has joined #oooq | 16:08 | |
| *** hubbot has quit IRC | 16:08 | |
| *** dmellado has quit IRC | 16:09 | |
| *** ccamacho has joined #oooq | 16:09 | |
| *** jaganathan has quit IRC | 16:19 | |
| *** lucasagomes is now known as lucas-afk | 16:23 | |
| *** lucas-afk is now known as lucasagomes | 16:23 | |
| *** trown is now known as trown|lunch | 16:29 | |
| myoung | amoralej: we hit a pip install issue in rdo2 that I resolved last week | 16:34 |
| amoralej | myoung, i was checking it with jpena | 16:34 |
| amoralej | how did you resolved it? | 16:35 |
| myoung | https://trello.com/c/5p4kvvPi/596-rdophase2-bm-jobs-failing-b-c-concurrent-pip-installs-are-failing-due-to-sharing-pip-cache | 16:36 |
| myoung | amoralej: it was when multiple executors on a jenkins node were running pip install in parallel, there was cache issues / failures (pip bug) | 16:36 |
| myoung | https://bugs.launchpad.net/tripleo/+bug/1772460 | 16:36 |
| openstack | Launchpad bug 1772460 in tripleo "rdo2: BM jobs failing b/c concurrent pip installs are failing due to sharing pip cache" [Critical,Fix released] - Assigned to Matt Young (halcyondude) | 16:36 |
| myoung | solution was to change jenkins master config (or could do it in job config...but we just did for all) to use a seperate pip cache for each executor | 16:37 |
| *** holser__ has quit IRC | 16:37 | |
| myoung | amoralej: not sure if that's what you're hitting or not... | 16:37 |
| amoralej | myoung, i think so | 16:38 |
| amoralej | looks similar | 16:38 |
| myoung | amoralej: if you're not able to access global jenkins config for ci.centos, this is what RH qe folk did (in this case it's baked into their groovy pipeline definition) but same thing could be done at script/wrapper level that we do control... https://code.engineering.redhat.com/gerrit/#/c/138531/3/vars/sh2.groovy | 16:39 |
| * myoung hunts for calories (lunch) so he does not perish and will biab | 16:39 | |
| amoralej | thanks a bunch myoung | 16:41 |
| *** udesale has quit IRC | 16:44 | |
| *** matbu has quit IRC | 16:48 | |
| *** sshnaidm has joined #oooq | 16:49 | |
| *** sshnaidm is now known as sshnaidm|off | 16:50 | |
| panda | myoung: so no meeting on 5/28 ? | 16:59 |
| myoung | is us holiday | 17:01 |
| myoung | no one from US will be online | 17:01 |
| myoung | red hat holiday too | 17:01 |
| panda | myoung: we are skipping or postponing to tuesday ? | 17:02 |
| panda | mmmh because there's not tripleo meeting | 17:02 |
| *** amoralej is now known as amoralej|off | 17:03 | |
| *** matbu has joined #oooq | 17:04 | |
| *** jfrancoa has quit IRC | 17:11 | |
| myoung | panda: I moved the meeting to tuesday before the tripleo meeting | 17:11 |
| *** dtantsur is now known as dtantsur|afk | 17:11 | |
| panda | myoung: ok | 17:11 |
| myoung | also earlier so sshnaidm|off doesn't have to talk to us when it's dark outside, and I increased to 60m as we'll be discussing results of design phase | 17:12 |
| *** hubbot has joined #oooq | 17:13 | |
| *** panda is now known as panda|off | 17:23 | |
| *** tesseract has quit IRC | 17:25 | |
| *** trown|lunch is now known as trown | 17:44 | |
| rlandy|rover | ocata is killing us | 17:50 |
| rlandy|rover | job fails in a different place each time:( | 17:51 |
| *** matbu has quit IRC | 17:54 | |
| rlandy|rover | arxcruz|ruck: you around? | 18:06 |
| rlandy|rover | need your help with https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans/131/console | 18:07 |
| rlandy|rover | Invalid input for operation: segmentation_id prohibited for flat provider network. | 18:08 |
| *** zoli is now known as zoli|gone | 18:08 | |
| *** zoli|gone is now known as zoli | 18:08 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 18:16 |
| *** matbu has joined #oooq | 18:26 | |
| rfolco | myoung, need your expertise real quick, would you have any tips on where I should look to switch repos from centos trunk to rhos ? I see rhos repos are installed, but not enabled. Not finding the variable that enables it for a osp job. Thoughts? | 18:51 |
| rfolco | myoung, https://softwarefactory.usersys.redhat.com/logs/95/95/7/check/osp-rhel-7-undercloud-oooq/2925bfe/logs/undercloud/etc/yum.repos.d/ | 18:52 |
| myoung | rfolco: is your job using rhos-release (like the older virt jobs?) | 18:52 |
| * myoung looks at the logs | 18:53 | |
| rfolco | myoung, I manually install it and run it for rhos-12 | 18:53 |
| rfolco | then later in the job I make quickstart_release=rhos12.yml from tripleo-env | 18:53 |
| rfolco | not finding the condition that enables rhos-12 repos instead of trunk/centos | 18:54 |
| myoung | rfolco: the existing jobs we have on rhos-dev-jenkins ("ospphase0") use this: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-12.yml#n35 | 18:55 |
| myoung | repo_cmd_after: | 18:55 |
| myoung | the SUPER old build jobs (that made UC images for OSP) did it like this, prior to the repo-setup role... | 18:56 |
| myoung | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/image-build/rhos-12/latest/base_repo_script.sh.j2 | 18:56 |
| myoung | TLDR: invoking "rhos-release {{ some_args }} will set the repos to enabled | 18:56 |
| myoung | and "rhos-release -x" disables everything (run it prior) | 18:56 |
| myoung | rfolco: I need to dash for a call, free in about half an hour | 18:57 |
| rfolco | myoung, cool thanks for the pointers. Perhaps my yml is not being read properly | 18:57 |
| myoung | rfolco: https://github.com/openstack/tripleo-quickstart/tree/master/roles/repo-setup | 18:59 |
| myoung | rfolco: https://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/templates/repo_setup.sh.j2#L111 | 18:59 |
| * myoung will biaf (30) | 19:00 | |
| *** matbu has quit IRC | 19:06 | |
| *** ChanServ has quit IRC | 19:07 | |
| *** ChanServ has joined #oooq | 19:12 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 19:12 | |
| *** ChanServ has quit IRC | 19:21 | |
| *** ChanServ has joined #oooq | 19:30 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 19:30 | |
| myoung | rfolco: did you get it sorted? avail now | 19:31 |
| rfolco | myoung, still looking. Apparently rhos-12.yml is being loaded, but I don't know why the repos are disabled | 19:32 |
| *** honza is now known as Guest60997 | 19:34 | |
| myoung | rfolco: same job link as before? | 19:34 |
| rlandy|rover | finally a running ocata job | 19:34 |
| * myoung can look | 19:34 | |
| myoung | rlandy|rover: was it the pip concurrency issue? | 19:34 |
| rlandy|rover | failing to get md5sum | 19:34 |
| rlandy|rover | not saying this will pass but the previous failure was not legit | 19:35 |
| rfolco | myoung, I noticed that rhos-12 is pike... so perhaps need to change job name and release to make the code path enter in the right ways... | 19:35 |
| rfolco | myoung, https://softwarefactory.usersys.redhat.com/logs/95/95/7/check/osp-rhel-7-undercloud-oooq/2925bfe/ | 19:35 |
| rlandy|rover | also holding thumbs for this one ... https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans/132/console | 19:35 |
| *** bandini has quit IRC | 19:36 | |
| myoung | rfolco: looks like this: https://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/templates/repo_setup.sh.j2#L111 | 19:36 |
| myoung | is never emitting repo_after_cmd | 19:37 |
| rfolco | myoung, I am injecting rhos-12.yml from tripleo-env repo with a ugly workaround... https://softwarefactory.usersys.redhat.com/r/95 | 19:37 |
| myoung | https://softwarefactory.usersys.redhat.com/logs/95/95/7/check/osp-rhel-7-undercloud-oooq/2925bfe/logs/undercloud/home/zuul/repo_setup.sh.txt.gz | 19:37 |
| * myoung looks at #95 | 19:37 | |
| *** bandini has joined #oooq | 19:38 | |
| *** rlandy|rover is now known as rlandy | 19:42 | |
| *** rlandy is now known as rlandy|rover | 19:43 | |
| myoung | rfolco: i think i see what's up | 19:43 |
| myoung | rfolco: do you have a sec to BJ? might be faster | 19:43 |
| rfolco | yep | 19:44 |
| rfolco | lets do it | 19:44 |
| rlandy|rover | myoung: if the current job on phase2 with envE passes, we can re-enable that platform | 19:45 |
| myoung | rlandy|rover: \o/ | 19:45 |
| rlandy|rover | do you know where we left it with the promotion criteria? | 19:45 |
| myoung | rfolco: joining bluejeans.com/matyoung | 19:45 |
| rlandy|rover | looks like queens is old | 19:45 |
| rlandy|rover | myoung: ^^ | 19:45 |
| rlandy|rover | I'll check | 19:45 |
| myoung | https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/queens.ini#L37 | 19:45 |
| myoung | looks like envE is disabled | 19:46 |
| rfolco | myoung, i am in | 19:46 |
| myoung | it's the BMU and virt fs20 | 19:46 |
| myoung | rfolco: inc sry | 19:46 |
| rfolco | wth is inc sry :) | 19:47 |
| *** matbu has joined #oooq | 20:09 | |
| myoung | rfolco: https://softwarefactory.usersys.redhat.com/logs/95/95/7/check/osp-rhel-7-undercloud-oooq/2925bfe/job-output.txt.gz#_2018-05-24_13_11_06_185014 | 20:11 |
| myoung | ^^ --extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/master.yml | 20:11 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 20:16 |
| *** trown is now known as trown|outtypewww | 20:27 | |
| rlandy|rover | arxcruz|ruck: are you still working on tripleo-ci-centos-7-scenario002-multinode-oooq-container? | 20:29 |
| rlandy|rover | tempest failure? | 20:29 |
| rlandy|rover | "overcloud_deploy_result": "failed" - oh dear ocata | 20:31 |
| rlandy|rover | myoung: hello there | 20:53 |
| rlandy|rover | ever seen anything like this: | 20:53 |
| rlandy|rover | 2018-05-24 20:30:32 | u'message': u"Failed to run action [action_ex_id=a140298a-97d4-47d0-bf48-02acdfa975d8, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'skip_deploy_identifier': False, u'container': u'overcloud', u'timeout': 90}']\n ERROR: Property error: : resources.ControlVirtualIP.properties.network: : Error validating value 'ctlplane': <html><body><h1>503 | 20:53 |
| rlandy|rover | Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n", | 20:53 |
| rlandy|rover | 2018-05-24 20:30:32 | u'status': u'FAILED'} | 20:53 |
| myoung | rlandy|rover: that specific error doesn't ring a bell...which job/release? | 20:57 |
| rlandy|rover | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-ocata-rdo_trunk-minimal-346/undercloud/home/stack/overcloud_deploy.log.gz | 20:57 |
| rlandy|rover | ocata | 20:57 |
| rlandy|rover | aws lokking if it was possibly connected to https://bugs.launchpad.net/tripleo/+bug/1773179 | 20:58 |
| openstack | Launchpad bug 1773179 in tripleo "Undercloud upgrades fails in ocata because nova service failure" [High,Triaged] - Assigned to Ronelle Landy (rlandy) | 20:58 |
| rlandy|rover | asking on #tripleo | 21:03 |
| rlandy|rover | myoung: omg - "tempest_status": "passed" on envE - we're getting there | 21:09 |
| myoung | wat | 21:18 |
| myoung | woot | 21:18 |
| myoung | i came up with not a lot on the ocata OC deploy 503...which service is unavail? | 21:18 |
| myoung | ahh neutron (this time) | 21:19 |
| *** matbu has quit IRC | 21:23 | |
| rlandy|rover | myoung: what did you do about it? | 21:38 |
| rlandy|rover | myoung: on better news ... https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans/ | 21:39 |
| rlandy|rover | we have a pass on envE | 21:39 |
| rlandy|rover | was that blocking the queens promotion? | 21:39 |
| rlandy|rover | queens phase 2 is 7 days old | 21:39 |
| rlandy|rover | queens has it removed | 21:40 |
| rlandy|rover | so why is queens not promoting? | 21:41 |
| rlandy|rover | this is ridiculous ... env D | 21:43 |
| rlandy|rover | myoung: ping | 21:47 |
| rlandy|rover | re: queens phase 2 promotion | 21:47 |
| rlandy|rover | https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/queens.ini | 21:47 |
| rlandy|rover | periodic-queens-rdo_trunk-featureset020-1ctlr_1comp_64gb and oooq-queens-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans passed | 21:48 |
| rlandy|rover | no promotion? | 21:48 |
| rlandy|rover | current-tripleo-rdo-internal | 21:50 |
| rlandy|rover | 302 MB | 21:50 |
| rlandy|rover | 2 days ago | 21:50 |
| rlandy|rover | http://rhos-release.virt.bos.redhat.com:3030/rhosp says 7 days | 21:51 |
| rlandy|rover | panda|off: ping re: promotions if yo come back | 21:52 |
| rlandy|rover | promoter server | 22:05 |
| rlandy|rover | I need to get onto the promoter server | 22:14 |
| rlandy|rover | 2018-05-24 20:05:08,742 22989 ERROR promoter Unable to acquire lock. Another promoter process is running. Aborting. | 22:14 |
| rlandy|rover | 2018-05-24 22:11:37,024 3409 ERROR promoter Unable to acquire lock. Another promoter process is running. Aborting. | 22:14 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 22:16 |
| *** matbu has joined #oooq | 22:42 | |
| rlandy|rover | arxcruz|ruck: ^^ fyi - when you get in in the morning - queen should have promoted | 22:57 |
| *** matbu has quit IRC | 23:11 | |
| *** matbu has joined #oooq | 23:21 | |
| *** rlandy|rover is now known as rlandy|rvr|bbl | 23:25 | |
| *** matbu has quit IRC | 23:30 | |
| *** jtomasek has joined #oooq | 23:31 | |
| *** matbu has joined #oooq | 23:36 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!