| *** dtantsur|afk has quit IRC | 00:00 | |
| *** dtantsur has joined #rdo | 00:01 | |
| *** morazi has quit IRC | 00:03 | |
| *** Son_Goku has joined #rdo | 00:07 | |
| *** Caterpillar has quit IRC | 00:08 | |
| *** kellyp has quit IRC | 00:14 | |
| *** mary_grace has quit IRC | 00:20 | |
| *** Son_Goku has quit IRC | 00:20 | |
| *** gildub has quit IRC | 00:21 | |
| *** oanson has quit IRC | 00:22 | |
| *** suresh12 has quit IRC | 00:23 | |
| *** oanson has joined #rdo | 00:24 | |
| *** suresh12 has joined #rdo | 00:33 | |
| *** noslzzp has quit IRC | 00:34 | |
| dmsimard | The maintenance is complete, please let me know if there are any issues. | 00:34 |
|---|---|---|
| *** noslzzp has joined #rdo | 00:35 | |
| *** gildub has joined #rdo | 00:38 | |
| *** eck`gone is now known as eck` | 00:43 | |
| *** vnogin has joined #rdo | 00:44 | |
| *** vnogin has quit IRC | 00:51 | |
| *** kellyp has joined #rdo | 00:51 | |
| *** brault has joined #rdo | 00:53 | |
| *** kellyp has quit IRC | 00:56 | |
| *** brault has quit IRC | 00:56 | |
| *** brault has joined #rdo | 00:57 | |
| *** jcoufal has quit IRC | 00:59 | |
| *** jcoufal has joined #rdo | 01:00 | |
| *** brault has quit IRC | 01:01 | |
| *** vnogin has joined #rdo | 01:06 | |
| *** vnogin has quit IRC | 01:10 | |
| *** vnogin has joined #rdo | 01:11 | |
| *** vnogin has quit IRC | 01:12 | |
| *** vnogin has joined #rdo | 01:16 | |
| *** vnogin has quit IRC | 01:21 | |
| *** noslzzp has quit IRC | 01:21 | |
| *** vnogin has joined #rdo | 01:26 | |
| *** Son_Goku has joined #rdo | 01:28 | |
| *** TikTok has quit IRC | 01:30 | |
| *** vnogin has quit IRC | 01:30 | |
| *** noslzzp has joined #rdo | 01:31 | |
| *** vnogin has joined #rdo | 01:31 | |
| *** suresh12 has quit IRC | 01:32 | |
| *** TikTok has joined #rdo | 01:33 | |
| *** liujiong has joined #rdo | 01:34 | |
| *** rdogerrit has joined #rdo | 01:34 | |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: WIP: Upgrade registry to OpenShift 3.7 https://review.rdoproject.org/r/10259 | 01:34 |
| *** rlandy|afk is now known as rlandy | 01:36 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: glare: http://trunk.rdoproject.org/centos7-master/report.html | 01:37 |
| *** Son_Goku has quit IRC | 01:43 | |
| *** jcoufal has quit IRC | 01:44 | |
| *** jcoufal has joined #rdo | 01:45 | |
| *** vnogin has quit IRC | 01:47 | |
| *** oanson has quit IRC | 01:52 | |
| *** oanson has joined #rdo | 01:52 | |
| *** rlandy has quit IRC | 01:53 | |
| rdogerrit | rdo-trunk created openstack/glare-distgit rpm-master: openstack-glare: failed to build b0b7b9f https://review.rdoproject.org/r/10394 | 01:57 |
| *** noslzzp has quit IRC | 01:59 | |
| *** jcoufal has quit IRC | 01:59 | |
| *** jcoufal has joined #rdo | 02:03 | |
| rdobot | [sensu] RESOLVED: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| No build failures detected: http://trunk.rdoproject.org/centos7-master/report.html | 02:07 |
| *** vnogin has joined #rdo | 02:12 | |
| *** suresh12 has joined #rdo | 02:14 | |
| *** jcoufal has quit IRC | 02:16 | |
| *** suresh12 has quit IRC | 02:18 | |
| *** paragan has joined #rdo | 02:22 | |
| *** Jeffrey4l has quit IRC | 02:26 | |
| *** Amita has joined #rdo | 02:27 | |
| *** Jeffrey4l has joined #rdo | 02:29 | |
| *** Son_Goku has joined #rdo | 02:34 | |
| *** jcoufal has joined #rdo | 02:36 | |
| *** Son_Goku has quit IRC | 02:43 | |
| *** Son_Goku has joined #rdo | 02:46 | |
| *** noslzzp has joined #rdo | 02:54 | |
| *** fragatina has quit IRC | 02:56 | |
| *** fragatina has joined #rdo | 02:57 | |
| *** Son_Goku has quit IRC | 02:57 | |
| *** fragatin_ has joined #rdo | 02:59 | |
| *** fragati__ has joined #rdo | 03:01 | |
| *** fragatina has quit IRC | 03:01 | |
| *** noslzzp has quit IRC | 03:03 | |
| *** noslzzp has joined #rdo | 03:03 | |
| *** fragatin_ has quit IRC | 03:03 | |
| *** fragati__ has quit IRC | 03:04 | |
| *** noslzzp has quit IRC | 03:04 | |
| *** imcleod has quit IRC | 03:05 | |
| *** fragatina has joined #rdo | 03:05 | |
| *** fragatina has quit IRC | 03:09 | |
| *** richm has quit IRC | 03:11 | |
| *** jcoufal has quit IRC | 03:13 | |
| *** imcleod has joined #rdo | 03:19 | |
| *** vnogin has quit IRC | 03:20 | |
| *** mary_grace has joined #rdo | 03:25 | |
| *** Son_Goku has joined #rdo | 03:26 | |
| *** kellyp has joined #rdo | 03:32 | |
| *** imcleod has quit IRC | 03:35 | |
| *** kellyp has quit IRC | 03:36 | |
| *** TikTok has quit IRC | 03:42 | |
| *** shreshtha has joined #rdo | 03:47 | |
| *** shreshtha has quit IRC | 03:47 | |
| *** shreshtha has joined #rdo | 03:48 | |
| *** udesale has joined #rdo | 03:48 | |
| *** coolsvap has joined #rdo | 03:58 | |
| *** jatanmalde has joined #rdo | 04:06 | |
| *** fragatina has joined #rdo | 04:07 | |
| *** anilvenkata has joined #rdo | 04:09 | |
| *** anilvenkata has quit IRC | 04:10 | |
| *** anilvenkata has joined #rdo | 04:10 | |
| *** fragatina has quit IRC | 04:11 | |
| *** Amita has quit IRC | 04:16 | |
| *** ykarel|away is now known as ykarel | 04:29 | |
| *** suresh12 has joined #rdo | 04:51 | |
| *** gildub has quit IRC | 04:52 | |
| *** brault has joined #rdo | 05:05 | |
| *** janki has joined #rdo | 05:08 | |
| *** gildub has joined #rdo | 05:09 | |
| *** brault has quit IRC | 05:09 | |
| *** Son_Goku has quit IRC | 05:13 | |
| *** gildub has quit IRC | 05:26 | |
| *** chlong has quit IRC | 05:26 | |
| *** mary_grace has quit IRC | 05:27 | |
| *** oanson has quit IRC | 05:34 | |
| *** oanson has joined #rdo | 05:34 | |
| *** cshastri has joined #rdo | 05:44 | |
| *** suresh12 has quit IRC | 05:45 | |
| *** ratailor has joined #rdo | 05:45 | |
| *** suresh12 has joined #rdo | 05:52 | |
| *** suresh12 has quit IRC | 05:52 | |
| *** aditya_r has joined #rdo | 06:00 | |
| *** cshastri has quit IRC | 06:01 | |
| *** cshastri has joined #rdo | 06:02 | |
| *** jprovazn has joined #rdo | 06:07 | |
| *** TikTok has joined #rdo | 06:08 | |
| *** aditya_r has quit IRC | 06:12 | |
| *** brault has joined #rdo | 06:20 | |
| *** jprovazn has quit IRC | 06:21 | |
| *** jprovazn has joined #rdo | 06:28 | |
| *** brault has quit IRC | 06:35 | |
| *** emalikov has joined #rdo | 06:37 | |
| *** Amita has joined #rdo | 06:37 | |
| *** paramite has joined #rdo | 06:40 | |
| *** aufi has joined #rdo | 06:44 | |
| *** brault has joined #rdo | 06:45 | |
| *** jaosorior has joined #rdo | 06:46 | |
| *** iOsX has joined #rdo | 06:46 | |
| *** jtomasek has joined #rdo | 06:50 | |
| *** threestrands has quit IRC | 06:53 | |
| *** anshul has joined #rdo | 06:59 | |
| *** sbonazzo has joined #rdo | 07:00 | |
| iOsX | Did anyone deploy Murano project into RDO? | 07:04 |
| *** spectr has joined #rdo | 07:05 | |
| *** egonzalez has joined #rdo | 07:05 | |
| *** hrw has quit IRC | 07:06 | |
| *** hrw has joined #rdo | 07:08 | |
| *** sbulage has joined #rdo | 07:19 | |
| *** pcaruana has joined #rdo | 07:24 | |
| *** mosulica has joined #rdo | 07:27 | |
| *** smeyer has joined #rdo | 07:29 | |
| *** vnogin has joined #rdo | 07:32 | |
| *** kellyp has joined #rdo | 07:32 | |
| *** ccamacho has joined #rdo | 07:34 | |
| *** vnogin has quit IRC | 07:36 | |
| *** kellyp has quit IRC | 07:37 | |
| *** jackNemrod has joined #rdo | 07:40 | |
| *** hseipp has joined #rdo | 07:40 | |
| *** vnogin has joined #rdo | 07:56 | |
| *** vnogin has quit IRC | 08:00 | |
| *** egonzalez has quit IRC | 08:10 | |
| *** tesseract has joined #rdo | 08:10 | |
| *** cylopez has joined #rdo | 08:12 | |
| number80 | iOsX: at least packages maintainers have | 08:12 |
| *** ratailor is now known as ratailor|Lunch | 08:13 | |
| *** ykarel is now known as ykarel|lunch | 08:15 | |
| *** gkadam has joined #rdo | 08:17 | |
| *** kellyp has joined #rdo | 08:25 | |
| *** egonzalez has joined #rdo | 08:26 | |
| *** kellyp has quit IRC | 08:29 | |
| *** jhershbe has joined #rdo | 08:30 | |
| *** amoralej|off is now known as amoralej | 08:38 | |
| amoralej | panda|ruck, all jobs passed in last execution https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/lastSuccessfulBuild/ | 08:39 |
| amoralej | could we promote? | 08:39 |
| *** _bsurfer has joined #rdo | 08:39 | |
| iOsX | number80: and keep it in a secret? :) rpm-packages are exist, systemd services are present as well. | 08:45 |
| iOsX | Nevertheless, no documentation :D | 08:45 |
| *** jhershbe has quit IRC | 08:46 | |
| *** Caterpillar has joined #rdo | 08:46 | |
| number80 | iOsX: not sure to understand, what are we keeping secret? | 08:46 |
| number80 | Fact is, core members can't be experts on every service, but if someone wants to write docs on murano, they're welcome | 08:47 |
| number80 | If there are any blockers to contribute to RDO, I'm more than happy to hear it and try solving issues | 08:48 |
| *** vnogin has joined #rdo | 08:48 | |
| *** jpena|off is now known as jpena | 08:49 | |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Bump rdoinfo queens-uc tags to latest upper-constraints https://review.rdoproject.org/r/10368 | 08:50 |
| jpena | good morning | 08:50 |
| number80 | o/ | 08:51 |
| *** MrAbaddon has quit IRC | 08:51 | |
| *** vnogin has quit IRC | 08:52 | |
| iOsX | number80: if I correct understand you, I need to find persons who build rpm-packages for Murano ask them | 08:57 |
| *** hewbrocca_afk is now known as hewbrocca | 08:57 | |
| panda|ruck | amoralej: let me look | 09:01 |
| panda|ruck | amoralej: can you stop current run, before we reupload images again ? | 09:04 |
| amoralej | panda|ruck, let me check | 09:05 |
| amoralej | panda|ruck, i'll abort all jobs for master | 09:05 |
| panda|ruck | amoralej: thanks, they just started | 09:06 |
| amoralej | panda|ruck, done | 09:07 |
| number80 | iOsX: yep, but it's all in rdoinfo database | 09:08 |
| panda|ruck | ok, I'll start the promotion now | 09:08 |
| number80 | https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml | 09:08 |
| *** vnogin has joined #rdo | 09:10 | |
| *** jpich has joined #rdo | 09:12 | |
| panda|ruck | link promoted, promotimg containers now | 09:12 |
| panda|ruck | this will take a while | 09:12 |
| *** ykarel|lunch is now known as ykarel | 09:12 | |
| panda|ruck | amoralej: ^ | 09:12 |
| amoralej | panda|ruck, ok | 09:13 |
| *** ratailor|Lunch is now known as ratailor | 09:13 | |
| panda|ruck | I'll make an announcement on openstack-dev | 09:14 |
| *** derekh has joined #rdo | 09:14 | |
| *** jtomasek has quit IRC | 09:15 | |
| *** vnogin has quit IRC | 09:16 | |
| *** sbulage has quit IRC | 09:19 | |
| *** aditya_r has joined #rdo | 09:20 | |
| *** sbulage has joined #rdo | 09:21 | |
| *** lucas-afk is now known as lucasagomes | 09:21 | |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Promote CBS tags update for pike-release https://review.rdoproject.org/r/10379 | 09:23 |
| amoralej | panda|ruck, we also need tu update https://images.rdoproject.org/master/delorean/current-tripleo/ | 09:24 |
| amoralej | to point to new hash | 09:24 |
| *** jtomasek has joined #rdo | 09:24 | |
| *** aditya_r has quit IRC | 09:24 | |
| amoralej | is it done after tagging containers? | 09:24 |
| panda|ruck | amoralej: yes | 09:25 |
| amoralej | ack | 09:25 |
| *** aditya_r has joined #rdo | 09:25 | |
| panda|ruck | amoralej: I realize we should change a bit the order to make things a bit more atomic | 09:25 |
| rdogerrit | Alfredo Moralejo created config master: Revert "Delay the execution of openstack-periodic" https://review.rdoproject.org/r/10395 | 09:26 |
| rdogerrit | Alfredo Moralejo created config master: Revert "temporarily disable containers build" https://review.rdoproject.org/r/10396 | 09:26 |
| rdogerrit | Alfredo Moralejo proposed config master: Revert "temporarily disable containers build" https://review.rdoproject.org/r/10396 | 09:26 |
| *** aditya_r has quit IRC | 09:27 | |
| *** aditya_r has joined #rdo | 09:28 | |
| rdogerrit | Alfredo Moralejo created rdo-infra/ci-config master: Revert "tripleo: fix the pin" https://review.rdoproject.org/r/10397 | 09:31 |
| *** ratailor is now known as ratailor|session | 09:31 | |
| amoralej | panda|ruck, can you review ^ ^^ | 09:31 |
| *** racedo has joined #rdo | 09:32 | |
| *** aditya_r has quit IRC | 09:32 | |
| *** ykarel is now known as ykarel|session | 09:32 | |
| rdogerrit | Javier Peña created config master: Find repo for rdoinfo-DLRN-check jobs in dlrn-gate-repository https://review.rdoproject.org/r/10398 | 09:35 |
| jpena | amoralej: ^^ | 09:35 |
| jpena | that will fix the weirdo tests for rdoinfo changes | 09:35 |
| *** jatanmalde has quit IRC | 09:36 | |
| *** TikTok has quit IRC | 09:37 | |
| *** mcornea has joined #rdo | 09:39 | |
| *** udesale__ has joined #rdo | 09:39 | |
| openstackgerrit | Merged openstack/packstack master: Update Cirros image to 0.3.5 https://review.openstack.org/517089 | 09:39 |
| *** udesale has quit IRC | 09:41 | |
| *** jparrill has joined #rdo | 09:42 | |
| panda|ruck | amoralej: promotion completed | 09:49 |
| panda|ruck | amoralej: sending email | 09:49 |
| amoralej | panda|ruck, thanks! | 09:49 |
| amoralej | panda|ruck, btw i'm connected to a node which is running the containers-build job | 09:49 |
| amoralej | to try to see why it's taking so much time | 09:50 |
| panda|ruck | amoralej: cool, can I get sucha node too ? :) I don't know what's the best way to test buil without the undercloud. Maybe I'll just take the image from rdocloud and run it here, then make the test locally | 09:51 |
| amoralej | panda|ruck, i think undercloud is not needed | 09:52 |
| amoralej | panda|ruck, i/o is pretty fast, the problem is not building the containers but pushing them | 09:52 |
| panda|ruck | amoralej: it's not per se, but brings in some dependencies, | 09:52 |
| amoralej | and it's swapping, we will fix that removing the undercloud deployment | 09:52 |
| amoralej | what kind of dependencies? | 09:53 |
| panda|ruck | amoralej: I don't remember, I remember that removing it was making the rest not working properly | 09:53 |
| panda|ruck | that's why I have to test | 09:53 |
| panda|ruck | mail sent | 09:54 |
| rdogerrit | Attila Darazs proposed rdo-infra/ci-config master: Compress undercloud image before upload https://review.rdoproject.org/r/10356 | 09:54 |
| amoralej | panda|ruck, there is a paremeter --max-concurrent-uploads=16 | 09:54 |
| amoralej | int the dockerd-current process | 09:55 |
| amoralej | maybe it's not enough | 09:55 |
| amoralej | i think concurrency in kolla is higuer | 09:55 |
| openstackgerrit | Javier Peña proposed openstack/packstack master: Fix neutron-vpnaas, re-enable it in tests https://review.openstack.org/517307 | 09:56 |
| *** pradk has joined #rdo | 09:57 | |
| *** links has quit IRC | 09:58 | |
| rdogerrit | Attila Darazs proposed rdo-infra/ci-config master: Compress undercloud image before upload https://review.rdoproject.org/r/10356 | 09:59 |
| *** liujiong has quit IRC | 10:06 | |
| *** daidv has quit IRC | 10:10 | |
| *** Son_Goku has joined #rdo | 10:10 | |
| *** apevec has joined #rdo | 10:13 | |
| *** apevec has quit IRC | 10:13 | |
| *** apevec has joined #rdo | 10:13 | |
| apevec | amoralej, panda|ruck - so we got luck with that Oct 31 hash? | 10:14 |
| amoralej | apevec, yes | 10:14 |
| amoralej | it's promoted | 10:14 |
| apevec | 4am run? | 10:14 |
| amoralej | i've started jobs in RDO-CI now | 10:14 |
| panda|ruck | apevec: define "luck" | 10:14 |
| amoralej | :) | 10:14 |
| panda|ruck | apevec: 12AM and 4AM | 10:14 |
| amoralej | panda|ruck, apevec wrt the container build jobs | 10:14 |
| apevec | 1st run earlier last night failed w/ timeouts in OC deploy | 10:15 |
| amoralej | i'm pretty sure the problem is performance of registry.rdoproject.org | 10:15 |
| amoralej | apevec, one failed and the other was aborted | 10:15 |
| apevec | amoralej, how come? it's _localhost_ timeout | 10:15 |
| amoralej | for review.r.o maintenance | 10:15 |
| apevec | ah you mean jobs testing w/ containers | 10:15 |
| amoralej | no, no | 10:15 |
| amoralej | containers build | 10:15 |
| apevec | didn't we disable it? | 10:15 |
| amoralej | i'm connected to a node running the job | 10:16 |
| amoralej | for pike | 10:16 |
| amoralej | it's running right now | 10:16 |
| apevec | ah ok | 10:16 |
| *** udesale__ has quit IRC | 10:16 | |
| *** udesale has joined #rdo | 10:16 | |
| amoralej | job build works fine | 10:16 |
| amoralej | it'd be better to avoid installing UC because it's swapping | 10:17 |
| amoralej | but io is very fast | 10:17 |
| amoralej | and building containers finish pretty fast | 10:17 |
| apevec | so where is the bottleneck? | 10:17 |
| amoralej | it take ages pushing them | 10:17 |
| amoralej | even for containers that doesn't fail | 10:17 |
| amoralej | and checking at opened connections | 10:17 |
| amoralej | is with rdo registry | 10:17 |
| amoralej | not localhost | 10:18 |
| panda|ruck | so the localhost is fake ? it's actually proxied response from dockerd ? | 10:19 |
| apevec | amoralej, what is the command pushing containers? | 10:19 |
| amoralej | it's kolla itself | 10:21 |
| amoralej | using docker python bindings | 10:21 |
| *** rcernin has joined #rdo | 10:21 | |
| amoralej | panda|ruck, i'm trying to understand that right now | 10:21 |
| amoralej | panda|ruck, i've put node on hold | 10:21 |
| amoralej | so when job finishes we can still reproduce it | 10:21 |
| apevec | good idea | 10:23 |
| apevec | I wonder if pushing using oc client would be any better | 10:23 |
| apevec | amoralej, that's all part of dmsimard's ansible role? | 10:23 |
| apevec | i.e. nothing oooq specific | 10:24 |
| *** Son_Goku has quit IRC | 10:24 | |
| amoralej | apevec, maybe but that'd be atomic registry specific | 10:24 |
| amoralej | this is using kolla feature itself | 10:24 |
| apevec | yeah... | 10:25 |
| amoralej | so | 10:25 |
| amoralej | connection is from | 10:25 |
| amoralej | docker-currentd local process to registry.r.o | 10:25 |
| *** vnogin has joined #rdo | 10:25 | |
| amoralej | so kolla somehow push the container via docker process?? | 10:26 |
| amoralej | that'd explain the localhost | 10:26 |
| *** jparrill has quit IRC | 10:29 | |
| *** Son_Goku has joined #rdo | 10:29 | |
| *** pgadiya has joined #rdo | 10:31 | |
| *** udesale__ has joined #rdo | 10:32 | |
| *** ihrachys has quit IRC | 10:33 | |
| *** vnogin has quit IRC | 10:33 | |
| *** ihrachys has joined #rdo | 10:33 | |
| apevec | yeah, looks like all goes via local docker | 10:33 |
| *** udesale has quit IRC | 10:34 | |
| amoralej | it uses overlay2 driver | 10:36 |
| amoralej | is that good? | 10:36 |
| apevec | it's better than DM | 10:36 |
| apevec | found some graphs https://camo.githubusercontent.com/2b239ce3d9f53da1aa88751d3477f977e8c9feb4/687474703a2f2f692e696d6775722e636f6d2f4a535565714c632e706e67 | 10:38 |
| *** Son_Goku has quit IRC | 10:39 | |
| apevec | it should be ~aufs (which is not available in el7) | 10:39 |
| amoralej | yeah, it's supported and recomended in 7.4 | 10:39 |
| apevec | so we have either network bottleneck on the path to the rdo registry or in the registry itself | 10:40 |
| amoralej | i'm thinking in creating another docker registry in a test server following https://docs.docker.com/registry/deploying/ to test | 10:40 |
| apevec | default docker one? | 10:40 |
| amoralej | yeah | 10:41 |
| amoralej | just to test | 10:41 |
| apevec | job is also pushing to dockerhub in parallel ? | 10:41 |
| amoralej | i don't think it's a problem of atomic registry software | 10:41 |
| amoralej | mmm | 10:41 |
| amoralej | i don't think so | 10:41 |
| amoralej | panda|ruck, ^ ? | 10:41 |
| apevec | or is that after pushing to rdo registry? | 10:41 |
| panda|ruck | the promoter script pushes to dockerhub | 10:46 |
| panda|ruck | one at a time | 10:46 |
| amoralej | so the job only push them to rdo registry | 10:47 |
| *** rfolco has joined #rdo | 10:51 | |
| apevec | ...which was supposed to be faster, since in the same network | 10:51 |
| apevec | do we have perf metrics for rdo registry? | 10:52 |
| *** vnogin has joined #rdo | 10:52 | |
| amoralej | apevec, what i see is slow IO | 10:54 |
| *** udesale__ has quit IRC | 10:54 | |
| amoralej | disk access | 10:54 |
| amoralej | could be another effect of ceph perf issue | 10:55 |
| *** jprovazn has quit IRC | 10:59 | |
| *** mosulica has quit IRC | 11:03 | |
| apevec | dang | 11:13 |
| apevec | dmanchad, ^ how does it look like on rdocloud Ceph side? | 11:13 |
| apevec | jpena, ^ any ideas what to monitor? | 11:14 |
| * apevec just noticed everybody involved is in Spain :) | 11:14 | |
| jpena | apevec: if the sysstat package is installed, we can check await times and compare with review.rdo and trunk-primary | 11:15 |
| apevec | maybe we could rename rdocloud to ESPcloud? :) | 11:15 |
| jpena | amoralej: is that slow IO happening in registry.rdo or in the slave VM? | 11:15 |
| amoralej | jpena, registry.rdo. | 11:16 |
| amoralej | slave is fast | 11:16 |
| *** Son_Goku has joined #rdo | 11:16 | |
| amoralej | it's local ssd | 11:16 |
| apevec | how big is the containers set? | 11:16 |
| apevec | maybe we could build w/ the local registry then push bulk to rdo openshift registry and dockerhub ? | 11:17 |
| apevec | panda|ruck, ^ | 11:17 |
| panda|ruck | apevec: not sure I undertand | 11:19 |
| panda|ruck | apevec: the container set size is 1-2G per build | 11:19 |
| apevec | panda|ruck, I mean install local registry on the node building containers and push there first | 11:20 |
| panda|ruck | apevec: because of the shared layers | 11:20 |
| apevec | then push to remote rdo and dockerhub registries | 11:20 |
| apevec | so 1-2GB could be on local ssd | 11:20 |
| *** anilvenkata has quit IRC | 11:20 | |
| panda|ruck | apevec: there is a local registry by default, the build pushes to it | 11:21 |
| apevec | hmm | 11:21 |
| amoralej | panda|ruck, but it pushes to rdo registry | 11:21 |
| apevec | but amoralej has seen connection to the remote ? | 11:21 |
| panda|ruck | amoralej: after they are created .. I think | 11:21 |
| amoralej | exactly | 11:21 |
| panda|ruck | so you should see all the images locally with docker images | 11:21 |
| amoralej | in docker, when it creates a container it's stored in the local docker | 11:21 |
| amoralej | not registry, docker process | 11:21 |
| apevec | ok, so what I'm proposing is already happening? | 11:22 |
| amoralej | and then it pushes to rdo registry | 11:22 |
| panda|ruck | yeah, well it's not a fully fledged registry, but it acts the same, only locally | 11:22 |
| apevec | ah, so there is or there isn't local registry? | 11:22 |
| *** mosulica has joined #rdo | 11:22 | |
| amoralej | it's not actually a registry | 11:22 |
| amoralej | i mean, not something listening | 11:22 |
| apevec | ok, got it | 11:22 |
| amoralej | that can be used by others | 11:22 |
| apevec | so all containers are stored locally first | 11:22 |
| amoralej | it's the own docker process | 11:22 |
| amoralej | exactly | 11:23 |
| amoralej | i've configured a basic registry | 11:23 |
| amoralej | and i'm testing the push from the node on hold | 11:23 |
| panda|ruck | why sould we need a local registry ? the nodes gets deleted after the build, so it's not that you can connect to it remotely after the bui;ld | 11:23 |
| amoralej | yeah, yeah | 11:24 |
| amoralej | i'm not saying we'd need it | 11:24 |
| apevec | it was just an idea, that bulk push would be more efficient | 11:24 |
| apevec | but probably not | 11:24 |
| panda|ruck | one thing we can check is if the kolla push retries the push, and how many times | 11:25 |
| amoralej | it tries 3 times | 11:25 |
| panda|ruck | dang | 11:25 |
| *** atoth has joined #rdo | 11:27 | |
| *** julim has quit IRC | 11:29 | |
| *** julim has joined #rdo | 11:30 | |
| *** dsneddon has quit IRC | 11:32 | |
| amoralej | panda|ruck, apevec using my new registry it took 10 minutes to upload all containers | 11:34 |
| amoralej | no failures | 11:34 |
| panda|ruck | #shipit | 11:35 |
| apevec | amoralej, where is your registry located? | 11:35 |
| amoralej | i'd say registry perf is the issue | 11:35 |
| amoralej | in rdo-cloud | 11:35 |
| apevec | your tenant? | 11:35 |
| amoralej | in my personal tenant | 11:35 |
| amoralej | yes | 11:35 |
| apevec | so your registry was empty | 11:35 |
| amoralej | yes | 11:36 |
| apevec | how full is rdo reg? | 11:36 |
| apevec | any logs on the rdo registry side which would tell where it's spending time? | 11:38 |
| amoralej | i'm checking about storage usage | 11:39 |
| apevec | BTW we had all poi failing in https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/214/ :( | 11:39 |
| *** sreichar has joined #rdo | 11:40 | |
| amoralej | yeah, let me fix that | 11:40 |
| amoralej | missing puppet-ssh_keygen | 11:40 |
| apevec | how come, new dep? | 11:41 |
| amoralej | yes | 11:41 |
| amoralej | not exactly | 11:41 |
| amoralej | added to p-o-i puppetfile | 11:41 |
| amoralej | https://github.com/openstack/puppet-openstack-integration/commit/9dc0a3b7d3a90ab8c7fbe6738a9809d86591235f | 11:42 |
| amoralej | not actually required in any puppet module yet | 11:42 |
| amoralej | let's deploy it from source to unblock | 11:43 |
| apevec | ok, so we don't need to package it yet | 11:43 |
| apevec | can we do that per module? | 11:43 |
| amoralej | apevec, yes | 11:43 |
| amoralej | in weirdo | 11:43 |
| *** janki has quit IRC | 11:45 | |
| apevec | is trello dead or just slow? | 11:46 |
| apevec | isitjustme | 11:46 |
| amoralej | apevec, root filesystem usage in registry is 600GB | 11:48 |
| amoralej | so ~550 for registry i guess | 11:48 |
| rdogerrit | Merged gating_scripts master: List packages which are not installed if rpm -q $LIST fails https://review.rdoproject.org/r/10298 | 11:49 |
| rdogerrit | Alfredo Moralejo created rdo-infra/ansible-role-weirdo-puppet-openstack master: Add puppet-ssh_keygen to the list of modules from source https://review.rdoproject.org/r/10399 | 11:49 |
| amoralej | apevec, ^ | 11:49 |
| dmsimard | amoralej, apevec: I investigated push performance issues before | 11:49 |
| *** shreshtha has quit IRC | 11:49 | |
| amoralej | dmsimard, nice | 11:49 |
| amoralej | what do you think about it? | 11:49 |
| dmsimard | There's a bunch of bugs opened about it in upstream docker | 11:50 |
| dmsimard | iirc it's around compression algos | 11:50 |
| rdogerrit | Merged rdo-infra/ci-config master: Compress undercloud image before upload https://review.rdoproject.org/r/10356 | 11:50 |
| apevec | dmsimard, amoralej did setup plain docker reg and push to there was 10min | 11:51 |
| dmsimard | If you tested a pure docker registry and it's faster, maybe there are fixes that have landed | 11:51 |
| apevec | amoralej, which version did you use? | 11:51 |
| amoralej | dmsimard, i tested with a pure docker registry | 11:51 |
| amoralej | but it's empty | 11:51 |
| amoralej | i deployed in a container | 11:51 |
| amoralej | oficial container | 11:51 |
| dmsimard | I'd need to look at irc logs to find the issues I mention | 11:51 |
| dmsimard | But I'm not really awake yet :p | 11:52 |
| dmanchad | apevec, dmsimard : I saw a spike on latency yesterday evening but that's all noticeable | 11:52 |
| dmanchad | https://imgur.com/a/NQq1O | 11:52 |
| dmsimard | It'd be interesting to see if openshift 3.7 is better | 11:53 |
| apevec | amoralej, so that's latest upstream docker version? | 11:53 |
| amoralej | i guess so, gimme a while | 11:54 |
| dmsimard | I'll look in around 1hr30 | 11:54 |
| dmsimard | Or 45 mins | 11:54 |
| dmsimard | How is Jenkins doing ? Any clone errors or slave disconnections ? | 11:55 |
| amoralej | ok | 11:55 |
| *** abregman|afk is now known as abregman | 11:55 | |
| apevec | dmsimard, have not hit any today, but I've not looked either | 11:56 |
| apevec | is console in review.rdo indexed yet? | 11:57 |
| dmsimard | Should be | 11:58 |
| * apevec tries kibana | 11:58 | |
| *** jayg|g0n3 is now known as jayg | 11:59 | |
| *** eck` is now known as eck`gone | 12:00 | |
| apevec | amoralej, panda|ruck - btw tripleo-ci-testing tag moved on? https://console.registry.rdoproject.org/registry#/images/master/centos-binary-aodh-base:tripleo-ci-testing | 12:01 |
| apevec | but pin was NOT removed in ci-config ... | 12:01 |
| apevec | can you double-check which containers did run in the perodic? | 12:04 |
| *** mosulica has quit IRC | 12:06 | |
| *** rhallisey has joined #rdo | 12:08 | |
| *** jlabarre has quit IRC | 12:08 | |
| *** ykarel|session is now known as ykarel | 12:09 | |
| *** ratailor|session has quit IRC | 12:09 | |
| amoralej | apevec, it seems the registry is not responding now | 12:11 |
| apevec | yeah, I was browsing via webui and it just hang | 12:11 |
| *** jcoufal has joined #rdo | 12:11 | |
| amoralej | docker ps hangs | 12:12 |
| apevec | where, are you logged in via ssh ? | 12:12 |
| amoralej | yes | 12:12 |
| amoralej | WarningFailedFailed to start container with docker id d38c67ff58b6 with error: operation timeout: context deadline exceeded | 12:14 |
| apevec | machine doesn't have any swap, so it's not swapping | 12:15 |
| *** jlabarre has joined #rdo | 12:15 | |
| *** kellyp has joined #rdo | 12:16 | |
| amoralej | apevec | 12:17 |
| amoralej | Out of memory: Kill process 36629 (cockpit-stub) score 1033 or sacrifice child | 12:17 |
| *** anshul has quit IRC | 12:17 | |
| apevec | ok, so we need to add some swap | 12:17 |
| amoralej | and disk is 100% | 12:17 |
| *** janki has joined #rdo | 12:18 | |
| amoralej | not capacity | 12:18 |
| apevec | so registry storage is all on vda1 and that's boot from volume? | 12:18 |
| amoralej | yeah | 12:19 |
| apevec | there is small vdb 50G for /var/lib/docker | 12:19 |
| amoralej | there is a process " du -s -B 1 /var/lib/origin/openshift.local.volumes/pods/ad4e6ebf-ae27-11e7-900a-fa163ec640a5/volumes/kubernetes.io~empty-dir/registry-storage" | 12:19 |
| amoralej | running for +20mins | 12:19 |
| apevec | wth is that | 12:19 |
| amoralej | but it's parent is openshift node process | 12:20 |
| amoralej | kubelet | 12:20 |
| apevec | I mean, really, shelling out?? | 12:20 |
| amoralej | so it could be kubernetes taking statistics on volume usage or something like that | 12:20 |
| *** kellyp has quit IRC | 12:20 | |
| amoralej | but with the size and number of dirs/files and using ceph | 12:20 |
| amoralej | that's bad | 12:21 |
| apevec | and killing IO... | 12:21 |
| *** lucasagomes is now known as lucas-hungry | 12:21 | |
| amoralej | but i'm afraid if i kill it, it may be worst | 12:21 |
| *** jprovazn has joined #rdo | 12:21 | |
| apevec | yeah, leave it | 12:22 |
| amoralej | apevec, https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/fs.go#L60 | 12:22 |
| amoralej | apevec, registry is back | 12:25 |
| panda|ruck | apevec: https://hub.docker.com/r/tripleomaster/centos-binary-aodh-api/tags/ | 12:25 |
| amoralej | and tags are correct | 12:26 |
| *** imcleod has joined #rdo | 12:28 | |
| apevec | hmm, tripleo-ci-testing 2 months ago? | 12:29 |
| apevec | ah that's old on dockerhub | 12:29 |
| apevec | so how do you tell in dockerhub which hash is "current-tripleo" now? | 12:30 |
| apevec | ok, comparing by size ... | 12:30 |
| apevec | good enough | 12:30 |
| amoralej | apevec, dmsimard kubelet has an option to disable the volume statistics and metrics, which is starting a "du" command on the whole directory | 12:33 |
| amoralej | https://kubernetes.io/docs/admin/kubelet/ | 12:33 |
| amoralej | --volume-stats-agg-period duration | 12:33 |
| amoralej | i'm not sure if that can be disabled in openshift | 12:33 |
| amoralej | iirc there is a whay to pass kubelet parameters | 12:33 |
| apevec | amoralej, how did you check tags are correct? | 12:34 |
| amoralej | apevec skopeo inspect docker://trunk.registry.rdoproject.org/master/centos-binary-nova-api:tripleo-ci-testing|grep rdo_version | 12:34 |
| apevec | I tried via console.registry.rdoproject.org but it became unresponsive | 12:34 |
| amoralej | i tried some | 12:34 |
| amoralej | du is running again | 12:34 |
| rdogerrit | Merged rdo-infra/ansible-role-weirdo-puppet-openstack master: Add puppet-ssh_keygen to the list of modules from source https://review.rdoproject.org/r/10399 | 12:34 |
| amoralej | by default is started every minute | 12:34 |
| amoralej | i think that's not good | 12:34 |
| apevec | so maybe console is killing it? | 12:34 |
| amoralej | registry seems to be working fine now | 12:35 |
| amoralej | let me check the console | 12:35 |
| amoralej | apevec, console is working fine for me | 12:36 |
| apevec | yeah, du stats every min seems too much | 12:36 |
| apevec | ok, then itisjustme | 12:37 |
| *** catintheroof has joined #rdo | 12:37 | |
| *** catintheroof has quit IRC | 12:37 | |
| *** catintheroof has joined #rdo | 12:37 | |
| apevec | amoralej, I'm sad, even skopeo failsforme :( FATA[0124] error parsing HTTP 400 response body: unexpected end of JSON input: "" | 12:39 |
| amoralej | let me try again | 12:39 |
| amoralej | yeah | 12:39 |
| amoralej | it's failing now | 12:39 |
| *** jcoufal_ has joined #rdo | 12:39 | |
| amoralej | docker ps is hang again | 12:39 |
| amoralej | i think that du is killing us | 12:40 |
| *** jcoufal has quit IRC | 12:41 | |
| *** pgadiya has quit IRC | 12:42 | |
| apevec | amoralej, so what about dirtyhack - replace du with a dummy emtpy shell script? | 12:42 |
| amoralej | apevec, i think it's better to disable it | 12:42 |
| amoralej | using configuration param | 12:42 |
| jpena | apevec: I'd rather do chmod -x :D | 12:43 |
| apevec | jpena, or that :) | 12:43 |
| amoralej | i'm trying to apply the config param | 12:43 |
| apevec | amoralej, yes, but for that we need restart | 12:43 |
| amoralej | i'm looking if there is a way to reload config dynamically | 12:43 |
| *** gkadam has quit IRC | 12:47 | |
| *** jcoufal_ has quit IRC | 12:47 | |
| *** jcoufal has joined #rdo | 12:48 | |
| *** toure_biab is now known as toure | 12:51 | |
| apevec | dmsimard, amoralej, ok, please leave updates in https://bugs.launchpad.net/tripleo/+bug/1729328 or CIX card, I've put summary of the current action in the card | 12:52 |
| openstack | Launchpad bug 1729328 in tripleo "Container images build in RDOcloud are randomly failing to be pushed to localhost" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 12:52 |
| * apevec goes to calls | 12:52 | |
| amoralej | apevec, dmsimard i've followed jpena idea | 12:53 |
| amoralej | chmod +x | 12:53 |
| apevec | -x ? | 12:53 |
| dmsimard | amoralej: I don't think we're collecting metrics in openshift | 12:53 |
| amoralej | dmsimard, yes it is | 12:53 |
| apevec | du is definitely running | 12:53 |
| amoralej | i've seen the du process | 12:54 |
| apevec | ...for long time | 12:54 |
| amoralej | i just killed it | 12:54 |
| *** lucas-hungry is now known as lucasagomes | 12:54 | |
| dmsimard | hmm, I'll take a look | 12:54 |
| amoralej | dmsimard, i've added the config option to /etc/origin/node/node-config.yaml | 12:54 |
| *** rcernin has quit IRC | 12:54 | |
| amoralej | but i was not brave to restart it :) | 12:54 |
| *** abregman is now known as abregman|afk | 12:54 | |
| dmsimard | says default false here.. https://github.com/openshift/openshift-ansible/tree/master/roles/openshift_hosted_metrics#role-variables | 12:56 |
| dmsimard | what's the setting called ? | 12:56 |
| dmsimard | there's nothing dangerous about restarting origin-node and origin-master | 12:56 |
| dmsimard | they're like restarting nova or keystone | 12:56 |
| dmsimard | it's docker you don't want to restart | 12:56 |
| amoralej | dmsimard, it's kubelet param volume-stats-agg-period: | 12:56 |
| *** jpena is now known as jpena|lunch | 12:57 | |
| dmsimard | don't see anything for that in trunk.. let me see in 3.5 | 12:57 |
| amoralej | dmsimard, https://kubernetes.io/docs/admin/kubelet/ | 12:57 |
| amoralej | i'm checking in kubernetes doc directly | 12:57 |
| amoralej | dmsimard, docker ps is unresponsive now | 12:58 |
| dmsimard | amoralej: it mostly always is :p | 12:58 |
| amoralej | wow | 12:59 |
| dmsimard | It was (more) responsive before the maintenance | 12:59 |
| dmsimard | but I haven't dug into it | 12:59 |
| *** myoung|bbl is now known as myoung | 13:00 | |
| dmsimard | amoralej: are you sure it's a node config and not a master config ? | 13:01 |
| amoralej | dmsimard, it's kubelet config | 13:01 |
| amoralej | so i guess it's node config | 13:01 |
| amoralej | i'm not expert at all, so... | 13:01 |
| dmsimard | I see it in both actually | 13:01 |
| *** kgiusti has joined #rdo | 13:01 | |
| dmsimard | https://github.com/openshift/openshift-ansible/blob/94a75431044d6124dcb22659cbecbb09c3cb3a38/roles/openshift_node/templates/node.yaml.v1.j2#L18 | 13:01 |
| dmsimard | https://github.com/openshift/openshift-ansible/blob/b74b78f37e01e1837843f236b51d3676174afdef/roles/openshift_master/defaults/main.yml#L90 | 13:02 |
| amoralej | let's change it in both then :) | 13:02 |
| dmsimard | so "kubelet_args" takes basically what is in https://github.com/rdo-infra/rdo-container-registry/blob/master/hosts#L12 | 13:02 |
| dmsimard | "openshift_node_labels" | 13:02 |
| amoralej | dmsimard, it's back | 13:03 |
| amoralej | docker, i mean | 13:04 |
| amoralej | it's responding and registry container is up again | 13:04 |
| dmsimard | amoralej: ohhhh https://github.com/openshift/openshift-ansible/commit/660bafe9cd0adbfa965bb3f8f31d6c043a0dc5d6#diff-eede3d17228609d613e2339187aa95d5 | 13:04 |
| dmsimard | so in 3.7 it's disabled by default | 13:04 |
| dmsimard | before that it was enabled by default | 13:05 |
| amoralej | but, those metrics are referring to the same | 13:05 |
| amoralej | there is some metrics staff in openshift | 13:05 |
| *** jcoufal has quit IRC | 13:05 | |
| dmsimard | it's two different things | 13:05 |
| amoralej | that deploys containers to store metrics | 13:05 |
| dmsimard | one role installs the metrics, the other roles configure the master/node to use them | 13:06 |
| dmsimard | at least afaict | 13:06 |
| dmsimard | This is the version we have deployed https://github.com/dmsimard/openshift-ansible/tree/rdo | 13:06 |
| *** jmelvin has joined #rdo | 13:06 | |
| dmsimard | https://github.com/dmsimard/openshift-ansible/blob/rdo/roles/openshift_metrics/defaults/main.yaml#L3 | 13:06 |
| dmsimard | installed by default | 13:06 |
| *** hrw has quit IRC | 13:07 | |
| *** jcoufal has joined #rdo | 13:08 | |
| *** eharney has joined #rdo | 13:08 | |
| *** hrw has joined #rdo | 13:08 | |
| amoralej | dmsimard, i'm testing to push kolla images again to registry.r.o | 13:09 |
| dmsimard | fyi, browsing logs, this is one of the docker push performance discussions I had noted back when I was looking into it https://github.com/moby/moby/issues/28732 | 13:09 |
| amoralej | now that du is not longer running | 13:09 |
| amoralej | why kolla doesn't timpestamp logs? | 13:09 |
| amoralej | it's ugly | 13:09 |
| dmsimard | and then there are other linked issues like https://github.com/moby/moby/issues/10181 and https://github.com/moby/moby/pull/17674 | 13:10 |
| dmsimard | basically the compression/decompression performance is terrible | 13:10 |
| dmsimard | but it doesn't explain why your registry would be so much faster | 13:11 |
| amoralej | yeah | 13:11 |
| *** udesale has joined #rdo | 13:11 | |
| *** coolsvap has quit IRC | 13:11 | |
| amoralej | dmsimard, let's merge the reverts for the new periodic job to do the right thing | 13:12 |
| amoralej | https://review.rdoproject.org/r/10395 https://review.rdoproject.org/r/10396 https://review.rdoproject.org/r/10397 | 13:12 |
| rdogerrit | David Moreau Simard proposed config master: Revert "temporarily disable containers build" https://review.rdoproject.org/r/10396 | 13:13 |
| *** hrw has quit IRC | 13:13 | |
| *** jmelvin_ has joined #rdo | 13:13 | |
| rdogerrit | Merged config master: Revert "Delay the execution of openstack-periodic" https://review.rdoproject.org/r/10395 | 13:13 |
| dmsimard | hmm why aren't job dequeuing immediately | 13:14 |
| *** jmelvin has quit IRC | 13:15 | |
| dmsimard | looks like a zuul merger bomb | 13:15 |
| dmsimard | all 3 mergers running | 13:16 |
| dmsimard | brb | 13:17 |
| *** dustins has joined #rdo | 13:17 | |
| *** hrw has joined #rdo | 13:18 | |
| *** mosulica has joined #rdo | 13:19 | |
| amoralej | apevec, dmsimard, still failing to push after removing the du | 13:20 |
| amoralej | ini fact the registry becomes down | 13:20 |
| amoralej | i try skopeo and it fails | 13:21 |
| *** jcoufal has quit IRC | 13:22 | |
| dmsimard | That's not normal... | 13:22 |
| *** jcoufal has joined #rdo | 13:22 | |
| dmsimard | I can look in 10 minutes.. | 13:23 |
| *** lucasagomes is now known as lucas-rellocate | 13:23 | |
| *** dneary has joined #rdo | 13:26 | |
| *** trown|outtypewww is now known as trown | 13:28 | |
| amoralej | dmsimard, taking a look to https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build/buildTimeTrend | 13:29 |
| amoralej | it seems job times increased on Oct 22 | 13:30 |
| *** udesale has quit IRC | 13:30 | |
| apevec | and keep growing | 13:30 |
| amoralej | was that the last maintenance? | 13:30 |
| dmsimard | 22 oct sounds like maintenance yeah | 13:34 |
| * dmsimard checks emails | 13:34 | |
| dmsimard | it was oct 25 actually | 13:35 |
| *** pkovar has joined #rdo | 13:35 | |
| dmsimard | I really want to just upgrade to 3.7.. but it's not released yet | 13:38 |
| number80 | oct 25 or dec 21, it's confusing to me (easy one but hey, it's friday!) | 13:38 |
| apevec | dmsimard, there is RC | 13:38 |
| dmsimard | apevec: yeah but it wasn't built until this week, I have been testing with a release from oct 4 | 13:38 |
| dmsimard | apevec: http://cbs.centos.org/koji/buildinfo?buildID=20557 wednesday (we posted on centos-devel to push a bit..) | 13:39 |
| dmsimard | I'll rebase on that.. | 13:39 |
| apevec | yeah I saw | 13:39 |
| apevec | peer pressure works :) | 13:39 |
| dmsimard | there's something wrong with the zuul mergers for now | 13:39 |
| dmsimard | it seems review.rdo is stuck doing a rebase bomb and zm01 and zm02 are idle | 13:40 |
| dmsimard | but they've been active before.. | 13:40 |
| *** eck`gone is now known as eck` | 13:40 | |
| apevec | dmsimard, btw in Kibana last GitCommandError was Nov 2 1800 | 13:40 |
| apevec | nothing in last 12h | 13:40 |
| dmsimard | apevec: nice, we should check for slave disconnection too | 13:40 |
| apevec | what would be the search keyword ? | 13:41 |
| *** jprovazn has quit IRC | 13:41 | |
| dmsimard | hmmmmm | 13:42 |
| dmsimard | java.io.IOException ? | 13:42 |
| dmsimard | actually "Slave went offline during the build" | 13:43 |
| dmsimard | last merge occured ~15 minutes ago... | 13:44 |
| *** jcoufal_ has joined #rdo | 13:45 | |
| *** jcoufal has quit IRC | 13:45 | |
| apevec | hmm, last slave offline was today 11:54 | 13:45 |
| dmsimard | so we're still htiting that issue then | 13:46 |
| apevec | https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-master-nv/1129/consoleText | 13:46 |
| weshay | apevec, dmsimard any idea why the rdo ci.centos are being aborted | 13:46 |
| apevec | weshay, like ^ | 13:46 |
| weshay | ah | 13:46 |
| apevec | looks like poor Jenkins cannot keep up | 13:46 |
| dmsimard | weshay: if it was last night there was a maintenance I sent an email | 13:46 |
| weshay | hrm. k | 13:47 |
| weshay | FYI.. they are no longer hooked into review.openstack | 13:47 |
| weshay | https://bugs.centos.org/view.php?id=14083 | 13:47 |
| apevec | so if it still happens after review.rdo resizing, we need to decrease nodepool size? | 13:48 |
| apevec | weshay, that's the ci.c.o jenkins? | 13:48 |
| dmsimard | weshay: you're mixing up ci.centos and review.rdo | 13:48 |
| apevec | we are talking about review.rdo Jenkins | 13:48 |
| apevec | what jobs do we still have in ci.c.o ? | 13:48 |
| dmsimard | amoralej, jpena|lunch: I increased merger verbosity on review.rdo and zm01/zm02 to understand what's going on | 13:48 |
| amoralej | ok | 13:49 |
| *** amoralej is now known as amoralej|lunch | 13:49 | |
| weshay | apevec, no.. review.openstack | 13:49 |
| apevec | weshay, didn't you move all oooq jobs to openstack-check in review.rdo ? | 13:49 |
| weshay | we trigger jobs on changes in review.openstack | 13:50 |
| apevec | weshay, but you linked to bugs.centos ? | 13:50 |
| weshay | so we trigger tq/tqe check jobs to test libvirt installs that run on ci.centos | 13:51 |
| apevec | so you can just drop https://ci.centos.org/view/rdo/view/tripleo-gate/ ? | 13:51 |
| weshay | no | 13:51 |
| weshay | I can't | 13:51 |
| *** jcoufal has joined #rdo | 13:51 | |
| weshay | well.. maybe I could | 13:51 |
| *** jcoufal_ has quit IRC | 13:51 | |
| weshay | but we'd rather not | 13:51 |
| apevec | what additional coverage does it have? | 13:51 |
| apevec | wrt review.rdo jobs | 13:51 |
| weshay | apevec, that checks the libvirt setup on a plain old virthost | 13:52 |
| apevec | a-ha | 13:52 |
| apevec | so we could do that with nested kvm in rdocloud? | 13:52 |
| weshay | we don't need a lot of coverage, but making sure that works for user experience is critical | 13:52 |
| *** jkilpatr has quit IRC | 13:52 | |
| weshay | apevec, ya.. I'm open to that | 13:52 |
| apevec | just need big enough node | 13:52 |
| *** emalikov has quit IRC | 13:52 | |
| weshay | heh | 13:52 |
| ykarel | apevec, 16GB + 16GB swap won't work? | 13:53 |
| apevec | dmsimard, ^ how can we have different nodepool node sizes for some jobs? | 13:53 |
| ykarel | would be slow | 13:53 |
| apevec | ykarel, dunno | 13:53 |
| apevec | weshay, what is the node size in cico for those oooq jobs? | 13:53 |
| dmsimard | apevec: it's a bit awkward to manage without nodepool v3 | 13:54 |
| dmsimard | apevec: 32gb ram | 13:54 |
| dmsimard | I don't think we'll get VMs of that size | 13:54 |
| dmsimard | realistically | 13:54 |
| apevec | now we have std size nodepool nodes 8G 8cpu | 13:54 |
| apevec | alright, so we need both zuul v3 in review.rdo and moar resources in CI tennant ? | 13:55 |
| *** chlong has joined #rdo | 13:55 | |
| dmsimard | apevec: the feature I'm talking about is not even merged in v3 yet, maybe after summit | 13:55 |
| dmsimard | it's this stack of patches https://review.openstack.org/#/c/504282/ | 13:56 |
| *** pkovar has quit IRC | 13:56 | |
| dmsimard | tl;dr, instead of setting max-servers, you set quotas and nodepool just maximizes quota usage | 13:56 |
| dmsimard | so it's easy to use different flavors | 13:56 |
| *** ykarel is now known as ykarel|away | 13:58 | |
| dmsimard | I don't understand what's going on with zuul... asked in #zuul | 13:58 |
| dmsimard | oh wait the two mergers JUST started picking up jobs | 13:58 |
| dmsimard | wth | 13:59 |
| *** egonzalez has quit IRC | 13:59 | |
| *** jpena|lunch is now known as jpena | 14:01 | |
| apevec | ah @bmw.de - runs on Zuul ? | 14:02 |
| mnaser | ^ thats cool | 14:03 |
| dmsimard | yeah, a lot | 14:03 |
| dmsimard | tobiash is on zuul/nodepool almost full time | 14:03 |
| apevec | so it's still feature/zuulv3 branch, not merged to master yet? | 14:03 |
| dmsimard | v3 hasn't merged back to master yet | 14:03 |
| dmsimard | mnaser: what's cool? :D | 14:04 |
| mnaser | bmw using zuul | 14:04 |
| * dmsimard nods | 14:04 | |
| dmsimard | apevec: there's a summary of the roadmap before release here http://lists.openstack.org/pipermail/openstack-infra/2017-November/005657.html | 14:04 |
| *** jcoufal_ has joined #rdo | 14:05 | |
| mnaser | also, pike promotion is something i believe we need for a puppet-ceilometer patch | 14:05 |
| mnaser | im not sure why it didnt happen because it doesnt seem like its stuck | 14:05 |
| *** ykarel|away has quit IRC | 14:05 | |
| *** jcoufal has quit IRC | 14:06 | |
| mnaser | https://review.rdoproject.org/r/#/c/10380/ <=> https://review.openstack.org/#/c/515911/ | 14:06 |
| apevec | mnaser, poi stable is using cloud sig stable repos? | 14:07 |
| dmsimard | it is | 14:07 |
| apevec | so we need to rebuid with Release: bump | 14:07 |
| mnaser | oh so it doesnt use stable/pike builds but the tagged packages? | 14:07 |
| mnaser | things i should know :( | 14:07 |
| dmsimard | mnaser: your patch hasn't made it to RDO stable repos yet | 14:08 |
| apevec | it uses rpms published on mirror.c.o | 14:08 |
| apevec | i.e. CBS Koji builds | 14:08 |
| *** jcoufal has joined #rdo | 14:08 | |
| apevec | which are triggered when you change NVR in distgit | 14:08 |
| apevec | your patch made it to rdo pike trunk | 14:08 |
| mnaser | ok so in the future should we bump NVR in the same patch or do 2 patches | 14:08 |
| apevec | amended backport in the same patch is fine if it is single change | 14:09 |
| apevec | now just send followup with Release + changelog | 14:09 |
| apevec | to pike-rdo | 14:09 |
| *** jcoufal_ has quit IRC | 14:10 | |
| *** jbrooks has quit IRC | 14:11 | |
| apevec | previous bump was an example of an amended backport from rpm-master https://review.rdoproject.org/r/#/c/9938/2/openstack-ceilometer.spec | 14:11 |
| mnaser | ok gotcha | 14:11 |
| *** jcoufal_ has joined #rdo | 14:12 | |
| *** jcoufal has quit IRC | 14:12 | |
| *** jkilpatr has joined #rdo | 14:12 | |
| *** bnemec is now known as beekneemech | 14:14 | |
| rdogerrit | Mohammed Naser created openstack/ceilometer-distgit pike-rdo: Bump Ceilometer release https://review.rdoproject.org/r/10400 | 14:18 |
| mnaser | ^ i think i did that right | 14:18 |
| * number80 looks | 14:20 | |
| number80 | Yep | 14:20 |
| number80 | I prefer splitting NVR bump in RDO packages distgit as it allows easier cherry-picks | 14:21 |
| *** jatanmalde has joined #rdo | 14:22 | |
| apevec | number80, if it is just a single change, squashing w/ backport is fine | 14:24 |
| number80 | apevec: yeah, just a preference | 14:24 |
| *** lucas-rellocate is now known as lucasagomes | 14:25 | |
| *** jbrooks has joined #rdo | 14:26 | |
| *** jprovazn has joined #rdo | 14:26 | |
| rdogerrit | Javier Peña proposed rdo-infra/puppet-dlrn master: Additional configuration to support SELinux=Enforcing https://review.rdoproject.org/r/7369 | 14:31 |
| number80 | Please help with reviewing => https://review.rdoproject.org/r/#/q/project:rdo-infra/puppet-dlrn+status:open | 14:32 |
| number80 | Some of them have already 2* +2 but I don't have +W :) | 14:33 |
| *** hrw has quit IRC | 14:34 | |
| rdogerrit | Javier Peña proposed rdo-infra/puppet-dlrn master: Add symlink from current-passed-ci to current-tripleo-rdo https://review.rdoproject.org/r/9158 | 14:34 |
| *** dneary has quit IRC | 14:35 | |
| number80 | dmsimard: is this change still relevant? https://review.rdoproject.org/r/#/c/7444/ | 14:39 |
| number80 | (no rush, either update or abandon it when you'll get time) | 14:40 |
| *** cshastri has quit IRC | 14:43 | |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: WIP: Upgrade registry to OpenShift 3.7 https://review.rdoproject.org/r/10259 | 14:48 |
| *** amoralej|lunch is now known as amoralej | 14:49 | |
| dmsimard | jpena: zuul merger still not catching up... | 14:57 |
| *** suresh12 has joined #rdo | 14:57 | |
| dmsimard | all 3 are busy merging things | 14:57 |
| dmsimard | they were stuck for a while but they're going now, almost 2hr backlogged now | 14:58 |
| jpena | dmsimard: I think it's nodepool | 14:59 |
| jpena | it's stuck for 2 hr | 14:59 |
| jpena | let me restart it | 14:59 |
| dmsimard | uh wait | 14:59 |
| *** hrw has joined #rdo | 15:01 | |
| *** jkilpatr has quit IRC | 15:01 | |
| rdogerrit | Merged rdo-infra/ci-config master: Revert "tripleo: fix the pin" https://review.rdoproject.org/r/10397 | 15:03 |
| *** suresh12 has quit IRC | 15:04 | |
| dmsimard | jpena: zm01 https://review.rdoproject.org/jenkins/job/config-check/4833/consoleText | 15:05 |
| rdogerrit | Merged rdo-infra/puppet-dlrn master: Sync symlinks from public to internal instance before purging https://review.rdoproject.org/r/10227 | 15:05 |
| dmsimard | jpena: zm02: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-master-nv/1136/consoleText | 15:05 |
| jpena | dmsimard: nice, then | 15:06 |
| jpena | we can now check if all errors happen when cloning from review.rdo, that might be something | 15:06 |
| dmsimard | jpena: apevec looked earlier and it hasn't occured since yesterday | 15:07 |
| dmsimard | so cleaning up the zuul merger cached repos might have done the trick | 15:08 |
| dmsimard | ¯\_(ツ)_/¯ | 15:08 |
| jpena | dmsimard: oh, nice | 15:08 |
| jpena | ¯\_(ツ)_/¯ | 15:08 |
| dmsimard | I got the idea because there was some git repo corruption in upstream infra | 15:09 |
| dmsimard | they ran a "git fsck" on all repos on all merged and deleted a few, it was due to an unclean restart | 15:09 |
| dmsimard | I didn't want to bother with fsck'ing everything though | 15:09 |
| *** sreichar has quit IRC | 15:12 | |
| *** Son_Goku has quit IRC | 15:13 | |
| *** spr has quit IRC | 15:14 | |
| *** jkilpatr has joined #rdo | 15:14 | |
| *** aditya_r has joined #rdo | 15:16 | |
| amoralej | panda|ruck, dmsimard i'm aborting all periodic jobs for master which started still with the pin applied | 15:20 |
| dmsimard | amoralej: make sure it's a soft abort | 15:20 |
| *** janki has quit IRC | 15:20 | |
| rdogerrit | Merged config master: Revert "temporarily disable containers build" https://review.rdoproject.org/r/10396 | 15:20 |
| amoralej | dmsimard, yes | 15:20 |
| *** janki has joined #rdo | 15:21 | |
| weshay | amoralej, ah thanks | 15:21 |
| panda|ruck | amoralej: thanks | 15:21 |
| *** janki has quit IRC | 15:22 | |
| *** karthiks has quit IRC | 15:22 | |
| *** suresh12 has joined #rdo | 15:23 | |
| *** jprovazn has quit IRC | 15:26 | |
| rdogerrit | Merged openstack/ceilometer-distgit pike-rdo: Bump Ceilometer release https://review.rdoproject.org/r/10400 | 15:27 |
| *** jaosorior has quit IRC | 15:27 | |
| rdogerrit | Merged rdo-infra/puppet-dlrn master: Add symlink from current-passed-ci to current-tripleo-rdo https://review.rdoproject.org/r/9158 | 15:30 |
| *** jackNemrod has quit IRC | 15:33 | |
| *** links has joined #rdo | 15:35 | |
| *** ykarel|away has joined #rdo | 15:39 | |
| *** suresh12 has quit IRC | 15:40 | |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: WIP: Upgrade registry to OpenShift 3.7 https://review.rdoproject.org/r/10259 | 15:41 |
| dmsimard | tripleo running at capacity.. 2017-11-03 15:45:01,211 WARNING nodepool.NodePool: Provider rdo-cloud-tripleo over-allocated: max-servers 100 but counted 101 | 15:45 |
| *** spectr has quit IRC | 15:49 | |
| *** jcoufal has joined #rdo | 15:50 | |
| *** spr has joined #rdo | 15:51 | |
| *** jcoufal_ has quit IRC | 15:53 | |
| *** jatanmalde has quit IRC | 15:55 | |
| *** jatanmalde has joined #rdo | 15:56 | |
| openstackgerrit | Javier Peña proposed openstack/packstack master: [WIP] Multinode CI job for Packstack https://review.openstack.org/517644 | 15:56 |
| openstackgerrit | Javier Peña proposed openstack/packstack master: [WIP] Multinode CI job for Packstack https://review.openstack.org/517644 | 16:02 |
| amoralej | panda|ruck, how often does the promoter script run? | 16:05 |
| amoralej | i think it should have promoted current-tripleo-rdo for master | 16:05 |
| panda|ruck | amoralej: I disabled it | 16:06 |
| panda|ruck | amoralej: did all the revert merged ? | 16:06 |
| amoralej | yes | 16:06 |
| rdogerrit | Alfredo Moralejo proposed rdo-infra/puppet-dlrn master: run-purge: do not remove symlinks created only on remote https://review.rdoproject.org/r/8077 | 16:07 |
| *** smeyer has quit IRC | 16:09 | |
| *** kellyp has joined #rdo | 16:11 | |
| panda|ruck | amoralej: ok, reneabling | 16:11 |
| amoralej | thanks | 16:11 |
| panda|ruck | amoralej: done | 16:11 |
| amoralej | it runs as a cron job? | 16:11 |
| rdogerrit | Merged rdo-infra/puppet-dlrn master: run-purge: do not remove symlinks created only on remote https://review.rdoproject.org/r/8077 | 16:11 |
| panda|ruck | yes ... next run is at :21 | 16:12 |
| panda|ruck | in 9 minutes | 16:12 |
| amoralej | ok | 16:12 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: WIP: Upgrade registry to OpenShift 3.7 https://review.rdoproject.org/r/10259 | 16:12 |
| *** mosulica has quit IRC | 16:14 | |
| *** ykarel|away has quit IRC | 16:16 | |
| *** dneary has joined #rdo | 16:19 | |
| *** jcoufal has quit IRC | 16:21 | |
| *** vnogin has quit IRC | 16:21 | |
| *** aditya_ra has joined #rdo | 16:22 | |
| *** jcoufal has joined #rdo | 16:23 | |
| *** aditya_r has quit IRC | 16:23 | |
| apevec | amoralej, https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/216/ didn't finish yet, so current-tripleo-rdo wouldn't be promoted yet? | 16:24 |
| amoralej | apevec, it should | 16:24 |
| amoralej | promoter script must be running right now | 16:24 |
| amoralej | let's wait a bit | 16:25 |
| apevec | ah I see poi from 216 + other jobs for 214 reported success | 16:25 |
| amoralej | yes | 16:25 |
| apevec | amoralej, dmsimard - back to registry issues: what is the status now? | 16:28 |
| amoralej | i'm monitoring current build job for pike | 16:28 |
| apevec | du is -x ? | 16:28 |
| amoralej | yes | 16:28 |
| *** jatanmalde has quit IRC | 16:29 | |
| apevec | parameter changed, origin restarted? | 16:29 |
| amoralej | nop | 16:29 |
| apevec | and mid-term, plan is to upgrade to origin 3.7 rc ? | 16:29 |
| amoralej | only du -x | 16:29 |
| apevec | ok | 16:29 |
| amoralej | i'm seeing the same behavior | 16:29 |
| apevec | 3.7 disables du by default | 16:29 |
| apevec | heh | 16:29 |
| amoralej | pushing is taking ages | 16:29 |
| amoralej | well, maybe a bit better | 16:30 |
| amoralej | in 50 minutes it has built all containers | 16:30 |
| amoralej | and pushed 50 | 16:30 |
| amoralej | out of 120 | 16:30 |
| amoralej | no failures so far | 16:30 |
| apevec | ok, so that is better | 16:31 |
| amoralej | maybe, but not much | 16:31 |
| dmsimard | amoralej, apevec: 3.5 enables logging and metrics by default, 3.7 disables them by default | 16:32 |
| dmsimard | ¯\_(ツ)_/¯ | 16:32 |
| dmsimard | discussed with internal openshift ops, they disable metrics as well | 16:32 |
| apevec | it was "oops" effect :) | 16:32 |
| rdogerrit | Merged rdoinfo master: Pin branchless projects for stable releases https://review.rdoproject.org/r/10328 | 16:32 |
| *** sbonazzo has quit IRC | 16:32 | |
| amoralej | IO seems pretty reasonable now, i'd say | 16:33 |
| amoralej | i'm lost | 16:33 |
| apevec | I've installed iotop on registry machine, | 16:33 |
| apevec | and doesn't look that bad | 16:33 |
| *** dneary has quit IRC | 16:33 | |
| amoralej | yeah | 16:33 |
| apevec | I would add swap on the box | 16:34 |
| apevec | since there was OOM | 16:34 |
| apevec | openshift process is 6G | 16:35 |
| amoralej | openshift master process is taking more that 6GB RSS | 16:35 |
| apevec | and machine has 7.6G total | 16:35 |
| amoralej | i wonder if this is normal | 16:35 |
| jpena | yes, there's not a lot of available RAM | 16:35 |
| jpena | let's add a swap volume, it's easy | 16:35 |
| apevec | except, not on the vda which already taxed by the registry storage :) | 16:36 |
| *** hewbrocca is now known as hewbrocca_afk | 16:36 | |
| apevec | I'd add new volume? | 16:36 |
| jpena | I was thinking on creating a new volume, just like we did with review.rdo | 16:36 |
| apevec | +1 | 16:36 |
| amoralej | we don't have updated kernel | 16:37 |
| *** cylopez has quit IRC | 16:37 | |
| amoralej | we are in 7.3 | 16:37 |
| apevec | oh | 16:38 |
| amoralej | nor updated docker | 16:38 |
| apevec | let's add swap + yum update asap | 16:38 |
| apevec | then schedule reboot also asap | 16:38 |
| amoralej | +1 | 16:39 |
| jpena | dmsimard: would you update affect the running operations? | 16:42 |
| jpena | ^^ | 16:42 |
| jpena | s/you/yum | 16:42 |
| *** aufi has quit IRC | 16:43 | |
| dmsimard | hard to say without testing it first | 16:44 |
| dmsimard | for RAM usage, it's because we're due to prune images | 16:44 |
| apevec | origin is all in containers so it should not? | 16:44 |
| jpena | swap is enabled | 16:44 |
| jpena | btw | 16:44 |
| dmsimard | it's a bit more than that | 16:45 |
| dmsimard | it'd update the origin packages, but you would also need to spin up new pods with the updated pod images | 16:45 |
| dmsimard | there's a set of pod images (i.e, docker-registry) for each origin release | 16:45 |
| apevec | dmsimard, I mean just yum update for OS packages | 16:45 |
| apevec | not origini | 16:45 |
| dmsimard | apevec: origin is a package :) | 16:46 |
| apevec | yeah, but we would update it right? | 16:46 |
| dmsimard | I don't think it should be a problem but it's hard to tell | 16:46 |
| apevec | err | 16:46 |
| apevec | would NOT | 16:46 |
| amoralej | could we just disable the repo for origin and update OS only? | 16:46 |
| apevec | enabled repos are only base, updates, extras | 16:47 |
| dmsimard | I have passing 3.7 integration tests: https://review.rdoproject.org/r/#/c/10259/ do we want to just try and deploy a "dev" registry to test it out ? | 16:47 |
| apevec | ah no, it's alos openstack and ceph | 16:47 |
| dmsimard | apevec: that's from opstools | 16:47 |
| dmsimard | apevec: I discussed that issue with them a while back.. I think it was only for like rabbitmq or something | 16:48 |
| apevec | dmsimard, you'd try 3.7 in a new instance, not update on the same machine? | 16:48 |
| * dmsimard looks at emails | 16:48 | |
| dmsimard | apevec: I would, yeah, it's an upgrade from 3.5 to 3.7.. lots of things changed and stuff might be hard to "fix" like metrics, logging, etc. | 16:48 |
| apevec | dmsimard, might be, there are other packages in openstack-ocata which update base packages... | 16:49 |
| dmsimard | apevec: we could run "registry-dev" and try it out. | 16:49 |
| apevec | few python-* and mariadb e.g. | 16:49 |
| apevec | dmsimard, ack, let's | 16:49 |
| amoralej | dmsimard, sounds good | 16:49 |
| apevec | but in the meantime let's update OS and reboot current one | 16:50 |
| dmsimard | mrunge: did you ever get around to cross-tag the necessary things from RDO repos to opstools repos ? | 16:50 |
| apevec | I'd disable all but updates and extras repos | 16:50 |
| dmsimard | mrunge: nevermind, let me revive the thread | 16:50 |
| apevec | amoralej, is containers push still running? | 16:51 |
| amoralej | yes | 16:51 |
| amoralej | pushed 67 out of 119 | 16:51 |
| dmsimard | I'll run a prune this weekend, see if that helps. | 16:51 |
| amoralej | dmsimard, how long takes a prune, i guess it's a heavy process, right? | 16:52 |
| dmsimard | with the volume of images we have and the fact that we're not pruning on a regular basis, it's a few hours yes | 16:53 |
| dmsimard | enough to make me fall asleep looking at the prune logs in Denver :) | 16:53 |
| *** tesseract has quit IRC | 16:54 | |
| amoralej | :) | 16:54 |
| *** links has quit IRC | 16:55 | |
| apevec | long enough to run it in screen(1) | 16:55 |
| dmsimard | I'm a screen guy, not a tmux guy :( | 16:55 |
| dmsimard | sorry | 16:55 |
| apevec | tmate is popular I heard | 16:56 |
| apevec | so I'll run: # yum-config-manager --disable centos-openshift-origin --disable centos-ceph-jewel --disable centos-openstack-ocata --disable centos-opstools-release --disable sensu | 16:56 |
| dmsimard | er | 16:56 |
| dmsimard | yum update --disablerepo="*" --enablerepo=centos-base,etc ? | 16:56 |
| *** jmelvin_ has quit IRC | 16:57 | |
| *** trown is now known as trown|lunch | 16:59 | |
| apevec | or that | 16:59 |
| amoralej | apevec, promotion of current-tripleo-rdo is done | 17:00 |
| apevec | but I'd prefer to freeze the box until we migrate to 3.7 | 17:00 |
| apevec | amoralej, \o/ | 17:00 |
| apevec | not green, but yellow is nice too | 17:00 |
| *** fragatina has joined #rdo | 17:00 | |
| apevec | compared to 55d ref :) | 17:00 |
| *** fragatina has quit IRC | 17:01 | |
| *** fragatina has joined #rdo | 17:01 | |
| openstackgerrit | Javier Peña proposed openstack/packstack master: [WIP] Multinode CI job for Packstack https://review.openstack.org/517644 | 17:01 |
| *** dneary has joined #rdo | 17:02 | |
| *** jcoufal has quit IRC | 17:05 | |
| *** brault has quit IRC | 17:07 | |
| *** brault has joined #rdo | 17:08 | |
| apevec | w/ only base/extras/updates Total download size: 342 M | 17:08 |
| apevec | it includes docker update | 17:08 |
| apevec | so we should run with origin stopped ? | 17:09 |
| *** suresh12 has joined #rdo | 17:09 | |
| *** jtomasek has quit IRC | 17:12 | |
| *** brault has quit IRC | 17:12 | |
| *** suresh12 has quit IRC | 17:13 | |
| apevec | amoralej, where do you watch containers push progress? | 17:15 |
| amoralej | apevec, i have a ssh session | 17:15 |
| amoralej | and look in the logs | 17:15 |
| amoralej | 88/119 right now | 17:15 |
| *** sbulage has quit IRC | 17:15 | |
| *** jprovazn has joined #rdo | 17:16 | |
| *** jkilpatr has quit IRC | 17:16 | |
| apevec | hmm yeah, https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-pike-containers-build/258/console doesn't show it :/ | 17:17 |
| amoralej | yeah | 17:18 |
| *** paragan has quit IRC | 17:19 | |
| *** suresh12 has joined #rdo | 17:19 | |
| *** aditya_ra has quit IRC | 17:19 | |
| *** mcornea has quit IRC | 17:19 | |
| *** suresh12 has quit IRC | 17:20 | |
| *** suresh12 has joined #rdo | 17:20 | |
| *** jmlowe has quit IRC | 17:20 | |
| *** newmember has quit IRC | 17:23 | |
| *** ykarel|away has joined #rdo | 17:24 | |
| *** jtomasek has joined #rdo | 17:25 | |
| *** lucasagomes is now known as lucas-afk | 17:27 | |
| *** jkilpatr has joined #rdo | 17:28 | |
| *** catintheroof has quit IRC | 17:31 | |
| *** mary_grace has joined #rdo | 17:32 | |
| *** catintheroof has joined #rdo | 17:33 | |
| *** chlong_ has joined #rdo | 17:34 | |
| *** hseipp has quit IRC | 17:34 | |
| rdogerrit | Javier Peña proposed rdo-infra/puppet-dlrn master: Add support for a public rsync option for workers https://review.rdoproject.org/r/6112 | 17:37 |
| rdogerrit | Javier Peña proposed rdo-infra/puppet-dlrn master: Set the right SQLAlchemy string in workers' alembic.ini https://review.rdoproject.org/r/9614 | 17:37 |
| rdogerrit | Javier Peña proposed rdo-infra/puppet-dlrn master: Support the release_numbering parameter in projects.ini https://review.rdoproject.org/r/10161 | 17:37 |
| *** dtantsur is now known as dtantsur|afk | 17:38 | |
| jpena | number80: any last comment before I merge https://softwarefactory-project.io/r/9858 ? | 17:38 |
| * number80 looks | 17:39 | |
| number80 | good for me | 17:39 |
| jpena | thx! | 17:39 |
| *** _bsurfer has quit IRC | 17:43 | |
| *** noslzzp has joined #rdo | 17:43 | |
| *** jprovazn has quit IRC | 17:44 | |
| *** vnogin has joined #rdo | 17:44 | |
| *** catintheroof has quit IRC | 17:45 | |
| *** vnogin has quit IRC | 17:45 | |
| *** pblaho has quit IRC | 17:45 | |
| *** vnogin has joined #rdo | 17:45 | |
| sfbender | Fabien Boucher created DLRN master: Some small fixes in doc/ and command lines help https://softwarefactory-project.io/r/10083 | 17:46 |
| *** ykarel|away has quit IRC | 17:49 | |
| *** vnogin has quit IRC | 17:52 | |
| *** vnogin has joined #rdo | 17:53 | |
| *** catintheroof has joined #rdo | 17:53 | |
| sfbender | Merged DLRN master: Support new method for package Release version https://softwarefactory-project.io/r/9858 | 17:53 |
| *** dsneddon has joined #rdo | 17:56 | |
| *** dustins has quit IRC | 17:57 | |
| rdogerrit | Javier Peña proposed rdo-infra/puppet-dlrn master: Add support for a public rsync option for workers https://review.rdoproject.org/r/6112 | 17:58 |
| *** jpich has quit IRC | 17:58 | |
| *** vnogin has quit IRC | 18:01 | |
| *** derekh has quit IRC | 18:04 | |
| apevec | amoralej, what's the counter now? On console it started tagging | 18:06 |
| amoralej | apevec, yes | 18:06 |
| amoralej | all containers are pushed | 18:06 |
| amoralej | it's almost finished | 18:06 |
| apevec | build and push was 15:38 - 17:50 ... | 18:07 |
| openstackgerrit | Javier Peña proposed openstack/packstack master: [WIP] Multinode CI job for Packstack https://review.openstack.org/517644 | 18:07 |
| amoralej | yes | 18:07 |
| apevec | and SUCCESS | 18:07 |
| amoralej | apevec, also note that usually container buils is done in parallel for master and pike | 18:07 |
| amoralej | this run is only pike | 18:07 |
| apevec | ah so that would be additional load | 18:08 |
| *** jmelvin_ has joined #rdo | 18:08 | |
| amoralej | yeah | 18:08 |
| amoralej | i hope the images prune will help | 18:08 |
| *** noslzzp has quit IRC | 18:09 | |
| *** trown|lunch is now known as trown | 18:09 | |
| apevec | still, why is this not web-scale :) | 18:09 |
| amoralej | :) | 18:09 |
| apevec | which jobs will now use pike containers? | 18:09 |
| amoralej | i expected container would fix ALL problems in the world by free | 18:09 |
| amoralej | ... | 18:09 |
| amoralej | the four in the queue | 18:10 |
| * jpena lols hard | 18:10 | |
| amoralej | 016-019 | 18:10 |
| dmsimard | I'm wrapping up a patch for 3.7 | 18:10 |
| dmsimard | then we can try it out | 18:10 |
| apevec | we'd need new jobs so we don't mess w/ promotion ones | 18:11 |
| *** catintheroof has quit IRC | 18:11 | |
| dmsimard | sure | 18:11 |
| apevec | also to have right comparison, we'd have to export all containers we have now | 18:12 |
| apevec | is there easy oc client command to copy between registries? | 18:12 |
| amoralej | apevec, ~ 500GB of images... | 18:13 |
| apevec | yeah, stress test :) | 18:13 |
| amoralej | :) | 18:13 |
| dmsimard | it's like 30k layers | 18:13 |
| apevec | so 016-019 are testing w/ containers | 18:13 |
| dmsimard | last I checked | 18:13 |
| amoralej | apevec, yes | 18:14 |
| dmsimard | I have this BZ I need to follow up on for RAM usage due to high amount of images/layers https://bugzilla.redhat.com/show_bug.cgi?id=1489501 | 18:14 |
| openstack | bugzilla.redhat.com bug 1489501 in Master "Memory usage for origin-master is abnormally high" [Low,New] - Assigned to mfojtik | 18:14 |
| apevec | "load all images into a cache" ! | 18:15 |
| *** mary_grace has quit IRC | 18:15 | |
| dmsimard | the metadata of the images :p | 18:16 |
| amoralej | souds as web scale, yeah | 18:17 |
| *** catintheroof has joined #rdo | 18:17 | |
| *** vnogin has joined #rdo | 18:18 | |
| *** dneary has quit IRC | 18:20 | |
| *** dneary has joined #rdo | 18:21 | |
| *** vnogin has quit IRC | 18:23 | |
| *** dneary has quit IRC | 18:26 | |
| jpena | have a nice weekend everyone! | 18:26 |
| *** dneary has joined #rdo | 18:26 | |
| *** jpena is now known as jpena|off | 18:27 | |
| *** dneary has quit IRC | 18:27 | |
| *** dneary has joined #rdo | 18:29 | |
| *** dneary has quit IRC | 18:29 | |
| *** weshay is now known as weshay_brb | 18:30 | |
| *** dneary has joined #rdo | 18:30 | |
| *** brault has joined #rdo | 18:30 | |
| *** vnogin has joined #rdo | 18:30 | |
| *** brault has quit IRC | 18:35 | |
| *** vnogin has quit IRC | 18:35 | |
| rdogerrit | rdo-trunk created openstack/ironic-distgit pike-rdo: Update to 9.1.2 https://review.rdoproject.org/r/10401 | 18:37 |
| rdogerrit | rdo-trunk created openstack/cinder-distgit pike-rdo: Update to 11.0.1 https://review.rdoproject.org/r/10402 | 18:39 |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Promote CBS tags update for pike-release https://review.rdoproject.org/r/10379 | 18:40 |
| *** cylopez has joined #rdo | 18:41 | |
| *** cylopez has quit IRC | 18:41 | |
| *** jmlowe has joined #rdo | 18:43 | |
| *** amoralej is now known as amoralej|off | 18:44 | |
| rdogerrit | Merged rdo-infra/puppet-dlrn master: Add rpmdevtools to required packages to install https://review.rdoproject.org/r/10253 | 18:48 |
| *** brault has joined #rdo | 18:49 | |
| *** kellyp has quit IRC | 18:49 | |
| rdogerrit | Alan Pevec proposed rdoinfo master: Promote CBS tags update for pike-testing https://review.rdoproject.org/r/10369 | 18:52 |
| *** brault has quit IRC | 18:53 | |
| rdogerrit | Alan Pevec proposed rdoinfo master: Promote CBS tags update for pike-testing https://review.rdoproject.org/r/10369 | 18:54 |
| rdogerrit | Alan Pevec proposed rdoinfo master: Promote CBS tags update for pike-testing https://review.rdoproject.org/r/10369 | 18:55 |
| *** dustins has joined #rdo | 19:00 | |
| *** brault has joined #rdo | 19:01 | |
| *** brault has quit IRC | 19:05 | |
| apevec | tripleo gate is >23h ... | 19:11 |
| *** suresh12 has quit IRC | 19:15 | |
| *** rwsu has quit IRC | 19:18 | |
| *** suresh12 has joined #rdo | 19:19 | |
| *** weshay_brb is now known as weshay | 19:20 | |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Update RDO registry playbooks to deploy OpenShift 3.7 https://review.rdoproject.org/r/10259 | 19:22 |
| *** brault has joined #rdo | 19:22 | |
| rdogerrit | David Moreau Simard created rdo-infra/rdo-container-registry master: Remove needless variable re-declaration https://review.rdoproject.org/r/10403 | 19:24 |
| dmsimard | amoralej|off: ^ I think we can use hostpath that way | 19:25 |
| *** jrist has quit IRC | 19:25 | |
| *** vnogin has joined #rdo | 19:25 | |
| *** brault has quit IRC | 19:26 | |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Update RDO registry playbooks to deploy OpenShift 3.7 https://review.rdoproject.org/r/10259 | 19:28 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Remove needless variable re-declaration https://review.rdoproject.org/r/10403 | 19:28 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Update RDO registry playbooks to deploy OpenShift 3.7 https://review.rdoproject.org/r/10259 | 19:30 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Remove needless variable re-declaration https://review.rdoproject.org/r/10403 | 19:30 |
| *** pcaruana has quit IRC | 19:32 | |
| *** jrist has joined #rdo | 19:36 | |
| *** jtomasek has quit IRC | 19:38 | |
| apevec | dmsimard, period is done, I'll yum update registry box now | 19:44 |
| apevec | then reboot, how long does it take? | 19:44 |
| apevec | i.e. do we need to announce short outage? | 19:45 |
| *** jrist_ has joined #rdo | 19:49 | |
| *** jrist_ has quit IRC | 19:49 | |
| *** jrist has quit IRC | 19:49 | |
| *** jrist has joined #rdo | 19:50 | |
| *** jrist has quit IRC | 19:50 | |
| *** jrist has joined #rdo | 19:50 | |
| *** panda|ruck is now known as panda|ruck|off | 19:56 | |
| dmsimard | it doesn't take long unless there's an issue :p | 19:56 |
| dmsimard | but wait first | 19:56 |
| dmsimard | ok there's no /.autorelabel :) | 19:56 |
| *** s34n has left #rdo | 19:56 | |
| dmsimard | apevec: I'm not very confident about this, did you get a list of packages that are going to be updated ? | 19:57 |
| *** catintheroof has quit IRC | 20:04 | |
| *** rfolco has quit IRC | 20:06 | |
| *** kgiusti has left #rdo | 20:06 | |
| rdogerrit | rdo-trunk created openstack/tripleo-image-elements-distgit newton-rdo: Update to 5.3.2 https://review.rdoproject.org/r/10404 | 20:08 |
| rdogerrit | rdo-trunk created openstack/tripleo-image-elements-distgit ocata-rdo: Update to 6.1.1 https://review.rdoproject.org/r/10405 | 20:08 |
| rdogerrit | rdo-trunk created openstack/instack-undercloud-distgit newton-rdo: Update to 5.3.3 https://review.rdoproject.org/r/10406 | 20:08 |
| rdogerrit | rdo-trunk created openstack/tripleo-heat-templates-distgit ocata-rdo: Update to 6.2.4 https://review.rdoproject.org/r/10407 | 20:08 |
| EmilienM | YES | 20:08 |
| rdogerrit | rdo-trunk created openstack/instack-undercloud-distgit ocata-rdo: Update to 6.1.2 https://review.rdoproject.org/r/10408 | 20:08 |
| rdogerrit | rdo-trunk created puppet/puppet-tripleo-distgit newton-rdo: Update to 5.6.5 https://review.rdoproject.org/r/10409 | 20:09 |
| rdogerrit | rdo-trunk created puppet/puppet-tripleo-distgit ocata-rdo: Update to 6.5.4 https://review.rdoproject.org/r/10410 | 20:09 |
| rdogerrit | rdo-trunk created openstack/tripleo-common-distgit newton-rdo: Update to 5.4.5 https://review.rdoproject.org/r/10411 | 20:09 |
| rdogerrit | rdo-trunk created openstack/tripleo-heat-templates-distgit newton-rdo: Update to 5.3.4 https://review.rdoproject.org/r/10412 | 20:09 |
| *** suresh12 has quit IRC | 20:10 | |
| *** vnogin has quit IRC | 20:12 | |
| *** rwsu has joined #rdo | 20:14 | |
| *** catintheroof has joined #rdo | 20:15 | |
| *** eck` is now known as eck`gone | 20:17 | |
| *** suresh12 has joined #rdo | 20:20 | |
| *** mary_grace has joined #rdo | 20:21 | |
| apevec | weshay, dmsimard - so looks like yum update killed docker :( | 20:22 |
| weshay | apevec, I was checking ot see if the pike promotion tagged the containers correctly | 20:23 |
| weshay | that may have been bad timing | 20:23 |
| dmsimard | apevec: what are we trying to fix with a yum update on a friday ? I know the machine is not up to date.. | 20:23 |
| apevec | yeah, I should have disabled extras too | 20:24 |
| *** suresh12 has quit IRC | 20:25 | |
| apevec | I'll try reverting to previous docker | 20:25 |
| apevec | Extras is just not same sla | 20:25 |
| dmsimard | apevec: I'll stop origin-master and origin-node so they're not thrown in for a loop | 20:25 |
| apevec | ok | 20:26 |
| dmsimard | apevec: just start them both once it looks like docker is starting | 20:26 |
| dmsimard | I'm working on 3.7 right now.. worst case scenario that'll be our backup plan | 20:27 |
| apevec | prev was 2:docker-1.12.6-28.git1398f24.el7 update today 2:docker-common-1.12.6-61.git85d7426.el7 | 20:27 |
| apevec | yum downgrade docker\* tries 2:docker-common-1.12.6-55.gitc4618fb.el7 which fails due to missing deps ... | 20:28 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Update RDO registry playbooks to deploy OpenShift 3.7 https://review.rdoproject.org/r/10259 | 20:30 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Remove needless variable re-declaration https://review.rdoproject.org/r/10403 | 20:30 |
| dmsimard | apevec: Error starting daemon: SELinux is not supported with the overlay2 graph driver on this kernel. Either boot into a newer kernel or disable selinux in docker (--selinux-enabled=false) | 20:30 |
| number80 | apevec: better using yum history undo #transaction to avoid that | 20:31 |
| dmsimard | apevec: wait, don't touch anything | 20:31 |
| dmsimard | apevec: I think the docker update flushed the docker config | 20:31 |
| apevec | I'm not | 20:31 |
| apevec | dmsimard, it shouldn't touch %config files ... | 20:32 |
| apevec | where is it? | 20:32 |
| apevec | dmsimard, but yes, we might need reboot to get 7.4 kernel | 20:32 |
| apevec | let's reboot, all is down anyway | 20:33 |
| dmsimard | apevec: https://github.com/rdo-infra/rdo-container-registry/commit/c35981fae5753ef91ca31717fe6346897f0a01cd | 20:33 |
| rdogerrit | rdo-trunk created openstack/tripleo-common-distgit pike-rdo: Update to 7.6.3 https://review.rdoproject.org/r/10413 | 20:33 |
| dmsimard | # grep selinux /etc/sysconfig/docker | 20:33 |
| dmsimard | OPTIONS=' --selinux-enabled --log-driver=journald' | 20:33 |
| rdogerrit | rdo-trunk created openstack/tripleoclient-distgit pike-rdo: Update to 7.3.3 https://review.rdoproject.org/r/10414 | 20:33 |
| rdogerrit | rdo-trunk created openstack/tripleo-image-elements-distgit pike-rdo: Update to 7.0.1 https://review.rdoproject.org/r/10415 | 20:33 |
| rdogerrit | rdo-trunk created openstack/tripleo-puppet-elements-distgit pike-rdo: Update to 7.0.1 https://review.rdoproject.org/r/10416 | 20:33 |
| rdogerrit | rdo-trunk created openstack/tripleo-heat-templates-distgit pike-rdo: Update to 7.0.3 https://review.rdoproject.org/r/10417 | 20:34 |
| dmsimard | docker starts now | 20:34 |
| rdogerrit | rdo-trunk created openstack/tripleo-ui-distgit pike-rdo: Update to 7.4.3 https://review.rdoproject.org/r/10418 | 20:34 |
| rdogerrit | rdo-trunk created openstack/instack-undercloud-distgit pike-rdo: Update to 7.4.3 https://review.rdoproject.org/r/10419 | 20:34 |
| rdogerrit | rdo-trunk created puppet/puppet-tripleo-distgit pike-rdo: Update to 7.4.3 https://review.rdoproject.org/r/10420 | 20:34 |
| dmsimard | might as well reboot at this point, yes | 20:34 |
| rdogerrit | rdo-trunk created openstack/tripleo-validations-distgit pike-rdo: Update to 7.4.2 https://review.rdoproject.org/r/10421 | 20:34 |
| dmsimard | apevec: you there ? | 20:35 |
| apevec | hmm, it is marked %config(noreplace) | 20:35 |
| apevec | dmsimard, yes, let's reboot | 20:35 |
| apevec | dmsimard, I logged off | 20:36 |
| dmsimard | reboot with fsck skipped started | 20:36 |
| apevec | ok, it's back w/ 7.4 kernel | 20:41 |
| dmsimard | pods are up and running, should be ok | 20:42 |
| dmsimard | going to run a few tests | 20:43 |
| *** jmlowe has quit IRC | 20:43 | |
| apevec | skope client works | 20:46 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Update RDO registry playbooks to deploy OpenShift 3.7 https://review.rdoproject.org/r/10259 | 20:47 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Remove needless variable re-declaration https://review.rdoproject.org/r/10403 | 20:47 |
| dmsimard | docker pulls are also working | 20:48 |
| dmsimard | apevec: so I asked earlier but didn't get an answer.. what is the update for ? other than generally updating things | 20:48 |
| dmsimard | I lost part of the discussion earlier today | 20:48 |
| *** brault has joined #rdo | 20:48 | |
| *** suresh12 has joined #rdo | 20:51 | |
| *** trown is now known as trown|outtypewww | 20:51 | |
| *** brault has quit IRC | 20:52 | |
| *** suresh12 has quit IRC | 20:55 | |
| rdogerrit | Merged openstack/tripleo-image-elements-distgit pike-rdo: Update to 7.0.1 https://review.rdoproject.org/r/10415 | 20:56 |
| rdogerrit | Merged openstack/tripleo-common-distgit pike-rdo: Update to 7.6.3 https://review.rdoproject.org/r/10413 | 20:58 |
| rdogerrit | Merged openstack/tripleo-puppet-elements-distgit pike-rdo: Update to 7.0.1 https://review.rdoproject.org/r/10416 | 20:59 |
| *** bswartz has quit IRC | 21:02 | |
| *** catintheroof has quit IRC | 21:04 | |
| apevec | dmsimard, so in general, running out of date public machine is not good idea, but also vague idea was that updated kernel could help perf | 21:04 |
| dmsimard | apevec: yes, updating machines is something we def should be doing -- I was at the office the other day with nhicher exactly to update review.rdo but we were not able to do it due to regressions | 21:05 |
| dmsimard | There's bugs preventing 2.6 from working with 7.4 | 21:05 |
| apevec | ansible? | 21:06 |
| dmsimard | I think it's something along the lines of 2.6 meant to work with RDO newton (they pull deps from us) and there was issues getting that to work on 7.4 and then 2.7 is designed to work with pike and is fine, but pike wouldn't work with 2.6 | 21:07 |
| dmsimard | I don't remember all the details | 21:07 |
| rdogerrit | Merged openstack/tripleoclient-distgit pike-rdo: Update to 7.3.3 https://review.rdoproject.org/r/10414 | 21:08 |
| rdogerrit | Merged puppet/puppet-certmonger-distgit ocata-rdo: Update to v1.1.1 tag official tag https://review.rdoproject.org/r/9920 | 21:08 |
| *** suresh12 has joined #rdo | 21:08 | |
| apevec | hmm, so newton is not working w/ 7.4 ? I thought we tested it, I'll check that next week | 21:08 |
| apevec | maybe we are missing update | 21:08 |
| dmsimard | SF, not OpenStack | 21:08 |
| *** suresh12 has quit IRC | 21:09 | |
| apevec | ok, I understood it was issue w/ deps coming from rdo newton? | 21:09 |
| dmsimard | apevec: yes, but specifically in the context of software factory | 21:09 |
| dmsimard | apevec: the packages themselves are fine, I mean, we ran jobs on newton (and still do?) with 7.4 | 21:10 |
| rdogerrit | Merged puppet/puppet-tripleo-distgit pike-rdo: Update to 7.4.3 https://review.rdoproject.org/r/10420 | 21:13 |
| rdogerrit | Merged openstack/tripleo-heat-templates-distgit pike-rdo: Update to 7.0.3 https://review.rdoproject.org/r/10417 | 21:13 |
| *** kellyp has joined #rdo | 21:14 | |
| apevec | weshay, dmsimard - looks like we are unlucky w/ cico in https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-pike-current-tripleo/107/ | 21:14 |
| apevec | 2 jobs failed in TASK [Wait until server is up and runnning] | 21:15 |
| apevec | or not, those jobs are not allocating cico nodes? | 21:15 |
| dmsimard | those are rdo cloud | 21:17 |
| dmsimard | weirdo jobs on cico are on rdo cloud | 21:17 |
| rdogerrit | Merged openstack/tripleo-validations-distgit pike-rdo: Update to 7.4.2 https://review.rdoproject.org/r/10421 | 21:17 |
| apevec | yeah, I could've read above: Create job virtual machine | 21:18 |
| dmsimard | yuck | 21:18 |
| dmsimard | need a patch for openshift-ansible | 21:18 |
| *** jayg is now known as jayg|g0n3 | 21:19 | |
| rdogerrit | Merged openstack/instack-undercloud-distgit pike-rdo: Update to 7.4.3 https://review.rdoproject.org/r/10419 | 21:22 |
| *** jayg|g0n3 has quit IRC | 21:23 | |
| rdogerrit | Merged openstack/tripleo-common-distgit newton-rdo: Update to 5.4.5 https://review.rdoproject.org/r/10411 | 21:28 |
| rdogerrit | Merged openstack/tripleo-image-elements-distgit newton-rdo: Update to 5.3.2 https://review.rdoproject.org/r/10404 | 21:35 |
| rdogerrit | Merged openstack/instack-undercloud-distgit newton-rdo: Update to 5.3.3 https://review.rdoproject.org/r/10406 | 21:40 |
| rdogerrit | Merged puppet/puppet-tripleo-distgit newton-rdo: Update to 5.6.5 https://review.rdoproject.org/r/10409 | 21:43 |
| *** jmlowe has joined #rdo | 21:43 | |
| rdogerrit | Merged puppet/puppet-tripleo-distgit ocata-rdo: Update to 6.5.4 https://review.rdoproject.org/r/10410 | 21:45 |
| rdogerrit | Merged openstack/tripleo-heat-templates-distgit ocata-rdo: Update to 6.2.4 https://review.rdoproject.org/r/10407 | 21:46 |
| rdogerrit | Merged openstack/tripleo-heat-templates-distgit newton-rdo: Update to 5.3.4 https://review.rdoproject.org/r/10412 | 21:46 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Update RDO registry playbooks to deploy OpenShift 3.7 https://review.rdoproject.org/r/10259 | 21:49 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Remove needless variable re-declaration https://review.rdoproject.org/r/10403 | 21:49 |
| dmsimard | that has a patch to openshift ansible.. let's see if it works... | 21:49 |
| * dmsimard going to eat | 21:49 | |
| *** paramite has quit IRC | 21:49 | |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Update RDO registry playbooks to deploy OpenShift 3.7 https://review.rdoproject.org/r/10259 | 21:50 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Remove needless variable re-declaration https://review.rdoproject.org/r/10403 | 21:50 |
| *** apevec has left #rdo | 21:51 | |
| rdogerrit | Merged openstack/tripleo-image-elements-distgit ocata-rdo: Update to 6.1.1 https://review.rdoproject.org/r/10405 | 21:52 |
| *** jmelvin_ has quit IRC | 21:52 | |
| rdogerrit | Merged openstack/instack-undercloud-distgit ocata-rdo: Update to 6.1.2 https://review.rdoproject.org/r/10408 | 21:53 |
| EmilienM | number80: need your help | 21:57 |
| number80 | hai | 21:57 |
| *** suresh12 has joined #rdo | 21:57 | |
| EmilienM | tripleo-ui tarball was renamed | 21:57 |
| EmilienM | http://tarballs.openstack.org/tripleo-ui/tripleo-ui-7.4.3..f4e1c6b.tar.gz | 21:57 |
| EmilienM | instead of http://tarballs.openstack.org/tripleo-ui/tripleo-ui-7.4.3.tar.gz | 21:58 |
| number80 | It looks like a blunder | 21:58 |
| EmilienM | I'm wondering if there is anything we can do in dist-git | 21:59 |
| EmilienM | yeah it's npm format now | 21:59 |
| number80 | Yeah, but they somehow messed up | 21:59 |
| number80 | I mean the two dots are wrong | 21:59 |
| EmilienM | k | 21:59 |
| jrist | yeah | 21:59 |
| jrist | it's the right hash tho! | 21:59 |
| *** dustins has quit IRC | 22:00 | |
| number80 | jrist: am I right that new archive name is tripleo-ui-$VERSION-$SHORT_HASH.tar.gz ? | 22:00 |
| number80 | (I can fix that in spec) | 22:00 |
| jrist | I think that's an option | 22:00 |
| jrist | but I think EmilienM would prefer no hash | 22:01 |
| jrist | just the version | 22:01 |
| EmilienM | I prefer nothing | 22:01 |
| EmilienM | ahah | 22:01 |
| number80 | I'd prefer too but up to you guys | 22:01 |
| jrist | EmilienM: so just tripleo-ui.tar.gz? | 22:01 |
| number80 | nah, keep the version! | 22:01 |
| jrist | let's just do version plz | 22:01 |
| EmilienM | I don't care at all tbh | 22:01 |
| jrist | ;( | 22:01 |
| EmilienM | as long as we have an rpm downstream | 22:01 |
| EmilienM | that was my goal for today :D | 22:01 |
| number80 | Well, package makers are opinionated | 22:01 |
| number80 | and hard-headed too | 22:02 |
| EmilienM | yeah I know one from Lyon he's tough | 22:02 |
| jrist | rofl | 22:02 |
| number80 | we want versioned tarballs as much as kids wanted candies during Halloween | 22:02 |
| EmilienM | jrist: go patch openstack-infra if you want to rename the tarball | 22:02 |
| jrist | rofl | 22:02 |
| jrist | link me? | 22:02 |
| EmilienM | otherwise, change distgit and accept new name | 22:02 |
| *** suresh12 has quit IRC | 22:02 | |
| EmilienM | gimme a tarball gimme a tarball | 22:02 |
| jrist | I don't understand what either of those statements mean | 22:02 |
| number80 | Well, I want predictible name scheme that doesn't change every release | 22:03 |
| jrist | number80: do you have it handy? | 22:03 |
| jrist | +1 | 22:03 |
| * number80 looks | 22:03 | |
| EmilienM | jrist: I have to leave soon to fly to SYD, I'll let you https://review.rdoproject.org/r/#/c/10418/ | 22:04 |
| *** suresh12 has joined #rdo | 22:04 | |
| *** yselkowitz has quit IRC | 22:04 | |
| EmilienM | jrist: it's the only thing remaining for tripleo stable release | 22:04 |
| EmilienM | everything else was released | 22:04 |
| EmilienM | if you ask me, I prefer the old format: tripleo-ui-$VERSION.tar.gz | 22:04 |
| EmilienM | more predictable | 22:04 |
| EmilienM | and also consistent with other openstack projects | 22:05 |
| *** d0ugal has quit IRC | 22:06 | |
| *** fragatina has quit IRC | 22:06 | |
| number80 | fun, npm pack works properly | 22:07 |
| * number80 needs to find where releases team moved code to generate tarballs | 22:07 | |
| *** suresh12 has quit IRC | 22:08 | |
| *** d0ugal has joined #rdo | 22:09 | |
| *** jrist has quit IRC | 22:11 | |
| *** jrist has joined #rdo | 22:15 | |
| *** jrist has quit IRC | 22:15 | |
| *** jrist has joined #rdo | 22:15 | |
| number80 | Damn, finding logs for a job in post pipeline is like searching for a needle | 22:16 |
| *** suresh12 has joined #rdo | 22:20 | |
| *** thrash is now known as thrash|g0ne | 22:24 | |
| dmsimard | number80: try git-os-log | 22:25 |
| dmsimard | number80: er, git os-job | 22:26 |
| dmsimard | https://doughellmann.com/blog/2017/10/03/git-os-job-1-1-1/ | 22:26 |
| number80 | Interesting | 22:29 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Update RDO registry playbooks to deploy OpenShift 3.7 https://review.rdoproject.org/r/10259 | 22:30 |
| rdogerrit | David Moreau Simard proposed rdo-infra/rdo-container-registry master: Remove needless variable re-declaration https://review.rdoproject.org/r/10403 | 22:30 |
| EmilienM | number80: we're debbugging it on #openstack-release, but you're not connected on this one. | 22:40 |
| EmilienM | fwiw, it's a good channel to be connected on | 22:40 |
| number80 | well, I used to be on that one but it's not on autoconnect | 22:41 |
| rdogerrit | Merged openstack/cinder-distgit pike-rdo: Update to 11.0.1 https://review.rdoproject.org/r/10402 | 22:44 |
| *** MrAbaddon has joined #rdo | 22:45 | |
| *** fragatina has joined #rdo | 22:50 | |
| *** mary_grace has quit IRC | 22:53 | |
| EmilienM | number80: also #openstack-infra - we discussed about it and found the root cause: https://review.openstack.org/517733 | 23:06 |
| EmilienM | I mean, https://review.openstack.org/517733 is a fix to the root cause | 23:06 |
| EmilienM | jrist: ^ fyi | 23:06 |
| EmilienM | ok I'm out now, ttyl folks | 23:06 |
| jrist | yup thanks | 23:07 |
| *** toure is now known as toure_biab | 23:15 | |
| number80 | ack | 23:25 |
| *** vnogin has joined #rdo | 23:27 | |
| *** links has joined #rdo | 23:32 | |
| *** Son_Goku has joined #rdo | 23:41 | |
| *** Son_Goku has quit IRC | 23:42 | |
| *** Son_Goku has joined #rdo | 23:44 | |
| *** Son_Goku has quit IRC | 23:48 | |
| *** Son_Goku has joined #rdo | 23:49 | |
| *** Son_Goku has quit IRC | 23:51 | |
| *** Son_Goku has joined #rdo | 23:55 | |
| *** Son_Goku has joined #rdo | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!