| pabelanger | dmsimard: is somebody doing ARM images for RDO and docker? | 00:06 |
|---|---|---|
| rdogerrit | Alan Pevec proposed rdo-infra/centos-release-openstack pike-rdo: Update the spec for centos-release-openstack-pike https://review.rdoproject.org/r/9140 | 00:06 |
| pabelanger | aarch64 and armhf for example | 00:06 |
| rdogerrit | Alan Pevec proposed rdo-infra/centos-release-openstack pike-rdo: Update the spec for centos-release-openstack-pike https://review.rdoproject.org/r/9140 | 00:07 |
| rdogerrit | rdo-trunk created rdoinfo master: Promote CBS tags update for newton-testing https://review.rdoproject.org/r/9169 | 00:07 |
| rdogerrit | rdo-trunk created rdoinfo master: Promote CBS tags update for pike-testing https://review.rdoproject.org/r/9170 | 00:07 |
| rdogerrit | rdo-trunk proposed rdoinfo master: Promote CBS tags update for ocata-testing https://review.rdoproject.org/r/8999 | 00:07 |
| dmsimard | pabelanger: not a good person to ask. number80 or tonyb might know. We don't have any of these right now, that's for sure but it might be planned. Pike is officially the first release with hopefully full ppc64le and aarch64 support. | 00:08 |
| pabelanger | dmsimard: cool | 00:08 |
| * tonyb reads scrool back | 00:08 | |
| pabelanger | just at docker meetup here in town and somebody was asking for docker images on arm | 00:08 |
| tonyb | pabelanger: I don't think so but I'd like to think that ppc64le and aarch64 get rolled into whatever workflow is decided upon for the other trippleo containers | 00:09 |
| pabelanger | tonyb: okay, cool. | 00:09 |
| pabelanger | figured I'd ask | 00:10 |
| tonyb | pabelanger: I'm *sure* anyone dowing a product would have them interanlly | 00:10 |
| pabelanger | Ya, persons was just asking is anybody was doing docker and aarch64, wasn't sure if RDO was or not | 00:11 |
| *** thegreenhundred has joined #rdo | 00:14 | |
| tonyb | pabelanger: fair enough. | 00:14 |
| apevec | dmsimard, I've updated https://review.rdoproject.org/r/#/c/9140/5 to enable testing repo by default, wdyt | 00:16 |
| *** sasha2 has quit IRC | 00:16 | |
| apevec | just to have something to run https://ci.centos.org/view/rdo/view/cbs-promote-pipelines/job/rdo-promote-cloudsig-testing-pike/ after pushing to testing | 00:16 |
| dmsimard | apevec: on my phone right now | 00:17 |
| apevec | ok, review when you can | 00:17 |
| dmsimard | But I vaguely remember us doing something similar in ocata perhaps | 00:17 |
| apevec | yeah | 00:17 |
| apevec | I thought new rdoinfo jobs would save us of that | 00:18 |
| apevec | but alas | 00:18 |
| number80 | pabelanger: ARM docker images for RDO, not that I heard of. But ping radez, he may have some info on that. | 00:20 |
| *** sasha2 has joined #rdo | 00:20 | |
| radez | nope, I've only got qcow images, no docker images | 00:21 |
| number80 | radez: ok, then nobody is working on docker images for aarch64 that I heard of :) | 00:22 |
| *** Son_Goku has joined #rdo | 00:31 | |
| pabelanger | actually, guy I was talking to in Ottawa, knows radez. So pointed him your way | 00:33 |
| pabelanger | from Dell | 00:33 |
| radez | I imagine I'll get to the containers eventually, | 00:33 |
| radez | but I'm tryign to finish up the tripleo work | 00:33 |
| pabelanger | Ya, thanks what I basically said | 00:33 |
| pabelanger | somebody likely working on something | 00:34 |
| radez | and it's been slow going functioning on centos 7.3 | 00:34 |
| radez | the 7.4 alt suppoort should help out a ton | 00:34 |
| *** sasha2 has quit IRC | 00:37 | |
| *** Son_Goku has quit IRC | 00:37 | |
| *** morazi has joined #rdo | 00:41 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: nova: http://trunk.rdoproject.org/centos7-master/report.html | 00:45 |
| dmsimard | apevec: +2 | 00:45 |
| dmsimard | apevec: but we still need to get it built and shipped to -extras before being able to properly test it | 00:45 |
| dmsimard | I think at least the weirdo jobs are looking good | 00:48 |
| dmsimard | but we need the oooq coverage.. | 00:48 |
| *** MrAbaddon has quit IRC | 00:54 | |
| *** spotz has quit IRC | 01:00 | |
| rdogerrit | Merged rdo-infra/centos-release-openstack pike-rdo: Update the spec for centos-release-openstack-pike https://review.rdoproject.org/r/9140 | 01:04 |
| number80 | And it failed again | 01:05 |
| *** MrAbaddon has joined #rdo | 01:07 | |
| number80 | Automation needs to be driven by AI to be super duper efficient | 01:10 |
| apevec | damn | 01:10 |
| apevec | so packstack all pass | 01:11 |
| *** Son_Goku has joined #rdo | 01:11 | |
| apevec | poi 2 and 3 failed | 01:11 |
| number80 | failure are unrelated for poi 2 | 01:13 |
| apevec | it's tempest failures | 01:14 |
| apevec | 1 object storage and 7 ec2api | 01:16 |
| number80 | yeah, but it shouldn;t happen | 01:16 |
| apevec | cannot tell until you root cause those failures | 01:16 |
| apevec | number80, ec2api errors must be incompatiblity between cross-tagged old ec2api release and pike clients: nova_floating_ip = nova.floating_ips.create() | 01:21 |
| apevec | 2017-08-31 00:42:06.186 30932 ERROR ec2api.api AttributeError: 'Client' object has no attribute 'floating_ips' | 01:21 |
| apevec | https://centos.logs.rdoproject.org/weirdo-generic-puppet-openstack-scenario002/2701/weirdo-project/logs/ec2api/ec2api.txt.gz | 01:21 |
| apevec | we'll need snapshot, they still don't have release from stable/pike | 01:22 |
| number80 | Yeah | 01:23 |
| number80 | Latest commit that successfully passed CI is 5061c009cfc83d0efc51a9ad4d2f0560046e0d80 | 01:24 |
| number80 | Erm no, it's 74906017f7bb7e273600b57e9f6131afa7ac5f7d | 01:25 |
| *** imcleod has joined #rdo | 01:25 | |
| apevec | number80, ack, that's stable/pike head | 01:27 |
| apevec | but again, how did they create stable/pike w/o rc1 tag... again self-managed?! | 01:28 |
| apevec | for such critical project | 01:28 |
| apevec | maybe poi should not consider ec2-api is worth including in scenario002 :( | 01:29 |
| apevec | is NOT worth* | 01:29 |
| apevec | anyway, snap it is | 01:29 |
| apevec | number80, can you do it? | 01:29 |
| apevec | I'm digging into other failures | 01:29 |
| rdogerrit | hguemar created openstack/ec2-api-distgit rpm-master: Requirement sync for Pike https://review.rdoproject.org/r/9171 | 01:33 |
| number80 | yep | 01:33 |
| number80 | I was already preparing that ^ | 01:33 |
| *** dmacpher has joined #rdo | 01:33 | |
| *** jhershbe has joined #rdo | 01:35 | |
| *** spotz has joined #rdo | 01:35 | |
| apevec | object storage failure is: | 01:38 |
| apevec | testtools.matchers._impl.MismatchError: 'tempest-d--67488436' not in ['tempest-a--852248077', 'tempest-b--1660248112', 'tempest-c--1046594035', 'tempest-e--2080095498', 'tempest-f--81986837'] | 01:38 |
| apevec | might be timing issue, since other swift test cases passed | 01:38 |
| number80 | CI passed after reqcheck, I can branch now | 01:39 |
| rdogerrit | hguemar created config master: Create pike-rdo branch for openstack-ec2api https://review.rdoproject.org/r/9172 | 01:41 |
| apevec | and in poi 003 we have Puppet (err): Failed to apply catalog: undefined method `[]' for nil:NilClass | 01:42 |
| apevec | looks like in puppet-murano https://centos.logs.rdoproject.org/weirdo-generic-puppet-openstack-scenario003/2708/weirdo-project/puppet.txt.gz | 01:42 |
| number80 | I haven't looked at poi3 | 01:42 |
| apevec | dmsimard, ^ | 01:42 |
| apevec | any ideas? | 01:42 |
| *** liujiong has joined #rdo | 01:42 | |
| dmsimard | Lemme get my laptop | 01:42 |
| apevec | puppet-murano is new in Pike | 01:43 |
| apevec | http://cbs.centos.org/koji/buildinfo?buildID=19512 | 01:43 |
| rdogerrit | Merged openstack/ec2-api-distgit rpm-master: Requirement sync for Pike https://review.rdoproject.org/r/9171 | 01:44 |
| number80 | at least, puppet-murano packaging is sane and it's the latest in pike branch (and passed CI), so it does not come from here | 01:44 |
| number80 | here = packaging | 01:45 |
| apevec | then it might be change in poi | 01:45 |
| apevec | ? | 01:46 |
| * dmsimard has a laptop | 01:46 | |
| rdogerrit | Merged config master: Create pike-rdo branch for openstack-ec2api https://review.rdoproject.org/r/9172 | 01:46 |
| apevec | /usr/share/openstack-puppet/modules/murano/lib/puppet/provider/murano_application/murano.rb:40:in `block in instances' | 01:46 |
| dmsimard | yeah looking now | 01:47 |
| apevec | so maybe instances creation failed, but still bad error reporting | 01:47 |
| *** jhershbe has quit IRC | 01:49 | |
| number80 | here we go | 01:50 |
| rdogerrit | hguemar created openstack/ec2-api-distgit pike-rdo: Pike update (74906017f7bb7e273600b57e9f6131afa7ac5f7d) https://review.rdoproject.org/r/9173 | 01:50 |
| number80 | I still need to figure out a proper version number not to conflict with ocata or future pike release | 01:53 |
| number80 | Maybe I'll go with 4.1.0 and pre-release style NVR | 01:54 |
| apevec | hmm | 01:54 |
| dmsimard | apevec, number80: unrelated but wtf ? https://centos.logs.rdoproject.org/weirdo-generic-puppet-openstack-scenario003/2708/weirdo-project/logs/syslog.txt.gz#_Aug_31_00_14_58 | 01:54 |
| dmsimard | looks like the horizon cache thing on restart that's going crazy | 01:54 |
| apevec | might be some xstatic dep missing? | 01:54 |
| dmsimard | dunno, worth looking into. Adding to todos | 01:55 |
| number80 | we have very pyscss => http://cbs.centos.org/koji/packageinfo?packageID=348 | 01:55 |
| number80 | *old | 01:55 |
| apevec | what's in u-c ? | 01:55 |
| dmsimard | I lost the etherpad, where is it ? | 01:55 |
| number80 | same version | 01:56 |
| apevec | https://review.rdoproject.org/etherpad/p/pike-release-preparation | 01:56 |
| dmsimard | thanks | 01:58 |
| rdogerrit | hguemar proposed openstack/ec2-api-distgit pike-rdo: Pike update (74906017f7bb7e273600b57e9f6131afa7ac5f7d) https://review.rdoproject.org/r/9173 | 01:59 |
| number80 | Similar ticket for mitaka => https://bugzilla.redhat.com/show_bug.cgi?id=1332457 | 02:00 |
| openstack | bugzilla.redhat.com bug 1332457 in openstack-horizon "Unable to start horizon (mitaka)" [High,Closed: notabug] - Assigned to iserrano | 02:00 |
| *** jmlowe has quit IRC | 02:00 | |
| * number80 notes pinging mrunge during CET morning to see if he has any hints | 02:01 | |
| *** jmlowe has joined #rdo | 02:01 | |
| dmsimard | got nothing so far on the murano thing, still looking | 02:01 |
| number80 | https://review.rdoproject.org/r/#/c/1811/ <= found the fix for the mitaka issue | 02:02 |
| number80 | s/fix/workaround/ | 02:02 |
| number80 | Not worthy to be called a fix | 02:02 |
| * number80 spit on soil when we increase timeouts to solve issues | 02:03 | |
| * apevec gives up for tonight | 02:05 | |
| apevec | see you in few h | 02:05 |
| *** apevec has quit IRC | 02:05 | |
| number80 | yeah | 02:07 |
| * number80 finishes current task | 02:07 | |
| number80 | otherwise I won't be able to sleep anyway | 02:07 |
| rdogerrit | hguemar proposed openstack/ec2-api-distgit pike-rdo: Pike update (74906017f7bb7e273600b57e9f6131afa7ac5f7d) https://review.rdoproject.org/r/9173 | 02:09 |
| *** sasha2 has joined #rdo | 02:18 | |
| rdogerrit | Merged openstack/ec2-api-distgit pike-rdo: Pike update (74906017f7bb7e273600b57e9f6131afa7ac5f7d) https://review.rdoproject.org/r/9173 | 02:28 |
| number80 | ^ | 02:35 |
| rdogerrit | hguemar proposed rdoinfo master: Push all pike-candidate to pike-testing https://review.rdoproject.org/r/9161 | 02:35 |
| number80 | ^ | 02:35 |
| * number80 off to sleep | 02:37 | |
| *** ykarel has joined #rdo | 02:44 | |
| *** imcleod has quit IRC | 02:47 | |
| *** MrAbaddon has quit IRC | 02:47 | |
| *** gtrxcb has joined #rdo | 02:54 | |
| *** cliles has quit IRC | 02:54 | |
| *** dmacpher has quit IRC | 03:00 | |
| *** MrAbaddon has joined #rdo | 03:02 | |
| *** scorcoran has joined #rdo | 03:04 | |
| *** ykarel has quit IRC | 03:13 | |
| *** jmlowe has quit IRC | 03:13 | |
| *** jmlowe has joined #rdo | 03:14 | |
| *** dmacpher has joined #rdo | 03:17 | |
| *** udesale has joined #rdo | 03:37 | |
| *** links has joined #rdo | 03:39 | |
| *** gkadam_ has joined #rdo | 03:40 | |
| *** scorcoran has quit IRC | 03:40 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: nova: http://trunk.rdoproject.org/centos7-master/report.html | 03:42 |
| *** dmacpher has quit IRC | 04:01 | |
| *** dmacpher has joined #rdo | 04:02 | |
| *** ykarel has joined #rdo | 04:07 | |
| *** janki has joined #rdo | 04:10 | |
| *** shreshtha has joined #rdo | 04:15 | |
| *** aludwar has quit IRC | 04:21 | |
| *** vijay3093 has joined #rdo | 04:21 | |
| *** vijay3093 has quit IRC | 04:22 | |
| *** anilvenkata|AFK is now known as anilvenkata | 04:23 | |
| *** jhershbe has joined #rdo | 04:23 | |
| ykarel | number80, hi | 04:23 |
| ykarel | i saw jobs are failing for 9161 | 04:23 |
| ykarel | packstack and poi | 04:24 |
| ykarel | Error is in launching VMs for jobs: Server Error for url: https://phx2.cloud.rdoproject.org:13774/v2.1/os-volumes_boot Gateway Time-out | 04:24 |
| ykarel | 3 poi and 3 packstack jobs failed with same reason | 04:26 |
| ykarel | Is someone looking into it | 04:26 |
| ykarel | in my tenant i am able to create VM from volume, so may be the issue resolved, recheck might help | 04:32 |
| *** masonjoyers__ has quit IRC | 04:39 | |
| *** masonjoyers__ has joined #rdo | 04:39 | |
| *** jlabarre has quit IRC | 04:41 | |
| *** sasha2 has quit IRC | 04:44 | |
| *** anshul has joined #rdo | 04:49 | |
| *** dpawar has joined #rdo | 04:51 | |
| ykarel | number80, posted recheck to see if issue resolved | 04:57 |
| ykarel | info: 1.5 hours ago newton promotion jobs also failed with same reason | 05:02 |
| *** rbartal has joined #rdo | 05:07 | |
| *** iranzo has joined #rdo | 05:15 | |
| *** ratailor has joined #rdo | 05:18 | |
| *** aditya_r has joined #rdo | 05:18 | |
| *** rbartal has quit IRC | 05:31 | |
| *** abregman has joined #rdo | 05:34 | |
| *** richm has quit IRC | 05:36 | |
| *** aditya_r has quit IRC | 05:40 | |
| *** aditya_r has joined #rdo | 05:40 | |
| *** mosulica has joined #rdo | 05:49 | |
| *** agurenko has joined #rdo | 05:50 | |
| *** pgadiya has joined #rdo | 05:52 | |
| *** sbonazzo has joined #rdo | 05:52 | |
| *** fzdarsky|afk has joined #rdo | 05:55 | |
| dpawar | apevec: Not sure is this correct place to ask this question, does RedHat support not tripleo based (manual) deployment officially for Pike ? for OSP11 it seems like it is supported https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/pdf/manual_installation_procedures/Red_Hat_OpenStack_Platform-11-Manual_Installation_Procedures-en-US.pdf. Any idea, where should i post this question if this is not appropriate palce? | 05:55 |
| dpawar | Hi, Not sure is this correct place to ask this question, does RedHat support not tripleo based (manual) deployment officially for Pike ? for OSP11 it seems like it is supported https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/pdf/manual_installation_procedures/Red_Hat_OpenStack_Platform-11-Manual_Installation_Procedures-en-US.pdf. Any idea, where should i post this question if this is not appropriate palce? | 05:56 |
| *** sbonazzo has quit IRC | 06:01 | |
| dpawar | dmsimard: ^^^ any idea ? | 06:02 |
| ykarel | number80, this time 5 vms created, one failed with error: 'message': u'Build of instance 83e377b0-a35e-4794-b6f3-702e0c2b1d34 aborted: Volume resource quota exceeded | 06:08 |
| ykarel | dmsimard, jpena|off apevec ^^ who can fix this | 06:08 |
| *** lkuchlan has joined #rdo | 06:10 | |
| *** lkuchlan is now known as lkuchlan_wfh | 06:11 | |
| *** sbonazzo has joined #rdo | 06:13 | |
| *** aditya_r has quit IRC | 06:15 | |
| *** links has quit IRC | 06:15 | |
| *** aditya_r has joined #rdo | 06:15 | |
| *** emalikov has joined #rdo | 06:19 | |
| *** pcaruana has joined #rdo | 06:22 | |
| *** mpeterson has joined #rdo | 06:26 | |
| *** dmacpher has quit IRC | 06:27 | |
| rdogerrit | chkumar246 created openstack/watcher-distgit pike-rdo: Remove tempest plugin reference from watcher project https://review.rdoproject.org/r/9174 | 06:30 |
| rdogerrit | chkumar246 created openstack/watcher-distgit pike-rdo: Removed reference of intree tempest plugin for pike https://review.rdoproject.org/r/9175 | 06:33 |
| *** jtomasek has quit IRC | 06:36 | |
| *** jtomasek has joined #rdo | 06:36 | |
| *** yfried has joined #rdo | 06:37 | |
| rdogerrit | chkumar246 created rdoinfo master: Switch pike source for tempest to tag 17.0.0 https://review.rdoproject.org/r/9176 | 06:37 |
| *** jtomasek has quit IRC | 06:39 | |
| *** dpawar has quit IRC | 06:39 | |
| *** jtomasek has joined #rdo | 06:40 | |
| *** rcernin has joined #rdo | 06:40 | |
| *** nyechiel has joined #rdo | 06:42 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: nova: http://trunk.rdoproject.org/centos7-master/report.html | 06:42 |
| *** sc has joined #rdo | 06:42 | |
| *** snarwade has joined #rdo | 06:49 | |
| *** oidgar has joined #rdo | 06:49 | |
| *** jprovazn has joined #rdo | 06:49 | |
| *** dpawlik has quit IRC | 06:53 | |
| *** tvignaud has joined #rdo | 06:54 | |
| *** danpawlik has joined #rdo | 06:57 | |
| *** apevec has joined #rdo | 06:57 | |
| *** apevec has quit IRC | 06:57 | |
| *** apevec has joined #rdo | 06:57 | |
| apevec | ykarel, for infra issues in rdocloud we need jpena|off | 07:00 |
| *** tshefi has joined #rdo | 07:00 | |
| *** dpawar has joined #rdo | 07:00 | |
| apevec | dpawar, re. OSP support: not right channel, and OSP12 based on Pike is not released yet, in general Red Hat will support issues with end-result cloud, regardless of deployment method but we recommend OSP Director i.e. productized TripleO. | 07:03 |
| dpawar | apevec: Thanks ! | 07:05 |
| *** dgurtner has joined #rdo | 07:07 | |
| *** dgurtner has joined #rdo | 07:07 | |
| *** fzdarsky|afk is now known as fzdarsky | 07:13 | |
| *** apetrich has quit IRC | 07:14 | |
| rdogerrit | chkumar246 proposed openstack/watcher-distgit pike-rdo: Remove tempest plugin reference from watcher project https://review.rdoproject.org/r/9174 | 07:15 |
| *** apetrich has joined #rdo | 07:15 | |
| *** larizmen has joined #rdo | 07:17 | |
| *** iranzo has quit IRC | 07:20 | |
| remix_tj | sorry hamzy i was AFK. Since our timezone are shifted, when you went out for lunch, i was going out to dinner plus rock festival :-D | 07:22 |
| *** gtrxcb has quit IRC | 07:22 | |
| remix_tj | hamzy: anyway i see resources.NovaCompute: Went to status ERROR due to \"Message: No valid host was found. There are not enough hosts available. So i suspect that previous stack cleanup hasn't been done very well. | 07:22 |
| *** iranzo has joined #rdo | 07:23 | |
| *** iranzo has quit IRC | 07:23 | |
| *** iranzo has joined #rdo | 07:23 | |
| *** sbulage has joined #rdo | 07:23 | |
| *** rwmjones has joined #rdo | 07:23 | |
| *** racedo has joined #rdo | 07:28 | |
| *** ragiman has joined #rdo | 07:28 | |
| *** links has joined #rdo | 07:28 | |
| *** egallen has joined #rdo | 07:28 | |
| *** jlibosva has joined #rdo | 07:31 | |
| *** tesseract has joined #rdo | 07:32 | |
| *** hjensas has joined #rdo | 07:32 | |
| *** apetrich has quit IRC | 07:35 | |
| rdogerrit | Yatin Karel created puppet/puppet-murano-distgit pike-rdo: Pike update (75837136f0c8a9c7b334d6a3d757425382058934) https://review.rdoproject.org/r/9177 | 07:36 |
| *** abregman is now known as abregman|mtg | 07:36 | |
| ykarel | apevec, poi scenario 003 failing in pike-testing for rdoinfo, hope this fixes it ^^ | 07:37 |
| *** rbartal has joined #rdo | 07:38 | |
| *** hjensas has quit IRC | 07:38 | |
| *** jpich has joined #rdo | 07:39 | |
| *** cylopez has joined #rdo | 07:41 | |
| *** apetrich has joined #rdo | 07:42 | |
| *** funzo_ has joined #rdo | 07:43 | |
| *** jpena|off is now known as jpena | 07:43 | |
| jpena | good morning | 07:43 |
| jpena | ykarel: are we having issues with RDO Cloud? Let me check | 07:44 |
| ykarel | jpena, Ok | 07:44 |
| ykarel | jpena, current issue: aborted: Volume resource quota exceede | 07:44 |
| *** egonzalez has joined #rdo | 07:45 | |
| jpena | we seem to be ok in terms of volumes now | 07:45 |
| jpena | oh wait | 07:45 |
| jpena | maybe it was another tenant | 07:45 |
| *** funzo has quit IRC | 07:46 | |
| chandankumar | number80: jpena ykarel apevec \o/ | 07:47 |
| jpena | ykarel: which job was having trouble? | 07:47 |
| chandankumar | jpena: number80 https://review.rdoproject.org/r/#/c/9174/ | 07:47 |
| jpena | o/ | 07:47 |
| number80 | o/ | 07:47 |
| chandankumar | to fix watcher project | 07:47 |
| * chandankumar is dreaming whole night about tempest plugin packaging during sleep | 07:48 | |
| jpena | chandankumar: well, the tempest plugin is still there in stable/pike, shouldn't we do it the other way around? I mean, only have the new watcher-tests-tempest plugin in queens and later | 07:48 |
| *** dpawar has quit IRC | 07:49 | |
| ykarel | jpena, https://review.rdoproject.org/jenkins/job/weirdo-validate-buildsys-tags/623/consoleText | 07:49 |
| jpena | ykarel: ok, so it was one of the ci.centos jobs, and those are running on the rdo-ci tenant | 07:49 |
| ykarel | https://ci.centos.org/job/weirdo-generic-packstack-scenario003/1253/consoleText | 07:49 |
| ykarel | jpena, ^^ one | 07:50 |
| chandankumar | jpena: that is also doable | 07:50 |
| chandankumar | jpena: then better to untag watcher tempest plugin | 07:50 |
| *** aufi has joined #rdo | 07:50 | |
| jpena | chandankumar: right, that was the plan. We untagged it from CBS yesterday, and my plan was to remove it from rdoinfo for dlrn today | 07:50 |
| chandankumar | or i can backport tempest plugin removal patch for pike | 07:50 |
| jpena | ykarel: now we should be ok, we have enough volume quota for at least 6 instances | 07:51 |
| jpena | however, I need to understand why some instances have a 80 GB volume and some others use 100 | 07:51 |
| chandankumar | because if any tests break, backporting to watcher project for pike is going to be hard | 07:51 |
| ykarel | jpena, Ok, | 07:52 |
| ykarel | jpena there is one more failure poi scenario 003 in murano | 07:52 |
| ykarel | i have pushed https://review.rdoproject.org/r/#/c/9177, | 07:52 |
| apevec | ykarel, have you figured out what's going on with murano in poi 003 ? | 07:53 |
| apevec | oh I see | 07:53 |
| jpena | ykarel: so the murano issue requires a later puppet-murano commit? | 07:54 |
| apevec | ykarel, do you have specific commit which fixes the issue? | 07:54 |
| chandankumar | jpena: number80 yesterday was a good learning for me for handling tempest plugin release | 07:54 |
| ykarel | apevec, to me it looks this: https://github.com/openstack/puppet-murano/commit/75837136f0c8a9c7b334d6a3d757425382058934 | 07:55 |
| number80 | :) | 07:55 |
| *** hewbrocca_afk is now known as hewbrocca | 07:55 | |
| number80 | ykarel: I don't any relevant changed compared to 11.3.0 | 07:57 |
| ykarel | number80, is https://github.com/openstack/puppet-murano/commit/75837136f0c8a9c7b334d6a3d757425382058934 there | 07:57 |
| *** florianf has joined #rdo | 07:58 | |
| number80 | https://paste.fedoraproject.org/paste/uFElKFx4deIS3XNHfVycnA | 07:58 |
| jpena | a change in the gemspec shouldn't justify the error ;./ | 07:59 |
| ykarel | i could see the difference in passing pike promotion jobs and failing jobs, so i thought this could be the reason | 08:00 |
| *** dpawar has joined #rdo | 08:00 | |
| *** egallen has quit IRC | 08:00 | |
| ykarel | there we have: puppet-murano-11.3.1-0.20170825180829.7583713.el7.centos.noarch | 08:00 |
| apevec | must be some other change | 08:01 |
| number80 | Well, I am myself skeptic but it could be some puppet wizardry | 08:01 |
| number80 | yeah, maybe in murano or muranoclient | 08:01 |
| apevec | gem definition should not matter with rpm packaged puppet module? | 08:01 |
| apevec | at least I hope, otherwise that means we are pulling from git with puppet-* RPMs ?! | 08:02 |
| *** jhershbe has quit IRC | 08:02 | |
| ykarel | apevec, Ok | 08:04 |
| *** egallen has joined #rdo | 08:04 | |
| apevec | ykarel, it'd be rather change in murano itself? | 08:04 |
| apevec | what' | 08:05 |
| ykarel | checking | 08:05 |
| apevec | s murano commit in passed-ci ? | 08:05 |
| ykarel | openstack-murano-api-4.0.0-0.20170825150325.94ba1de.el7.centos.noarch | 08:05 |
| ykarel | ^^ same with 4.0.0 | 08:07 |
| jpena | apevec: the puppet-murano commit we have in passed-ci is the latest in stable/pike | 08:08 |
| rdogerrit | Javier Peña created rdo-infra/ci-config master: Use the same volume size in all VM creation attempts https://review.rdoproject.org/r/9178 | 08:13 |
| jpena | ^^ will fix the 80 vs 100 GB discrepancy | 08:13 |
| jpena | now for murano | 08:13 |
| apevec | ykarel, right, 94ba1de == tag 4.0.0 == stable/pike head | 08:13 |
| apevec | ykarel, but what is the actual failure, anything in murano logs? | 08:14 |
| apevec | and who owns murano? let's reach out to them | 08:14 |
| jpena | apevec: the error is at the end of https://centos.logs.rdoproject.org/weirdo-generic-puppet-openstack-scenario003/2712/weirdo-project/puppet.txt.gz | 08:14 |
| jpena | but it doesn't make a lot of sense to me | 08:14 |
| apevec | maintainers: | 08:14 |
| apevec | - mathieu.velten@cern.ch | 08:14 |
| apevec | - asteroid566@gmail.com | 08:14 |
| apevec | jpena, yeah, that's on puppet side | 08:15 |
| jpena | is there another error on murano? | 08:15 |
| apevec | number80, ^ know anyone of maintaners? | 08:15 |
| ykarel | jpena, i couldn't see any | 08:15 |
| apevec | since we don't have murano expertise, I would drop it from release | 08:16 |
| apevec | unless maintainers show up | 08:16 |
| number80 | apevec: yeah drop it from GA, and I'll ping Igor | 08:16 |
| apevec | ykarel, is murano unconditional included in poi ? | 08:16 |
| jpena | I'm going to try a local VM install with this repo, so I can play around | 08:17 |
| ykarel | apevec, https://github.com/openstack/puppet-openstack-integration/blob/44a78a81fa47e93501ce4af22f803f909c346ff9/fixtures/scenario003.pp#L32 | 08:17 |
| *** openstackgerrit has quit IRC | 08:17 | |
| rdogerrit | Merged rdo-infra/ci-config master: Use the same volume size in all VM creation attempts https://review.rdoproject.org/r/9178 | 08:17 |
| *** mcornea has joined #rdo | 08:18 | |
| *** abregman|mtg is now known as abregman | 08:18 | |
| apevec | ykarel, jpena - so we need to override scenario003.pp - is there a way in weirdo to do that? | 08:18 |
| *** jhershbe has joined #rdo | 08:20 | |
| ykarel | apevec, don't think so, checking | 08:20 |
| jpena | apevec: not that I'm aware of. We could propose a patch to https://github.com/openstack/puppet-openstack-integration/blob/master/fixtures/scenario003.pp#L32, but that would require acceptance from the puppet people | 08:21 |
| apevec | yeah, and backport to stable/pike so unlikely | 08:21 |
| apevec | hey, why don't we package poi ? | 08:22 |
| apevec | so it's part of promoted set | 08:22 |
| apevec | now it's always from git | 08:22 |
| apevec | so poi changes could break us | 08:22 |
| apevec | with poi rpm, we could have patches like this | 08:22 |
| *** dpawar has quit IRC | 08:23 | |
| apevec | but also, having a place in weirdo to apply workaround would be good | 08:23 |
| apevec | otherwise, like in this case RDO cannot define what we want to ship | 08:24 |
| *** gfidente has joined #rdo | 08:25 | |
| ykarel | apevec, looks good | 08:29 |
| *** lucas-afk is now known as lucasagomes | 08:30 | |
| *** jhershbe has quit IRC | 08:30 | |
| apevec | ykarel, what looks good, there's a way to patch poi already? | 08:31 |
| apevec | I was looking at ansible-role-weirdo* but didn't find it yet | 08:32 |
| ykarel | apevec, to have rpm for poi | 08:32 |
| ykarel | apevec, i too didn't find | 08:32 |
| ykarel | apevec, apevec, to have rpm for poi is good to apply patches. | 08:32 |
| jehane | hi, I'm having a quick (and probably stupid) question | 08:33 |
| jehane | we have installed RDO Mitaka with ceph hammer | 08:33 |
| jehane | we are planning to upgrade ceph to jewel but we can't upgrade OpenStack yet. Is this a problem ? | 08:34 |
| jehane | is an openstack version locked on a ceph version ? | 08:35 |
| apevec | jehane, it's not tested in CI but iirc somebody reported running Mitaka w/ jewel | 08:35 |
| jehane | apevec: ok | 08:35 |
| jehane | do you remember if they were using ceph for nova and glance ? | 08:35 |
| number80 | jehane: you could try hitting the mailing-list | 08:37 |
| jehane | number80: thanks, I will try | 08:37 |
| number80 | what would be interesting is that people fill a page with their setup so we know what are the combination that have been known to work | 08:38 |
| apevec | jehane, found it, it was in one meeting minutes: http://eavesdrop.openstack.org/meetings/rdo_meeting___2017_01_18/2017/rdo_meeting___2017_01_18.2017-01-18-15.00.log.html | 08:39 |
| apevec | "Jewel is known to work with Mitaka but we are not testing it anywhere" | 08:39 |
| *** sfbender has quit IRC | 08:40 | |
| *** snarwade has quit IRC | 08:40 | |
| apevec | iirc it was mnaser running jewel with mitaka | 08:40 |
| apevec | jehane, so yes, ask on rdo-list, people will chime in | 08:40 |
| apevec | ykarel, jpena - we could https://github.com/rdo-infra/ansible-role-weirdo-puppet-openstack/blob/master/tasks/main.yml#L49 to insert patched poi, | 08:42 |
| apevec | where is refspec coming from, and could we override it in the job ? | 08:43 |
| apevec | it is normally via gerrit env vars right? | 08:43 |
| apevec | what if just set them and point to a poi fork? | 08:43 |
| *** tosky has joined #rdo | 08:44 | |
| *** karimb has joined #rdo | 08:45 | |
| *** yfried has quit IRC | 08:45 | |
| jpena | I'm trying to see where that comes from | 08:45 |
| jpena | I can't find that option in https://ci.centos.org/job/weirdo-generic-wrapper | 08:46 |
| *** jhershbe has joined #rdo | 08:46 | |
| *** nyechiel has quit IRC | 08:48 | |
| *** liujiong has quit IRC | 08:53 | |
| *** nyechiel has joined #rdo | 08:56 | |
| jpena | apevec: for a very quick hack, we could try and change https://github.com/rdo-infra/ansible-role-weirdo-puppet-openstack/blob/master/defaults/main.yml#L22 to point to a fork disabling murano... but I'm not so sure about that | 08:57 |
| *** MrAbaddon has quit IRC | 08:58 | |
| *** vnogin has joined #rdo | 09:08 | |
| * chandankumar is reformatting my laptop, too much hanging problem | 09:08 | |
| chandankumar | *me will be not available for few moments | 09:08 |
| *** MrAbaddon has joined #rdo | 09:11 | |
| jpena | I have reproduced the p-o-i scenario003 issue locally, now investigating | 09:11 |
| ykarel | jpena, Great | 09:14 |
| *** abregman is now known as abregman|mtg | 09:15 | |
| jpena | apevec: have we ever tested the new python-urllib3 package? | 09:16 |
| number80 | jpena: do you have any hints of failure? | 09:17 |
| number80 | I remember that python-requests broke openstack client builds in Fedora but not affecting runtime | 09:17 |
| *** milan has joined #rdo | 09:18 | |
| *** ushkalim has joined #rdo | 09:18 | |
| *** tosky has quit IRC | 09:18 | |
| jpena | number80: the p-o-i run fails when it tries to run a package prefetch for murano. When I ran "murano package-list" manually, urllib3 spit out some warning messages before showing its output, which *might* confuse the puppet provider (I saw that in the past) | 09:18 |
| *** snarwade has joined #rdo | 09:19 | |
| jpena | an earlier urllib3 version doesn't show this behavior | 09:19 |
| number80 | let's try remove urllib3/requests from GA review | 09:19 |
| number80 | apevec: ^ | 09:19 |
| jpena | https://review.rdoproject.org/etherpad/p/murano-failure <- this is the message I get | 09:20 |
| *** tosky has joined #rdo | 09:20 | |
| jpena | I'll check if it's possible to ignore the warnings in puppet-murano, I remember seeing something like this in the past | 09:21 |
| *** apetrich has quit IRC | 09:21 | |
| apevec | number80, no, requests/urllib3 are g-r ! | 09:21 |
| apevec | we need those | 09:21 |
| apevec | if they don't work, then min g-r is wrong | 09:21 |
| apevec | and we need to update to u-c | 09:22 |
| number80 | ack, but I wouldn't be surprised that they behave differently in our environment and spew araning | 09:22 |
| number80 | especially with certificates handling which is patched | 09:22 |
| number80 | it can be a separate review that we won't merge just to run CI jobs | 09:23 |
| number80 | (and compare results to acertain that is the root cause) | 09:23 |
| apevec | jpena, warning msg affecting puppet provider is a bug in puppet module | 09:23 |
| apevec | number80, ok, let's drop requests urllib3 from GA | 09:24 |
| apevec | we'll push it separately post GA | 09:24 |
| number80 | ack | 09:24 |
| apevec | after puppet module(s) are fixed | 09:24 |
| rdogerrit | hguemar proposed rdoinfo master: Push all pike-testing to pike-release https://review.rdoproject.org/r/9162 | 09:24 |
| number80 | ^ unrelated to the topic we just discussed | 09:24 |
| * apevec checks upstream u-c | 09:25 | |
| apevec | pike u-c: requests===2.18.2 urllib3===1.22 | 09:26 |
| apevec | I'll get those in cbs and prepare pike-update | 09:26 |
| apevec | but let's just get this pike-testing push out | 09:26 |
| apevec | we already missed regular 9am centos sign&push :( | 09:27 |
| *** vnogin has quit IRC | 09:32 | |
| *** vnogin has joined #rdo | 09:33 | |
| number80 | ok, so if we get new update, I can just submit reviews | 09:33 |
| * number80 was trying to split it up | 09:33 | |
| rdogerrit | hguemar proposed rdoinfo master: Push all pike-candidate to pike-testing https://review.rdoproject.org/r/9161 | 09:33 |
| rdogerrit | hguemar proposed rdoinfo master: Push all pike-testing to pike-release https://review.rdoproject.org/r/9162 | 09:33 |
| apevec | number80, yeah, not going to push this into GA ! | 09:36 |
| jpena | number80: the latest patchset for 9161 is re-adding some previous errors (puppet-stdlib and watcher-tests-tempest) | 09:37 |
| apevec | one rdoinfo CI run takes 2-3h | 09:37 |
| apevec | number80, you have out of date local review, git-review -d 9161 | 09:38 |
| apevec | then edit | 09:38 |
| apevec | hm, but now you need to restore ps5 first | 09:39 |
| apevec | jpena, go ahead and push the edit, since you did ps5 | 09:39 |
| rdogerrit | Javier Peña proposed rdoinfo master: Push all pike-candidate to pike-testing https://review.rdoproject.org/r/9161 | 09:40 |
| *** zz9pzza has quit IRC | 09:40 | |
| apevec | jpena, ec2-api was reverted too https://review.rdoproject.org/r/#/c/9161/6..7/rdo.yml | 09:41 |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: nova: http://trunk.rdoproject.org/centos7-master/report.html | 09:42 |
| rdogerrit | Javier Peña proposed rdoinfo master: Push all pike-candidate to pike-testing https://review.rdoproject.org/r/9161 | 09:42 |
| apevec | ok ps9 is good | 09:43 |
| *** dpawar has joined #rdo | 09:47 | |
| *** MrAbaddon has quit IRC | 09:49 | |
| *** tumble has quit IRC | 09:51 | |
| apevec | number80, your comment about watcher-tests-tempest under TODO https://review.rdoproject.org/etherpad/p/pike-release-preparation - that's resolved now, right? | 09:52 |
| jpena | apevec: I'm running some tests, can confirm the murano issue is due to the urllib3 warning confusing puppet-murano | 09:53 |
| jpena | I'll work on a patch | 09:53 |
| apevec | thanks | 09:53 |
| apevec | but, this will be post GA | 09:53 |
| apevec | I also wonder why it doesn't show upstream? | 09:53 |
| jpena | they don't have the updated urllib3 maybe? | 09:54 |
| apevec | we'll see if u-c version is better | 09:54 |
| apevec | jpena, upstream should run with even higher version from u-c | 09:54 |
| *** wolverineav has joined #rdo | 09:54 | |
| jpena | apevec: but not p-o-i, that runs with rpms | 09:55 |
| number80 | yeah, it's solved now | 09:55 |
| apevec | ok, then murano gate | 09:55 |
| apevec | ah nm, that's not testing puppet... | 09:55 |
| apevec | so yes, puppet was never tested with updated requests/urllib3 | 09:56 |
| apevec | jpena, and I guess debian has older versions too? | 09:56 |
| apevec | correction, ubuntu | 09:56 |
| jpena | apevec: even better, they don't run murano tests on ubuntu | 09:57 |
| jpena | https://github.com/openstack/puppet-openstack-integration/blob/master/fixtures/scenario003.pp#L23 | 09:57 |
| apevec | a-ha | 09:57 |
| apevec | oh well, so we don't care that much about murano but will end up fixing it | 09:58 |
| *** sbonazzo is now known as sbonazzo|afk | 09:59 | |
| * apevec sees opportunity for a motivational blog targeting package mainters: "80% of life is showing up" | 10:00 | |
| apevec | correction, 80% of _success_ :) https://quoteinvestigator.com/2013/06/10/showing-up/ | 10:02 |
| *** tumble has joined #rdo | 10:05 | |
| *** egallen has quit IRC | 10:05 | |
| *** abregman|mtg is now known as abregman | 10:09 | |
| jpena | apevec: https://review.openstack.org/499546 should help with the puppet-murano issue, let's see | 10:15 |
| *** apetrich has joined #rdo | 10:16 | |
| rdogerrit | Javier Peña proposed rdoinfo master: Push all pike-candidate to pike-testing https://review.rdoproject.org/r/9161 | 10:21 |
| *** udesale has quit IRC | 10:27 | |
| *** karimb has quit IRC | 10:28 | |
| *** richm has joined #rdo | 10:29 | |
| * chandankumar is back | 10:33 | |
| chandankumar | number80: apevec Anything blocker on tempest plugin side for RDO-PIke release? | 10:33 |
| number80 | chandankumar: I'm trying to figure if there aren't any hidden issue with them like the one we had yesterday | 10:35 |
| number80 | It may be faster if you check if they have no upgrade path issue | 10:36 |
| chandankumar | number80: sure | 10:36 |
| *** karimb has joined #rdo | 10:39 | |
| *** jkilpatr has quit IRC | 10:41 | |
| *** apetrich has quit IRC | 10:42 | |
| *** MrAbaddon has joined #rdo | 10:46 | |
| *** aditya_r has quit IRC | 10:48 | |
| *** rhallisey has joined #rdo | 10:50 | |
| *** raissa has quit IRC | 10:51 | |
| *** vnogin has quit IRC | 10:52 | |
| *** raissa has joined #rdo | 10:52 | |
| *** jhershbe has quit IRC | 10:52 | |
| chandankumar | number80: keystone barbican safe, i found no other issues with tempest plugins | 10:53 |
| number80 | excellent so we're good | 10:53 |
| * number80 now looking at CI "realtime" logs | 10:54 | |
| chandankumar | number80: one more thing we untagged watcher tempest plugin ? just to confirm na. | 10:54 |
| *** agurenko has quit IRC | 10:54 | |
| number80 | chandankumar: I haven't untagged it from -candidate, but it's safe | 10:54 |
| chandankumar | https://review.rdoproject.org/r/#/c/9161/10/rdo.yml i am not seeing here | 10:55 |
| chandankumar | so we are safe | 10:55 |
| chandankumar | number80: let's do the final push then we are good to go | 10:55 |
| apevec | number80, let's untagg it in CBS, otherwise bot will re-propose it | 10:56 |
| *** gszasz has joined #rdo | 10:56 | |
| *** agurenko has joined #rdo | 10:58 | |
| *** vnogin has joined #rdo | 10:59 | |
| *** vnogin has quit IRC | 10:59 | |
| *** Caterpillar has joined #rdo | 11:00 | |
| chandankumar | time to grab some food | 11:00 |
| *** jkilpatr has joined #rdo | 11:01 | |
| *** vnogin has joined #rdo | 11:01 | |
| *** gcerami has joined #rdo | 11:01 | |
| number80 | excellent idea | 11:02 |
| apevec | yes! food++ | 11:02 |
| *** sbonazzo|afk has quit IRC | 11:03 | |
| *** abregman is now known as abregman|mtg | 11:03 | |
| *** sbonazzo has joined #rdo | 11:04 | |
| apevec | jpena, https://review.rdoproject.org/jenkins/job/validate-buildsys-tags/710/ is running 44min already and still in setup phase... | 11:06 |
| jpena | apevec: it shouldn't take long | 11:06 |
| *** dpawar has quit IRC | 11:07 | |
| * apevec looks at timestamps in console | 11:07 | |
| apevec | started :21 | 11:07 |
| *** pkovar has joined #rdo | 11:07 | |
| apevec | downloading rpms from cbs until :57 | 11:08 |
| apevec | now it's doing test install all of them in chroot | 11:09 |
| *** Caterpillar has quit IRC | 11:09 | |
| *** cylopez has quit IRC | 11:10 | |
| *** jistr is now known as jistr|biab | 11:12 | |
| *** Caterpillar has joined #rdo | 11:12 | |
| ykarel | jpena, also check if there is enough volume quota for running weirdo jobs | 11:13 |
| ykarel | otherwise we have to wait longer | 11:14 |
| jpena | yep, the rdo-ci tenant is empty now | 11:14 |
| ykarel | jpena, ack | 11:14 |
| jpena | validate-buildsys-tags is done! | 11:15 |
| jpena | now for weirdo jobs | 11:15 |
| *** Caterpillar has joined #rdo | 11:16 | |
| *** Caterpillar has quit IRC | 11:16 | |
| *** morazi has quit IRC | 11:17 | |
| *** Caterpillar has joined #rdo | 11:22 | |
| jpena | damn, p-o-i scenario002 has failed due to a network issue -> https://ci.centos.org/job/weirdo-generic-puppet-openstack-scenario002/2706/console | 11:23 |
| *** jhershbe has joined #rdo | 11:25 | |
| *** eaguilar has quit IRC | 11:34 | |
| *** egallen has joined #rdo | 11:35 | |
| *** thrash|g0ne is now known as thrash | 11:41 | |
| *** amuller has joined #rdo | 11:41 | |
| chandankumar | rbowen: hello | 11:43 |
| chandankumar | rbowen: we need some help in advertizing about https://opensourceindia.in/osidays/open-source-india-2017/ from RDO side. | 11:43 |
| chandankumar | rbowen: once i get the content, i will pass it to you. | 11:44 |
| apevec | jpena, again socket.error: [Errno 104] Connection reset by peer in wait_for :( | 11:44 |
| *** lucasagomes is now known as lucas-hungry | 11:44 | |
| jpena | apevec: let's increase the delay in https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/create-vm.sh#L151 ? | 11:46 |
| jpena | to 30 seconds, for example | 11:46 |
| jpena | that would give us some breathing room | 11:46 |
| *** shreshtha has quit IRC | 11:51 | |
| *** abregman|mtg is now known as abregman | 11:52 | |
| *** adarazs is now known as adarazs_brb | 11:53 | |
| *** apetrich has joined #rdo | 11:54 | |
| *** thegreenhundred has quit IRC | 11:55 | |
| jpena | dmsimard: around? | 11:55 |
| dmsimard | jpena: half awake | 11:55 |
| *** eaguilar has joined #rdo | 11:55 | |
| dmsimard | Sure, go for 30 | 11:55 |
| jpena | dmsimard: quick question, then. Why are we using search_regex in https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/create-vm.sh#L150 ? | 11:55 |
| number80 | jpena: that's a good idea | 11:56 |
| jpena | I understand we're trying to make sure openssh is avaliable, but... Wouldn't it be enough just to check for the port? | 11:56 |
| number80 | (the 30s thing) | 11:56 |
| dmsimard | Because that's how Ansible documents it should be done | 11:56 |
| dmsimard | Thte fact that the port is open is probably not enough | 11:57 |
| sshnaidm | hi, all, I wonder why hash of "consistent" repo on tripleo (https://trunk.rdoproject.org/centos7/consistent/delorean.repo) doesn't change during last 5 days, does somebody know? | 11:57 |
| dmsimard | I dunno. | 11:57 |
| dmsimard | sshnaidm: build failures and everyone is busy shipping pike | 11:57 |
| dmsimard | Consistent only moves forward when there are no build failures | 11:58 |
| jpena | ok, let's start with the 30 seconds delay, then I'll play with search_regex | 11:58 |
| sshnaidm | dmsimard, ok.. | 11:58 |
| dmsimard | sshnaidm: http://dlrn.readthedocs.io/en/latest/repositories.html | 11:58 |
| rdogerrit | Javier Peña created rdo-infra/ci-config master: Increate delay when waiting for a vm to boot to 30 https://review.rdoproject.org/r/9179 | 11:58 |
| dmsimard | sshnaidm: http://tinyurl.com/y8hxewm2 rdo/rdo | 11:59 |
| dmsimard | Feel free to contribute to fixing the builds :) | 11:59 |
| sshnaidm | dmsimard, and what to look there? CHECK-DELOREAN-MASTER-CURRENT ? | 12:00 |
| sshnaidm | dmsimard, I have barely idea how this works.. | 12:00 |
| dmsimard | sshnaidm: yeah: https://uchiwa.monitoring.rdoproject.org/uchiwa/#/client/uchiwa/master.monitoring.rdoproject.org?check=check-delorean-master-current | 12:01 |
| *** nijaba has quit IRC | 12:01 | |
| sshnaidm | dmsimard, ok, I see nova fails. And where all logs, builds, etc..? | 12:01 |
| dmsimard | sshnaidm: https://trunk.rdoproject.org/centos7-master/report.html | 12:01 |
| dmsimard | See failures | 12:01 |
| dmsimard | I need to afk for a while, be back later | 12:02 |
| * number80 thinks it'd be nice to have a log server with "realtime" visualisation | 12:02 | |
| *** vnogin has quit IRC | 12:02 | |
| *** nyechiel has quit IRC | 12:02 | |
| number80 | Jenkins UI sucks big time for that purpose | 12:02 |
| dmsimard | number80: ELK coming to review.rdo | 12:03 |
| *** mvk has joined #rdo | 12:03 | |
| sshnaidm | oook.. so we have "ImportError: No module named blockdiag.imagedraw", seems like we need dependency on python-blockdiag | 12:03 |
| number80 | dmsimard: will it be public accessible? | 12:03 |
| rdogerrit | Merged rdo-infra/ci-config master: Increate delay when waiting for a vm to boot to 30 https://review.rdoproject.org/r/9179 | 12:03 |
| number80 | sshnaidm: which package? | 12:03 |
| *** jkilpatr has quit IRC | 12:03 | |
| *** morazi has joined #rdo | 12:04 | |
| sshnaidm | number80, openstack-nova | 12:04 |
| number80 | ... | 12:04 |
| sshnaidm | number80, ? | 12:04 |
| number80 | I don't see where it's importing blockdiag and it was never detected by trunk CI | 12:05 |
| * number80 grepped nova code and no reference to blockdiag | 12:05 | |
| sshnaidm | number80, hmm.. I look here trunk.rdoproject.org/centos7-master/e9/da/e9daa28bae07fff7e617f03ef6b4956148db17f2_c34374a8/rpmbuild.log | 12:05 |
| * number80 will look | 12:05 | |
| sshnaidm | number80, oh, maybe you're right.. | 12:06 |
| sshnaidm | number80, it happens when /usr/bin/python2 setup.py build_sphinx | 12:06 |
| number80 | likely sphinx or a sphinx extension issue | 12:07 |
| *** karimb has quit IRC | 12:07 | |
| sshnaidm | so it's actually sphinx problem, yeah | 12:07 |
| rdogerrit | chkumar246 created openstack/tempest-distgit pike-rdo: Switch to tempest-17.0.0 https://review.rdoproject.org/r/9180 | 12:07 |
| sshnaidm | number80, shouldn't we still add this as dep..? I don't know what is the process.. | 12:08 |
| number80 | it's not oslo.sphinx | 12:08 |
| jpena | p-o-i scenario003 ended up successfully, so the lower urllib3 version did the trick for murano | 12:08 |
| chandankumar | myoung: tosky tbarron jschlueter https://review.rdoproject.org/r/#/c/9180/ and https://review.rdoproject.org/r/9176 tempest-17.0.0 let's make manila happy | 12:08 |
| dmsimard | number80: sure | 12:08 |
| number80 | sshnaidm: first investigating which package actually needs it, if it is sphinx or an extension, it might come back in another package build later | 12:09 |
| *** jkilpatr has joined #rdo | 12:09 | |
| *** jbadiapa has quit IRC | 12:09 | |
| *** vnogin has joined #rdo | 12:10 | |
| *** vnogin has quit IRC | 12:11 | |
| *** adarazs_brb is now known as adarazs | 12:11 | |
| *** jkilpatr has quit IRC | 12:12 | |
| sshnaidm | number80, it's sphinxcontrib-blockdiag extension for sphinx | 12:12 |
| *** jkilpatr has joined #rdo | 12:12 | |
| sshnaidm | number80, not sure I understood " it might come back in another package build later" | 12:12 |
| *** jlabarre has joined #rdo | 12:12 | |
| *** vnogin has joined #rdo | 12:12 | |
| ykarel | number80, sshnaidm https://review.openstack.org/#/c/476159/ introduced blockdiag | 12:12 |
| *** scorcoran has joined #rdo | 12:13 | |
| *** vnogin has quit IRC | 12:14 | |
| sshnaidm | ykarel, hmm.. shouldn't it be added to extensions in beginning of file (line 36)? extensions = ['sphinx.ext.autodoc' ... | 12:14 |
| number80 | ykarel: so, it's not pike | 12:15 |
| * number80 pushes it to the background of the queue | 12:15 | |
| number80 | Pike GA first, then post-GA, and the rest later :) | 12:15 |
| ykarel | number80, yes it's not in pike | 12:15 |
| ykarel | number80, yeah | 12:16 |
| jpena | I'm going to grab lunch. Someone please recheck https://review.rdoproject.org/r/9161 once it finishes the current run | 12:16 |
| *** jpena is now known as jpena|lunch | 12:16 | |
| ykarel | number80, can we disable doc and man building for nova to move forward | 12:16 |
| ykarel | promote consitent | 12:16 |
| *** jkilpatr has quit IRC | 12:17 | |
| number80 | ykarel: no add the dependency in nova package | 12:17 |
| ykarel | number80, i think it's not there yet | 12:18 |
| ykarel | i think we need to update deps.yml | 12:18 |
| ykarel | isn't it | 12:18 |
| *** jayg|g0n3 is now known as jayg | 12:18 | |
| *** vnogin has joined #rdo | 12:19 | |
| ykarel | looks like it's there: http://cbs.centos.org/koji/packageinfo?packageID=5029 | 12:19 |
| number80 | yeah, but it's a question of hours | 12:19 |
| number80 | it will be included in pike GA push | 12:19 |
| *** vnogin has quit IRC | 12:19 | |
| number80 | I don't think that a lack of Queens promotions at this moment will be blocking anyone | 12:20 |
| *** jistr|biab is now known as jistr | 12:20 | |
| dmsimard | Yeah, let's focus on getting Pike out | 12:20 |
| dmsimard | If we miss today's opportunity to push and sign, we're delaying to monday and that would be awful | 12:20 |
| dmsimard | And also the first time in years that we've been delayed so much | 12:21 |
| *** vnogin has joined #rdo | 12:21 | |
| dmsimard | Usually RDO is out within hours of upstream release, we're already the day before | 12:21 |
| dmsimard | Queens is not important right now, sorry. | 12:21 |
| number80 | damn job failed after 2 hours | 12:21 |
| number80 | it's poi 2 | 12:22 |
| *** vnogin has quit IRC | 12:22 | |
| dmsimard | Dropping kid at school and I'll try to help even though my allergies are killing me | 12:22 |
| *** imcleod has joined #rdo | 12:22 | |
| *** vnogin has joined #rdo | 12:23 | |
| *** jprovazn has quit IRC | 12:24 | |
| *** pchavva has joined #rdo | 12:24 | |
| ykarel | number80, i think jpena's fix will work this time | 12:24 |
| *** jhershbe has quit IRC | 12:24 | |
| dmsimard | Be back in ~30 mins or so | 12:24 |
| *** vnogin has quit IRC | 12:25 | |
| number80 | ok | 12:26 |
| *** jbadiapa has joined #rdo | 12:26 | |
| *** dprince has joined #rdo | 12:26 | |
| *** jprovazn has joined #rdo | 12:27 | |
| *** jkilpatr has joined #rdo | 12:30 | |
| *** noslzzp has joined #rdo | 12:30 | |
| *** imcleod has quit IRC | 12:31 | |
| *** vnogin has joined #rdo | 12:33 | |
| *** vnogin has quit IRC | 12:34 | |
| *** eharney has joined #rdo | 12:34 | |
| *** vnogin has joined #rdo | 12:35 | |
| *** rlandy has joined #rdo | 12:35 | |
| *** vnogin has quit IRC | 12:36 | |
| *** jaosorior has quit IRC | 12:37 | |
| *** pgadiya has quit IRC | 12:37 | |
| *** vnogin has joined #rdo | 12:38 | |
| *** thegreenhundred has joined #rdo | 12:38 | |
| *** jaosorior has joined #rdo | 12:38 | |
| *** vnogin has quit IRC | 12:39 | |
| *** vnogin has joined #rdo | 12:39 | |
| *** gkadam_ has quit IRC | 12:40 | |
| *** vnogin has quit IRC | 12:40 | |
| *** nyechiel has joined #rdo | 12:41 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: nova: http://trunk.rdoproject.org/centos7-master/report.html | 12:42 |
| rbowen | What's the status this morning. I gather it's not going as well as last time, hmm? | 12:42 |
| *** vnogin has joined #rdo | 12:42 | |
| apevec | rbowen, still various CI issues | 12:42 |
| apevec | jpena|lunch, have you increased timeout i.e. can I recheck 9161 ? | 12:43 |
| ykarel | apevec, yes | 12:43 |
| ykarel | apevec, https://review.rdoproject.org/r/#/c/9179/ | 12:43 |
| ykarel | here ^^ increase | 12:43 |
| *** jaosorior has quit IRC | 12:44 | |
| *** jaosorior has joined #rdo | 12:45 | |
| *** jaosorior has quit IRC | 12:45 | |
| *** jaosorior has joined #rdo | 12:46 | |
| apevec | ok, /me rechecked | 12:46 |
| apevec | damn timeouts | 12:47 |
| *** rhallisey has quit IRC | 12:47 | |
| *** noslzzp has quit IRC | 12:47 | |
| apevec | imho such atomic timeouts should be ad-inf, then have external watchdog for the whole job | 12:48 |
| *** noslzzp has joined #rdo | 12:49 | |
| rdogerrit | Attila Darazs created config master: tripleo: rename images-build to containers build https://review.rdoproject.org/r/9181 | 12:50 |
| *** jmelvin has joined #rdo | 12:50 | |
| *** lucas-hungry is now known as lucasagomes | 12:51 | |
| *** raissa has quit IRC | 12:52 | |
| *** abregman is now known as abregman|mtg | 12:53 | |
| dmsimard | apevec: hey, I mean, the cloud infrastructure we're using could be stable and reliable too | 12:54 |
| dmsimard | That'd be great :) | 12:54 |
| dmsimard | Anyone want to give me a tl;dr of the situation ? | 12:54 |
| dmsimard | KB's deadline seems scary | 12:54 |
| *** aditya_r has joined #rdo | 12:55 | |
| *** trown|outtypewww is now known as trown | 12:57 | |
| *** raissa has joined #rdo | 12:58 | |
| apevec | dmsimard, what's the deadline? | 12:58 |
| dmsimard | apevec: see scrollback in #centos-devel | 12:59 |
| apevec | dmsimard, tl;dr just rechecked with increased timeout | 12:59 |
| apevec | dmsimard, don't see it | 12:59 |
| apevec | when was it? | 12:59 |
| *** udesale has joined #rdo | 13:00 | |
| apevec | I only see number80 and kbsingh discussing how little sleep they both had last night :) | 13:00 |
| dmsimard | apevec: http://paste.openstack.org/raw/620058/ | 13:00 |
| dmsimard | that was 2h30 ago | 13:01 |
| apevec | dmsimard, yeah, 4h is refering to the sleep length | 13:01 |
| *** emalikov has quit IRC | 13:01 | |
| dmsimard | oh | 13:01 |
| * dmsimard tired | 13:01 | |
| apevec | but realistical deadline is indeed in next few hours | 13:01 |
| dmsimard | didn't get much sleep either, I'M going to die of allergies | 13:01 |
| apevec | so current recheck MUST succeed... | 13:01 |
| *** sasha2 has joined #rdo | 13:02 | |
| dmsimard | What is failing ? CI jobs ? | 13:02 |
| *** sasha2 has quit IRC | 13:02 | |
| number80 | apevec: there will be no automated push for few days so we have to ping KB (that was the main topic of the sleepless folks discussion ) | 13:02 |
| apevec | murano was fixed by removing requests/urllib3 update | 13:02 |
| *** sasha2 has joined #rdo | 13:02 | |
| apevec | jpena has the fix in puppet-murano | 13:02 |
| apevec | we'll push that after ga | 13:02 |
| dmsimard | I can hold nodes as required | 13:02 |
| dmsimard | so that we can troubleshoot | 13:02 |
| apevec | +1 | 13:03 |
| dmsimard | that's what we did last night for a couple things | 13:03 |
| dmsimard | the validate buildsys job isn't very good at telling us what's wrong, I added a todo | 13:03 |
| apevec | last failure was wait_for issue which increased timeout should cover | 13:03 |
| *** Goneri has joined #rdo | 13:03 | |
| apevec | yeah, you need to look into each "child" weirdo job | 13:03 |
| kbsingh | apevec: my lack of sleep was an openshift problem :) but happy to keep an eye on things for you as well | 13:04 |
| *** scorcoran is now known as scorcoran_biab | 13:05 | |
| dmsimard | apevec: did oooq work ? | 13:05 |
| dmsimard | did we ship a centos-release pkg ? | 13:05 |
| *** imcleod has joined #rdo | 13:05 | |
| apevec | I've tagged it for extras | 13:06 |
| *** jprovazn has quit IRC | 13:06 | |
| apevec | lemme check if it got pushed, not sure if that's automated | 13:06 |
| *** jprovazn has joined #rdo | 13:06 | |
| *** lkuchlan_wfh has quit IRC | 13:06 | |
| apevec | it's -0 w/ testing enabled, for now | 13:06 |
| *** rhallisey has joined #rdo | 13:08 | |
| number80 | it is automated | 13:08 |
| * number80 thinks it's checked every 4h or so | 13:09 | |
| dmsimard | number80: we got told yesterday extras was managed by hughes | 13:09 |
| number80 | Yep, Johnny | 13:09 |
| dmsimard | so not sure if he has to do something | 13:09 |
| number80 | as long as it's tagged, it's ok | 13:10 |
| apevec | it is tagged http://cbs.centos.org/koji/buildinfo?buildID=19640 | 13:11 |
| *** imcleod has quit IRC | 13:11 | |
| mnaser | apevec we were indeed running jewel with mitaka | 13:11 |
| apevec | number80, but not in http://mirror.centos.org/centos/7/extras/x86_64/Packages/?C=M;O=D | 13:12 |
| mnaser | we had to manually add the repos because centos-release-openstack-mitaka pulls in centos-release-ceph-hammer and causes a conflict when trying to install centos-release-ceph-jewle | 13:12 |
| apevec | mnaser, thanks for confirming, so my brain cells still somewhat work :) | 13:12 |
| dmsimard | mnaser: oi, btw 10.2.9 is coming soon, sometime next week hopefully | 13:13 |
| dmsimard | mnaser: and someone is working on luminous | 13:13 |
| mnaser | apevec np :) | 13:14 |
| mnaser | dmsimard nice! im excited for bluestore | 13:14 |
| * number80 holding breath as validate-buildsys-tags is almost finished weirdo-validate-buildsys-tags is on starting block | 13:14 | |
| *** udesale has quit IRC | 13:15 | |
| dmsimard | I'll lock all the weirdo nodes | 13:15 |
| dmsimard | so that they don't get released | 13:15 |
| *** artom has quit IRC | 13:15 | |
| *** pradk has joined #rdo | 13:15 | |
| *** dustins has joined #rdo | 13:16 | |
| mnaser | are tagged releases out? i mean we're running the trunk ones which are the same bit for bit but just having clean release names is nice :p | 13:16 |
| apevec | mnaser, running CI on them | 13:17 |
| apevec | builds are done in cbs | 13:17 |
| mnaser | oh sweet :) | 13:17 |
| apevec | mnaser, watch https://review.rdoproject.org/r/9161 for results | 13:18 |
| mnaser | thank you i'll add myself to that review to get free email updates :p | 13:18 |
| dmsimard | mnaser: some of us are sleep deprived but we'll get it out | 13:18 |
| mnaser | no rush... the packages are all out there already and you guys have technically been building pike everyday anyways :p | 13:19 |
| mnaser | we wouldn't have been able to upgrade so early / quickly if trunk.rdoproject.org wasn't a thing | 13:19 |
| *** jpena|lunch is now known as jpena | 13:20 | |
| apevec | "We shall CI our release, whatever the cost may be!" | 13:20 |
| jpena | so the previous run only had a failure in p-o-i scenario002, we're in good shape | 13:20 |
| * jpena crosses fingers | 13:20 | |
| *** jprovazn has quit IRC | 13:20 | |
| * apevec knocks wood | 13:20 | |
| dmsimard | mnaser: ah I was wondering how you upgraded so quickly | 13:21 |
| dmsimard | mnaser: be warned that upgrades from trunk to stable aren't tested, however, you're on your own for that | 13:21 |
| mnaser | dmsimard releases.openstack.org -- grab the hash of the release, get it from trunk, install pike deps repo and repo for that hash, install, voila | 13:22 |
| jpena | mnaser: while we wait for the CI job to finish, how's your Software Factory installation going? :) | 13:22 |
| dmsimard | mnaser: for example, you might end up rolling back packages because you're getting things that landed in stable/pike but were not released in the tarballs | 13:22 |
| mnaser | dmsimard i see. shouldn't be too much of an issue (we'll test it out) | 13:22 |
| dmsimard | mnaser: the "deps" repo contains -testing which is not populated with openstack builds during the cycle but it is fully populated before we tag -release | 13:23 |
| mnaser | jpena good! we took the zuul route so took the jenkins out of it and it "just works" .. took a little while for everyone to get used to it but its all good | 13:23 |
| dmsimard | mnaser: i.e, -testing is not self-contained and meant to work with trunk repos until ~release | 13:23 |
| dmsimard | mnaser: but when release draws near, you can use -testing by itself | 13:23 |
| apevec | dmsimard, not until we push 9161 | 13:24 |
| *** kgiusti has joined #rdo | 13:24 | |
| apevec | it's still only deps atm in pike-testing | 13:24 |
| jpena | nice | 13:24 |
| *** ratailor has quit IRC | 13:24 | |
| dmsimard | oh hey I just got this email | 13:24 |
| mnaser | dmsimard i'm not too worried. as long as the service itself is built from the same hash that we're running, libraries won't be that much of an issue | 13:24 |
| dmsimard | Congratulations David, We're celebrating 2 Years of your service and dedication to Red Hat and our mission. Thanks for contributing to our success and helping to make Red Hat the best place to make a difference together! | 13:24 |
| dmsimard | \o/ | 13:24 |
| mnaser | :D | 13:25 |
| dmsimard | mnaser: oh, yeah, it's just a disclaimer kind of statement | 13:25 |
| jpena | congrats dmsimard ;) | 13:25 |
| dmsimard | mnaser: fine print if you will :) | 13:25 |
| *** emalikov has joined #rdo | 13:25 | |
| mnaser | dmsimard i thought my SLA covered full support for any RDO issues (ha-ha-ha) | 13:25 |
| *** gszasz has quit IRC | 13:29 | |
| dmsimard | mnaser: who's SLA, yours to your customers? :P | 13:29 |
| mnaser | lols | 13:29 |
| *** scorcoran_biab is now known as scorcoran | 13:30 | |
| *** scorcoran is now known as scorcoran_mtg | 13:30 | |
| sshnaidm | how do I submit changed into https://github.com/rdo-infra/ci-config? I try via https://review.rdoproject.org/r/#/admin/projects/rdo-infra/ci-config but it show me a lot of commits to send.. I'm confused | 13:31 |
| rbowen | dmsimard: 2 years!? Awesome. Congratulations. | 13:31 |
| jpena | sshnaidm: just git review, it should work | 13:32 |
| * number80 considering to rewrite some jobs in rust | 13:32 | |
| number80 | more realistically, it would benefit them to be rewritten in python | 13:33 |
| number80 | some stuff could easily be made concurrent | 13:33 |
| *** jcoufal has joined #rdo | 13:34 | |
| dmsimard | sshnaidm: you clone it, do a commit and do a git review ? shouldn't be any different than any other gerrit project | 13:35 |
| *** jhershbe has joined #rdo | 13:35 | |
| *** abregman|mtg is now known as abregman | 13:36 | |
| sshnaidm | dmsimard, seems I didn't submit changes yet to new gerrit.. | 13:39 |
| *** MrAbaddon has quit IRC | 13:39 | |
| jpena | weirdo jobs starting | 13:39 |
| remix_tj | rbowen: a quick question about the contributor list in pike release announcement: is list of people that specifically has worked on packaging for RDO? | 13:40 |
| rbowen | It *should* be anybody that did anything on RDO that touched git in some way. However, we did note that some people were missing. | 13:40 |
| rbowen | Notably myself and chandankumar. | 13:40 |
| dmsimard | jpena: thanks, locking the VMs | 13:41 |
| rbowen | remix_tj: Is someone else missing? | 13:41 |
| *** fultonj has joined #rdo | 13:42 | |
| rdogerrit | Merged config master: tripleo: rename images-build to containers build https://review.rdoproject.org/r/9181 | 13:42 |
| jpena | all 6 VMs have been created without issue \o/ | 13:43 |
| *** cylopez has joined #rdo | 13:43 | |
| apevec | yay | 13:43 |
| apevec | jpena, did I imagine or there was a blog from you or Alfredo about deps management in RDO ? | 13:44 |
| apevec | or is it in docs somewhere? | 13:44 |
| *** imcleod has joined #rdo | 13:44 | |
| jpena | apevec: I guess Alfredo did it | 13:44 |
| * jpena checks | 13:44 | |
| remix_tj | rbowen: i don't know. I was only looking at the list. Every time i see a contributor list on something i use, i grep for my name or nick to check if i forgot that i made some contribution :-P | 13:45 |
| *** cylopez1 has joined #rdo | 13:45 | |
| *** shreshtha has joined #rdo | 13:45 | |
| *** gszasz has joined #rdo | 13:45 | |
| *** cylopez has quit IRC | 13:47 | |
| *** danpawlik has quit IRC | 13:50 | |
| *** ihrachys has joined #rdo | 13:50 | |
| rdogerrit | Sagi Shnaidman created rdo-infra/ci-config master: Check URL of promotion every hour https://review.rdoproject.org/r/9182 | 13:50 |
| chandankumar | remix_tj: you can grep with your username, if it is missing, feel free to add. | 13:51 |
| jpena | apevec: I can't find the blog post, maybe Alfredo didn't send it | 13:51 |
| remix_tj | i think i haven't done any contribution directly to RDO, but only to tripleo | 13:52 |
| apevec | jpena, yeah, not sure where it was | 13:53 |
| chandankumar | dmsimard: https://www.youtube.com/watch?v=yVHviEAZ1xg it will refresh your mind. :-) | 13:53 |
| *** mpeterson has quit IRC | 13:53 | |
| rdogerrit | Sagi Shnaidman proposed rdo-infra/ci-config master: Check URL of promotion every hour https://review.rdoproject.org/r/9182 | 13:53 |
| *** gszasz has quit IRC | 13:53 | |
| *** gszasz has joined #rdo | 13:55 | |
| *** josecastroleon has quit IRC | 13:57 | |
| *** jaosorior has quit IRC | 13:57 | |
| *** jaosorior has joined #rdo | 13:59 | |
| *** vnogin has quit IRC | 13:59 | |
| *** josecastroleon has joined #rdo | 13:59 | |
| apevec | sshnaidm, number80 - I would leave https://review.rdoproject.org/r/9030 (nova master ftbfs) to Nova maintaers to act on | 14:03 |
| apevec | otherwise, they've never an opportunity when we fix their stuff | 14:03 |
| apevec | they all got gerrit notification | 14:04 |
| apevec | (opportunity to learn about deps.yml etc.) | 14:05 |
| *** snarwade has quit IRC | 14:05 | |
| *** MrAbaddon has joined #rdo | 14:05 | |
| *** jobewan has joined #rdo | 14:05 | |
| *** scorcoran_mtg is now known as scorcoran | 14:07 | |
| *** scorcoran is now known as scorcoran_biab | 14:07 | |
| sshnaidm | apevec, is there somebody I can ping them..? | 14:08 |
| dmsimard | apevec: lol have you ever seen compute team fix a nova ftbfs ? | 14:10 |
| *** vnogin has joined #rdo | 14:10 | |
| jpena | apevec: regarding the nova ftbfs, I think it's sphinxcontrib.actdiag, not blockdiag | 14:10 |
| jpena | DEBUG: Could not import extension sphinxcontrib.actdiag (exception: No module named actdiag) | 14:11 |
| *** marrusl has quit IRC | 14:11 | |
| dmsimard | ooo pike failed on missing release package again | 14:13 |
| dmsimard | I guess it's not out yet | 14:13 |
| number80 | apevec: well, speaking for myself, I'm not even looking to master | 14:14 |
| * number80 looked only to be sure if it wasn't affecting Pike | 14:14 | |
| jpena | 2/6 succeeded, 4 still going | 14:15 |
| *** scorcoran_biab is now known as scorcoran | 14:16 | |
| *** abregman is now known as abregman|mtg | 14:16 | |
| *** jhershbe has quit IRC | 14:17 | |
| mnaser | jpena i see im not the only one that has a bunch of tabs open :) | 14:18 |
| dmsimard | trown: can we manually get some oooq coverage going for pike release testing ? | 14:18 |
| *** imcleod has quit IRC | 14:18 | |
| dmsimard | trown: we're struggling to get it to work because with our current implementation it's expecting centos-release-openstack-pike to exist, which does not (yet) | 14:19 |
| trown | dmsimard: what do you mean by manual? | 14:19 |
| dmsimard | trown: anything that isn't in the gate for https://review.rdoproject.org/r/#/c/9161/ | 14:20 |
| *** imcleod has joined #rdo | 14:20 | |
| *** Amita has joined #rdo | 14:22 | |
| trown | dmsimard: I need more to go on... I should try to figure out how to get the pike pre-release bits on my local machine and test? or I need to setup some quick job? not exactly sure what the ask is | 14:22 |
| dmsimard | apevec, jpena, number80: I guess we could compromise on at least tagging -testing without ooo(q) coverage to at least get the bits in -testing and making it easier to test ooo(q) out of band | 14:23 |
| jpena | +1 for me | 14:23 |
| jpena | all packstack jobs went fine, now waiting for p-o-i | 14:24 |
| trown | +1 here too | 14:24 |
| dmsimard | We can have reasonable expectations that if weirdo is fully passing, ooo(q) will pass -- pike has been generally green in the pipeline for a while | 14:24 |
| number80 | dmsimard: that's already the case | 14:24 |
| dmsimard | but it's still required before we tag for release | 14:24 |
| number80 | Yeah, but GA means we still have to tag them in -release | 14:24 |
| hamzy | remix_tj, FYI there is definitely an issue with ironic booting the nodes with ppc64le and petitboot, so I am investigating how to do it properly... | 14:26 |
| number80 | ack | 14:26 |
| mnaser | so while we're waiting for p-o-i... magnum in pike relies on some stuff that far down the dependency chain require python-backports-ssl_match_hostname 3.5.0 | 14:27 |
| mnaser | or else it raises exceptions in the log all day long | 14:27 |
| mnaser | is there a way we can get that into rdo (el7 seems to ship 3.4.0) | 14:27 |
| number80 | mnaser: is it breaking everything? | 14:27 |
| number80 | if it's only cluttering logs, it can be a 0day update | 14:27 |
| mnaser | number80 the feature of retrieving stats doesn't work that's all | 14:28 |
| mnaser | but that's not even fully integrated | 14:28 |
| mnaser | so its not critical | 14:28 |
| number80 | ok, let's document it and I'll prepare that update | 14:28 |
| jpena | anyway, 3.5.0 is in fedora | 14:28 |
| mnaser | let me get you more info | 14:28 |
| *** imcleod has quit IRC | 14:29 | |
| * number80 is running a scratch build | 14:30 | |
| mnaser | i have the exact patch that fixed the issue for us, let me get you info | 14:30 |
| number80 | Yep | 14:30 |
| number80 | chandankumar, rbowen: can you point me to the release notes? Let's document known issues | 14:31 |
| chandankumar | number80: https://github.com/redhat-openstack/website/blob/master/source/blog/2017-08-30-rdo-pike-released.html.md | 14:32 |
| rbowen | chandankumar: https://github.com/redhat-openstack/website/blob/master/source/blog/2017-08-30-rdo-pike-released.html.md | 14:32 |
| *** Son_Goku has quit IRC | 14:32 | |
| rbowen | number80, rather: https://github.com/redhat-openstack/website/blob/master/source/blog/2017-08-30-rdo-pike-released.html.md | 14:32 |
| number80 | ok | 14:32 |
| mnaser | number80 https://bitbucket.org/brandon/backports.ssl_match_hostname/diff/backports/ssl_match_hostname/__init__.py?diff2=a8ef5d616d92&at=default | 14:32 |
| rbowen | It is currently "published: false" | 14:32 |
| mnaser | that was the patch that applying locally resolved the issue | 14:32 |
| rbowen | We just need to make that true to push it live. | 14:32 |
| number80 | mnaser: thanks :) | 14:33 |
| number80 | rbowen: well, we're still waiting for the CI to give us green light | 14:33 |
| number80 | we still have another run after the current one if it passes | 14:33 |
| *** sfbender has joined #rdo | 14:33 | |
| sfbender | Javier Peña created DLRN master: Complete Python 3 support https://softwarefactory-project.io/r/9515 | 14:33 |
| *** dgurtner has quit IRC | 14:34 | |
| dmsimard | jpena, apevec, number80: only one job remaining, all green so far | 14:34 |
| sfbender | Javier Peña proposed DLRN master: Complete Python 3 support https://softwarefactory-project.io/r/9515 | 14:35 |
| jpena | go CI job, you can do it! | 14:36 |
| *** artom has joined #rdo | 14:36 | |
| mnaser | would be nice if ansible streamed output back :p | 14:36 |
| apevec | dmsimard, re. push to -testing, w/o ooo: that's what I proposed last night, then run https://ci.centos.org/view/rdo/view/cbs-promote-pipelines/job/rdo-promote-cloudsig-testing-pike/ to get current DoD satisfied | 14:36 |
| dmsimard | mnaser: *cough* zuul v3 *cough* | 14:36 |
| dmsimard | apevec: ok, so we don't need trown to manually test things then | 14:36 |
| apevec | but yes, we need that centos-release pushed | 14:37 |
| apevec | just tagging for extras was not enough | 14:37 |
| apevec | I guess it's manual ? | 14:37 |
| dmsimard | We might need to ping hughes | 14:37 |
| dmsimard | I think he has something to do | 14:37 |
| apevec | http://cbs.centos.org/koji/buildinfo?buildID=19640 | 14:37 |
| apevec | http://mirror.centos.org/centos/7/extras/x86_64/Packages/?C=M;O=D | 14:37 |
| number80 | jpena: poi scenario 2 passed and it was the one failing :) | 14:39 |
| number80 | now remain poi 1 who passed previously | 14:40 |
| jpena | poi 1 is now collecting logs, it's passed | 14:40 |
| ykarel | Finished: SUCCESS | 14:42 |
| jpena | \o/ | 14:42 |
| *** agurenko has quit IRC | 14:43 | |
| *** cylopez1 has quit IRC | 14:43 | |
| apevec | number80, what is failing ?! | 14:43 |
| apevec | ah _was_ failing | 14:43 |
| number80 | Yeah | 14:43 |
| apevec | phew | 14:43 |
| number80 | :) | 14:43 |
| number80 | now, we can stress again waiting it to get merged | 14:44 |
| *** mpeterso1 has joined #rdo | 14:45 | |
| number80 | mnaser: can you open a bugzilla ticket for python-backports-ssl_match_hostname? | 14:46 |
| jpena | dmsimard: have you unlocked the VMs? | 14:46 |
| number80 | we still need one to override base OS system | 14:46 |
| dmsimard | jpena: nope, I'll go ahead and unlock/delete them now | 14:46 |
| number80 | *package | 14:46 |
| jpena | cool | 14:46 |
| * apevec piles up his +2 on 9161 | 14:46 | |
| dmsimard | jpena: VMs are gone | 14:47 |
| number80 | remember that true GA is => https://review.rdoproject.org/r/#/c/9162/8 | 14:47 |
| chandankumar | yay!!! | 14:47 |
| number80 | if it's in not in release repo, it ain't GA :) | 14:47 |
| *** marrusl has joined #rdo | 14:47 | |
| *** iranzo has quit IRC | 14:47 | |
| apevec | number80, right, we'll run ci.centos pipeline in parallel | 14:47 |
| apevec | with merging 9162 | 14:48 |
| *** josecastroleon has quit IRC | 14:48 | |
| apevec | then hold announcement until we have pass in rdo-promote-cloudsig-testing-pike | 14:48 |
| number80 | *nods* | 14:48 |
| *** iranzo has joined #rdo | 14:49 | |
| *** iranzo has quit IRC | 14:49 | |
| *** iranzo has joined #rdo | 14:49 | |
| mnaser | number80 i will do that | 14:49 |
| number80 | thanks :) | 14:49 |
| number80 | mnaser_ | 14:49 |
| number80 | mnaser++ | 14:49 |
| number80 | ah right, the bot is muted | 14:49 |
| sfbender | Javier Peña proposed DLRN master: Complete Python 3 support https://softwarefactory-project.io/r/9515 | 14:50 |
| mnaser | number80 under which product/component ideally? | 14:51 |
| number80 | distribution | 14:51 |
| *** josecastroleon has joined #rdo | 14:51 | |
| number80 | product is RDO | 14:51 |
| *** aditya_r has quit IRC | 14:52 | |
| mnaser | number80 thank you, that was 99% of the work figuring that part out :p | 14:52 |
| *** imcleod has joined #rdo | 14:52 | |
| number80 | Well, I like when I get bug reports that includes the fix :) | 14:52 |
| *** josecastroleon has quit IRC | 14:53 | |
| *** strigazi has joined #rdo | 14:53 | |
| number80 | ah, I was wondering why it wasn't merged yet, but there's still a non-voting job running | 14:54 |
| *** mpeterso1 has quit IRC | 14:54 | |
| *** jprovazn has joined #rdo | 14:55 | |
| *** janki has quit IRC | 14:55 | |
| mnaser | https://bugzilla.redhat.com/show_bug.cgi?id=1487296 done | 14:55 |
| openstack | bugzilla.redhat.com bug 1487296 in distribution "Old version of python-backports-ssl_match_hostname being shipped" [Unspecified,New] - Assigned to lars | 14:55 |
| chandankumar | waiting for a job to finish | 14:55 |
| dmsimard | apevec: do we have an announcement draft somewhere ? | 14:56 |
| dmsimard | rbowen: ^ | 14:56 |
| rbowen | https://github.com/redhat-openstack/website/blob/master/source/blog/2017-08-30-rdo-pike-released.html.md | 14:57 |
| dmsimard | I just want to make sure there's some sort of note about RDO releasing *now* but how it will hit the CentOS mirrors in a while | 14:57 |
| *** cylopez has joined #rdo | 14:57 | |
| dmsimard | We always get questions about that because we fire the announcement before it hits the mirrors | 14:57 |
| rbowen | There isn't. Can you add a sentence/paragraph to that effect? Or give me one to add? | 14:57 |
| dmsimard | rbowen: looking | 14:57 |
| *** mpeterson has joined #rdo | 14:58 | |
| *** jobewan is now known as jobewan_away | 14:58 | |
| *** vnogin has quit IRC | 14:59 | |
| *** rpioso has joined #rdo | 15:00 | |
| *** daidv has quit IRC | 15:03 | |
| *** aditya_r has joined #rdo | 15:03 | |
| *** daidv has joined #rdo | 15:04 | |
| *** anshul has quit IRC | 15:04 | |
| *** abregman|mtg is now known as abregman | 15:06 | |
| *** emalikov has quit IRC | 15:08 | |
| *** mpeterson has quit IRC | 15:09 | |
| number80 | https://github.com/redhat-openstack/website/pull/1059 | 15:09 |
| *** nyechiel has quit IRC | 15:09 | |
| number80 | ^ add a known issues section in release notes | 15:09 |
| dmsimard | number80, apevec, jpena: rbowen and I worked out an additional node in the header portion of the release notes "The release is making its way out to the CentOS mirror network, and should be on your favorite mirror site momentarily." | 15:11 |
| dmsimard | looks okay to you ? | 15:11 |
| dmsimard | s/node/note/ | 15:11 |
| jpena | +1 | 15:11 |
| *** dprince has quit IRC | 15:11 | |
| number80 | "is currently being replicated on CentOS mirrors, it will be shortly available on our favorite mirror: | 15:12 |
| dmsimard | "is currently" is sort of meh | 15:12 |
| *** mosulica has quit IRC | 15:12 | |
| dmsimard | if I read that announcement tomorrow | 15:12 |
| dmsimard | :P | 15:12 |
| dmsimard | but just bikeshedding | 15:12 |
| *** thrash is now known as thrash|biab | 15:16 | |
| *** mpeterson has joined #rdo | 15:16 | |
| number80 | right is being replicated should be enough | 15:17 |
| *** imcleod has quit IRC | 15:18 | |
| * number80 thinks that roman idioms speakers tend to like longer phrases | 15:18 | |
| dmsimard | that last rdoinfo job is collecting logs right now | 15:19 |
| dmsimard | not entirely sure why it's even running, it's master, not pike | 15:19 |
| dmsimard | did we bump master things in that rdoinfo patch ? | 15:19 |
| number80 | no | 15:19 |
| *** ragiman has quit IRC | 15:20 | |
| sshnaidm | apevec, do we have nova issue on https://trello.com/b/j4IcIomh/production-chain-escalation ? | 15:20 |
| *** mpeterson has quit IRC | 15:21 | |
| jpena | it's now in the gate | 15:21 |
| *** iranzo has quit IRC | 15:22 | |
| *** imcleod has joined #rdo | 15:23 | |
| *** vnogin has joined #rdo | 15:25 | |
| apevec | jpena, 9162 will need rebase | 15:26 |
| *** shreshtha has quit IRC | 15:27 | |
| jpena | apevec: yep, I'm waiting for the merge, it looks like a simple "rebase" on gerrit doesn't work | 15:27 |
| *** ushkalim has quit IRC | 15:29 | |
| *** scorcoran is now known as scorcoran_food | 15:31 | |
| *** pcaruana has quit IRC | 15:32 | |
| *** ykarel has quit IRC | 15:32 | |
| *** sbonazzo has quit IRC | 15:33 | |
| *** abregman has quit IRC | 15:34 | |
| *** tshefi has quit IRC | 15:35 | |
| *** cylopez has quit IRC | 15:36 | |
| *** eaguilar has quit IRC | 15:38 | |
| *** eaguilar has joined #rdo | 15:39 | |
| *** jobewan_away is now known as jobewan | 15:40 | |
| *** jackNemrod has joined #rdo | 15:41 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: nova: http://trunk.rdoproject.org/centos7-master/report.html | 15:42 |
| *** apetrich has quit IRC | 15:45 | |
| *** sreichar has joined #rdo | 15:47 | |
| *** thrash|biab is now known as thrash | 15:50 | |
| *** aditya_ra has joined #rdo | 15:53 | |
| *** aditya_r has quit IRC | 15:53 | |
| *** imcleod has quit IRC | 15:54 | |
| *** Amita has quit IRC | 15:57 | |
| *** mvk has quit IRC | 15:59 | |
| *** ykarel has joined #rdo | 16:00 | |
| *** egonzalez has quit IRC | 16:02 | |
| *** mpeterson has joined #rdo | 16:03 | |
| *** lucasagomes is now known as lucas-hungry | 16:04 | |
| *** mpeterson has quit IRC | 16:16 | |
| *** jpich has quit IRC | 16:17 | |
| rdogerrit | trown created config master: Install ansible in container build job https://review.rdoproject.org/r/9183 | 16:18 |
| *** jlibosva has quit IRC | 16:20 | |
| *** aufi has quit IRC | 16:21 | |
| dmsimard | we really sleeping 10 seconds between tags? :/ | 16:21 |
| * dmsimard following https://review.rdoproject.org/jenkins/job/update-buildsys-tags/173/console | 16:21 | |
| jpena | that's what I thought | 16:22 |
| dmsimard | It's too late to change for this run but we might want to sleep, I dunno, 5 seconds for release tagging ? | 16:22 |
| dmsimard | We're tagging a /lot/ of packages | 16:22 |
| jpena | number80: do you know if there's any reason why we should wait? Like avoiding DoS? | 16:23 |
| *** mpeterson has joined #rdo | 16:24 | |
| *** MrAbaddon has quit IRC | 16:24 | |
| dmsimard | jpena: it sounds like we're making sure the tag went through | 16:24 |
| dmsimard | we send the tag command | 16:24 |
| *** jobewan has quit IRC | 16:24 | |
| dmsimard | and then retry for as long as the package isn't showing up as tagged | 16:24 |
| dmsimard | a tag command is a koji task so it needs to run | 16:25 |
| dmsimard | and I /guess/ it could take a bit but is usually near instantaneous | 16:25 |
| dmsimard | I don't think retrying every 5 seconds is bad | 16:25 |
| number80 | not sure why it's done like this | 16:25 |
| rdogerrit | David Moreau Simard created config master: Sleep 5 seconds instead of 10 https://review.rdoproject.org/r/9184 | 16:26 |
| dmsimard | ^ | 16:26 |
| rdogerrit | David Moreau Simard proposed config master: Sleep 5 seconds instead of 10 https://review.rdoproject.org/r/9184 | 16:26 |
| number80 | When I was doing it manually, I did it in a single command line | 16:26 |
| number80 | I am looking to rewrite it in python where I can optimize thing more easily | 16:27 |
| number80 | like retrieving all those info in a single API call | 16:27 |
| number80 | that's what graffiti does | 16:27 |
| rdogerrit | trown created config master: Reintroduce undercloud install to container image build https://review.rdoproject.org/r/9185 | 16:28 |
| *** mcornea has quit IRC | 16:28 | |
| number80 | If a package is not registered in a tag, it can do it for you | 16:29 |
| trown | gcerami: https://review.rdoproject.org/r/9185 | 16:29 |
| dmsimard | Ok but let's at least put that at 5 seconds for the time being | 16:29 |
| *** dprince has joined #rdo | 16:29 | |
| dmsimard | Now is not the time to rewrite the job :) | 16:29 |
| *** masonjoyers__ has quit IRC | 16:30 | |
| *** mpeterso1 has joined #rdo | 16:30 | |
| rdogerrit | David Moreau Simard proposed config master: Sleep 5 seconds instead of 10 https://review.rdoproject.org/r/9184 | 16:31 |
| dmsimard | jpena: fixed | 16:31 |
| *** jistr is now known as jistr|afk | 16:31 | |
| dmsimard | actually | 16:31 |
| *** larizmen has quit IRC | 16:31 | |
| dmsimard | hang on a second | 16:31 |
| dmsimard | we're sleeping 5 seconds | 16:31 |
| dmsimard | before checking if things have been tagged | 16:31 |
| *** scorcoran_food is now known as scorcoran | 16:32 | |
| dmsimard | we should probably check first, and then sleep | 16:32 |
| dmsimard | we might be sleeping for nothing | 16:32 |
| *** mpeterson has quit IRC | 16:32 | |
| *** noslzzp has quit IRC | 16:33 | |
| *** aditya_ra has quit IRC | 16:33 | |
| *** noslzzp has joined #rdo | 16:33 | |
| *** hewbrocca is now known as hewbrocca_afk | 16:34 | |
| *** trown is now known as trown|lunch | 16:35 | |
| rdogerrit | David Moreau Simard proposed config master: Don't sleep for nothing and sleep for 5 seconds instead of 10 https://review.rdoproject.org/r/9184 | 16:36 |
| * dmsimard says we shouldn't rewrite the job | 16:36 | |
| * dmsimard rewrites the job | 16:37 | |
| dmsimard | number80, jpena ^ | 16:37 |
| *** mpeterso1 has quit IRC | 16:39 | |
| number80 | dmsimard: shell is limiting us and many features are in graffiti already or can be fixed in graffiti | 16:42 |
| dmsimard | number80: that's fair, I just want to make sure we're not sleeping 10 seconds for absolutely nothing for each package we're tagging | 16:43 |
| dmsimard | we could be nice and sleep 2 seconds if we get a successful tag back without retries | 16:44 |
| rdogerrit | David Moreau Simard proposed config master: Don't sleep for nothing and sleep for 5 seconds instead of 10 https://review.rdoproject.org/r/9184 | 16:44 |
| dmsimard | there, sleep 2 added so we don't potentially spam cbs if everything was tagged | 16:44 |
| dmsimard | number80, jpena ^ | 16:44 |
| number80 | I guess Alfredo had issues with throttling | 16:45 |
| dmsimard | number80: the logic was reversed -- when you implement a retry/sleep | 16:46 |
| dmsimard | you don't sleep before trying at least once | 16:46 |
| *** morazi has quit IRC | 16:46 | |
| dmsimard | that's flawed | 16:46 |
| dmsimard | you retry if you fail | 16:46 |
| dmsimard | you don't wait 10 seconds before trying the first time | 16:46 |
| *** milan has quit IRC | 16:47 | |
| dmsimard | ~280 packages at 10 second forced sleep is 45 minutes | 16:48 |
| *** morazi has joined #rdo | 16:48 | |
| number80 | it's 5 min usually | 16:49 |
| number80 | (in one call) | 16:49 |
| dmsimard | number80: look at the console https://review.rdoproject.org/jenkins/job/update-buildsys-tags/173/console | 16:50 |
| number80 | saw that | 16:50 |
| dmsimard | it's been going for far more than 5 minutes | 16:50 |
| dmsimard | over an hour now, so we should be done soon I think | 16:50 |
| *** aditya_r has joined #rdo | 16:50 | |
| *** mpeterson has joined #rdo | 16:51 | |
| number80 | dmsimard: I have even more extreme optimization | 16:55 |
| number80 | cbs buildinfo <list of nvr> | 16:55 |
| number80 | then, split output | 16:55 |
| dmsimard | number80: I don't want to change the existing job too much | 16:55 |
| dmsimard | number80: we can optimize after release is out | 16:56 |
| dmsimard | use graffiti or whatever else | 16:56 |
| number80 | it's no big change, but it'll wait sync to release anyway | 16:56 |
| dmsimard | number80: need to merge https://review.rdoproject.org/r/#/c/9184/ so it's effective for release job | 16:57 |
| rdogerrit | Merged config master: Don't sleep for nothing and sleep for 5 seconds instead of 10 https://review.rdoproject.org/r/9184 | 16:59 |
| *** anilvenkata has quit IRC | 17:03 | |
| *** mpeterson has quit IRC | 17:03 | |
| *** fzdarsky is now known as fzdarsky|afk | 17:04 | |
| *** Son_Goku has joined #rdo | 17:06 | |
| *** sshnaidm is now known as sshnaidm|off | 17:09 | |
| *** rcernin has quit IRC | 17:12 | |
| *** tesseract has quit IRC | 17:14 | |
| *** jkilpatr has quit IRC | 17:16 | |
| *** sbulage has quit IRC | 17:16 | |
| *** egallen has quit IRC | 17:18 | |
| *** cylopez has joined #rdo | 17:22 | |
| apevec | dang (re. tagging one by one) | 17:22 |
| apevec | and sleep... | 17:23 |
| apevec | number80, I also assumed grafitti is used, but I guess we have not been reviewing enough all that :( | 17:23 |
| *** jmelvin has quit IRC | 17:23 | |
| apevec | it really should be one cbs call | 17:24 |
| *** jackNemrod has quit IRC | 17:24 | |
| *** jmelvin has joined #rdo | 17:24 | |
| apevec | how it's running over 2h | 17:25 |
| rdogerrit | Merged rdoinfo master: Push all pike-candidate to pike-testing https://review.rdoproject.org/r/9161 | 17:25 |
| jpena | finished! | 17:26 |
| jpena | now I'll update 9162 | 17:26 |
| apevec | yay | 17:26 |
| *** gfidente has quit IRC | 17:26 | |
| ykarel | great | 17:26 |
| *** cylopez has quit IRC | 17:27 | |
| rdogerrit | Javier Peña proposed rdoinfo master: Push all pike-testing to pike-release https://review.rdoproject.org/r/9162 | 17:28 |
| *** jkilpatr has joined #rdo | 17:29 | |
| apevec | now we need to wait for buildlogs sync | 17:30 |
| chandankumar | how much buildlogs sync takes generally? | 17:30 |
| apevec | it's quick, but runs only every 2h | 17:31 |
| apevec | on even UTC hours | 17:31 |
| apevec | in ~45min | 17:31 |
| *** trown|lunch is now known as trown | 17:33 | |
| apevec | jpena, dmsimard - anything before enabling https://ci.centos.org/view/rdo/view/cbs-promote-pipelines/job/rdo-promote-cloudsig-testing-pike/ | 17:33 |
| apevec | ? | 17:33 |
| rdogerrit | Merged config master: Reintroduce undercloud install to container image build https://review.rdoproject.org/r/9185 | 17:34 |
| apevec | do we do it via ci-config or directly in jenkins UI ? | 17:34 |
| jpena | I think it has to be done in the Jenkins UI, I can't find anything in the jjb description that allows us to enable/disable | 17:36 |
| *** imcsk8 has joined #rdo | 17:37 | |
| dmsimard | apevec: yeah I sent a patch to improve the tagging perf by several orders of magnitude, we can further improve later -- didn't want to risk introducing bugs | 17:39 |
| dmsimard | apevec: also, we should check with bstinson | 17:40 |
| *** aditya_ra has joined #rdo | 17:40 | |
| dmsimard | buildlogs from inside cico != outside cico | 17:40 |
| dmsimard | So the content might already be there (or not) | 17:40 |
| apevec | ah that would be good! | 17:41 |
| *** kambiz has quit IRC | 17:41 | |
| *** aditya_r has quit IRC | 17:41 | |
| *** vnogin has quit IRC | 17:42 | |
| *** vnogin has joined #rdo | 17:42 | |
| *** pkovar has quit IRC | 17:44 | |
| *** eaguilar has quit IRC | 17:44 | |
| *** kambiz has joined #rdo | 17:45 | |
| chandankumar | dmsimard: do we use openstack-health in rdo? https://github.com/openstack/openstack-health | 17:49 |
| dmsimard | chandankumar: no | 17:49 |
| dmsimard | The stack to make it work is quite complicated | 17:50 |
| *** jistr|afk is now known as jistr | 17:50 | |
| *** gcerami has quit IRC | 17:51 | |
| apevec | dmsimard, I'm merging https://review.rdoproject.org/r/9137 | 17:51 |
| *** aramteke has joined #rdo | 17:51 | |
| apevec | multiarch can be followup | 17:51 |
| dmsimard | Ok | 17:51 |
| *** aramteke has quit IRC | 17:52 | |
| *** links has quit IRC | 17:52 | |
| *** aditya_ra has quit IRC | 17:53 | |
| *** eharney has quit IRC | 17:56 | |
| *** vnogin has quit IRC | 17:57 | |
| chandankumar | dmsimard: you mean npm stuffs in openstack-health or something else? | 17:59 |
| *** ykarel_ has joined #rdo | 17:59 | |
| *** imcleod has joined #rdo | 17:59 | |
| *** gszasz has quit IRC | 18:00 | |
| *** rwsu has quit IRC | 18:00 | |
| *** Caterpillar has quit IRC | 18:00 | |
| number80 | openstack-health? | 18:01 |
| number80 | Now OpenStack has a health service? | 18:01 |
| *** Caterpillar has joined #rdo | 18:01 | |
| *** ykarel has quit IRC | 18:01 | |
| *** eaguilar has joined #rdo | 18:02 | |
| chandankumar | number80: http://status.openstack.org/openstack-health/ | 18:02 |
| *** imcleod has quit IRC | 18:05 | |
| *** eze has joined #rdo | 18:06 | |
| EmilienM | number80: it's old | 18:07 |
| EmilienM | status.openstack.org/openstack-health/#/ | 18:07 |
| EmilienM | it's to track gate jobs results | 18:07 |
| EmilienM | and know which tempest tests fail, etc | 18:07 |
| *** eaguilar has quit IRC | 18:07 | |
| *** wolverineav has quit IRC | 18:08 | |
| EmilienM | see http://status.openstack.org/openstack-health/#/job/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container for example | 18:08 |
| *** wolverineav has joined #rdo | 18:08 | |
| EmilienM | it tracks periodic and gate (not check) | 18:08 |
| EmilienM | you can even subscribe to RSS per project or per job | 18:08 |
| number80 | meh, I wanted to joke about Everything as a Service | 18:08 |
| number80 | I haven't done one for long :) | 18:09 |
| *** jdennis has quit IRC | 18:09 | |
| jpena | 18:11:32 DEBUG util.py:417: Error: Package: python2-pykafka-2.6.0-0.1.dev2.el7.x86_64 (temp-cloud7-openstack-common-release) | 18:11 |
| jpena | 18:11:32 DEBUG util.py:417: Requires: python-tabulate | 18:11 |
| jpena | number80: ^^ | 18:11 |
| jpena | we have an error in the common deps, it seems | 18:12 |
| dmsimard | chandankumar: openstack-health retrieves subunit files from jobs, it doesn't work on it's own and there's different components to make it work | 18:13 |
| dmsimard | chandankumar: there is a story on sf.io about it somewhere | 18:13 |
| dmsimard | jpena: hmmm, I think I remember python-tabulate showing up in one of the jobs yesterday | 18:14 |
| * dmsimard checks logs | 18:14 | |
| jpena | apevec: ^^ | 18:15 |
| *** MrAbaddon has joined #rdo | 18:15 | |
| jpena | dmsimard: I think I saw it in one of the upstream rpm-packaging reviews | 18:16 |
| dmsimard | jpena: http://paste.openstack.org/raw/619950/ | 18:16 |
| dmsimard | iirc that was at the end of one of the buildsys tagging jobs | 18:16 |
| *** rwsu has joined #rdo | 18:16 | |
| dmsimard | I think we're good on public buildlogs https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-pike/?C=M;O=D | 18:17 |
| dmsimard | but I'm probably late in the news | 18:17 |
| apevec | no, you are ahead :) | 18:18 |
| dmsimard | w00t | 18:19 |
| apevec | cdn edge node I'm hitting still has old content | 18:19 |
| apevec | but it is usually refresh around :17 | 18:19 |
| dmsimard | I hit a recheck on p-o-i https://review.openstack.org/#/c/499162/ | 18:19 |
| number80 | jpena: ack btw | 18:19 |
| dmsimard | chandankumar: openstack-health: https://softwarefactory-project.io/storyboard/#!/story/202 | 18:20 |
| apevec | ok, what's with python-tabulate ? | 18:20 |
| dmsimard | apevec: jpena knows but I did see these warnings at the end of a run yesterday http://paste.openstack.org/raw/619950/ | 18:21 |
| *** jistr is now known as jistr|off | 18:21 | |
| dmsimard | might or might not be related | 18:21 |
| chandankumar | dmsimard: thanks :-) | 18:21 |
| apevec | hm, ovs 2.6.1 in pike | 18:21 |
| jpena | it looks like it is only required for monasca-common http://codesearch.openstack.org/?q=pykafka&i=nope&files=&repos= | 18:21 |
| jpena | and we're not shipping it in RDO, it's just there for rpm-packaging | 18:22 |
| jpena | so I'll remove it from the review, ok? | 18:22 |
| apevec | but it's blocking us | 18:22 |
| apevec | jpena, ack | 18:22 |
| jpena | that's something we can fix with more time | 18:22 |
| dmsimard | jpena: could it be a transient dep ? | 18:23 |
| dmsimard | a dep of a dep | 18:23 |
| dmsimard | so, i.e, not in reqs.txt | 18:23 |
| dmsimard | or something | 18:23 |
| number80 | jpena: drop it, have it as a post-GA thing | 18:23 |
| * dmsimard thinking out loud | 18:23 | |
| * number80 is working on it | 18:23 | |
| rdogerrit | Javier Peña proposed rdoinfo master: Push all pike-testing to pike-release https://review.rdoproject.org/r/9162 | 18:24 |
| * number80 is building python-tabulate | 18:25 | |
| *** jdennis has joined #rdo | 18:26 | |
| *** abregman has joined #rdo | 18:29 | |
| apevec | https://review.rdoproject.org/r/#/c/9162/10..11/deps.yml looks good | 18:30 |
| apevec | ok buildlogs cdn refreshed for me too | 18:32 |
| apevec | running https://ci.centos.org/view/rdo/view/cbs-promote-pipelines/job/rdo-promote-cloudsig-testing-pike/2/ | 18:32 |
| *** jtomasek has quit IRC | 18:37 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: nova: http://trunk.rdoproject.org/centos7-master/report.html | 18:42 |
| *** scorcoran is now known as scorcoran_biab | 18:51 | |
| *** imcleod has joined #rdo | 18:54 | |
| *** scorcoran_biab has quit IRC | 18:56 | |
| *** eharney has joined #rdo | 18:57 | |
| *** jkilpatr has quit IRC | 18:57 | |
| *** jkilpatr has joined #rdo | 18:57 | |
| *** scorcoran has joined #rdo | 18:58 | |
| *** jmelvin has quit IRC | 18:59 | |
| *** jmelvin has joined #rdo | 19:00 | |
| *** Son_Goku has quit IRC | 19:00 | |
| *** rcernin has joined #rdo | 19:01 | |
| *** ykarel_ has quit IRC | 19:02 | |
| *** imcleod has quit IRC | 19:04 | |
| *** iranzo has joined #rdo | 19:11 | |
| jschlueter | apevec: just notice os-win has 2.2.0 tag coming off of stable/pike but master last tag is 2.1.0 | 19:14 |
| apevec | yes, known issue: until first round of Queens releases... on master, PBR does not see newer tags pushed to stable/pike | 19:15 |
| apevec | I'll repropose workaround in dlrn | 19:15 |
| apevec | too bad I didn't manage to work on it before this critical period | 19:16 |
| dmsimard | apevec: the build failed | 19:16 |
| jschlueter | apevec: also looks like the tarball vs branch is missing the ;sys_platform=win32 entries from requirements.txt | 19:16 |
| dmsimard | ERROR testing new configuration for cloud7-openstack-pike-release | 19:16 |
| dmsimard | We should had held the node .. | 19:17 |
| apevec | so what failed? | 19:18 |
| dmsimard | apevec, number80, jpena: https://review.rdoproject.org/jenkins/job/validate-buildsys-tags/713/console | 19:18 |
| dmsimard | 19:04:09 DEBUG util.py:417: http://mirror.centos.org/centos/7/cloud/x86_64/openstack-pike/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found | 19:18 |
| dmsimard | wtf ? | 19:18 |
| apevec | failure: repodata/repomd.xml from pike: [Errno 256] No more mirrors to try. | 19:19 |
| apevec | 19:04:09 DEBUG util.py:417: http://mirror.centos.org/centos/7/cloud/x86_64/openstack-pike/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found | 19:19 |
| apevec | yeah, something is not quite right | 19:19 |
| apevec | we are trying to push there | 19:19 |
| *** iranzo has quit IRC | 19:19 | |
| apevec | I guess edge-case not covered, the initial push | 19:19 |
| jpena | apevec number80: we have a few unresolved deps -> https://review.rdoproject.org/etherpad/p/unresolved-deps | 19:19 |
| *** eze is now known as eaguilar|afk | 19:19 | |
| dmsimard | those are the same I pasted earlier | 19:20 |
| number80 | jpena: normal | 19:20 |
| apevec | yeah, repoclosure has few know issues, hence it's just warning | 19:20 |
| jpena | same in newton | 19:20 |
| number80 | wait | 19:20 |
| *** eaguilar|afk is now known as eze | 19:20 | |
| number80 | are we trying to openstack-tripleo-ui-deps to -release? | 19:20 |
| *** eze is now known as eaguilar|afk | 19:20 | |
| number80 | this is BR only | 19:20 |
| jpena | also "error testing new configuration for cloud7-openstack-pike-release" | 19:21 |
| jpena | that's probably the key, I'll check | 19:21 |
| dmsimard | I will fire a recheck immediately and hold the node | 19:21 |
| dmsimard | worst case we kill the job | 19:21 |
| apevec | dmsimard, hold on | 19:21 |
| number80 | seems not | 19:21 |
| dmsimard | apevec: ? | 19:22 |
| apevec | number80, ah yes we need to keep few BRs in -testing for dlrn | 19:22 |
| number80 | btw I fixed matplotlib dependency but I can't remember if I submitted a build or not | 19:22 |
| number80 | I did http://cbs.centos.org/koji/buildinfo?buildID=19635 | 19:22 |
| apevec | dmsimard, all clear, wanted to check tripleo-ui-deps is NOT pushed | 19:22 |
| jpena | so the "repo not found" for openstack-pike seems to be the issue | 19:23 |
| apevec | number80, post-GA cleanup | 19:23 |
| apevec | jpena, right | 19:23 |
| *** eaguilar|afk is now known as eze | 19:23 | |
| jpena | apevec: so what's the next step? | 19:23 |
| apevec | jpena, looks like initial push is not covered | 19:23 |
| apevec | manual +2 Verified | 19:23 |
| number80 | apevec: yeah, that was the plan, it's not even used in openstack | 19:23 |
| number80 | (the -qt4 subpackage I mean) | 19:24 |
| dmsimard | I put a logs.tar.gz here https://logs.rdoproject.org/62/9162/11/check/validate-buildsys-tags/Z6f88c0bcfeac435ba07765601c131cc5/ | 19:24 |
| dmsimard | with the full logs | 19:24 |
| dmsimard | (for grep and things)( | 19:24 |
| apevec | then run https://ci.centos.org/view/rdo/view/cbs-promote-pipelines/job/rdo-promote-cloudsig-stable-ocata/ | 19:24 |
| *** pchavva has quit IRC | 19:25 | |
| apevec | oh wait, manual +Verified will skip gate where cbs tagging happens | 19:25 |
| dmsimard | apevec, number80, jpena: ssh jenkins@38.145.34.104 | 19:25 |
| apevec | number80, can you get grafitti yaml from job logs? | 19:25 |
| number80 | apevec: it should be | 19:26 |
| jschlueter | apevec: dlrn tarball has the dropped requirements | 19:26 |
| dmsimard | apevec: this? https://logs.rdoproject.org/62/9162/11/check/validate-buildsys-tags/Z6f88c0bcfeac435ba07765601c131cc5/info/tags.yaml | 19:26 |
| apevec | correction: then run https://ci.centos.org/view/rdo/view/cbs-promote-pipelines/job/rdo-promote-cloudsig-stable-pike/ - bad paste earlier ... | 19:26 |
| dmsimard | apevec: we can forcefully get zuul to enqueue a gate job | 19:27 |
| apevec | jschlueter, aha, so we have too old setuptools there | 19:27 |
| dmsimard | apevec: regardless if it's passed check or not | 19:27 |
| jschlueter | apevec: ack | 19:27 |
| apevec | dmsimard, how does that work? | 19:27 |
| dmsimard | apevec: https://docs.openstack.org/infra/zuul/client.html#enqueue | 19:27 |
| apevec | that's admin action? | 19:28 |
| dmsimard | directly on the zuul server node yeah. | 19:28 |
| apevec | at least https://ci.centos.org/view/rdo/view/cbs-promote-pipelines/job/rdo-promote-cloudsig-testing-pike/ is promising, build-images passed | 19:29 |
| dmsimard | wait, so we don't actually care about the check gate for the release then | 19:29 |
| dmsimard | if we have cico | 19:29 |
| apevec | ideally we'd care if it worked :) | 19:29 |
| dmsimard | oh, yeah we do | 19:29 |
| dmsimard | because it validates tags and stuff | 19:29 |
| apevec | yeah, but there's a problem b/c it expects released repo | 19:30 |
| apevec | it's designed for updates, not initial push | 19:30 |
| apevec | unless someone can debug that quickly | 19:30 |
| apevec | and add a special case when release repo is not there yet | 19:31 |
| dmsimard | this is what the job runs http://paste.openstack.org/show/620134/ | 19:31 |
| *** amuller has quit IRC | 19:31 | |
| dmsimard | gating_scripts here https://review.rdoproject.org/r/gitweb?p=gating_scripts.git;a=tree | 19:32 |
| * dmsimard thinks all those bash scripts should be re-written to ansible for clarity | 19:34 | |
| dmsimard | it probably has to do with this block ? https://review.rdoproject.org/r/gitweb?p=gating_scripts.git;a=blob;f=buildsys-tags/validate_updates.sh;h=debebb5ff7813c72a7014e732dc687738c5e5e1c;hb=HEAD#l79 | 19:35 |
| dmsimard | the repo gets enabled but there's nothing in it | 19:35 |
| dmsimard | jpena, apevec, number80: do we want to hack it ^ ? | 19:38 |
| dmsimard | or ninja it | 19:38 |
| dmsimard | we're probably going to have the same issue in the gate | 19:38 |
| dmsimard | so I think we have to hack it | 19:38 |
| rdogerrit | Jon Schlueter created openstack/os-win-distgit rpm-master: Let RPM handle requirements https://review.rdoproject.org/r/9186 | 19:39 |
| *** jkilpatr has quit IRC | 19:40 | |
| rbowen | So, can someone give a quick status update? | 19:40 |
| number80 | dmsimard: ninja but we'll have to tag them manually | 19:40 |
| dmsimard | rbowen: we've jumped the first big hurdle | 19:41 |
| jpena | a *big* hack we could try is to change baseurl in https://review.rdoproject.org/r/gitweb?p=gating_scripts.git;a=blob;f=buildsys-tags/mock-config.cfg;h=152c54557e1196ffcbd5c1f1f4168f6a8ef74c48;hb=HEAD | 19:41 |
| dmsimard | rbowen: now it's more of a tooling issue more than a packaging one | 19:41 |
| jpena | line 53 | 19:41 |
| jpena | so when we enable the pike repo, we're actually using the buildlogs one instead of mirror.centos.org | 19:42 |
| jpena | do it once, get it tagged, undo | 19:42 |
| dmsimard | jpena: or just not enable it ? | 19:42 |
| dmsimard | but yeah, I guess we could do that | 19:42 |
| rbowen | Are we likely to have Pike today? | 19:42 |
| jschlueter | ssbarnea: https://review.rdoproject.org/r/9186 | 19:43 |
| rdogerrit | rdo-trunk created openstack/tempest-distgit rpm-master: openstack-tempest: failed to build f1fbb3f https://review.rdoproject.org/r/9187 | 19:43 |
| dmsimard | apevec: I fired https://ci.centos.org/job/weirdo-testing-pike/1/ manually, it was disabled and didn't fire in the pipeline | 19:45 |
| *** spanglywires has joined #rdo | 19:45 | |
| dmsimard | apevec: oooq cbs testing jobs failed | 19:46 |
| dmsimard | fail on undercloud install | 19:46 |
| dmsimard | rbowen: things aren't looking particularly good right now | 19:46 |
| rbowen | ok. Thanks for the update. | 19:46 |
| dmsimard | I know there's never a good time for PTO but this is an awful time for the guy who wrote a major portion of the release automation :/ | 19:47 |
| number80 | I can run graffiti to produce same result if we decide to ninja-merge for now | 19:50 |
| dmsimard | number80: CI jobs aren't passing | 19:50 |
| number80 | just stating an option | 19:50 |
| number80 | not saying we should do that | 19:50 |
| number80 | if it's a tooling only issue | 19:51 |
| *** jackNemrod has joined #rdo | 19:51 | |
| dmsimard | 2017-08-31 19:34:18 | 2017-08-31 19:34:18,170 INFO: [1;31mError: Execution of '/bin/yum -d 0 -e 0 -y install openstack-kolla' returned 1: Error: Nothing to do[0m | 19:51 |
| dmsimard | 2017-08-31 19:34:18 | 2017-08-31 19:34:18,170 INFO: [1;31mError: /Stage[main]/Tripleo::Profile::Base::Docker_registry/Package[openstack-kolla]/ensure: change from purged to present failed: Execution of '/bin/yum -d 0 -e 0 -y install openstack-kolla' returned 1: Error: Nothing to do[0m | 19:52 |
| dmsimard | 2017-08-31 19:45:16 | Reason: os-refresh-config failed. See log for details. | 19:52 |
| jpena | let's think about the options | 19:54 |
| *** egonzalez has joined #rdo | 19:54 | |
| dmsimard | http://paste.openstack.org/show/620135/ | 19:54 |
| jpena | if we ninja-merge and apply graffiti, we'll have a most-probably broken repo, but we can then do small incremental fixes | 19:54 |
| dmsimard | jpena: nothing we can't do with the existing -testing | 19:54 |
| jpena | mmm right. Did this error happen with -testing? | 19:55 |
| dmsimard | we could not run ooo in the rdoinfo gate | 19:55 |
| dmsimard | before tagging to -testing | 19:55 |
| jpena | yes, tes | 19:55 |
| jpena | yes | 19:55 |
| jpena | I mean | 19:56 |
| jpena | the ooo error happened with the -testing repo | 19:56 |
| dmsimard | yes, those are from here | 19:56 |
| jpena | ok | 19:56 |
| dmsimard | https://ci.centos.org/view/rdo/view/cbs-promote-pipelines/job/rdo-promote-cloudsig-testing-pike/ | 19:56 |
| jpena | so once we fix that one, we could ninja-merge to avoid the chicken&egg problem we have now | 19:56 |
| dmsimard | weirdo is running separately in https://ci.centos.org/job/weirdo-testing-pike/1/ | 19:56 |
| *** jayg has quit IRC | 19:57 | |
| dmsimard | I'm not sure the issue is just kolla | 19:57 |
| dmsimard | those os-refresh/collect config errors look like a problem | 19:57 |
| dmsimard | there was something about python requests/urllib earlier ? | 19:57 |
| dmsimard | both jobs have the same failures | 19:58 |
| jpena | yes, we had the puppet-murano issue with the newer urllib3 | 19:58 |
| dmsimard | so we downgraded it or something ? | 19:59 |
| jpena | we kept the same version we had till now | 20:00 |
| dmsimard | we have requests 2.11 | 20:00 |
| dmsimard | and os-collect-config has >2.14 | 20:00 |
| dmsimard | not saying that's the issue but /me shrug | 20:00 |
| dmsimard | let me look at what we're using in a passing job | 20:00 |
| dmsimard | yeah it's passing with 2.11 so that's not it | 20:01 |
| jpena | dmsimard: wait, we have no kolla in -testing (!?) | 20:01 |
| jpena | https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L4680-L4691 | 20:01 |
| dmsimard | jpena: that's apparently one of the things it's complaining about yes | 20:01 |
| jpena | there's no build for it ? | 20:01 |
| dmsimard | how come | 20:01 |
| dmsimard | we missed it for some reason ? | 20:01 |
| jpena | yep | 20:02 |
| *** mpeterson has joined #rdo | 20:02 | |
| dmsimard | adding that is probably a good start | 20:02 |
| dmsimard | I'm trying to compare the failing job with a passing one | 20:02 |
| *** jayg has joined #rdo | 20:02 | |
| jpena | there's no pike-rdo branch for kolla-distgit, so we missed it during branching | 20:03 |
| jpena | number80: ^^ | 20:03 |
| *** jkilpatr has joined #rdo | 20:03 | |
| number80 | ack | 20:03 |
| apevec | we don't have stable containers, why do we care about kolla? | 20:04 |
| dmsimard | pkg diff between failing (cbs -testing) and passing (trunk pike promotion) https://www.diffchecker.com/xu7xjBND | 20:04 |
| dmsimard | apevec: tripleoclient uses kolla | 20:04 |
| dmsimard | that prepare something image thing stuff imports and wraps kolla-build | 20:04 |
| dmsimard | so it's probably a dep in one of those tripleo-common packages | 20:04 |
| apevec | ah | 20:04 |
| *** vnogin has joined #rdo | 20:05 | |
| dmsimard | the versions in trunk for the series of os-*-config packages aren't the same | 20:06 |
| dmsimard | I don't know if we missed a tag/bump | 20:06 |
| dmsimard | could just be pbr/dlrn autobump I dunno | 20:06 |
| jpena | dmsimard: that's autobump | 20:06 |
| dmsimard | ah, yeah, i think that's just autobump | 20:06 |
| dmsimard | the pkgdiff is not very conclusive | 20:08 |
| *** imcleod has joined #rdo | 20:08 | |
| rdogerrit | hguemar created openstack/kolla-distgit rpm-master: Pike requirements sync https://review.rdoproject.org/r/9188 | 20:08 |
| dmsimard | it's possible that due to kolla installation failure puppet failed dependencies and lead to the os-*-config failure | 20:08 |
| number80 | ^ | 20:08 |
| jpena | makes sense | 20:08 |
| jpena | "always go for the first error" | 20:09 |
| dmsimard | I hate puppet's "let's fail as late as possible" | 20:09 |
| dmsimard | thankfully ansible is "fail as early as possible" :) | 20:09 |
| dmsimard | if kolla is the only issue | 20:09 |
| dmsimard | the weirdo jobs should come back green | 20:10 |
| dmsimard | weirdo jobs should tell us if there's anything else in the meantime | 20:10 |
| dmsimard | so if we're very optimistic, we're looking at a minimum 3hrs if everything is green all the way through (rdoinfo check, rdoinfo check, ci.centos, tagging, buildlogs sync for kolla, etc.) | 20:10 |
| dmsimard | are we going to push for a release today ? otherwise we're delaying until monday | 20:11 |
| dmsimard | everyone is tired but that would break my heart :( | 20:11 |
| rbowen | Monday is Labo(u)r day | 20:11 |
| dmsimard | that's correct, so tuesday even | 20:12 |
| dmsimard | ugh :( | 20:12 |
| apevec | let's fix that kolla issue first | 20:12 |
| apevec | did they have pike release tag? | 20:12 |
| apevec | or we need snap? | 20:12 |
| *** mpeterson has quit IRC | 20:13 | |
| apevec | milestone3 is kolla last tag | 20:13 |
| jpena | they haven't branched yet, the newest tag they have is 5.0.0.0b3 from Jul 27 | 20:13 |
| apevec | so snpa? | 20:13 |
| jpena | I'd say snap | 20:13 |
| apevec | snap | 20:13 |
| *** rpioso has quit IRC | 20:14 | |
| apevec | I guess we should revisit feasibility of current DoD with deployment project | 20:15 |
| apevec | ooo is also rc1 only | 20:15 |
| apevec | we need to define staged GA | 20:15 |
| apevec | with weirdo only DoD for GA +2h | 20:15 |
| apevec | and ooo, kolla etc. +2 weeks | 20:16 |
| dmsimard | even weirdo depends on packstack and p-o-i | 20:16 |
| apevec | i.e. Pike cycle-trailing release deadline which is Sep11-15 | 20:16 |
| dmsimard | which are not technically released | 20:16 |
| apevec | but in better shape | 20:17 |
| apevec | usually | 20:17 |
| dmsimard | no failures in https://ci.centos.org/job/weirdo-testing-pike/1/console yet | 20:17 |
| apevec | ok, who is preparing kolla snap ? | 20:17 |
| jpena | I'm going for dinner, brb | 20:17 |
| *** jpena is now known as jpena|brb | 20:18 | |
| apevec | I'll take it | 20:18 |
| rdogerrit | Merged openstack/kolla-distgit rpm-master: Pike requirements sync https://review.rdoproject.org/r/9188 | 20:20 |
| *** imcleod has quit IRC | 20:21 | |
| apevec | number80, ^ are you sending pike-rdo patch in sf config? | 20:23 |
| number80 | yeah | 20:23 |
| apevec | ok, I'm on standy for reviews | 20:23 |
| rdogerrit | hguemar created config master: Create pike-rdo branch for kolla https://review.rdoproject.org/r/9189 | 20:25 |
| number80 | apevec: ^ | 20:25 |
| *** egallen has joined #rdo | 20:25 | |
| *** egallen has quit IRC | 20:25 | |
| *** kgiusti has left #rdo | 20:27 | |
| rdogerrit | Merged config master: Create pike-rdo branch for kolla https://review.rdoproject.org/r/9189 | 20:27 |
| rdogerrit | David Moreau Simard created gating_scripts master: Temporarily not enable the release repo https://review.rdoproject.org/r/9190 | 20:27 |
| dmsimard | apevec, jpena|brb, number80 ^ wdyt | 20:27 |
| dmsimard | hah, hang on | 20:28 |
| rdogerrit | David Moreau Simard proposed gating_scripts master: Temporarily not enable the release repo https://review.rdoproject.org/r/9190 | 20:28 |
| dmsimard | take two | 20:28 |
| apevec | number80, pike passed-ci is bbeda30a9df9021bf463b90d864e39cd6d7870df | 20:28 |
| *** vnogin has quit IRC | 20:29 | |
| number80 | okalternative would be testing if release repo is existing but whatever | 20:29 |
| dmsimard | last rdoinfo check failed on same 404 | 20:29 |
| dmsimard | number80: that's a hack that will be reverted | 20:29 |
| dmsimard | I still have the node held if anyone wants to look | 20:30 |
| apevec | dmsimard, not sure about COMMON_TESTING change | 20:30 |
| dmsimard | apevec: it's copy pasta from the block right above | 20:31 |
| dmsimard | it's a if/elif | 20:31 |
| number80 | dmsimard: yeah, I gave it a +2 | 20:31 |
| dmsimard | apevec: if we don't put that there, -testing won't be enabled | 20:31 |
| apevec | yeah but here we are actually testing common-release | 20:31 |
| dmsimard | as far as I can undertand | 20:31 |
| apevec | yeah, but COMMON_RELEASE will be set iiuc ? | 20:31 |
| * apevec reads more bash | 20:31 | |
| dmsimard | but common release doesn't exist | 20:31 |
| dmsimard | right ? | 20:31 |
| apevec | this is envvar | 20:32 |
| apevec | repo is temp-common | 20:32 |
| dmsimard | oh | 20:32 |
| apevec | it's coming from changed_repos.txt | 20:33 |
| apevec | i.e. what changes are detected in rdoinfo | 20:33 |
| apevec | so in our release review COMMON_RELEASE will be set | 20:33 |
| rdogerrit | David Moreau Simard proposed gating_scripts master: Temporarily not enable the release repo https://review.rdoproject.org/r/9190 | 20:33 |
| dmsimard | like this ^ ? | 20:33 |
| apevec | yep that should work | 20:34 |
| *** tosky has quit IRC | 20:35 | |
| dmsimard | we need to ninja merge that, there's no gate jobs | 20:35 |
| apevec | or not... we might confuse further logic if it sees builds we are trying to push in the enabled repo? | 20:35 |
| rdogerrit | Merged gating_scripts master: Temporarily not enable the release repo https://review.rdoproject.org/r/9190 | 20:35 |
| apevec | let's try, how long that job takes? | 20:35 |
| rdogerrit | hguemar created openstack/kolla-distgit pike-rdo: Pike update 5.0.0 pre-release (bbeda30a9df9021bf463b90d864e39cd6d7870df) https://review.rdoproject.org/r/9191 | 20:35 |
| dmsimard | apevec: ~1hr | 20:35 |
| dmsimard | but we need the kolla fix | 20:36 |
| dmsimard | before doing a recheck | 20:36 |
| apevec | right | 20:36 |
| dmsimard | and I guess rebase | 20:36 |
| apevec | almost there | 20:36 |
| apevec | yes, we need kolla in testing first | 20:36 |
| dmsimard | number80: added a comment in your review | 20:37 |
| dmsimard | number80: nevermind | 20:38 |
| *** jayg is now known as jayg|g0n3 | 20:38 | |
| dmsimard | 3/6 green in https://ci.centos.org/job/weirdo-testing-pike/ | 20:38 |
| dmsimard | remaining 3 in progress | 20:38 |
| apevec | number80, hguemar@benihime.seireitei - is that a trap for spammers? :) | 20:39 |
| number80 | ah | 20:39 |
| dmsimard | 4/6 green | 20:39 |
| number80 | forgot that | 20:39 |
| number80 | it just my laptop crashed and my emacs config is half-broken | 20:39 |
| * number80 is on temporary laptop | 20:39 | |
| apevec | oh fun | 20:40 |
| rdogerrit | hguemar proposed openstack/kolla-distgit pike-rdo: Pike update 5.0.0 pre-release (bbeda30a9df9021bf463b90d864e39cd6d7870df) https://review.rdoproject.org/r/9191 | 20:40 |
| *** peter-os has joined #rdo | 20:40 | |
| number80 | Yeah, I haven't find yet why my config is broken | 20:40 |
| *** scorcoran is now known as scorcoran_biab | 20:41 | |
| apevec | number80, need autosetup -S git | 20:43 |
| dmsimard | apevec, number80, EmilienM: we have full upstream p-o-i passing on standalone -testing https://review.openstack.org/#/c/499162/ | 20:43 |
| EmilienM | dmsimard: do you want us to merge it? | 20:44 |
| rdogerrit | hguemar proposed openstack/kolla-distgit pike-rdo: Pike update 5.0.0 pre-release (bbeda30a9df9021bf463b90d864e39cd6d7870df) https://review.rdoproject.org/r/9191 | 20:44 |
| apevec | EmilienM, hold on | 20:44 |
| apevec | actually you don't care about kolla | 20:44 |
| EmilienM | apevec: ok | 20:45 |
| *** eharney has quit IRC | 20:45 | |
| apevec | so up to you | 20:45 |
| dmsimard | EmilienM: for the time being I'm using it as a sanity check | 20:45 |
| EmilienM | no I don't | 20:45 |
| EmilienM | in that context I mean | 20:45 |
| EmilienM | dmsimard: ok | 20:45 |
| apevec | yeah, in poi | 20:45 |
| EmilienM | anyway let me know | 20:45 |
| dmsimard | apevec: so that means weirdo is all green | 20:45 |
| dmsimard | only ooo now... | 20:45 |
| dmsimard | hopefully it's that kolla issue | 20:45 |
| *** scorcoran_biab has quit IRC | 20:46 | |
| apevec | yes, review needs one more fix and we're good to merge it | 20:46 |
| *** artom has quit IRC | 20:47 | |
| *** florianf has quit IRC | 20:50 | |
| number80 | apevec: you fixed the tagging review? | 20:51 |
| *** spanglywires has quit IRC | 20:51 | |
| apevec | number80, no, fix what? | 20:51 |
| apevec | we need kolla build | 20:51 |
| apevec | then push to testing | 20:51 |
| apevec | then rebase release review on that | 20:52 |
| number80 | this one + the release one | 20:52 |
| number80 | ok | 20:52 |
| number80 | ohttp://cbs.centos.org/koji/taskinfo?taskID=218456 | 20:52 |
| number80 | damn | 20:52 |
| number80 | fixed | 20:52 |
| rdogerrit | hguemar created rdoinfo master: Add kolla to pike-testing https://review.rdoproject.org/r/9192 | 20:55 |
| *** imcleod has joined #rdo | 20:56 | |
| apevec | number80, cbs build is not finished, ^ will fail | 20:57 |
| number80 | yeah | 20:57 |
| number80 | I somehow did things in parallel and my fingers just submitted reviews on their own | 20:57 |
| *** eze has quit IRC | 20:58 | |
| * number80 has fingers with their own will | 20:58 | |
| apevec | it's good test for CI :) | 20:58 |
| apevec | it did fail | 20:58 |
| number80 | \o/ | 20:58 |
| number80 | I had to create a noop cbs alias to avoid manually tagging builds w/o thinking after we added automation ... | 21:00 |
| *** scorcoran has joined #rdo | 21:03 | |
| *** julim_ has quit IRC | 21:04 | |
| *** trown is now known as trown|outtypewww | 21:04 | |
| *** wolverineav has quit IRC | 21:04 | |
| *** wolverineav has joined #rdo | 21:05 | |
| *** imcleod has quit IRC | 21:06 | |
| rdogerrit | Merged openstack/kolla-distgit pike-rdo: Pike update 5.0.0 pre-release (bbeda30a9df9021bf463b90d864e39cd6d7870df) https://review.rdoproject.org/r/9191 | 21:07 |
| *** wolverineav has quit IRC | 21:09 | |
| *** jprovazn has quit IRC | 21:11 | |
| apevec | dmsimard, https://review.rdoproject.org/r/9137 doesn't want into gate ? | 21:12 |
| apevec | messing with votes confused zuul? | 21:12 |
| dmsimard | apevec: https://review.rdoproject.org/r/#/c/9153/1 | 21:12 |
| dmsimard | Parent needs to merge | 21:13 |
| *** Goneri has quit IRC | 21:13 | |
| apevec | a-ha | 21:13 |
| rdogerrit | Merged rdo-infra/rdo-release pike-rdo: Use HTTPS for the testing and pending repositories https://review.rdoproject.org/r/9153 | 21:14 |
| rdogerrit | Merged rdo-infra/rdo-release pike-rdo: Enable the release repository for RDO Pike https://review.rdoproject.org/r/9137 | 21:14 |
| rdogerrit | Merged rdo-infra/rdo-release ocata-rdo: Use HTTPS for the testing and pending repositories https://review.rdoproject.org/r/9154 | 21:14 |
| dmsimard | apevec, number80: do we care about check queue for kolla rdoinfo | 21:15 |
| rdogerrit | Merged rdo-infra/rdo-release newton-rdo: Use HTTPS for the testing and pending repositories https://review.rdoproject.org/r/9155 | 21:15 |
| dmsimard | cause I'll enqueue gate right away | 21:15 |
| *** jpena|brb is now known as jpena | 21:15 | |
| *** dustins has quit IRC | 21:15 | |
| dmsimard | otherwise we're gonna wait >1hr for weirdo and ooo to come back | 21:16 |
| dmsimard | just for kolla | 21:16 |
| dmsimard | which is gonna get tested in release anyway | 21:16 |
| apevec | let's just push it | 21:16 |
| dmsimard | no | 21:16 |
| dmsimard | don't push/submit | 21:16 |
| dmsimard | needs to go through gate | 21:16 |
| dmsimard | submitting gate now. | 21:16 |
| apevec | ok | 21:16 |
| rdogerrit | Alan Pevec created rdo-infra/centos-release-openstack pike-rdo: Enable the release repository for RDO Pike https://review.rdoproject.org/r/9193 | 21:17 |
| *** paramite has quit IRC | 21:18 | |
| dmsimard | hmmm, not seeing the job in queue gate | 21:19 |
| dmsimard | but I see in the logs that it should have worked | 21:19 |
| dmsimard | 2017-08-31 21:18:16,578 INFO zuul.Scheduler: Adding rdoinfo, <Change 0x7fcc340f1e50 9192,1> to <Pipeline gate> | 21:20 |
| dmsimard | oh, it needs verified | 21:20 |
| jpena | yes, it's running weirdo tests | 21:20 |
| *** imcleod has joined #rdo | 21:21 | |
| apevec | so what time do we save ? | 21:21 |
| dmsimard | I guess it doesn't work :/ | 21:21 |
| dmsimard | pabelanger: it's possible to bypass check queue right ? | 21:21 |
| dmsimard | pabelanger: enqueuing directly to gate ? | 21:21 |
| dmsimard | "zuul enqueue --trigger gerrit --pipeline gate --project rdoinfo --change 9192,1" doesn't generate any errors, I see "2017-08-31 21:20:54,650 INFO zuul.Scheduler: Adding rdoinfo, <Change 0x7fcc340f1e50 9192,1> to <Pipeline gate>" in logs | 21:22 |
| dmsimard | but there's nothing in the gate queue | 21:23 |
| dmsimard | I manually +2/+v and +W the patch | 21:23 |
| dmsimard | apevec: we're saving the time it takes to run the weirdo and ooo jobs | 21:24 |
| dmsimard | apevec: which are needless for the kolla tag | 21:24 |
| dmsimard | but I guess we're stuck with that | 21:24 |
| dmsimard | and then we'll have to run the /same/ weirdo and ooo jobs in rdoinfo GA | 21:24 |
| dmsimard | we could disable the jobs temporarily | 21:25 |
| dmsimard | and rely on cico instead ? | 21:25 |
| rdogerrit | rdo-trunk created openstack/tempest-distgit rpm-master: openstack-tempest: failed to build 5ee9e19 https://review.rdoproject.org/r/9194 | 21:25 |
| dmsimard | I mean, weirdo is known to be fully green already | 21:25 |
| apevec | yeah | 21:25 |
| apevec | for testing | 21:25 |
| apevec | unknown for proposed release tags | 21:26 |
| pabelanger | dmsimard: you can, but I usually like to make sure it passed checked first | 21:26 |
| dmsimard | I guess.. | 21:26 |
| dmsimard | I need to lie down a bit, my allergy pills are knocking me out | 21:27 |
| dmsimard | afk | 21:27 |
| *** spanglywires has joined #rdo | 21:27 | |
| *** racedo has quit IRC | 21:33 | |
| *** karimb has joined #rdo | 21:35 | |
| *** abregman has quit IRC | 21:35 | |
| *** egonzalez has quit IRC | 21:38 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: nova, tempest: http://trunk.rdoproject.org/centos7-master/report.html | 21:42 |
| *** imcleod has quit IRC | 21:42 | |
| *** oidgar has quit IRC | 21:42 | |
| *** morazi has quit IRC | 21:53 | |
| *** rlandy is now known as rlandy|bbl | 21:58 | |
| *** jackNemrod has quit IRC | 21:58 | |
| *** [1]rpioso has joined #rdo | 22:00 | |
| *** peter-os has quit IRC | 22:01 | |
| rdogerrit | Merged rdo-infra/centos-release-openstack pike-rdo: Enable the release repository for RDO Pike https://review.rdoproject.org/r/9193 | 22:01 |
| *** apevec has quit IRC | 22:02 | |
| *** spanglywires has quit IRC | 22:06 | |
| *** scorcoran has quit IRC | 22:06 | |
| *** nyechiel has joined #rdo | 22:10 | |
| *** pradk has quit IRC | 22:13 | |
| *** apevec has joined #rdo | 22:15 | |
| *** apevec has quit IRC | 22:15 | |
| *** apevec has joined #rdo | 22:15 | |
| *** jcoufal has quit IRC | 22:16 | |
| *** nyechiel has quit IRC | 22:16 | |
| *** dprince has quit IRC | 22:20 | |
| number80 | so what should we do now? | 22:24 |
| *** rbowen has quit IRC | 22:25 | |
| number80 | "ERROR testing new configuration for cloud7-openstack-pike-release" | 22:25 |
| number80 | as-is, automation is blocked | 22:25 |
| jpena | https://review.rdoproject.org/r/9192 is finishing tests, then it will merge. After that, we can update 9162 with kolla, and while it's running test the promotion as well | 22:25 |
| number80 | oh, I missed that one | 22:26 |
| jpena | dmsimard sent a hack for the ci scripts | 22:26 |
| * number80 is confused | 22:26 | |
| jpena | let me recap | 22:26 |
| number80 | I guess I will be out of commission tomorrow morning :) | 22:27 |
| jpena | kolla is being tagged in -testing | 22:27 |
| jpena | still in ci | 22:27 |
| *** tosky has joined #rdo | 22:27 | |
| jpena | after that, we can update the big-fat -release change in rdoinfo and try again | 22:27 |
| jpena | CI scripts will behave a bit different because of https://review.rdoproject.org/r/9190 | 22:27 |
| jpena | in parallel, we'll test the updated -testing repo with https://ci.centos.org/view/rdo/view/cbs-promote-pipelines/job/rdo-promote-cloudsig-testing-pike/ | 22:28 |
| jpena | if everything goes fine, we'll be in one of two situations | 22:28 |
| jpena | a) we don't have to ninja-merge anything and we're happy | 22:29 |
| jpena | b) repos work but we have to ninja-merge the rdoinfo change and tag manually | 22:29 |
| *** vnogin has joined #rdo | 22:29 | |
| number80 | ack | 22:29 |
| number80 | as for b) we can reuse the cbs_actions generated by CI to have consistent results | 22:30 |
| number80 | so not too bad | 22:30 |
| jpena | nice | 22:30 |
| *** rcernin has quit IRC | 22:30 | |
| number80 | Yeah, that's why there was a two-step tagging, it's easier to review things or manually fix things | 22:31 |
| * number80 had learnt this tricks from Releng who has to deal with this kind of stuff | 22:31 | |
| number80 | always allow manual intervention while keep things consistent with automation | 22:31 |
| jpena | it's going to be a while until rdoinfo-tripleo-pike-testing-centos-7-multinode-1ctlr-featureset005-nv finishes, I'll be away for a few mins | 22:32 |
| *** vnogin has quit IRC | 22:34 | |
| *** thegreenhundred has quit IRC | 22:38 | |
| *** aludwar has joined #rdo | 22:38 | |
| *** [1]rpioso has quit IRC | 22:40 | |
| *** tosky has quit IRC | 22:41 | |
| *** richm has quit IRC | 22:42 | |
| *** richm has joined #rdo | 22:42 | |
| mnaser | dmsimard fyi yum update was all that was needed once stuff was out in pike | 22:52 |
| apevec | jpena, number80 - we should have just tagged kolla snap manuall for -testing | 23:00 |
| apevec | we missed few 2h cycles for buildlogs sync already :( | 23:00 |
| apevec | next one is in ~1h | 23:00 |
| number80 | arf | 23:00 |
| apevec | but ok, now we have actual oooq job running in https://review.rdoproject.org/jenkins/job/rdoinfo-tripleo-pike-testing-centos-7-multinode-1ctlr-featureset005-nv/15/ | 23:01 |
| apevec | it's non-voting | 23:01 |
| apevec | but result will be interesting to have | 23:02 |
| apevec | nm, it failed :) | 23:02 |
| jpena | it has failed in tempest run | 23:02 |
| jpena | mmm | 23:02 |
| apevec | sort: write failed: standard output: Broken pipe ?! | 23:03 |
| apevec | config_tempest failed | 23:04 |
| rdogerrit | Merged rdoinfo master: Add kolla to pike-testing https://review.rdoproject.org/r/9192 | 23:06 |
| jpena | ok, I'll update 9162 | 23:07 |
| apevec | ack | 23:07 |
| rdogerrit | Javier Peña proposed rdoinfo master: Push all pike-testing to pike-release https://review.rdoproject.org/r/9162 | 23:07 |
| *** jmelvin has quit IRC | 23:08 | |
| apevec | number80, jpena - then we stop and continue in the morning, another all nighter will not help anyone | 23:08 |
| *** eaguilar has joined #rdo | 23:08 | |
| number80 | yeah | 23:09 |
| number80 | at some point, my brain stopped working | 23:09 |
| jpena | ok, let's get some sleep | 23:09 |
| jpena | see you tomorrow | 23:09 |
| *** jpena is now known as jpena|off | 23:09 | |
| *** ihrachys has quit IRC | 23:09 | |
| *** apevec has quit IRC | 23:10 | |
| *** vnogin has joined #rdo | 23:13 | |
| *** eaguilar has quit IRC | 23:14 | |
| *** eaguilar has joined #rdo | 23:14 | |
| *** vnogin has quit IRC | 23:27 | |
| *** artom has joined #rdo | 23:50 | |
| rdogerrit | Jon Schlueter proposed openstack/os-win-distgit rpm-master: Let RPM handle requirements https://review.rdoproject.org/r/9186 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!