Thursday, 2016-03-17

*** ajmiller has quit IRC00:00
funginibalizer: it's utc midnight now, so free to start running that script at any time. people are now past the deadline for adjusting their preferred e-mail addresses in gerrit00:00
*** akshai has joined #openstack-infra00:00
*** tzn has joined #openstack-infra00:01
*** lucas-afk has quit IRC00:02
*** rhallisey has quit IRC00:03
*** tzn has quit IRC00:03
*** achanda has joined #openstack-infra00:03
pabelangerokay, so debian-jessie dib was successful with uploads happening now00:04
pabelangerwe should be able to check rax shortly00:04
dhellmannis etherpad ok? I can't get a couple of pads to load00:04
clarkbdhellmann: db backups happen at 0000UTC00:04
dhellmannclarkb : ok, thanks00:05
fungipabelanger: in that case mind copying the qcow2 into /var/log/nodepool/images/ and letting us know the download url so zigo can grab a copy to mount locally and inspect?00:05
clarkbpabelanger: rax is the longest uploads, osic and bluebox have the image uploaded though00:05
pabelangerfungi: sure, 1 min00:05
pabelangerclarkb: ya, we are only launching on rax and ovh currently00:06
fungipabelanger: no rush, i just know he's eager to give it a once over and see if there's more needed on the dib side of things00:06
*** darrenc_ has joined #openstack-infra00:06
*** fverbs has joined #openstack-infra00:06
*** rm_work has quit IRC00:06
*** mjb has quit IRC00:06
*** SumitNaiksatam has joined #openstack-infra00:06
*** darrenc has quit IRC00:07
*** mcarden has quit IRC00:07
*** flwang has quit IRC00:07
*** gus has quit IRC00:07
*** vkramskikh has quit IRC00:07
*** mcarden has joined #openstack-infra00:07
*** Jeffrey4l has quit IRC00:08
*** rm_work has joined #openstack-infra00:08
clarkbpabelanger: oh, why?00:08
*** lucasagomes has joined #openstack-infra00:08
*** mjb has joined #openstack-infra00:08
*** gus has joined #openstack-infra00:08
*** dims has joined #openstack-infra00:08
pabelangerclarkb: not sure. I believe that was the only clouds originally tested00:09
pabelangerclarkb: so, I figure if rax comes only, we can enable the rest of them00:09
*** flwang1 has joined #openstack-infra00:09
*** dims_ has quit IRC00:09
pabelangerfungi: zigo: is the debian-jessie dib00:09
*** dims has quit IRC00:10
clarkbok one is booting in ovh now00:10
pabelangerworking on updating puppet-apply to get debian experimental00:11
*** vkramskikh has joined #openstack-infra00:13
*** mtanino has quit IRC00:13
clarkband its ready00:13
pabelangerwill have patch in 2mins00:14
*** hichihara has joined #openstack-infra00:14
*** achanda has quit IRC00:15
openstackgerritPaul Belanger proposed openstack-infra/system-config: Run puppet-apply on debian jessie jenkins slaves
*** jokke_ has quit IRC00:16
*** sridhar_ram has quit IRC00:16
*** jokke_ has joined #openstack-infra00:16
clarkbI logged in disk looks good00:16
*** yamahata has joined #openstack-infra00:18
clarkbpabelanger: that change needs a corresponding one to run the apply test on jessie00:19
clarkbwhich might need to go atop my change or under it, I don't care which order but reabsing likely necessary00:20
pabelangerclarkb: Yup, just running tox now on project-config00:20
openstackgerritMerged openstack-infra/release-tools: add convenience script for finding unreleased stable branch changes
*** timothyb89 has quit IRC00:20
*** fawadkhaliq has quit IRC00:20
*** fawadkhaliq has joined #openstack-infra00:21
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental puppet-apply job for debian-jessie
pabelangerand experimental job00:23
pabelangerwe can likely add bindep-fallback too00:23
pabelangerbut need a drink first00:23
*** flwang1 has quit IRC00:23
*** timothyb89 has joined #openstack-infra00:25
*** darrenc_ is now known as darrenc00:25
*** flepied has joined #openstack-infra00:26
*** flwang1 has joined #openstack-infra00:26
*** baoli has joined #openstack-infra00:28
openstackgerritMerged openstack-infra/system-config: Run puppet-apply on debian jessie jenkins slaves
*** baoli has quit IRC00:30
tonybnibalizer: Feel free to generate the rolls anytime from now (for the next 23.5 hours)00:30
*** Swami has quit IRC00:35
*** adrian_otto has quit IRC00:35
*** asettle has joined #openstack-infra00:35
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental debian-jessie job for bindep-fallback
*** zz_dimtruck is now known as dimtruck00:36
*** thorst has joined #openstack-infra00:37
*** salv-orl_ has quit IRC00:40
funginibalizer: also, as i said, i'm happy to walk you through it, and run it myself again too so we can compare results if you want00:40
fungii've been running it... a lot00:40
*** yamahata has quit IRC00:41
fungii just want to be certain that, as a ptl candidate in a contested race, i'm nowhere in the chain of custody of the electorate data between gerrit and the officials00:41
mordredI miss anything good?00:42
clarkbwe are almost off of devstack-trusty00:42
clarkbwe have debian slaves now00:42
mordredeveryone is awesome00:42
fungicats and dogs living together00:43
fungimass hysteria00:43
clarkbjackfruit beer is surprisingly good00:43
clarkbthtas about all I have00:43
fungimordred: we're going to have an actual infra ptl election for, like, the first time ever00:43
*** cloudtrainme has joined #openstack-infra00:45
mordredfungi: I'm excited!00:47
mordredfungi: we're like a real thing00:47
fungiall projecty and stuff00:48
mordredfungi: does this mean we have to start using rabbit for more things? especially for things it wasn't designed to be used for?00:48
fungimordred: we're going to use it as our distributed lock manager. that's already been decided00:49
mordredfungi: AWESOME00:49
fungior maybe as a syslog replacement00:49
fungiooh, rabbit as an init system!00:49
fungitake that, lennart!00:49
*** fawadkhaliq has quit IRC00:51
*** fawadkhaliq has joined #openstack-infra00:51
bkeroI'm sure he'll be devastated.00:51
*** dims has joined #openstack-infra00:52
ianwdoes anyone know how to get iced-tea to stop warning me about the terrible things the rax java console will do to my system00:52
clarkbmix it iwth lemonade00:53
ianwit makes it almost impossible to get the early boot output, since rax likes to disconnect the damn thing every time you reboot00:53
fungiyes, click the okay button (every single time)00:53
*** sputnik13 has quit IRC00:53
fungiand yeah, definitely a pain that you get disconnected when the instance drops its vga00:53
fungii wonder if nova console works there? or does xen not support that?00:54
*** chlong has joined #openstack-infra00:55
*** asettle has quit IRC00:56
*** baoli has joined #openstack-infra00:56
bkeroianw: 2>/dev/null? :)00:57
*** fverbs has quit IRC00:58
ianw <- so yeah, i'm not crazy, it is a bug00:59 bug 2189 in Plugin "Unable to remove security pop up" [Enhancement,Resolved: fixed] - Assigned to jvanek00:59
*** baoli has quit IRC00:59
*** akshai_ has joined #openstack-infra01:01
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again
*** ams_ has quit IRC01:04
*** kzaitsev_mb has quit IRC01:04
*** odyssey4me has quit IRC01:04
*** bpokorny has quit IRC01:04
*** baoli has joined #openstack-infra01:05
*** akshai has quit IRC01:05
*** ams_ has joined #openstack-infra01:06
*** odyssey4me has joined #openstack-infra01:06
*** baoli has quit IRC01:06
openstackgerritMerged openstack-infra/system-config: Cleanup of nodepool builder logging
*** smarcet has quit IRC01:07
*** ramishra has quit IRC01:07
*** zhurong has joined #openstack-infra01:07
*** ramishra has joined #openstack-infra01:08
*** cloudtrainme has quit IRC01:11
*** jpr has joined #openstack-infra01:13
*** tphummel has quit IRC01:15
*** chlong has quit IRC01:18
*** asettle has joined #openstack-infra01:20
*** sflanigan has quit IRC01:20
*** IlyaG has quit IRC01:20
*** sdake has quit IRC01:22
openstackgerritJamie Lennox proposed openstack-infra/project-config: Refactor the v3 only job
*** piet has quit IRC01:22
openstackgerritJamie Lennox proposed openstack-infra/project-config: Run the v3 only job in neutron
*** baoli has joined #openstack-infra01:23
*** hrybacki has quit IRC01:25
openstackgerritTony Breeds proposed openstack-infra/project-config: Add devstack-plugin-additional-pkg-repos
*** achanda has joined #openstack-infra01:26
*** hrybacki has joined #openstack-infra01:26
*** apoorvad has quit IRC01:27
*** piet has joined #openstack-infra01:28
*** Daisy has joined #openstack-infra01:30
*** chlong has joined #openstack-infra01:31
*** Daisy has quit IRC01:32
*** Daisy has joined #openstack-infra01:33
*** jpr1 has joined #openstack-infra01:36
*** jpr2 has joined #openstack-infra01:37
*** jpr has quit IRC01:37
*** Daisy has quit IRC01:37
*** eliqiao_ has joined #openstack-infra01:40
*** jpr1 has quit IRC01:40
*** alex_xu has quit IRC01:42
*** eliqiao has quit IRC01:43
*** baoli has quit IRC01:43
*** alex_xu has joined #openstack-infra01:44
*** tqtran-afk has quit IRC01:48
*** piet has quit IRC01:49
*** alex_xu has quit IRC01:51
*** fawadkhaliq has quit IRC01:52
*** alex_xu has joined #openstack-infra01:53
*** fawadkhaliq has joined #openstack-infra01:53
openstackgerritTony Breeds proposed openstack-infra/project-config: WIP: add job to test Nova with latest libvirt code
*** woodster_ has quit IRC01:57
*** fawadkhaliq has quit IRC01:58
*** fawadkhaliq has joined #openstack-infra01:58
*** Daisy has joined #openstack-infra01:58
*** kzaitsev_mb has joined #openstack-infra02:00
*** HeOS has quit IRC02:01
*** Daisy has quit IRC02:01
*** [HeOS] has joined #openstack-infra02:01
*** asselin_ has joined #openstack-infra02:01
*** Daisy has joined #openstack-infra02:02
*** bryan_att has quit IRC02:02
openstackgerritTony Breeds proposed openstack-infra/project-config: Add devstack-plugin-additional-pkg-repos
openstackgerritTony Breeds proposed openstack-infra/project-config: WIP: add job to test Nova with latest libvirt code
*** salv-orlando has joined #openstack-infra02:03
*** asselin__ has joined #openstack-infra02:04
*** asselin_ has quit IRC02:06
DaisyAJaeger: there ?02:08
*** sflanigan has joined #openstack-infra02:08
*** ramishra has quit IRC02:09
*** thorst has quit IRC02:09
*** rabi has joined #openstack-infra02:09
*** rabi has quit IRC02:09
*** gildub has quit IRC02:09
*** thorst has joined #openstack-infra02:10
*** bmjen has quit IRC02:10
ianwis there anyone with their key in the built images who could log into and reset the root password for me?  i don't want to reboot it, because i want to see what glean has done on its first boot02:11
*** Hunner has quit IRC02:12
*** salv-orlando has quit IRC02:12
mordredianw: you want root password or your key added to it?02:13
ianwmordred: if you just set the root password i can log on in the console02:13
*** eliqiao_ is now known as eliqiao02:14
ianwit's my vm, but i can't log into it :)02:14
mordredianw: password asdf02:14
*** aeng has quit IRC02:14
mordredianw: of course, now EVERYONE knows your root password ... :)02:14
* jroll logs in and reboots02:14
*** aeng has joined #openstack-infra02:15
mordredianw: for nexttime - there is an additional element you can add, devuser, that will add a user to the image with the ssh public key of the person building the image02:15
mordredianw: you might find that useful if your'e debugging dib images and stuff02:15
*** fawadkhaliq has quit IRC02:15
ianwmordred: yeah, i know, but this is the official build image pabelanger shared with me02:15
mordredah. gotcha02:15
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Add devstack-plugin-additional-pkg-repos
*** fawadkhaliq has joined #openstack-infra02:16
mordredclarkb: I feel like you told me about mirror related patches yesterday and then I went drinking or something ... do you need me to look at osmething?02:16
clarkbmordred yes one sec02:16
*** aeng has quit IRC02:16
clarkbmordred and its parent02:17
*** oanson has quit IRC02:17
*** oanson has joined #openstack-infra02:18
*** thorst has quit IRC02:18
*** ramishra has joined #openstack-infra02:18
ianwhmm, maybe we have a race condition between network.service and glean02:19
*** odyssey4me has quit IRC02:19
*** lucasagomes has quit IRC02:19
*** darrenc has quit IRC02:19
*** mriedem1 has quit IRC02:19
*** mattoliverau has quit IRC02:19
*** aspiers has quit IRC02:19
*** treaki__ has quit IRC02:19
*** binbincong has quit IRC02:19
*** yolanda has quit IRC02:19
*** sileht has quit IRC02:19
*** jraim has quit IRC02:19
*** vmbrasseur has quit IRC02:19
*** mtreinish has quit IRC02:19
*** tsufiev has quit IRC02:19
*** Douhet has quit IRC02:19
*** fkautz has quit IRC02:19
*** Ng has quit IRC02:19
*** zxiiro has quit IRC02:19
*** fdegir has quit IRC02:19
*** simonmcc has quit IRC02:19
*** ashtokolov has quit IRC02:19
*** dansmith has quit IRC02:19
*** yonglihe has quit IRC02:19
*** evgenyl has quit IRC02:19
*** mwhahaha has quit IRC02:19
*** Volundr has quit IRC02:19
*** mescanef has quit IRC02:19
*** cfarquhar has quit IRC02:19
mordredianw: yay!02:20
*** dims has quit IRC02:20
*** treaki__ has joined #openstack-infra02:20
*** MarkAtwood has joined #openstack-infra02:21
*** IlyaG has joined #openstack-infra02:21
*** zhurong has quit IRC02:22
*** Daisy_ has joined #openstack-infra02:22
*** Daisy has quit IRC02:22
*** akshai_ has quit IRC02:22
*** Qiming has joined #openstack-infra02:23
Daisy_Hi, I think something wrong with OpenStack ID server. I cannot login to translate website. defstack website is not able to log in neither, an error code 500 returned.02:24
*** lucasagomes has joined #openstack-infra02:24
*** odyssey4me has joined #openstack-infra02:25
*** darrenc has joined #openstack-infra02:25
*** mriedem1 has joined #openstack-infra02:25
*** mattoliverau has joined #openstack-infra02:25
*** aspiers has joined #openstack-infra02:25
*** binbincong has joined #openstack-infra02:25
*** yolanda has joined #openstack-infra02:25
*** sileht has joined #openstack-infra02:25
*** jraim has joined #openstack-infra02:25
*** vmbrasseur has joined #openstack-infra02:25
*** mtreinish has joined #openstack-infra02:25
*** tsufiev has joined #openstack-infra02:25
*** Douhet has joined #openstack-infra02:25
*** fkautz has joined #openstack-infra02:25
*** Ng has joined #openstack-infra02:25
*** zxiiro has joined #openstack-infra02:25
*** fdegir has joined #openstack-infra02:25
*** simonmcc has joined #openstack-infra02:25
*** ashtokolov has joined #openstack-infra02:25
*** dansmith has joined #openstack-infra02:25
*** yonglihe has joined #openstack-infra02:25
*** evgenyl has joined #openstack-infra02:25
*** mwhahaha has joined #openstack-infra02:25
*** Volundr has joined #openstack-infra02:25
*** mescanef has joined #openstack-infra02:25
*** cfarquhar has joined #openstack-infra02:25
*** aviau has quit IRC02:25
clarkbDaisy_: it has been having trouble today, smarcet was working on it02:25
Daisy_Thanks for the information.02:26
*** aviau has joined #openstack-infra02:26
*** fawadkhaliq has quit IRC02:26
Daisy_have to wait then.02:26
*** aviau has quit IRC02:26
*** fawadkhaliq has joined #openstack-infra02:26
*** IlyaG has quit IRC02:26
*** aviau has joined #openstack-infra02:26
ianwmordred: db1afd64 (Monty Taylor 2015-06-27 12:49:23 -0400  3) After=network.service network.target02:26
ianwmaybe that's what is intended02:26
*** madhu_ak has quit IRC02:26
*** johnsom_ has joined #openstack-infra02:28
*** johnsom has quit IRC02:28
*** kzaitsev_mb has quit IRC02:28
*** zhurong has joined #openstack-infra02:29
mordredclarkb: landed parent patch. child patch failing tests - also put a -1 on it with a couple of comments02:29
mordredclarkb: if we're going to have this parameterized, do we need automatoin aroud creating afs volumes?02:30
mordredianw: well ... assume my systemd is shaky at best02:30
clarkbmordred: no idea02:31
clarkbI guess we arent doing large aggregate volume because that isnt how afs works02:31
*** tphummel has joined #openstack-infra02:32
mordredright. we have volume-per-mirror so that we can release each one when we update it02:32
*** akshai has joined #openstack-infra02:32
openstackgerritMerged openstack-infra/system-config: Make reprepro useable on multiple mirrors
*** Daisy_ has quit IRC02:36
*** Daisy has joined #openstack-infra02:36
*** fawadkhaliq has quit IRC02:38
*** fawadkhaliq has joined #openstack-infra02:38
*** piet has joined #openstack-infra02:40
*** fawadkhaliq has quit IRC02:43
ianwmordred: this would explain why it comes up much faster the second boot -- at that point glean has overwritten the network scripts.  the first time, it waits for network.service which tries to dhcp configure the interfaces.  did you possibly hit something about doing ifup before "network.service" was up?02:44
*** ianychoi3 is now known as ianychoi02:45
mordredianw: possibly? I know there are screwy things related to having interfaces configured to listen for dhcp in the image as it relates to udev and stuff02:45
mordredI want to say you have to have _something_ configured for the interfaces or something in udev doesn't fire properly - but if you have dhcp, then the udev rule will ifup it before glean can rewrite it02:46
mordredbut it works a little different on centos and ubuntu02:46
mordredianw: consider that I'm perhaps just stupid02:47
*** achanda has quit IRC02:48
*** achanda has joined #openstack-infra02:50
*** jpr2 has quit IRC02:50
ianwwell, just me booting the offiical image it took the network about 2 minutes to come up, but it did come up.  i don't know if that's long enough to timeout the node getting registered02:50
*** Daisy_ has joined #openstack-infra02:51
*** Daisy has quit IRC02:51
SamYapledont you people ever sleep?02:52
*** achanda has quit IRC02:52
greghayneswe sleep when waiting for test results02:53
SamYaple"its compiling"02:54
SamYaplei see no reason to stop using that excuse02:54
*** jamesmcarthur has joined #openstack-infra02:54
*** achanda has joined #openstack-infra02:55
*** asettle has quit IRC02:56
*** Qiming has quit IRC02:57
*** Qiming has joined #openstack-infra02:57
*** asettle has joined #openstack-infra02:59
EmilienMclarkb: ty for
*** aspiers has quit IRC03:02
EmilienMianw: can you look at please?03:03
*** asettle has quit IRC03:04
*** jamesmcarthur has quit IRC03:04
*** armax has joined #openstack-infra03:09
*** piet has quit IRC03:10
prometheanfiregreghaynes: again?03:12
prometheanfireneed it for the project-config stuff03:12
*** dimtruck is now known as zz_dimtruck03:13
openstackgerritMerged openstack-infra/project-config: Add puppet-pacemaker to tripleo project list
EmilienMianw: thx03:14
*** markvoelker has quit IRC03:15
*** thorst has joined #openstack-infra03:16
*** salv-orlando has joined #openstack-infra03:17
*** yamahata has joined #openstack-infra03:22
*** thorst has quit IRC03:23
*** verdurin has quit IRC03:23
*** kzaitsev_mb has joined #openstack-infra03:25
*** salv-orlando has quit IRC03:26
mordredclarkb, SamYaple: well I'll be damned - task import works on vexxhost03:29
*** claudiub|2 has quit IRC03:29
*** verdurin has joined #openstack-infra03:31
*** kzaitsev_mb has quit IRC03:31
mordredclarkb: AND it converted the format AND it boots ... and it looks like vexxhost supports qcow2 and raw formats03:33
mordredonce again, mnaser's cloud is awesome03:33
clarkbya we direct upload qcow2 so its fine there03:34
mordredjust saying - it does seem to be a working function in stock glance03:34
clarkbovh and rax are the ones03:34
mordredclarkb: I will now try ovh03:34
clarkband I bet rax wont do qcow2 to vhd03:34
mordredclarkb: it seems to have worked at ovh03:35
mordredtesting booting from it03:36
clarkbit boggles the mind why these are tasks03:36
mordredI know03:36
mordredand also undocumented03:36
mordredit seems that if we just put our images on a web server, we could only publish qcow2 images and reduce our size cost by at least a raw03:37
*** IlyaG has joined #openstack-infra03:37
mordredSamYaple: you win the gold star for teaching me even more crazy things about glance03:38
*** verdurin has quit IRC03:39
clarkbI mean why is each task not its own api verb03:39
*** gildub has joined #openstack-infra03:40
mordredclarkb: yah. I know03:40
*** mriedem1 has quit IRC03:40
mordredclarkb: I have booted a server in ovh from the image that I imported with the import task that convered form qcow2 to raw03:40
*** jamesmcarthur has joined #openstack-infra03:41
*** IlyaG has quit IRC03:42
mordredclarkb: yay. this means I get to add YET ANOTHER set of branching conditoins to create_image in shade03:42
mordredclarkb: because in v1 there is a direct option to tell glance to import from a url03:43
mordredwhich people have wanted, but we've never supported it in shade because there was no v2 equiv03:43
clarkbcanyou only convert on import not upload?03:43
mordredexcept it seems there IS A v2 equiv03:43
mordredI do not know the answer to that03:43
mordredbecause tasks are undocumented03:43
mordredclarkb: I'm going to poke at rax now03:44
mordredand also go read the glance source code03:44
*** links has joined #openstack-infra03:45
mordred| message    | Unsupported element in image_properties, please consult the documentation.       |03:46
mordredclarkb: there is a test case in glance that uses importing qcow2 into vhd format as the example payload03:50
clarkbwow its a thing you cant do with any toolI know of03:51
mordredwell, I mean it shows the syntax for asking the cloud to do the conversion for you03:51
mordredsimliar to the 'please convert this qcow2 image to raw format' which worked on vexxhost and ovh03:51
*** piet has joined #openstack-infra03:51
mordredclarkb: ok - I can't make it work on rax03:53
mordredclarkb: I'll ask rosmaita about it tomorrow03:54
*** adrian_otto has joined #openstack-infra03:54
mordredclarkb: are we still having rax issues?03:54
clarkbI dont think so03:54
clarkbnew problem is with internap03:55
*** baoli has joined #openstack-infra03:55
mordredoh. do we know what's up therE? just can't upload?03:55
clarkbthe token is only good for an hour and has to be valid at end of upload because glance uses it to talk to something else03:55
clarkbso leystoneauth is useless03:55
mordredoh. wow03:55
fungibasically today we got 3 of 4 images uploaded before the token expired03:56
mordredmaybe we should get a new token before each upload?03:56
fungithe first 3 took around an hour and so the token was expired when or shortly after we tried to upload the 4th03:56
*** Daisy_ has quit IRC03:56
clarkbmordred: that was mgagne 's suggested worlaround03:57
clarkbthis should also be fixed in glance03:57
clarkbbut I dont have enough details to file a bug03:57
*** rguillebert has quit IRC03:57
fungitrusts apparently fixes it, if you're on new enough openstack03:58
*** johnsom_ is now known as johnsom03:59
mordredoh good03:59
*** baoli has quit IRC03:59
mordredfungi, clarkb: image import-from-url works on internap04:00
clarkbwe should make sure zanata and openstackid are working again soon with translations happening now and all that04:01
*** fawadkhaliq has joined #openstack-infra04:01
mordredimport-from works on bluebox04:02
mordredso it seems that it works on all of our clouds04:02
mordredexcept rax04:02
clarkber osic04:02
mordredosic is pretty recent, right?04:03
mordred(but testing)04:03
clarkbI dont know what they run but its a new cloud built by rax04:03
mordredclarkb: works on osic04:04
mordredclarkb: I will add support to shade for this04:05
clarkbunfortunately you cant kill tasks iirc04:06
mordredyou cannot kill tasks - you can only poll for their status04:07
StevenKHuh, I have no idea what is going on with openstackid and zanata04:08
clarkbStevenK: well earlier today smarcet was doing things04:09
clarkband there were reports of broken ipv604:09
clarkbI am guessing some.combination of these factors is related to the current behavior04:09
clarkbfungi ^ do you know what the things were?04:09
fungiclarkb: afaik implementing new features for summit apps support04:10
fungipretty sure the ipv6-related dns issues were back on the 15th04:10
fungii'm going to guess something in the recent openstackid patches has broken authenticating anything besides www.openstack.org04:11
fungiany idea if groups.o.o authentication is similarly broken? i think it's the only other thing using openstackid currently04:12
fungiunfortunately this is just going to make the tipit requests for ssh access to the production server all the more urgent04:12
fungibecause apparently developing web apps isn't like developing software04:13
clarkbI havent seen any complaints for groups logins04:13
*** Jeffrey4l has joined #openstack-infra04:13
clarkbwhile I have no problem with windows users writing windows docs, why a video?04:14
clarkblooks like the concerns have been raisedand asked for actual text docs to fit with everything else04:15
*** Daisy has joined #openstack-infra04:17
fungii think it's that windows users don't read?04:20
*** piet has quit IRC04:20
*** thorst has joined #openstack-infra04:21
*** adrian_otto has quit IRC04:24
*** asettle has joined #openstack-infra04:27
*** kzaitsev_mb has joined #openstack-infra04:27
*** thorst has quit IRC04:28
*** treaki_ has joined #openstack-infra04:29
*** jamesmcarthur has quit IRC04:31
*** asettle has quit IRC04:32
*** kzaitsev_mb has quit IRC04:32
*** treaki__ has quit IRC04:32
*** armax has quit IRC04:33
*** flepied has quit IRC04:33
*** bexelbie has quit IRC04:34
*** Daisy has quit IRC04:36
*** IlyaG has joined #openstack-infra04:38
*** vgridnev has joined #openstack-infra04:43
*** IlyaG has quit IRC04:43
*** fedexo has joined #openstack-infra04:47
*** flepied has joined #openstack-infra04:47
*** salv-orlando has joined #openstack-infra04:48
*** aspiers has joined #openstack-infra04:49
*** salv-orlando has quit IRC04:51
*** weshay has quit IRC04:55
*** armax has joined #openstack-infra04:59
*** gnuoy has quit IRC05:00
*** jamespage has quit IRC05:00
*** gnuoy has joined #openstack-infra05:00
*** jamespage has joined #openstack-infra05:01
*** Jeffrey4l has quit IRC05:03
*** Daisy has joined #openstack-infra05:03
*** verdurin has joined #openstack-infra05:07
*** ianychoi2 has joined #openstack-infra05:09
*** ianychoi has quit IRC05:09
armaxclarkb: ping?05:09
prometheanfireubuntu-trusty seems to be missing uuidgen05:12
prometheanfireshould be fixed in this review though05:12
*** adrian_otto has joined #openstack-infra05:13
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add testing for the Gentoo element
*** markvoelker has joined #openstack-infra05:15
*** baoli has joined #openstack-infra05:15
*** adrian_otto has quit IRC05:16
*** adrian_otto has joined #openstack-infra05:16
*** bexelbie has joined #openstack-infra05:16
*** ianychoi2 is now known as ianychoi05:18
*** baoli has quit IRC05:20
*** markvoelker has quit IRC05:21
*** fawadkhaliq has quit IRC05:21
*** harlowja_at_home has quit IRC05:24
*** harlowja_at_home has joined #openstack-infra05:25
*** thorst has joined #openstack-infra05:26
*** _nadya_ has joined #openstack-infra05:27
*** asselin_ has joined #openstack-infra05:32
*** thorst has quit IRC05:33
*** asselin__ has quit IRC05:33
*** achanda has quit IRC05:37
openstackgerritIan Wienand proposed openstack-infra/glean: [WIP] Bring up glean before network.service
*** asettle has joined #openstack-infra05:43
*** sdake has joined #openstack-infra05:44
*** [HeOS] has quit IRC05:45
*** yamahata has quit IRC05:45
*** sc68cal has quit IRC05:45
*** HeOS has joined #openstack-infra05:45
*** sdake_ has joined #openstack-infra05:45
*** abregman has joined #openstack-infra05:46
*** sdake has quit IRC05:49
openstackgerritMichael Johnson proposed openstack-infra/project-config: neutron-lbaas-dashboard project is missing perms
*** MarkAtwood has quit IRC05:50
*** sdake_ is now known as sdake05:51
ianwprometheanfire: can you put that in a separate change and i can quick approve it05:56
prometheanfireianw: the cleanup?05:58
ianwthe new package for ubuntu05:58
prometheanfireoh :(05:58
ianwit's breaking generically, right?05:59
prometheanfireI'm actually kinda far from my gpg/ssh key05:59
prometheanfireit is05:59
prometheanfireit just needs that package (I think)05:59
ianwyes, devstack hit the same issue.  i'll push it in06:00
prometheanfireI'm fine if you take that fix06:00
*** Qiming_ has joined #openstack-infra06:00
prometheanfirenot gonna stay up as early as last night06:00
*** Qiming has quit IRC06:01
*** Daisy has quit IRC06:01
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add uuid-runtime for Trusty
*** Daisy has joined #openstack-infra06:02
*** esikachev has joined #openstack-infra06:04
*** rlandy has quit IRC06:06
*** chlong has quit IRC06:08
*** esikachev has quit IRC06:09
armaxI wonder if any of you has any recommendation on how to add a missing package into the image to allow neutron tests to work correctly06:09
armaxit seems that after change
armaxthe netcat-openbsd package is no longer available06:09
armaxand this breaks a neutron job06:10
armaxthis is from a good run:06:10
armaxthis is from a bad run06:10
*** _nadya_ has quit IRC06:10
armaxI’ll try poking at devstack itself, but a heads-up on this would be nice06:11
*** oanson has quit IRC06:12
*** salv-orlando has joined #openstack-infra06:13
*** Daisy has quit IRC06:17
*** roxanagh_ has joined #openstack-infra06:17
ianwarmax: you can add netcat-openbsd to files/debs/neutron in devsatck06:17
*** oanson has joined #openstack-infra06:18
armaxianw: that’s what I am about to do06:18
armaxianw: do you think that’s the right thing to do ?06:18
ianwswitching these nodes is a bit of a pain, but OTOH us getting the dependencies explicitly listed is a good idea06:18
armaxianw: if so, I am happy to proceed :)06:18
armaxianw: ack06:18
ianwarmax: yes.  i can quick approve that if it gets it going06:18
ianwi did similar for uuidgen06:18
armaxianw: posting quickly hang on06:19
*** chlong has joined #openstack-infra06:21
*** roxanagh_ has quit IRC06:22
armaxjohnsom: ^06:22
armaxianw: I wonder if this must be backported06:22
armaxI suppose it must, no?06:22
ianwprobably, yes06:23
*** salv-orlando has quit IRC06:23
armaxthough I am not quite sure how we’re broken on neutron stable/mitaka and yet the stable/mitaka branch is not yet open for devstack06:23
ianwhang on, hang, that's functional tests06:24
armaxianw: right, I think we can try some other way06:28
*** kzaitsev_mb has joined #openstack-infra06:29
armaxianw: I am still wrapping my head around this06:29
ianwarmax: actually, no, that's the right fix ->
ianwneutron functional tests uses the devstack package list06:29
armaxthat’s where I am right now, but as you can see, we don’t maintain a list of deps in tree06:30
armaxor am I supposed to add that deps to a different file in devstack06:30
*** thorst has joined #openstack-infra06:31
ianwarmax: so it seems like the functional tests use the devstack package list, but also has a "post" section where it installs a few other things06:32
armaxianw: do you think it’s best if we kept a list for test deps in the neutron tree?06:32
ianwarmax: i think you kind of have it, see my second comment06:32
*** Jeffrey4l has joined #openstack-infra06:32
*** flepied has quit IRC06:33
*** BigWillie has joined #openstack-infra06:33
ianwso there's a few too many choices to just push in a quick fix for this at this point.  i think it mostly depends on what neutron wants to do06:33
armaxianw: I think this lies with the functional testing setup06:34
*** ociuhandu has quit IRC06:34
ianwarmax: yes, i'd propose it in that fn at line 217 there i think06:34
armaxianw: if we can help it06:34
*** BigWillie has quit IRC06:34
*** kzaitsev_mb has quit IRC06:35
armaxianw: let me abandon the devstack fix06:35
armaxand thanks for the extra pair of eyes06:36
*** ociuhandu has joined #openstack-infra06:36
ianwarmax: heh, cool, i have to go now too.  probably not till us east time does devstack people come back06:36
tobiashjhesketh: would you agree to make the parameters of GerritWatcher.__init__ mandatory? This hides forgotten configuration values of this class (see again the keepalive stuff...)06:36
armaxianw: thanks again!06:37
*** thorst has quit IRC06:38
jheskethtobiash: I think having a default on the parameter is okay and in line with the other params06:40
jheskethtobiash: it's only called from the GerritConnection (afaik) so ensuring the config value is passed through correctly there there shouldn't be any problems06:40
*** fedexo has quit IRC06:41
tobiashjhesketh: As you wish, but making all of these parameters mandatory would make this part more robust against future refactoring mistakes as this would automatically enforce that all of the the config values are passed through.06:45
jheskethsure, I think both ways are fine... it depends on whether we expect GerritWatcher to be used elsewhere and if it were if there was a need for sane defaults on the parameters (and thereofre an expectation that not every instance of GerritWatcher is being configured by a config file)06:47
jheskethhowever since it is only the one place, removing the defaults is fine and possibly a good idea06:47
jheskethtobiash: in other words, feel free to change them if you'd like06:47
jheskeththe only tricky part is that we don't want too much into master while v3 is being worked on. But since that line is changing anyway it shouldn't hurt06:48
*** kingia_ has quit IRC06:48
tobiashwell, this default parameters stuff is not critical and could be done directly on the v3 branch06:49
tobiashjhesketh: btw, did you take the ownership of this keepalive patch?06:49
tobiashsaw that the latest versions are from you06:50
*** kingia has joined #openstack-infra06:50
jheskethtobiash: not ownership, just helping rebasing it when doing the connection work06:52
tobiashah, ok06:52
*** markvoelker has joined #openstack-infra06:52
openstackgerritTravis Tripp proposed openstack-infra/project-config: Adding searchlight-ui repo and jobs
*** akaszuba has joined #openstack-infra06:57
*** markvoelker has quit IRC06:58
*** jaosorior has joined #openstack-infra06:58
*** Daisy has joined #openstack-infra06:59
*** HeOS has quit IRC07:00
*** _nadya_ has joined #openstack-infra07:01
*** HeOS has joined #openstack-infra07:01
*** HeOS has quit IRC07:02
openstackgerritTravis Tripp proposed openstack-infra/project-config: Adding searchlight-ui repo and jobs
*** Daisy has quit IRC07:03
*** HeOS has joined #openstack-infra07:03
*** esikachev has joined #openstack-infra07:08
*** alex_xu has quit IRC07:10
*** jiangliucheng has quit IRC07:12
*** jiangliucheng has joined #openstack-infra07:12
*** alex_xu has joined #openstack-infra07:12
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Enable voting for gate jobs in sahara-tests
*** mikelk has joined #openstack-infra07:15
*** tesseract has joined #openstack-infra07:15
*** asettle has quit IRC07:15
*** tesseract is now known as Guest5718207:15
openstackgerritIan Wienand proposed openstack-infra/project-config: Use pip installed dib for functional tests
ianwgreghaynes / SpamapS : so i think that will be required to get dib functional tests working again07:16
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty
*** vgridnev has quit IRC07:17
*** _nadya_ has quit IRC07:18
*** Jeffrey4l has quit IRC07:22
*** jogo has quit IRC07:22
*** sridhar_ram has joined #openstack-infra07:23
ianwAJaeger: there seems to be quite a bit of fallout from these changes, i wonder if we can do things in a more controlled fashion?07:24
ianwi don't know what that is, although i agree we end up in a better place having dependencies specified in the long run07:24
*** claudiub|2 has joined #openstack-infra07:24
*** jogo has joined #openstack-infra07:25
*** jogo has joined #openstack-infra07:25
*** zeih has joined #openstack-infra07:29
*** kzaitsev_mb has joined #openstack-infra07:30
*** gongysh has joined #openstack-infra07:31
*** scheuran has joined #openstack-infra07:31
openstackgerritIan Wienand proposed openstack-infra/project-config: Use pip installed dib for functional tests
*** salv-orlando has joined #openstack-infra07:32
*** _nadya_ has joined #openstack-infra07:32
*** mikelk has quit IRC07:33
openstackgerritAndreas Jaeger proposed openstack/requirements: uc: openstackdocstheme==1.3.0
*** Daisy has joined #openstack-infra07:34
*** kzaitsev_mb has quit IRC07:35
*** thorst has joined #openstack-infra07:36
*** rcernin has joined #openstack-infra07:39
*** salv-orlando has quit IRC07:40
*** ihrachys has joined #openstack-infra07:41
*** kingia has quit IRC07:42
*** thorst has quit IRC07:43
*** flepied has joined #openstack-infra07:44
*** kingia has joined #openstack-infra07:44
*** jiangliucheng has quit IRC07:47
*** k4n0 has joined #openstack-infra07:48
*** Daisy has quit IRC07:48
*** Daisy has joined #openstack-infra07:49
*** camunoz has quit IRC07:50
*** Daisy has quit IRC07:53
*** jlanoux has joined #openstack-infra07:54
*** chlong has quit IRC07:57
AJaegerianw: the bare-trusty went so smooth and the initial first test and first conversion of devstack-trusty that clarkb converted them all in one.07:59
AJaegerianw: We expected less problems here ;(07:59
*** HeOS has quit IRC08:00
abregmancan anyone give me newbie tasks in openstack-infra projects? :(08:00
*** cody-somerville has quit IRC08:01
*** jlanoux has quit IRC08:03
*** Daisy has joined #openstack-infra08:04
*** jlanoux has joined #openstack-infra08:04
*** HeOS has joined #openstack-infra08:05
AJaegerabregman: start with reviewing ;)08:05
*** k4n0 has quit IRC08:06
abregmanAJaeger: any specific projects?08:06
abregmanis there even a way to filter only openstack-infra reviews?08:06
*** hichihara has quit IRC08:07
*** amrith is now known as _amrith_08:10
AJaegerabregman: you can filter using the search box, e.g. enter: "project:openstack-infra/project-config" and add also "status:open" - this will give you
AJaegerYou can also filter for all projects like "projects:openstack-infra/"08:11
abregmanAJaeger: ack. I'll do that. thanks!08:11
AJaegerabregman: where to start? It depends on what you want to do...08:11
abregmanAnsible, python..08:12
*** mikelk has joined #openstack-infra08:12
AJaegerabregman: our current specs are up at
AJaegerand you can even filter on those if you're interested in a topic for review like
AJaegerabregman: for ansible, system-config is more appropriate08:13
AJaegerI suggest you talk with rest of infra team during US office hours08:14
AJaegerothers might give you some more pointers on newbie tasks08:14
AJaegerabregman: Sorry, I forgot: Glad to hear that you're interested!08:15
*** zeih has quit IRC08:15
abregmanAJaeger: thanks for the tips :) I'll ask again during US office hours and look on some changes in gerrit08:15
AJaegerabregman: is our central docu starting page08:15
*** vgridnev has joined #openstack-infra08:16
DaisyAJaeger: Hi. Just let you know. I'm not able to login to translate website. So I didn't make any versions yet.08:17
*** sridhar_ram1 has joined #openstack-infra08:17
AJaegerDaisy: I've seen - yeah this is really bad timing ;(08:17
DaisyAJaeger: yeah. Do you have any ideas about the problem ?08:18
AJaegeropenstackid integration is broken08:18
*** zeih has joined #openstack-infra08:18
AJaegerHope they get it fixed quickly08:18
DaisyAJaeger: Translators can login to with Openstack ID, but cannot login to Zanata.08:19
*** sridhar_ram has quit IRC08:19
AJaegerif it's not working tomorrow morning, please come back directly08:19
*** sflanigan has quit IRC08:19
* AJaeger will try to ping people to fix it but haven't seen anyone around yet08:19
zigoAJaeger: Hi there. When I a job is ran, what parameter does it get in the env? Namely, do I get the name of the gerrit repo? How does it get the CR # and such?08:20
DaisyAJaeger: I think one day can be tolerant. I don't think we can wait for two days. It's busy time for translators.08:20
*** kingia_ has joined #openstack-infra08:20
AJaegerzigo, you can check on jenkins what parameters it gets and check your log file as well.08:22
AJaegerDaisy: I know, I fear both of us need to ping here ;(08:22
*** kingia has quit IRC08:23
*** salv-orlando has joined #openstack-infra08:23
DaisyAJaeger: understand. I will ping when people wake up.08:23
AJaegerzigo: We normally set jobs up so that they have a repo that contains the change applied.08:23
*** pcaruana has joined #openstack-infra08:24
DaisyAJaeger: hope it's not too difficult and they can fix it today.08:24
*** nmagnezi has joined #openstack-infra08:25
AJaegerinfra-root: according to status.o.o/zuul the periodic queue is not moving forward, did we lose the proposal slave?08:26
*** esikachev has quit IRC08:27
*** arxcruz has joined #openstack-infra08:27
*** yamamoto_ has joined #openstack-infra08:29
* AJaeger will be back later08:30
*** kzaitsev_mb has joined #openstack-infra08:31
zigoAJaeger: I don't have a job yet, so I don't really know where to look.08:35
*** fhubik has joined #openstack-infra08:35
*** esikachev has joined #openstack-infra08:36
zigoAJaeger: I believe I understood the thingy with jenkins/jobs/* and jenkins/scripts/* stuff.08:36
*** ifarkas has joined #openstack-infra08:36
*** kzaitsev_mb has quit IRC08:36
zigoAJaeger: Now, I'm trying to understand what will be passed to my script...08:36
*** vincentll has joined #openstack-infra08:36
ogelbukh_hi team, looks like this change has sign offs needed:
ogelbukh_or this is the wrong place to ask for reviews?08:39
*** sshnaidm has quit IRC08:40
*** johnsom has quit IRC08:40
*** mrmartin has joined #openstack-infra08:40
*** thorst has joined #openstack-infra08:40
xiangxinyongAJaeger: ping08:41
*** johnsom has joined #openstack-infra08:41
xiangxinyongAJaeger: could i ask a question about the ci?08:42
*** sdake has quit IRC08:43
*** SumitNaiksatam has quit IRC08:43
*** SumitNaiksatam_ has joined #openstack-infra08:43
*** sdake has joined #openstack-infra08:43
zigoAJaeger: Under what users are script executed?08:44
*** rossella_s has joined #openstack-infra08:44
*** fhubik has quit IRC08:45
*** hashar has joined #openstack-infra08:48
*** thorst has quit IRC08:48
*** jcoufal has joined #openstack-infra08:48
*** kzaitsev_mb has joined #openstack-infra08:48
*** fhubik has joined #openstack-infra08:50
*** sdake has quit IRC08:50
*** cody-somerville has joined #openstack-infra08:52
*** links has quit IRC08:53
*** bexelbie has quit IRC08:53
*** bexelbie has joined #openstack-infra08:53
*** jordanP has joined #openstack-infra08:55
*** gildub has quit IRC08:58
*** dingyichen has quit IRC08:59
*** cody-somerville has quit IRC09:01
*** cody-somerville has joined #openstack-infra09:02
*** ihrachys has quit IRC09:02
*** rossella_s has quit IRC09:03
*** rossella_s has joined #openstack-infra09:03
*** onovy has quit IRC09:07
*** peterlisak has quit IRC09:07
*** salv-orlando has quit IRC09:07
openstackgerritHua Wang proposed openstack-infra/devstack-gate: SERVICE_HOST should not be
*** zhurong has quit IRC09:10
*** fhubik is now known as fhubik_brb09:10
*** fhubik_brb is now known as fhubik09:10
yolandagood mornign09:12
yolandaAJaeger, i'll take a look at zuuo09:12
*** d0ugal has quit IRC09:13
*** d0ugal has joined #openstack-infra09:14
*** akaszuba has quit IRC09:16
*** akaszuba has joined #openstack-infra09:19
*** asettle has joined #openstack-infra09:19
yolandaAJaeger so yes, proposal is offline, i'm checking the reasons now09:22
yolandaDisconnected by anonymous : Offline due to Gearman request09:22
yolandathere should be some job incorrectly configured09:22
yolandai put back online now09:23
yolandaAJaeger, according to build history, the latest job run was
rcarrillocruzAJaeger, yolanda:
*** Daisy has quit IRC09:26
rcarrillocruzwas it agreed to install npm as a dep for all slaves, rather than downloading per-job basis09:26
*** jistr has joined #openstack-infra09:26
*** Daisy has joined #openstack-infra09:26
*** Daisy_ has joined #openstack-infra09:28
yolandai don't know, we'll need to check with another roots in US09:28
rcarrillocruzi mean, functionally is correct, but would like to understand if that was agreed upon prior to giving +109:29
*** kzaitsev_mb has quit IRC09:29
rcarrillocruzsounds like the typical thing that was at least discussed here on in the infra meeting09:30
yolandai'd say that this is done based on agreement, but cannot ensure 100%09:30
*** onovy has joined #openstack-infra09:31
*** peterlisak has joined #openstack-infra09:31
*** Daisy has quit IRC09:31
*** samuelBartel has joined #openstack-infra09:32
*** sshnaidm has joined #openstack-infra09:34
*** fverbs has joined #openstack-infra09:37
yolandaAJaeger, i'd say that the propose-translation-update-mitaka jobs are breaking proposal, going to take a look09:38
*** esikachev has quit IRC09:40
*** achanda has joined #openstack-infra09:44
*** matrohon has joined #openstack-infra09:45
*** achanda has quit IRC09:45
*** thorst has joined #openstack-infra09:45
*** esikachev has joined #openstack-infra09:46
*** sridhar_ram1 has quit IRC09:46
*** achanda has joined #openstack-infra09:46
*** achanda has quit IRC09:46
*** _degorenko|afk is now known as degorenko09:47
*** Daisy_ has quit IRC09:47
tobiashjhesketh: just curious, how many zuul-mergers do you have running for the openstack infrastructure?09:48
*** Daisy has joined #openstack-infra09:48
rcarrillocruzif memory serves, in gozer we had reusable-node for jobs using static slaves09:51
rcarrillocruzyolanda: ^09:51
*** sdague has joined #openstack-infra09:51
yolandai'm looking at upstream and i cannot see that reusable-node being used09:51
yolandaanywhere in layout.yaml09:51
yolandaso not sure if that's done in a different way09:52
*** kelepirci has joined #openstack-infra09:52
kelepircihello all09:52
*** Daisy has quit IRC09:52
*** thorst has quit IRC09:52
rcarrillocruzgrep does not show that on zuul's code09:53
yolandathere is something wrong in the mitake jobs, the other translation jobs are fine09:53
rcarrillocruzthat's strange09:53
rcarrillocruzlet's see the git log09:53
yolandabut when a mitaka-update job runs, it turns proposal off09:53
yolandabut i don't see anything different than other translation jobs09:55
rcarrillocruzthey used that flag at some point09:55
rcarrillocruzso if that's not on layout.yaml anymore09:55
rcarrillocruzsomething changed09:55
rcarrillocruzbut def. they used it09:56
rcarrillocruzand for proposal09:56
yolandano parameter-function anymore...09:56
rcarrillocruzparameter-function does exist on zuul's code tho09:57
yolandaso again , turned proposal offline. Something is wrong with these jobs09:57
*** dims has joined #openstack-infra09:57
*** dguitarbite has joined #openstack-infra10:00
yolandaso looking at job env vars, i see offline_node_when_complete set to 1 in the jobs that fails. The other jobs that run, are not sending that env var10:00
dguitarbitehello, I am not able to login to gerrit from for some reason10:00
rcarrillocruzah right10:00
yolandabut i don't know how that is configured10:00
rcarrillocruzthe function10:00
rcarrillocruzis defined on project_config10:00
yolandai see10:01
yolandabased on a regex10:01
yolandarcarrillocruz, do you want to send a fix to add mitaka?10:01
*** _nadya_ has quit IRC10:01
rcarrillocruzsure, where's the line of the regex10:02
*** jamesmcarthur has joined #openstack-infra10:03
*** nadya has joined #openstack-infra10:04
*** tpsilva has joined #openstack-infra10:04
*** jamesmcarthur has quit IRC10:08
yolandaso zuul/openstack_functions, you'll see the proposal_re10:08
yolandaneed to add update-mitaka there10:09
rcarrillocruzyup, grepped, thx10:09
rcarrillocruzi'll push shortly10:09
*** Qiming_ has quit IRC10:11
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Add mitaka to proposal regex
*** daemontool has joined #openstack-infra10:14
*** jaosorior has quit IRC10:14
*** jaosorior has joined #openstack-infra10:15
rcarrillocruzAJaeger, yolanda: ^10:15
*** yamamoto_ has quit IRC10:17
kelepirciI am having some problem with loging in to
kelepirciwhere should I get support fot this?10:17
yolandakelepirci, what problem do you have? sso problems?10:17
*** placko_ has joined #openstack-infra10:18
kelepirciI am trying to login to wiki with OpenID10:18
kelepircithe page redirecting me to ubuntu page and I alsrady have laucnhpad account.10:18
*** esikachev has quit IRC10:18
kelepirciit asks permission and I confirm.10:18
kelepirciafter that I am see a page Log in / create account with open id10:19
kelepirciWhen I click on the button. nothings happens10:19
kelepircithan I stuck with this URL
yolandagoing to check myself10:21
mrmartinyolanda: wiki.o.o is down?10:21
yolandai cannot enter10:22
yolandait takes long time10:22
mrmartinhigh load?10:22
mrmartinI'm trying to login too10:22
*** tzn has joined #openstack-infra10:22
mrmartin28.8 load10:23
yolandai could login successfully and is faster now10:23
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Add the log of package list for pip3
*** ociuhandu has quit IRC10:24
*** rguillebert has joined #openstack-infra10:25
yolandakelepirci, can you try again?10:25
mrmartinok the load is much better now10:26
mrmartinI can imagine it got another attack.10:26
kelepircilet me try10:27
*** yamamoto has joined #openstack-infra10:27
kelepircinope samethings happens10:27
*** bradjones has quit IRC10:27
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
yolandakelepirci, what's your ID for launchpad, so i can check logs?10:28
yolandai see several requests on logs10:28
*** links has joined #openstack-infra10:29
*** kzaitsev_mb has joined #openstack-infra10:29
*** daemontool has quit IRC10:31
*** adrian_otto has quit IRC10:31
yolandaAJaeger, or project-config cores, will fix the proposal slave problem10:33
krotscheckAnyone around who knows things about cloud-config?10:34
*** kzaitsev_mb has quit IRC10:34
* krotscheck is curious about the format, because it might be a good data structure by which he can configure javascript webclient cloud apps.10:34
kelepirciAll users need a nickname; you can choose one from the options below.10:34
kelepirciI am getting following error10:34
krotscheckmordred: When you wake up, I'd like to pick your brain about cloudconfig things.10:36
*** bradjones has joined #openstack-infra10:36
*** bradjones has quit IRC10:36
*** bradjones has joined #openstack-infra10:36
kelepirciI think we are hitting following bug10:36
kelepirciwhere should I a report this bug?10:37
yolandakelepirci, you can file bugs on!/project_group/5510:38
yolandamaybe pabelanger can help you more when he comes online10:39
kelepirciok than I will mage the bug report and wait for pabelanger10:40
rcarrillocruzmordred: need to chat with you about ansible modules and region handling, gimme a ping when you're around10:43
openstackgerritSean Dague proposed openstack-infra/project-config: add uuid-runtime to base image builds
*** BobBall_AWOL is now known as BobBall10:45
*** e0ne has joined #openstack-infra10:50
*** thorst has joined #openstack-infra10:51
*** esikachev has joined #openstack-infra10:51
AJaegeryolanda, rcarrillocruz thanks a lot for fixing the proposal problem that I introduced ;(10:52
yolandaAJaeger np10:53
*** Jeffrey4l has joined #openstack-infra10:53
*** placko_ has quit IRC10:55
AJaegerkelepirci: is this the first time you try to login to the wiki? We disabled new account creation it at one point due to spam - not sure whether it's enabled again10:55
*** placko_ has joined #openstack-infra10:55
openstackgerritMerged openstack-infra/project-config: neutron-lbaas-dashboard project is missing perms
*** esikachev has quit IRC10:56
*** ociuhandu has joined #openstack-infra10:57
*** asettle has quit IRC10:57
*** kzaitsev_mb has joined #openstack-infra10:57
*** thorst has quit IRC10:58
openstackgerritMerged openstack-infra/project-config: Add mitaka to proposal regex
*** zeih has quit IRC11:01
*** kzaitsev_mb has quit IRC11:02
openstackgerritMerged openstack-infra/storyboard-webclient: Edit task notes from board
*** cfarquhar has quit IRC11:03
*** acabot has quit IRC11:03
*** ihrachys has joined #openstack-infra11:04
*** kgiusti has left #openstack-infra11:07
*** salv-orlando has joined #openstack-infra11:07
*** dguitarbite has quit IRC11:07
*** Qiming has joined #openstack-infra11:12
*** boris-42 has joined #openstack-infra11:14
*** salv-orlando has quit IRC11:18
*** vgridnev has quit IRC11:19
*** smarcet has joined #openstack-infra11:20
yolandaAJaeger, are you ok with approving ?11:21
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: Added mising link to config file
yolandapuppet jobs to ubuntu-trusty11:21
*** gongysh has quit IRC11:23
smarcetfungi: yolanda : morning , when u get a change please review , thx !11:25
doug-fishAJaeger: Hi - I see that Zanata is still unavailable this morning - any update?11:29
AJaegeryolanda: yes, I am - but it needs probably manual jenkins update11:29
AJaegerdoug-fish: let's ask smarcet ^11:29
yolandai haven't checked how many jobs, AJaeger. What's the number?11:29
AJaegeryolanda: lots of ;( It did not download the log file completely for me...11:30
AJaegersmarcet: will 293586 fix openstackid for Zanata?11:30
yolandai put a +2 but not approved11:30
pabelangerkelepirci: yolanda: Ya, we've disabled new users on the wiki for the moment. We have a patch up to re-enable it, we need 287232 to land11:30
pabelangeralso, morning11:30
yolandagood morning pabelanger11:31
*** acabot has joined #openstack-infra11:31
pabelangeryolanda: AJaeger: do you have a moment to review 293767? Adds puppet-apply job for debian-jessie, which was brought online last night11:32
pabelangerand experimental job11:33
AJaegerpabelanger: the translation check site needs someone to help moving forward, is this something you have time for? See - might be quite involved but important for release11:33
AJaegerpabelanger: will review shortly11:33
pabelangerAJaeger: sure, I can dive into that this afternoon11:33
*** dalgaaf has joined #openstack-infra11:33
*** yamamoto has quit IRC11:34
AJaegerpabelanger: I made a online fix for (puppet ubuntu-trusty change)11:34
openstackgerritMerged openstack-infra/project-config: Remove gate-tempest-dsvm-full-ceph job
AJaegeryolanda: can you restart the proposal salve again, please?11:34
Zaraanyone have time to look at (storyboard email address config)? it has some +1s but no +2 or +A yet; should be the last step.11:34
yolandai have the window for it opened :)11:34
openstackgerritMerged openstack-infra/project-config: Add devstack-plugin-additional-pkg-repos
*** rhallisey has joined #openstack-infra11:35
AJaegerpabelanger: will conflict with 293719. Could you rebase it on top of each other, please?11:36
*** ldnunes has joined #openstack-infra11:36
AJaegeriff we want to move forward today with 293719...11:37
openstackgerritMerged openstack-infra/project-config: Create new networking-ovn-native-l3-nv check job
openstackgerritMerged openstack-infra/project-config: puppet-pacemaker: don't run beaker tests on ubuntu
*** kzaitsev_mb has joined #openstack-infra11:39
*** mrmartin has quit IRC11:40
openstackgerritMerged openstack-infra/project-config: Add javascript jobs to fuel-ui project
openstackgerritMerged openstack-infra/project-config: Add Barbican & Magnum roles to OpenStack-Ansible
*** thorst has joined #openstack-infra11:42
*** yamamoto has joined #openstack-infra11:42
mattthi all, we have a gate job initiated by tox (see, when i run this locally stuff installed by ansible happens /outside/ of tox's functional venv, but in the gate it installs everything inside the venv11:42
mattti literally invoke 'tox -e functional' in my test env, should i be invoking tox in a different manner?11:42
*** vgridnev has joined #openstack-infra11:43
*** thorst has quit IRC11:43
*** thorst has joined #openstack-infra11:43
*** thorst_ has joined #openstack-infra11:44
*** placko_ has quit IRC11:44
*** placko_ has joined #openstack-infra11:45
pabelangermattt: what version of tox do you have locally?11:47
*** _skolekonov is now known as skolekonov11:48
*** chlong has joined #openstack-infra11:48
matttpabelanger: 1.6.011:48
*** thorst has quit IRC11:48
*** openstackgerrit has quit IRC11:48
matttpabelanger: i installed that from apt, let me try from pip11:48
*** openstackgerrit has joined #openstack-infra11:48
*** Daisy has joined #openstack-infra11:48
pabelangermattt: I also see testenv doing things in your homedir too, so that is an issue11:49
matttpabelanger: what version of tox is in the infra images, do you know off hand ?11:49
AJaegermattt: look at the log file, it shows all versions11:50
pabelanger2.x for sure11:50
*** zeih has joined #openstack-infra11:51
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again
*** mtanino has joined #openstack-infra11:52
AJaegersmarcet: will fix the openstackid/ problems?11:53
*** Daisy has quit IRC11:54
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again
AJaegeryolanda: do you still have the proposal restart window open and can restart it, please? Might have been too soon to restart it ;(11:55
matttpabelanger: tox upgraded in my test env still uses /usr/local/lib, rather than .tox/function/lib11:56
AJaegerthanks again, yolanda11:57
yolandano problem, that's easy :)11:57
* AJaeger will be back later11:58
AJaegeryolanda: works now! 2nd job running ;)11:58
yolandacool. I didn't know what reusable slave function worked in that way now11:58
yolandain downstream we had a parameter_function: reusable-slave per job11:59
doug-fishsmarcet: I'm still concerned about the problems. It's not clear to me that the fix is related to it (maybe it is?)12:01
*** HeOS has quit IRC12:01
doug-fishis the fix related, or are you still working on separately?12:01
pabelangerokay, pushing 293719 now.12:01
pabelangerwill monitor JJB12:01
pabelangermattt: sorry, I don't have time to help debug this morning. Hopefully somebody else will be able to lend a hand12:02
matttpabelanger: np, thanks anyway!12:02
yolandaah, pabelanger has been the brave one to approve puppet moves to ubuntu-trusty :)12:03
*** HeOS has joined #openstack-infra12:05
pabelangeryolanda: Ya, I looked it over last night.12:05
pabelangernow to have JJB work :)12:06
yolandapabelanger, care approving ? to fix a devstack-gate bug12:06
*** yamamoto has quit IRC12:14
*** weshay has joined #openstack-infra12:14
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again
pabelangeryolanda: AJaeger: fungi lets reduce devstack-trusty to 1 for 29350612:16
pabelangerso we can reduce the existing devstack-trusty nodes12:16
nibalizerfungi: okay, running the thing12:17
*** thorst has joined #openstack-infra12:17
*** baoli has joined #openstack-infra12:18
*** _amrith_ is now known as amrith12:18
*** gordc has joined #openstack-infra12:18
openstackgerritMerged openstack-infra/project-config: The puppet-cookiecutter need gems to works.
openstackgerritPaul Belanger proposed openstack-infra/project-config: Change min-ready for ubuntu-trusty/devstack-trusty
*** edmondsw has joined #openstack-infra12:18
pabelangeryolanda: AJaeger: fungi: these land that first, then move devstack-trusty to 0 after we land puppet JJB12:19
doug-fishMaybe smarcet is off now? Is anyone else looking at the problem? Many (all?) translators have been unable to translate for the last 12 hours or so.12:19
pabelangeras we have 159 ready devstack-trusty images12:19
pabelangerdoug-fish: sadly, I don't know the status atm12:19
*** yamamoto has joined #openstack-infra12:20
*** armax has quit IRC12:20
*** hashar is now known as hasharAway12:21
*** aysyd has joined #openstack-infra12:21
*** baoli_ has joined #openstack-infra12:21
*** thorst_ has quit IRC12:21
*** jpr has joined #openstack-infra12:22
doug-fishpabelanger: understood - I guess I'm looking for at least some acknowledgment that it's recognized as a significant problem and is a priority for somebody.12:22
*** BobBall_ has joined #openstack-infra12:23
ianychoidoug-fish, I have tried to find why openstackid is not working, but I am not an infra guy, so it is hard to identify this bug..12:23
*** BobBall_ is now known as Guest4719812:23
ianychoiFor openstack-id, smarcet is the person who commits into openstackid, see:
ianychoiI think he would be the best person to solve this important problem.12:24
*** baoli has quit IRC12:24
pabelangerdoug-fish: I can follow up, like I'm sure you will, but is a mystery to me. Hopefully another infra-root knows a little more12:25
doug-fishianychoi: good to know - thanks for the background! Are other systems unavailable as well? I was only aware of this as a problem12:25
*** dims has quit IRC12:25
*** BobBall has quit IRC12:25
*** rfolco has joined #openstack-infra12:25
ianychoitranslate-dev.o.o and refstack.o.o also do not work now.12:26
doug-fishI see.12:26
*** vgridnev has quit IRC12:26
nibalizerpabelanger: i don't kno wmuch either12:27
*** tzn has quit IRC12:27
*** automagically_ has joined #openstack-infra12:30
pabelangeryolanda: looks like 293719 will need some more work, failing gate now.  So, removing +A atm, as I might have to step away12:30
*** automagically has joined #openstack-infra12:31
pabelangernibalizer: do you have a moment to approve 293506? we're reducing devstack-trusty nodes and increasing ubuntu-trusty12:31
*** vgridnev has joined #openstack-infra12:32
openstackgerritMerged openstack-infra/project-config: Use pip installed dib for functional tests
*** automagically_ has quit IRC12:33
*** asettle has joined #openstack-infra12:34
*** electrofelix has joined #openstack-infra12:34
*** dguitarbite has joined #openstack-infra12:37
*** automagically has left #openstack-infra12:37
*** Keedya has joined #openstack-infra12:37
mnaserunrelated but i feel like infra team might have a thought here, anyone know of a tool which can show iops per process in linux?  iotop does bandwidth per process which isn't answering our specific question12:38
odyssey4meclarkb mordred Based on it looks like we could start to add additional mirrors into the apt mirror? Would it be acceptable, for instance, to add a MariaDB mirror?12:39
*** kgiusti has joined #openstack-infra12:39
*** rlandy has joined #openstack-infra12:39
evgenylHi, kindly ask to consider reviewing/merging patch on repo creation , thanks.12:40
openstackgerritDarragh Bailey proposed openstack-infra/system-config: Remove http.maxWait hack as fixed in Gerrit 2.5
*** piet has joined #openstack-infra12:42
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Refactoring
*** hasharAway is now known as hashar12:42
*** armax has joined #openstack-infra12:43
openstackgerritMerged openstack-infra/project-config: Change min-ready for ubuntu-trusty/devstack-trusty
*** hashar has quit IRC12:45
*** Keedya has quit IRC12:45
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Reduce exceptions noise
*** eharney has joined #openstack-infra12:47
openstackgerritMerged openstack-infra/openstackid-resources: Reduce exceptions noise
*** hashar has joined #openstack-infra12:47
electrofelixregarding irc logs of openstack related channels, is that reserved for just big tent projects, or is it something that any of the projects under the openstack umbrella can avail of?12:47
*** salv-orlando has joined #openstack-infra12:48
*** yamamoto has quit IRC12:48
*** Daisy has joined #openstack-infra12:50
openstackgerritIan Y. Choi proposed openstack-infra/system-config: Openstackid Production Release rollback to 1.0.13
pabelangerokay, stepping away for a few hours. Running over the to USA!12:51
ianychoismarcet, fungi, could you review ? It is so important problem I think.. Please review it and if it is not the solution, would you find a solution for openstackid problems?12:51
openstackgerritIan Y. Choi proposed openstack-infra/system-config: Openstackid Production Release rollback to 1.0.13
openstackgerritIan Y. Choi proposed openstack-infra/system-config: Openstackid Production Release rollback to 1.0.13
Daisyianychoi: Thank you for working on the login problem.12:53
*** esikachev has joined #openstack-infra12:54
pabelangerianychoi: +2 to rollback, if openstackid stopped working after the patch I approved landed12:54
pabelangerassuming rollbacks work12:55
ianychoipabelanger, thanks and really sorry for your review removement.12:55
*** amrith is now known as _amrith_12:55
ianychoiI tried to add an reviewer: smarcet but mis-removed you..12:56
Daisysmarcet, fungi: please help to fix the login problem of translation website as soon as possible. If the patch ianychoi submitted can work as a temp solution, please approve it. We need the translation website workable. It's a busy time for translators. And they have been stand by for a whole day.12:56
smarcethey Daisy12:56
smarcetsorry which is the issue?12:56
ianychoismarcet, would you see ?12:57
DaisyWe cannot login to http://www.openstackorg with Openstack ID now.12:57
*** amotoki has joined #openstack-infra12:57
pabelangerinfra-root, if you have a moment, could use some feedback on our ongoing openid issue ^12:58
*** yamamoto has joined #openstack-infra12:58
*** salv-orlando has quit IRC12:58
smarcetDaisy : mmm login is working for me12:58
Zarasomeone has made a story here if it's useful to update things there:!/story/200051312:59
Daisysmarcet: sorry wrong website12:59
Daisysmarcet: try this: http://translate.openstack.org12:59
Zara(either way, when things are fixed, please someone let the author know.)13:00
smarcetDaisy: should i ask are u using DH association type?13:00
smarcetbc i am seeing a lot of errors related to that13:01
smarcet'Zend\Crypt\Exception\InvalidArgumentException' with message 'Invalid13:01
smarcetparameter; not a positive natural number greater than 1' in13:01
smarcetStack trace: #013:01
smarcetZend\Crypt\PublicKey\DiffieHellman->setGenerator('1') #113:01
Daisysmarcet: I don't know, sorry. pleia2 should know the answer.13:01
smarcetbefore rollback openstackid13:02
smarcetplease let me know that13:02
smarcetbc i think that request are passing wroing parameters13:02
Daisysmarcet: I have to join i18n meeting now. talk to you later.13:03
smarcetwe are not using DH association on www.openstack.org13:03
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Improve TripleO testing coverage for Puppet OpenStack CI
*** katomo has joined #openstack-infra13:03
openstackgerritMerged openstack-infra/openstackid: Refactoring
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Improve TripleO testing coverage for Puppet OpenStack CI
smarcetpabelanger:ianychoi: need to know before rollback which is the openid association type that site is using13:05
ianychoismarcet, it was working so well yesterday.13:06
ianychoiI tried to identify what have occurred for one day, and just found that openstackid version was changed..13:07
*** mtanino has quit IRC13:07
smarcetwell thing is that i need to know which kind of association are u using13:07
smarcetto identify the problem13:07 site is working fine so far, but no use DH assoc13:08
*** asettle has quit IRC13:08
pabelangersmarcet: I'm leaving for a few hours, but will be back. So another root will take over the patch.  We should likely have some sort of recap once we figure out the issue so we don't hit the problem again.13:08
ianychoiHmm.. is not also working.13:09
ianychoiWould something with old openstackid access method would generate some problems?13:09
smarcetianychoi: thing is that 1.0.14 did touch anything related the openid flow13:10
smarcetonly related to oauth213:10
*** markvoelker has joined #openstack-infra13:11
AJaegerpabelanger: I'll rebase 293719 now...13:11
* Zara reads logs, sees openid story was generated thanks to suggestions in this channel, wanders off again :)13:12
pabelangerAJaeger: great.13:12
*** jpr has quit IRC13:13
*** asettle has joined #openstack-infra13:13
pabelangerokay, finally stepping away. devstack-trusty is at min-ready: 1 now. but we have 148 ready ATM. That will go down until puppet JJB change changes13:13
pabelangerwill back in a few hours13:13
*** asettle has quit IRC13:13
*** salv-orlando has joined #openstack-infra13:15
*** ajmiller has joined #openstack-infra13:16
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty
*** pradk has joined #openstack-infra13:18
*** akshai has quit IRC13:19
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Move puppet jobs to ubuntu-trusty
*** amitgandhinz has joined #openstack-infra13:19
*** thiagop_ has joined #openstack-infra13:19
*** bdemers has quit IRC13:19
*** placko_ has quit IRC13:21
AJaegerpabelanger, clarkb, yolanda, this passes local testing now. I rebased first and then fixed the issue that was introduced due to other change getting approved ^13:21
*** placko_ has joined #openstack-infra13:22
openstackgerritSam Betts proposed openstack-infra/glean: [WIP] Enable support for configuring bonded interfaces
*** mrmartin has joined #openstack-infra13:23
*** thiagop has quit IRC13:23
smarcetyolanda: fungi: could i get access to server13:23
smarcetso i could take a look13:23
*** thiagop_ is now known as thiagop13:24
*** mriedem has joined #openstack-infra13:24
*** asettle has joined #openstack-infra13:25
AJaegerARgh, proposal slave is down again ;( yolando, could you try again and check the logs, please?13:27
*** links has quit IRC13:28
*** yamamoto has quit IRC13:28
*** oanson has quit IRC13:29
*** asettle has quit IRC13:30
*** _amrith_ is now known as amrith13:31
SamYaplemordred: w00t more crazy glance things hoooo13:31
SamYapleshould fix the OVH upload issues yes clarkb ?13:32
*** akshai has joined #openstack-infra13:32
*** moravec has quit IRC13:32
*** yamamoto has joined #openstack-infra13:34
amrithhi, how does one get a custom dashboard installed for a project. I'm looking to add some dashboards and would like them to, maybe, show up in,dashboards. I'll likely create them with gerrit-dash-creator and would like the link that it creates to show up here ... is that possible?13:35
*** krtaylor has quit IRC13:36
*** _ari_ has joined #openstack-infra13:36
*** jamesmcarthur has joined #openstack-infra13:36
AJaegeramrith: best talk with jeblair and drag rossella_s in - this would fullfill a request by neutron as well. But it's not clear yet how to do. jeblair gave some suggestions last Friday (see eavesdrop.o.o) but this needs more work13:37
openstackgerritThomas Goirand proposed openstack-infra/project-config: WIP: Add a job to build packages
*** regXboi has joined #openstack-infra13:38
*** hashar has quit IRC13:38
rossella_samrith, I've create a script for neutron...
rossella_samrith, I am looking for a way to have custom dashboard if you find out how to do that, let me know :)13:39
*** hamzy has quit IRC13:40
rossella_sAJaeger, thanks a lot for pinging me!13:40
*** yamamoto has quit IRC13:40
*** jamesmcarthur has quit IRC13:40
*** ddieterly has joined #openstack-infra13:40
*** fhubik has quit IRC13:40
*** jamesmcarthur has joined #openstack-infra13:40
zigoAJaeger: Am I right that the Debian image was successfully built?13:41
*** shardy has joined #openstack-infra13:41
amrithAJaeger, did you say jeblair gave you directions on this channel?13:41
amrithrossella_s, thanks.13:41
AJaegerzigo: no idea ;( I don't have access to these systems13:41
*** ajmiller has quit IRC13:42
AJaegeramrith: yes, he did - read logs from last friday on eavesdrop13:42
SamYaplezigo: w00t debian13:42
*** katomo has left #openstack-infra13:42
amrithAJaeger, thanks13:42
*** oanson has joined #openstack-infra13:42
amrithI'm trying to do one other thing. Find a way to tag reviews into groups. like Mitaka-3, or Mitaka-RC1. Right now the best I'm able to do is star the reviews and then share a link that includes starredby:amrith13:42
amriththis seems lame.13:42
amrithI don't know how to tag reviews.13:42
amrithif rossella_s or AJaeger know how to do that, please share ...13:42
zigoAJaeger: I'm double guessing that it did work, because someone pushed the .qcow2 image at as I asked for (so I can inspect the image and design my script).13:43
zigoAJaeger: Is it correct to say that the logs at are combined logs, and that may contain multiple build attempts before logrotate kicks in?13:43
AJaegerzigo: Sorry, I cannot really help with image building, please wait for other infra folks and discuss with them13:44
AJaegeramrith: one way: Use the same topic ;)13:44
zigoAJaeger: Thanks, will do.13:44
rossella_samrith, this is the irc logs
*** moravec has joined #openstack-infra13:45
*** piet has quit IRC13:45
amriththanks, I was having a hard time finding it ...13:45
amrithAJaeger, use the same topic is an interesting approach13:45
jamesmcarthurfungi: groups is working fine as is www.o.o13:45
amrithI guess I can go and edit the topics13:45
amrithand include something common.13:46
amrithstill seems hokey ;)13:46
jamesmcarthurtalking with smarcet: this morning and it seems like we’ve got two separate errors13:46
openstackgerritRob Crittenden proposed openstack-infra/project-config: Add experimental job to test TLS proxy in devstack
jamesmcarthurrefstack atually gets you to the point of logging in, but then throws a 50013:46
*** hashar has joined #openstack-infra13:46
*** sigmavirus24_awa is now known as sigmavirus2413:47
*** nwkarsten has joined #openstack-infra13:47
jamesmcarthurThese errors started rolling in before the latest patch was approved for OpenStackID. This is just the first time we’ve been able to see the logs.13:47
*** krtaylor has joined #openstack-infra13:48
jamesmcarthurfungi: Try logging into refstack and you get teh proper openid assertion13:48
AJaegerjamesmcarthur: what about That is release critical13:48
jamesmcarthurbut refstack throws this error - {"code": 500, "title": "Internal Server Error"}13:48
smarcetAjaeger: seems that there is some problem on DH that they are seing13:49
jamesmcarthurAjaeger: translate is throwing a different error13:50
jamesmcarthur[2016-03-17 13:49:21] dev.ERROR: exception13:50
jamesmcarthur'Zend\Crypt\Exception\InvalidArgumentException' with message 'Invalid13:50
jamesmcarthurparameter; not a positive natural number greater than 1' in13:50
jamesmcarthurStack trace: #013:50
smarcetthey seems to sending a bad crypto param13:50
AJaegerjamesmcarthur: plesae use for such long pastes13:50
smarcetwe expect greater or equal to 213:50
smarcetthey send 113:50
AJaegersmarcet, jamesmcarthur can we go back to a working version, debug further and then continue?13:51
*** yamamoto has joined #openstack-infra13:51
AJaegerThe translators need access to to translate the Mitaka release13:52
smarcetAJaeger: would be helpfull if we got some examples of the openid requests that translate is doing13:52
*** ifarkas has quit IRC13:52
AJaegersmarcet: I'm not an infra-root and can't help on that front13:52
*** tzn has joined #openstack-infra13:53
*** ddieterly is now known as ddieterly[away]13:53
smarcetme neither ;)13:53
*** dims has joined #openstack-infra13:53
*** piet has joined #openstack-infra13:53
jamesmcarthurAjaegar: we are in a precarious position here b/c we are right in the middle of testing iOS apps. If we roll back, they break and we can’t submit app for approval in time for summit.13:53
smarcetsmarcet: IHMO, we started to see those errors a long before than 1.0.14 was released13:53
*** yamamoto has quit IRC13:53
yolandasmarcet sorry i was out13:54
*** xyang1 has joined #openstack-infra13:54
jamesmcarthursmarcet introduced a patch to get root access to the OpenStackID server. I think we could get this straightened out fairly quickly if we could just get access to this server.13:54
jamesmcarthurif we roll back, we’re not going to be able to figure out what the error was13:54
yolandasmarcet, i +2 your change, it needs another infra-root there13:55
jamesmcarthuryolanda: thank you!13:55
*** sshnaidm has quit IRC13:55
smarcetyolandaL thx u!13:55
smcginnisGerrit permissions question - I wasn't in cinder-milestone so I didn't have +2/+A on the new stable/mitaka branch.13:56
*** bdemers has joined #openstack-infra13:56
smcginnisI've been added now, but after logging out and back in I still only have -1/+1.13:56
smcginnisDoes it take a bit for those permissions to propogate?13:56
*** mtanino has joined #openstack-infra13:57
amrithAJaeger, rossella_s ... this may help.
jamesmcarthurAjaegar: I think where we’re also stuck is there were 0 changes on openid b/w releases. It’s not impossible that something in this release broke, but the fact that these errors are coming from openid, not oAuth2 doesn’t poitn to that. Hopefully if smarcet can get another +2, we can see what’s happening pretty quickly.13:58
Daisysmarcet: I'm back.13:59
AJaegeramrith: the question is how to *push* it there. neutron wants to update the dashboard daily13:59
Daisysmarcet: any progress with the login issue ?13:59
*** e0ne has quit IRC13:59
amrithAJaeger, I guess one finds how to do it once, then use the formula found on any bottle of shampoo ...13:59
amrithI'm still trying to figure out how to do it the first time.14:00
*** e0ne has joined #openstack-infra14:00
amrithright now, having trouble understanding how to get to /refs/metadata/...14:00
rossella_samrith, that's in the server where gerrit runs I think14:01
*** jamesmcarthur has quit IRC14:01
amrithand this airline wifi is flakey ;(14:01
smarceti dont have access to server14:01
*** jamesmcarthur has joined #openstack-infra14:01
smarcetmore over i dont have any sample requests from translate14:01
amrithwell rossella_s the things in that chris wang blog post actually do get me to the config section of the project.14:01
amrithi.e. if'm doing this on a clone of the trove project14:01
smarceti am suspect that translate is sending openid.dh_gen = 114:01
smarcetand that is the problem14:01
amrithand when I git fetch ... I end up seeing the project config14:02
amrithI guess one has to push that14:02
amrithand for that one would probably need to be an admin of some kind14:02
rossella_samrith, no idea how infra manages that, sorry14:02
fungi500 lines of scrollback. should catch up shortly14:02
*** ddieterly[away] is now known as ddieterly14:03
amrithwell, a step in the right direction takes you a step closer to the end point.14:03
amriththanks rossella_s AJaeger14:03
rossella_samrith, you should have a loot at jeepyb14:03
*** ifarkas has joined #openstack-infra14:03
AJaegerfungi, basically two problems right now: 1) openstackid and not working yet and 2) proposal slave goes offline. We thought it was fixed but seems not permanent ;(14:03
amrithgot it,
Daisysmarcet: cannot we just rollback ? I don't think our translators can wait. We are in busy time now.14:04
rossella_samrith, didn't have much time till now but that's my next step to try to understand how to make the dashboard available14:04
* AJaeger has to step out14:04
rossella_samrith, I found this old review, not sure if we can still use something from it
fungiAJaeger: is indeed offline. it looks like the last job it ran was django_openstack_auth-propose-translation-update-mitaka and that offlined it, so i bet we're missing an adjustment to the zuul parameter script to recognize the mitaka translations as a job name pattern to not offline after14:04
fungiyolanda: ^14:04
fungioh, looks like there's a fix14:05
fungiwhich is merged, but... did anyone re-online the worker? it's still offline14:05
rcarrillocruzis it down again? cos I pushed a change earlier that got merged14:06
yolandai put online a bunch of times14:06
yolandagoing to do it again14:06
*** nwkarsten has quit IRC14:06
yolandaso maybe i didn't replicate to jenkins masters14:06
yolandastable-mitaka are turning the node off on each run14:07
smarcetDaisy: i dont have the permissions to do that14:07
*** nwkarsten has joined #openstack-infra14:07
*** infra-red has joined #openstack-infra14:08
*** infra-red has quit IRC14:08
jamesmcarthurDaisy: based on the changes that went through and the errors we’re seeing from translate it seems we might be dealign with another issue.14:08
*** elmiko has joined #openstack-infra14:08
openstackgerritMichael McCune proposed openstack-infra/irc-meetings: correct data processing meeting times
jamesmcarthurWe are all under major crunch time. If smarcet can  get root accesss to the server, we can help troubleshoot pretty quickly.14:09
Daisysmarcet, fungi, AJaeger : As to the translation website login issue, I think the issue can be resolved by pleia, Zanata developers, and smarcet together. But because of the time difference, Zanata developers are not available now. I don't have the right to access that server too. But it's the most busy time in a release for translators. We have only 1 week left. We need to have the server...14:09
Daisy...available ASAP. Please, find a way to bring the server back to translators. please.14:10
*** piet has quit IRC14:10
ianychoiPlease.. +1 for Daisy's request14:10
*** nwkarsten has quit IRC14:11
openstackgerritMerged openstack-infra/puppet-openstackid: Added mising link to config file
*** moravec has quit IRC14:14
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** jaypipes has quit IRC14:15
*** piet has joined #openstack-infra14:15
*** placko_ has quit IRC14:16
fungismarcet: jamesmcarthur: i believe pleia2 and StevenK worked on the openid configuration for zanata on but they're in singapore and australia right now i think. we can collectively start reverse-engineering, but it looks like we're just passing an openid url to a zanata puppet module red hat provided. i'm digging up the details now14:16
*** placko_ has joined #openstack-infra14:16
*** tonytan4ever has joined #openstack-infra14:16
jamesmcarthurfungi: would you be able to +2 smarcet’s request for root access so he can review the request in the logs?14:17
fungijamesmcarthur: which one? i already approved one a little bit ago14:17
*** esikachev has quit IRC14:18
jamesmcarthurah - that was it :)14:18
jamesmcarthuroh, not that was not it ;)14:18
jamesmcarthurit’s the one Yolanda +2’d14:18
*** ajmiller has joined #openstack-infra14:19
fungioh, the one where we're giving up on separation of concerns14:19
fungianyway, let's go ahead and roll back. if it turns out that our need for a stable openid provider for our community systems is incompatible with the development model for the marketing website then we should probably plan to fork the codebase temporarily and stand up separate servers while we work out whether there's a simpler openid implementation we can migrate to14:19
*** dims has quit IRC14:19
*** amrith is now known as _amrith_14:19
Daisy+1 to fungi . Thank you.14:20
fungiit doesn't sound like we can take the time for a lengthy debate with the root team right now about whether we're comfortable giving foundation contractors root accounts on the authentication system which was intended to serve as the security linchpin of all our production systems14:20
ianychoi+1 thanks a lot, fungi14:21
*** jaypipes has joined #openstack-infra14:22
*** jamesmcarthur_ has joined #openstack-infra14:22
*** rbrndt has joined #openstack-infra14:22
*** dims has joined #openstack-infra14:22
fungiseparately, we can dig into what refstack, keystone and zanata are collectively doing which is incorrect in light of the recent changes to openstackid and try to get their respective upstreams to address those issues14:23
jamesmcarthur_fungi: “oh, the one where we're giving up on separation of concerns” - I’m not sure I understand that. It’s pretty valuable for a dev team to have access to logs.14:23
*** yamamoto has joined #openstack-infra14:23
fungijamesmcarthur_: we set up a complete copy of the platform for that exact purpose14:23
*** jamesmcarthur has quit IRC14:23
*** jamesmcarthur_ is now known as jamesmcarthur14:23
AJaegerjamesmcarthur_: access to logs != root access14:23
*** Guest47198 is now known as BobBall14:23
jamesmcarthurfungi: but real traffic doesn’t hit the copy of the platform14:23
fungithere's a which is supposed to be authenticating against openstackid-dev.openstack.org14:24
AJaegerjamesmcarthur: we have an translate-dev as well14:24
AJaegerfungi, so going forward with by ianychoi ?14:24
fungithe intent was to use that to be able to test new changes to before updating it, so we could make sure we don't break the translation site authenticationm14:24
*** dims has quit IRC14:25
fungialso if you need logs data from i'm happy to provide it to you directly14:25
jamesmcarthurfungi: this is the first time we’ve heard of translate-dev or that translate is using openstackID. I’m glad they are :) but in the meantime, rolling back is going to break active fixes in our own development14:25
*** yamamoto has quit IRC14:26
jamesmcarthurfungi: so smarcet won’t get approved for root access to that server?14:26
ianychoiYes, and translate-dev.o.o is not also working now. Maybe Zanata developers and pleia2 would help for logs on translate-dev.o.o and the integration between translate-dev and openstackid-dev later (I am not sure whether translate-dev uses openstackid or openstackid-dev)14:27
*** mrmartin has quit IRC14:27
smarcettranslate dev is poiting to o_014:27
*** yamamoto has joined #openstack-infra14:27
*** gongysh has joined #openstack-infra14:27
*** pcaruana has quit IRC14:28
fungiianychoi: pleia2 is travelling and i don't know whether she sees irc which is why we need to reverse engineer this14:28
Daisyagree, ianychoi . +114:28
*** mrmartin has joined #openstack-infra14:28
fungijamesmcarthur: i'm not going to make an on the spot decision to grant root access to servers to people who aren't already heavily involved in maintaining any of our other systems, particularly when that's the authentication system for several important community (not foundation) platforms14:28
jamesmcarthurfungi: we’ve been asking for this for months, so it’s not something brand new14:29
ianychoifungi, would you approve ? It is for rollback from openstackid 1.0.14 to 1.0.13, the previous version which had been used up to one day ago.14:29
*** achanda has joined #openstack-infra14:29
*** tongli has joined #openstack-infra14:29
jamesmcarthurif you want to wait, I understand, but it feels like we are constantly battling against this, when it’s a very reasonable request to give the most knowledgeable dev access to logs in real time14:29
ianychoiUp to one day ago, translate.o.o was working so well.14:30
fungijamesmcarthur: and so far what opinion there has been from the group has been against that idea. having a sudden breakage of the platform doesn't instill confidence that also granting sysadmin access for that to the developer team is wise14:30
*** nwkarsten has joined #openstack-infra14:30
ianychoiIt is different from root access or log access issue..14:30
smarcetianychoi: Daisy:fungi: translate-dev is poiting to openstackid production14:30
smarcetwe are only asking root access for server14:30
*** esikachev has joined #openstack-infra14:30
fungismarcet: yes, i believe that was from when it was being used to test a different zanata release, so it needs to be switched back to authenticating against openstackid-dev again14:31
AJaegerjamesmcarthur: so, what can we do to get you more involved so that you can learn about the complete setup and realizes how systems are connected and gain the trust by the root team to include you or smarcet?14:31
*** nwkarste_ has joined #openstack-infra14:31
*** placko_ has quit IRC14:31
ogelbukh_please, review the project add patch: looks like all relevant signoffs are given14:31
jamesmcarthurAjaegar: we are not asking for sysadmin access. We are asking for root access to a single server.14:31
*** nwkarste_ has quit IRC14:32
smarcetfungi: there is any ref stack dev instance?14:32
*** placko_ has joined #openstack-infra14:32
*** _amrith_ is now known as amrith14:32
*** nwkarste_ has joined #openstack-infra14:32
jamesmcarthurAjaegar: as far as getting us more involved, it would defintiely be helpful if we knew of dependencies on openstackID. groups.o.o is obviouisly still working just fine as is www.o.o.14:32
*** gongysh has quit IRC14:32
jamesmcarthurThose are dependcies we check for when testing.14:32
fungismarcet: i encouraged the refstack maintainers to put one together but there isn't a dev platform for it yet, no14:33
jamesmcarthurrefstack and translate were unkown to us14:33
*** hamzy has joined #openstack-infra14:33
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Switch translate-dev to openstackid-dev
*** oanson has quit IRC14:33
AJaegerfungi, let's fix the translate-dev system ^14:33
fungialready approved14:33
jamesmcarthurfungi: so it looks like there are lots of things that might have caused this - no dev server for refstack and translate-dev pointing to openstackid production. I think it’s misplaced to put this on our development.14:34
Daisywhen will it take affect ?14:34
DaisyI mean, translate-dev.o.o ?14:34
ianychoiDaisy, not yet unfortunately..14:34
DaisyI can help to test.14:34
*** nwkarsten has quit IRC14:34
fungithis should hopefully give us insight into what's wrong between the expectations of zanata and openstackid master while we have temporarily rolled back14:34
*** tonytan4ever has quit IRC14:34
ianychoiWow, good to hear that14:34
AJaegerDaisy: translate-dev: After merge and next puppet run - expect an hour...14:34
*** gongysh has joined #openstack-infra14:35
Daisyok. 1 hour. wait.14:35
*** piet has quit IRC14:37
*** gongysh has quit IRC14:38
fungijamesmcarthur: yes, i'm just looking for an expedient means of identifying the problem for now without altering the workflow. i agree having translate-dev left authenticating against production was not great, but i also don't know that anyone bothered to try it since i agree the changelog for the new openstackid release didn't imply any behavior changes to the openid protocol handler14:38
AJaegeryolanda, fungi Is there a jenkins that is still taking down the proposal slave? It seems to be gone missing again ;(14:38
jamesmcarthurfungi: right - our concern is that this has been potentially going on longer than we thought14:38
fungiAJaeger: horizon-propose-translation-update-mitaka took it offline this time14:39
*** gongysh has joined #openstack-infra14:39
jamesmcarthurand again, translate wasn’t on our list of things to test with openstackID14:39
*** tonytan4ever has joined #openstack-infra14:39
*** sdake has joined #openstack-infra14:39
*** sigmavirus24 is now known as sigmavirus24_awa14:40
AJaegerfungi, so is the regex wrong? Or not updated?14:40
*** achanda has quit IRC14:40
*** erlon has joined #openstack-infra14:40
fungijamesmcarthur: right, i think whatever's broken translate logins was in the last 24 hours because they've been using it heavily. i don't know the frequency with which refstack is used but would need to hear more from its users to know a rough timeframe for when the issue started. also they wrote their own openid consumer so i wouldn't be surprised if it's generating some error messages14:40
*** sigmavirus24_awa is now known as sigmavirus2414:41
* AJaeger simplifies that regexes...14:41
fungiAJaeger: i've not had time to review the regex update change so i don't know, but my guess would be that it's wrong14:41
sdaguefungi: fyi, this has broken a bunch of folks -
sdagueor, the lack of that14:42
*** Qiming_ has joined #openstack-infra14:42
*** kaisers_ has joined #openstack-infra14:42
openstackgerritMerged openstack-infra/system-config: Openstackid Production Release rollback to 1.0.13
*** piet has joined #openstack-infra14:42
anteayafungi: fires today, trying to find a way to help, so far it all looks like root things14:43
*** jaosorior is now known as jaosorior_away14:43
fungisdague: that's fine for now, though we should probably move that entire list into devstack's early bootstrap phase at some point14:43
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Update wanted_ci_job to reflect current status of things
clarkbsdague: there is a different change to fix it in dib direvtly based on scrollback14:44
clarkbhowever this shpuld go in devstack...14:44
AJaegeranteaya: can you tell me what'S wrong with the regex in ?14:44
clarkbyou dont need uuidgen as a dep to install packages right?14:44
fungiideally devstack should be able to work on as minimal a base system as possible and install all the things it knows it needs14:45
anteayaAJaeger: I can look, I don't profess that regexes are my strength14:45
AJaegerSo, regex changes from (constraints-.*|updates?|update-liberty|plugins-list))$ to (constraints-.*|updates?|update-(liberty|mitaka)|plugins-list))$ - and does not catch update-mitaka ;(14:45
* anteaya looks14:45
AJaegeranteaya: see above ^14:45
*** flepied has quit IRC14:45
AJaegerclarkb: I had to rebase and fix your devstack-trusty puppet patch, hope it's fine now...14:45
AJaegerfix=fix after other changes merged14:45
*** Qiming has quit IRC14:46
sdaguefungi: maybe, I also think that the goal in dib was to build common images that work for things, and in current definitions is about making all tools have to assume different minimums than in the real world14:46
*** jamesmcarthur_ has joined #openstack-infra14:46
sdagueclarkb: I don't have scrollback, so that's the best I got. People were still complaining about the breaks this morning.14:47
*** jamesmcarthur has quit IRC14:47
*** jamesmcarthur_ is now known as jamesmcarthur14:47
fungismarcet: jamesmcarthur: so digging into the zanata puppet module, the openid url is being passed directly into zanata's xml config file:
*** Daisy_ has joined #openstack-infra14:49
* AJaeger just tested locally the regex from 293923 and it matches on 'horizon-propose-translation-update-mitaka'14:49
AJaegerfungi, clarkb: What kind of regex bug was on the bare-precise, does this hit us here perhaps as well?14:49
*** Sukhdev has joined #openstack-infra14:49
fungismarcet: jamesmcarthur: there's also a security-domain name="zanata.openid" block which is basically just turning on the openid support in zanata and providing the provider url, so i think any alterations in dh options are likely going to imply changes in the zanata codebase14:50
*** gongysh has quit IRC14:50
*** yamahata has joined #openstack-infra14:50
fungired hat's devs working on zanata are in australia i think, so it will take a little time to pull them into the discussion14:50
fungii'm going to start digging up the zanata source code, though java makes me go cross-eyed after not too long14:51
Daisyfungi: yes, there are in Australia.14:51
*** tonytan4ever has quit IRC14:51
AJaegerDAisy, can you send an email to Alex and co and ask them to join us here once they are awake, please?14:52
Daisyok. I will do it now.14:52
DaisyI can send him short message.14:52
clarkbsdague: uuid-runtime doesnt appear to be a valid fedora package name, util-linux is where that is hiding?14:52
openstackgerritMerged openstack-infra/devstack-gate: Fix bug in populating multiline env vars on reproduce
sdagueclarkb: yeh14:52
AJaegerthanks, Daisy14:52
sdagueit's only an ubuntu issue14:52
clarkbyes but that package install is for everything14:53
sdagueok, so is the issue the meta definitions in dib don't map all the way through?14:53
anteayaAJaeger: I don't see anything wrong with the new regex, the one that has update-(liberty|mitaka) in it14:53
*** Daisy_ has quit IRC14:53
anteayaAJaeger: so perhaps it is as you say and there is a bug with the parser14:54
clarkbyou didnt update the meta def I dont know that they ship a complete one for you14:54
fungiAJaeger: zuul's status page says "Last reconfigured: Thu Mar 17 2016 12:39:07 GMT+0000 (UTC)" so was that before the regex update merged?14:54
clarkbor ship one at all14:54
AJaegerfungi, that should be the time the regex update merged14:55
*** piet has quit IRC14:56
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build packages
fungiyep, merged at 10:59 utc so should be well before14:56
*** piet has joined #openstack-infra14:56
*** Zhongjun_ has joined #openstack-infra14:56
*** bdemers has quit IRC14:57
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Update wanted_ci_job to reflect current status of things
*** tonytan4ever has joined #openstack-infra14:58
*** amitgandhinz has quit IRC14:58
fungiAJaeger: it's possible these jobs were already in the gearman queue with the old parameter14:58
*** amitgandhinz has joined #openstack-infra14:58
fungiclarkb: ^ yeah?14:58
*** nadya has quit IRC14:59
AJaegerfungi, yes, they were14:59
AJaegerfungi, they were already in zuul since 6:00 UTC14:59
fungiparameters are going to be in the work request, i believe, so if they were queued in gearman before the reconfigure i think they retain their parameters14:59
*** nadya has joined #openstack-infra14:59
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages
AJaegerSo, let's restart the workers once the queue is empty?15:00
*** ifarkas_ has joined #openstack-infra15:00
fungiwhich means our jenkins admins will probably just keep onlining the worker until those flush through, unless we want to restart zuul15:00
AJaegerThere'S one more job with -mitaka in the queue and then we should be good15:00
AJaegerThus one more restart I hope ;)15:00
*** zz_dimtruck is now known as dimtruck15:01
*** nadya has quit IRC15:02
*** nadya has joined #openstack-infra15:02
clarkbsdague: ya doesnt look like the pkg-map element has any default mappings.15:02
*** ifarkas has quit IRC15:03
*** sshnaidm has joined #openstack-infra15:03
fungismarcet: it looks like most of the base openid options are being inherited from
AJaegerTime now for the final restart of the proposal slave, please15:03
*** achanda has joined #openstack-infra15:03
clarkbwe just need to provide a pkg-map file and should be good15:03
*** hashar is now known as hasharMeeting15:03
AJaegerDaisy, ianychoi, doug-fish, fungi, smarcet, jamesmcarthur : I can access again!15:04
Daisyhow did you make it happen ?15:05
fungismarcet: i'm looking through some of the subclassing of that now to see what they do for some of the specific providers, as that mighth indicate overrides we'd be able to use15:05
DaisyAJaeger, I 'm in too.15:05
zarosdague: here's the project for integrating ci test results to Gerrit
smarcetDaisy: ok i will test dev version of openstackid with dev version of translate once its up15:06
DaisyAJaeger: I have sent email to Alex, cc you. I also leave him a short message. I think he will come here when he sees it.15:06
doug-fishAJaeger: that's great!15:06
smarcetfungi: i am still seeing the error of DH params15:06
smarcetseems that wasnt translate after all15:06
ianychoiMe also works translate.o.o well, thanks all!15:07
zarosdague: only backend stuff so far, we are hoping to realize the displaying pretty soon.15:07
smarcetfungi: waiting to test on dev15:07
ianychoiAlso, refstack works well15:07
sdaguezaro: cool15:07
funginibalizer: did you have any luck with the run? instructions clear enough?need any help cross-checking the results?15:08
Daisycan I announce the server is back to i18n mailing list ?15:08
Daisyor you still want to do some more test ?15:08
*** achanda has quit IRC15:08
fungiDaisy: should be able to go back to using it for now. we're going to figure out the issue by testing between our translate and openstackid dev instances before trying to upgrade openstackid production again15:09
Daisywe have Europe translators. They are in daytime and they are waiting for the server back too.15:09
*** IlyaG has joined #openstack-infra15:09
DaisyGot it, fungi. Thank you. I will let our translators know.15:09
*** gongysh has joined #openstack-infra15:09
anteayahow does patchbot keep getting into #openstack-meeting?15:11
openstackgerritSean Dague proposed openstack-infra/project-config: update nova skipif because of tox name changes
*** dprince has joined #openstack-infra15:11
sdagueanteaya: it's notmyname's right?15:11
fungismarcet: in the apache error log, it looks like we were getting "Call to a member function postConsent() on a non-object in /srv/openstackid/slot1/app/controllers/openid/UserController.php on line 288" up until a little after 21:00 utc but no sign of that since15:11
anteayasdague: that is my understanding15:11
notmynameanteaya: I probably forgot to part it yesterday15:11
anteayaand jeblair removed patchbot last time15:11
sdagueAJaeger: I think one renaming issue happened -
anteayanotmyname: jeblair did suggest puppeting it15:12
anteayanotmyname: have you considered that?15:12
notmynameanteaya: I'm not sure what you mean by that15:12
anteayainfra works on the basis that all our services are automated15:12
fungismarcet: though that looks infrequent so probably just crawlers hitting something normal users don't15:13
anteayawe have puppet modules for all our services, so that we don't have to manually add and remove things15:13
AJaegersdague: let'S approve ;)15:13
anteayanotmyname: jeblair suggested that if you want to add things to infra services you adhere to infra workflow for doing so15:13
Zara(storyboard email sender config still waiting in review: , mentioning again since I think I picked a bad time earlier because of different timezones)15:13
*** IlyaG has quit IRC15:14
anteayanotmyname: we use puppet for the modules and ansible for the orchestration layer15:14
notmynameanteaya: I'm not going to expend any effort to move/run patchbot to -infra. the code's out there if someone else wants too. it's a useful tool I built for me that some other people also find useful15:14
anteayanotmyname: as puppet is not great at orchestration15:14
*** Qiming_ has quit IRC15:14
notmynameanteaya: that being said, I don't see anyone in -meeting complaining about patchbot (ie in the current meeting). is it causing a problem for you in there?15:14
anteayawell we do administer devlopement tools15:15
anteayaincluding channel administration15:15
sdagueAJaeger: sure15:15
smarcetfungi: i think so15:15
anteayaand meeting channels are shared space15:16
sdagueAJaeger: will need skipif changes as well, right?15:16
AJaegersdague: the change is (wrongly) in
*** gongysh has quit IRC15:16
AJaegerSo, I'll rebase now and fix...15:17
notmynameanteaya: I'm not asking about the Way Things Are or whatever. is it causing *you* any particular pain?15:17
sdagueAJaeger: ok, cool15:17
notmynameanteaya: I've talked to jeblair about it, and I don't have it lurk in any common meeting channel any more.15:17
*** dprince has quit IRC15:17
anteayaI see it as part of my role to proactively notice things that infra may be called on to address15:17
*** claudiub|2 has quit IRC15:17
anteayaand behaviour of meeting channels is one of them15:18
nibalizerfungi: ohai15:18
nibalizeri am running it now15:18
nibalizerim only just now getting back to real wifi15:18
funginibalizer: it may take longer than the estimate in the comments when gerrit is under load since it depends a lot on rest api call response times, but shouldn't take more than an hour tops15:19
*** jamesmcarthur has quit IRC15:19
nibalizeri am running it from france too15:19
*** jamesmcarthur has joined #openstack-infra15:19
nibalizerso thats an effect15:19
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs
AJaegersdague: rebased, could you review, please ? ^15:20
funginibalizer: hopefully not a huge impact since i did everything i could to batch/reduce individual api calls15:20
fungibut rtt will still play some into it15:20
AJaegerfungi, yolanda it's time now for the final restart of the proposal slave. Could either of you do it, please?15:20
fungiAJaeger: done15:21
fungii've been watching/kicking it as i notice it go back offline each time15:21
*** moravec has joined #openstack-infra15:21
*** flepied has joined #openstack-infra15:21
yolandaso what was going on? changes not applying on jjb?15:21
fungiyolanda: the parameter function tells zuul what parameters to include in the gearman work request15:22
*** dprince has joined #openstack-infra15:22
AJaegeryolanda: and it's applied the first time it enters the queue15:22
*** pvaneck has joined #openstack-infra15:22
nibalizerfungi: ok ill keep you posted15:23
AJaegeryolanda: so, the new regex only got applied on *new* jobs but not the once already in periodic queue15:23
fungiyolanda: so anything _new_ it enqueued in gearman after that config change took effect would no longer get the offline parameter, but jobs it had already queued in gearman still did15:23
yolandaok understood, and seems there was quite a long queue15:23
anteayaso as a data point we have a whole lot of available nodes, we have 175 available and 350 used:
fungioptions were to wait it out and keep kicking the worker back online until those flushed through, or restart zuul to clear the gearman queue (which unfortunately would have also lost everything from the post pipeline as well)15:24
*** bdemers has joined #openstack-infra15:24
anteayaand it appears our surplus of available nodes has been evident for about 12 hours15:24
*** mrmartin has quit IRC15:24
fungianteaya: yes, those are devstack-trusty nodes which aren't getting used but which nodepoold is building due to perceived demand because it still sees gearman registrations for old jobs running on devstack-trusty15:24
fungianteaya: that will continue until we restart zuul15:25
anteayaah sorry if this is an old itme15:25
anteayavery good15:25
fungiwhich will clear out any stale job registrations in gearman15:25
AJaegerwe should merge some time today if possible - it moves the old puppet jobs to ubuntu-trusty15:25
openstackgerritMerged openstack-infra/project-config: Skip nova-tox-db-functional job on doc only changes
clarkb has a slight bug in it in that activate must be sourced15:27
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib
*** Sukhdev has quit IRC15:28
AJaegerclarkb: good find. Will you fix or shall I?15:28
clarkbAJaeger: go for it I am still migrating to proper computer15:29
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix dib virtualenv setup
AJaegerclarkb: done ^15:30
fungismarcet: in the openstackid application log, this debug line looks like all that was being logged when someone would try to authenticate with translate.o.o:
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages
openstackgerritMerged openstack-infra/system-config: Switch translate-dev to openstackid-dev
*** jamesmcarthur has quit IRC15:33
*** angdraug has joined #openstack-infra15:33
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages
*** jpr has joined #openstack-infra15:36
*** jamesmcarthur has joined #openstack-infra15:36
*** Hunner has joined #openstack-infra15:36
*** Hunner has quit IRC15:36
*** Hunner has joined #openstack-infra15:36
*** yamahata has quit IRC15:37
*** nwkarsten has joined #openstack-infra15:37
AJaegersdague: are the skip-ifs now fine on and ?15:40
fungismarcet: jamesmcarthur: also, the rollback does seem to have gotten authentication for working again as well15:40
jamesmcarthurfungi: indeed - unfortunately we don’t know why15:41
*** nwkarste_ has quit IRC15:41
jamesmcarthuras soon as refstack and translate point to our openstackID-dev server we can sort it out15:41
nibalizerfungi: i think its made it to the 'd' section15:41
jamesmcarthurmeanwhile, our apps are broken :\15:41
AJaegerjamesmcarthur: the change for translate-dev has merged, not sure whether it is deployed already15:41
*** Swami has joined #openstack-infra15:42
fungiAJaeger: it'll still be a few minutes15:42
jamesmcarthurAjaeger: thanks sir15:42
zigoAJaeger: If I don't use "name", then I use what?15:42
jamesmcarthurwe’ll take a look as soon as it’s ready15:42
fungijamesmcarthur: new apps, or non-backward-compatible lockstep changes between the apps and the openid provider?15:42
*** amrith is now known as _amrith_15:43
zigoAJaeger: Just: "- pkgdeb-build-pkg" ?15:43
jamesmcarthurnew apps15:43
fungigot it15:43
AJaegerzigo: yes15:43
zigoCheers, doing it.15:43
zigoAJaeger: Same for gate:, I guess, right?15:43
AJaegerzigo: yes15:43
zigoOk, doing it, thanks.15:44
smarcetfungi: after that we resolve the issue with translate can we redeploy?15:44
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Correct branch limitations for ironic-inspector
zigofungi: Is it you who uploaded 176625.qcow2?15:45
fungismarcet: yes, i'm less concerned about refstack at the moment since the real urgency is translators getting their work merged in our very short string freeze window prior to release. if the upgrade re-breaks refstack i think we take our time troubleshooting it by pointing it at openstackid-dev and iterating there (since they don't have a dev deployment they can use)15:45
fungizigo: i did not, but i asked pabelanger to do so since he was the one keeping an eye on the image builds yesterday15:46
zigofungi: Ah, thanks so much. I'm nearly done downloading it! :)15:46
zigoIt's BIG...15:46
*** jpr has quit IRC15:47
fungiwe gave you a heads up on the url to it in channel, but you probably didn't see (it's been busy in here today)15:47
fungizigo: as soon as diskimage-builder tags a new release, we expect the qcow2 sizes to drop dramatically15:47
smarcetfungi: roger15:47
openstackgerritEric Harney proposed openstack-infra/project-config: Add NFS devstack plugin
clarkbfungi: not really15:47
fungiclarkb: no?15:48
clarkbfungi: the ~5.6GB size is about what we expect. We shouldn't get 11 or 12GB qcow2s anymore though15:48
fungioh, i thought we were still waiting for the fstrim change in a new release15:48
clarkbfungi: we are, the problem fstrim solves is the 11ish GB image sizes which happen for reasons I still haven't been able to fiure out15:49
fungialso i didn't look at the size of that image, but if it's in the 5-6gb range then that's correct for our images yes15:49
openstackgerrityolanda.robla proposed openstack-infra/project-config: Add non voting job to test k8s with dib atomic image
*** sridhar_ram has joined #openstack-infra15:49
clarkbmost of our qcow2s end up in the 6GB range and I don't think we can get them much smaller than that without deleting content15:49
*** apoorvad has joined #openstack-infra15:49
fungiright, that i agree with15:49
zigoIt still takes a long time to download 6GB with wget... :P15:50
fungiand i just checked, this image is indeed 5.6G so shouldn't expect it to get any smaller15:50
zigoBut I got the image now.15:50
*** ybathia has joined #openstack-infra15:50
*** bpokorny has joined #openstack-infra15:50
fungiyep, i agree it's a big file ;)15:50
clarkbmost of that is the git repo cache, then the OS install then the jenkins/devstack images15:50
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs
*** piet has quit IRC15:51
*** piet has joined #openstack-infra15:51
clarkbone of the larger repos is the activity site repo15:52
*** nmagnezi has quit IRC15:52
clarkbbut its still less than 200MB15:52
clarkbanteaya: the large number of ready nodes are devstack-trusty hwich are mostly unused at this point (puppet is still using them)15:52
*** nadya has quit IRC15:53
zigoTrying to use kpartx, it doesn't seem to show any partition...15:53
anteayaclarkb: thanks15:53
jmccroryhi all, I have an approved change that depends-on a change in another repo. The depended on change has merged, but I can't seem to get the first to go. Any ideas on how to get it moving?15:53
clarkbzigo: there should be a single / ext4 filesystem15:53
anteayapuppet patches should be happy right now15:53
clarkbzigo: not sure how that ends up in a partition table with qcow215:53
zigoclarkb: So I can just mount it using loopback?15:53
anteayajmccrory: link to patch in question?15:53
AJaegerjmccrory: somebody needs to approve it again - or toggle the +A.15:53
clarkbzigo: yes, do whatever the dance is for qcow2 images15:54
*** jpr has joined #openstack-infra15:54
zigoclarkb: oh, I'm silly, I should convert it first to a raw format using qemu-utils ...15:54
jmccroryAJaeger: ah ok, thanks!15:54
anteayajmccrory: as AJaeger said15:54
anteayajmccrory: the dependency was +A'd before the depends-on was merged15:55
clarkbsdague: were you going to write the pkg-map file for the uuid-runtime change? should someone else push that patch?15:55
jmccroryanteaya: yep looks like that's what happened, thanks. missed that section when reading through earlier15:55
*** krtaylor has quit IRC15:55
anteayajmccrory: a new +A on the dependency is needed for zuul to see that as a trigger, since at the time of the first +A the conditions weren't met15:55
anteayajmccrory: yup, zuul has no concept of time, it lives in the ever present now15:56
sdagueclarkb: it was not on my radar, as I'm in another stack of code15:56
anteayait is a rather zen tool15:56
zigoBAM, 17 GB... :P15:56
openstackgerritMerged openstack-infra/project-config: Fix dib virtualenv setup
zigoFun, you guys have your ssh keys in a $home! :)15:59
*** bmjen has joined #openstack-infra15:59
openstackgerritRoman Podoliaka proposed openstack-infra/project-config: oslo.db: don't run tests with SQLAlchemy 0.9.x
fungizigo: yeah, that's one of the reasons those images aren't generally reusable16:00
*** placko_ has quit IRC16:00
*** jamesmcarthur has quit IRC16:01
zigofungi: Jenkins currently complains about a syntax error in jenkins/scripts/, which I didn't even touch...16:01
*** IlyaG has joined #openstack-infra16:02
zigo[E] E040: Syntax error: syntax error near unexpected token `fi': ''16:02
fungiAJaeger: ^ known issue with the translation update script?16:02
clarkbgreghaynes: for dibs pkg-map stuff is there are precanned map? I cannot find one anywhere so guessing not?16:02
AJaegerLet me check...16:02
AJaegerzigo, have a link for me?16:02
*** adrian_otto has joined #openstack-infra16:03
abregmanhi everyone. I asked this before, but I'll try again: any tips for openstack-infra newbie on how to start? (beside reviewing)16:03
AJaegerzigo: you misparsed bashate ;(16:03
AJaeger"- /home/jenkins/workspace/gate-project-config-linters/jenkins/scripts/ : L88"16:03
AJaegerthat's the error16:03
*** mattymo_ is now known as mattymo16:04
*** samuelBartel has quit IRC16:05
anteayaabregman: hello, well infra is large and takes a long time to learn, so welcome16:05
anteayaabregman: and as you are learning patience is indispensible here16:05
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages
*** arxcruz has quit IRC16:06
anteayaabregman: have you any interest in any aspect of infra that you have seen?16:06
abregmananteaya: thank you. any suggestions on where to start?16:06
anteayaabregman: it is often helpful to start with one tool and then work from there16:06
*** zeih has quit IRC16:06
abregmananteaya: would tool considered as a good start for beginners?16:06
anteayaabregman: do you have any prior experience with any deployment tools, sysadmin, code review tools16:06
anteayaabregman: that depends on you16:07
*** tphummel has quit IRC16:07
anteayaabregman: you at least can irc I see16:07
anteayawhat is easy for one person is hard for another16:07
anteayais there any aspect that you see that you like yet?16:07
anteayaI'm trying to help, but it isn't a one size fits all answer16:08
pabelangerand back16:08
abregmananteaya: I have experience as sys admin and python developer. I'm familiar with Ansible, Jenkins, Python, Linux :)16:08
*** jamesmcarthur has joined #openstack-infra16:08
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add networking-generic-switch-dsvm job
smarcetfungi:Ajaeger: the proposed change on is wrong16:08
anteayawhat experience have you with jenkins things?16:08
smarcetfungi: the rite uri is
anteayaare you familiar at all with gearman?16:08
anteayapabelanger: welcome back16:08
AJaegersmarcet: OOpos ;(16:08
AJaegersmarcet: will fix...16:08
smarcetok cool thx :)16:08
fungioh, yep--sorry for not spotting that16:09
*** aeng has joined #openstack-infra16:09
smarcetnp :)16:09
*** mylu has joined #openstack-infra16:09
*** mattt has left #openstack-infra16:09
*** krtaylor has joined #openstack-infra16:09
aengping Daisy16:09
abregmananteaya: I setup several jobs that deploys OpenStack using packstack (by writing jjbs) in various topologies16:09
anteayaabregman: awesome16:09
Daisyaeng: Hi16:09
pabelangerAJaeger: do you mind helping 293767 land?  I'd like to do some testing on debian-jessie with puppet apply jobs16:09
aeng:), just saw your message16:09
abregmananteaya: I'm not familiar with gearman16:10
anteayaabregman: okay good16:10
DaisyAJaeger, fungi : aeng is Alex Eng from Zanata dev team.16:10
*** vgridnev has quit IRC16:10
pabelangerfungi: or if you have a sec for 29376716:10
aenglooks like the login works now16:10
*** zeih has joined #openstack-infra16:10
Daisyaeng: It's your midnight, I think.16:10
aengyup, 2am16:10
anteayaabregman: have you seen this image before?
Daisyaeng: yes. fungi and his team rollback the openstack id.16:11
aengoh... so in fact some changes happened in openid16:11
abregmananteaya: yes and I wrote this:
*** mylu has quit IRC16:11
Daisyaeng: I think they want your help to make the new OpenStack ID version workable.16:11
abregmananteaya: I simply don't know what to work on :) there are so many projects16:12
aengDaisy, even if we made the changes, it wont deploy soon.. we need to coordinate the deployment date16:12
fungiDaisy: aeng: yeah, we're working on getting our translate-dev site authenticating against the dev deployment of our openid provider so we can troubleshoot further while not blocking the translation team16:12
anteayaabregman: yup, I am trying to figure out how to help you16:13
fungiaeng: i think we're mostly just looking for help/insight into what the openid consumer implementation in zanata expects so we can make sure we cater to that in our provider implementation16:13
anteayaabregman: have you seen this before?
aengfungi, good point16:13
* aeng checking now16:13
yolandaAJaeger, just to confirm, any job that ends with -nv is non-voting right? no need to specify it16:13
pabelangeranteaya: AJaeger okay, approved 293719 and will be monitoring JJB16:13
abregmananteaya: yes I did16:14
AJaegeryolanda: corrrect16:14
fungiaeng: since some thought to be non-impacting adjustments between openstackid 1.0.13 and 1.0.14 seem to have caused zanata to no longer be able to authenticate against it16:14
greghaynesclarkb: There arent really pre-canned maps, no16:14
clarkbgreghaynes: and behavior if one distro doesn't have the package name is to skip or fail?16:14
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Fix openstack-id location for translation-dev
greghaynesclarkb: hah, the behavior is itll try and install the package. Whether that succeeds or fails is up to the package installer16:15
*** zeih has quit IRC16:15
anteayapabelanger: ack16:15
*** MarkAtwood has joined #openstack-infra16:15
clarkbso ya we should write a pkg-map to avoid surprises16:15
anteayapabelanger: ping if you spot a bug and need help with a fix16:15
AJaegersmarcet, fungi : 294131 is the fix - is it fine now?16:15
greghaynesclarkb: yep16:15
anteayaabregman: yes, I got to that place in your blog post, nicely written16:15
pabelangeranteaya: will do16:15
abregmananteaya: thank you16:16
fungiAJaeger: yes, that's the correct domain name16:16
anteayaabregman: okay, so as you surmised one of the hardest things is to have an overview16:16
fungiAJaeger: should be able to load it in a browser to confirm too16:16
AJaegerfungi: confirmed, I can load it16:16
AJaegerpabelanger: thanks for taking care of the puppet devstack-trusty16:17
*** vgridnev has joined #openstack-infra16:17
anteayaabregman: while it seems like it might be a whole lot of work for not much gain, personally what I would give you as a first assignment is read the scrollback for the past two days and give me a summary of what you think were items of importance that came up and were acted on16:17
*** vgridnev has quit IRC16:17
aengfungi, all I can tell is that we are using generic openid impl16:17
smarcetAJaeger: yes its is16:18
anteayaabregman: we have a lot of people working on a lot of different things, but the amount of people who are understanding how the pieces all fit together and co-ordinate are small16:18
AJaegerpabelanger: 293767 will conflict with the puppet change you just approved. Once it's rebased, I'll help land it...16:18
pabelangerAJaeger: ya, I am fine with that16:18
AJaegerbut now I need to step out for two hours...16:18
anteayaabregman: a large part of our work is seeing what needs doing and doing it, we don't have time to give folks direction16:18
aengfungi, what changes in the upgrades?16:19
*** nwkarsten has quit IRC16:19
abregmananteaya: just to understand I get it right, you mean I should read the history of this irc channel for the past two days?16:19
anteayaabregman: now, learning how to parse the backscroll and interpret it can take time, but I think that would be time well spent and I'm willing to hellp you with that16:19
anteayaabregman: yes, that is what I am saying16:19
*** ybathia has quit IRC16:19
anteayaabregman: now if you don't want to do that, the other option is stand up a gerrit on your own16:19
anteayaabregman: to have that experience16:20
abregmananteaya: k then! let's do it!16:20
anteayaabregman: awesome16:20
anteayaabregman: do you know how to find the logs?16:20
abregmananteaya: first question - where can I read all the channel history?16:20
*** nwkarsten has joined #openstack-infra16:20
anteayaabregman: great question16:20
*** moravec has quit IRC16:20
anteayathat is the link to all the logs for all of our channels as well as all our meeting times16:21
*** jamesmcarthur has quit IRC16:21
fungiaeng: short changelog is readable at
anteayaabregman: see the link " IRC Channel Logs"?16:21
*** hasharMeeting has quit IRC16:21
anteayaabregman: you should see an index with #openstack-infra listed16:21
abregmananteaya: yes16:21
anteayaabregman: link me when you find the infra logs16:21
abregmananteaya: found it16:22
fungiaeng: the 6 commits which implemented that can be seen linked from the top of the list at
anteayaabregman: great16:22
*** pcaruana has joined #openstack-infra16:22
anteayaabregman: this is where you get to work on your patience and persistence skills16:22
*** doug-fish has quit IRC16:22
jprabergman, anteaya: I've been dormant for a many weeks (other work...feh) but this is where I left off.  standing up a gerrit.  would feel good to jump in an collaborate on overview and sharing other notes from my work in jan.16:22
openstackgerritMerged openstack-infra/project-config: Move puppet jobs to ubuntu-trusty
anteayaabregman: ask questions as you have them, use paste and etherpad and the channel log urls to ensure I understand exactly what item you are referenceing when you reference something16:22
anteayaabregman: and again, welcome16:23
*** doug-fish has joined #openstack-infra16:23
anteayajpr: awesome, welcome16:23
*** Daisy has quit IRC16:23
abregmananteaya: great! what do you usually use to paste text and share it?16:23
anteayajpr: I didnt' set abregman's first task as gerrit mostly due to my percieved lack of people available to respond to questions as they arise16:24
*** nwkarste_ has joined #openstack-infra16:24
openstackgerritDmitry Tyzhnenko proposed openstack-infra/git-review: Add reviewers by group alias on upload
anteayaabregman: paste.openstack.org16:24
*** andymaier has joined #openstack-infra16:24
jpranteaya: yes, gerrit is not the place to start. :)  That's where I left off. :)16:24
anteayajpr abregman but if there are two of you, that is an ideal situation since I find a great first learning step is to compare notes with others16:24
*** ifarkas_ has quit IRC16:25
*** nwkarsten has quit IRC16:25
anteayajpr: do you want to work on the same task I set for abregman?16:25
jpranteaya: agreed16:25
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing
anteayajpr abregman and then at the very least, you have each other to talk to?16:25
anteayaand the rest of us can support as we have time?16:25
AJaegercould some project-config cores please review - to start with the python-db migration, please?16:25
abregmanyup. hi jpr :) what is your timezone?16:25
*** salv-orlando has quit IRC16:25
anteayaAJaeger: looking16:26
jpranteaya: ragarding task, i'd like to get my notes squared up and shared on getting started.  I ran into a lot of little things that took some debugging to understand16:26
*** salv-orlando has joined #openstack-infra16:26
* AJaeger steps out...16:26
AJaegerthanks, anteaya16:26
jprabergman: cst (central us)16:26
*** tphummel has joined #openstack-infra16:26
abregmananteaya: thank you for my first assignment!  /me exicted. Once I finish it, I'll ping you16:27
anteayaAJaeger: awesome so mriedem is already aware of this16:27
anteayaabregman: oh don't wait until you finish, ask as you go along16:27
openstackgerritClark Boylan proposed openstack-infra/project-config: add uuid-runtime to base image builds
clarkbgreghaynes: ^ did I do that right?16:27
*** doug-fish has quit IRC16:27
anteayajpr: as you are comfortable, everyone learns with a different style16:27
abregmanjpr: mine is ist (Israel)16:27
*** fernnest has quit IRC16:28
*** jamesmcarthur has joined #openstack-infra16:28
anteayajpr abregman I'm just thinking how fortuitous it is that you both are here, as I find learning with someone at the same level really helps both learning and retention16:28
anteayaso thank you both16:28
aengfungi, I will need to check with the team tomorrow for more details and let you guys know what is expected from Zanata side.16:28
*** ddieterly is now known as ddieterly[away]16:29
aengwhats the best way to communicate this with you?16:29
abregmanthank you anteaya!16:29
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental puppet-apply job for debian-jessie
fungiaeng: thanks. we're going to continue troubleshooting from the provider end as well and try to spot the mismatch (or worst case bisect changes until we pull out the offending one)16:29
abregmanjpr: what do  you plan to do now?16:29
pabelangerAJaeger: rebased ^16:29
jpranteaya: yes.  an fortuitous that I was itching to read the IRC channel again at this time. :)16:29
anteayaAJaeger: nit, you have pep8, docs in check and docs, pep8 in gate, I know you would care about having consistent order in the python-db-jobs zuul template16:30
fungiaeng: you can get up with us in here or on the mailing list16:30
*** mikelk has quit IRC16:30
openstackgerritPaul Van Eck proposed openstack-infra/puppet-refstack: Make puppet pull releases from Pypi
anteayaabregman jpr wonderful! :)16:30
*** rhallisey has quit IRC16:30
jprabergman: i'm going to review your blog and then start pulling together my notes and experience from a few weeks ago.  I have a working (then) pipeline of zuul, nodepool, jenkins16:30
*** rhallisey has joined #openstack-infra16:31
*** nwkarste_ has quit IRC16:31
*** ddieterly[away] is now known as ddieterly16:31
abregmanjpr: cool. I'm going to work on Anita task. probably will have a ton of questions on the way. will try to summarize it in the blog for newbies as me.16:32
*** doug-fish has joined #openstack-infra16:38
anteayaAJaeger: +2 on 292472, if you decide to address the nit, will +2 again16:38
*** doug-fish has quit IRC16:38
*** doug-fish has joined #openstack-infra16:38
*** lakshmiS has joined #openstack-infra16:39
jproops ignore16:39
pabelangerclarkb: fungi what is the preferred way to delete all the ubuntu-trusty ready nodes in nodepool these days?  we are at min-ready 1 but have 130 currently ready16:39
pabelangerclarkb: fungi: I assume nodepool delete16:40
clarkbpabelanger: you mean devstack-trusty? they will accumulate until we switch puppet away from them and restart zuul/geard16:40
pabelangerclarkb: sorry, have devstack-trusty16:40
harlowja_at_homeclarkb as requested16:40
fungipabelanger: we need a zuul restart or they'll keep coming back16:40
openstackgerritgreghaynes proposed openstack/diskimage-builder: Make ubuntu-core support releases
clarkbpabelanger: first step is get puppet switched, but wanted to make sure we were relatively stable with other things before we do that, then switch puppet and restart zuul16:41
pabelangerclarkb: fungi: Ya, JJB running right now for puppet switch16:41
pabelangercurrently monitoring16:41
lakshmiSwe at searchlight project need elasticsearch installed for our functional tests to run on zuul. How do we get elasticsearch running on zuul test environment?16:43
clarkblakshmiS: install it as part of your test jobs16:43
lakshmiSIs there an example on any other software like mysql being installed like that?16:44
*** sigmavirus24 is now known as sigmavirus24_awa16:45
*** sigmavirus24_awa is now known as sigmavirus2416:45
*** scheuran has quit IRC16:45
*** doug-fish has quit IRC16:45
clarkblakshmiS: yes, postgres and mysql are both installed as part of the jobs now. In devstack it installs all of those deps, for unittests the jenkins job config does it16:46
*** doug-fish has joined #openstack-infra16:46
*** doug-fish has quit IRC16:46
clarkblakshmiS: do your functional tests rely on devstack? if so I would have it install elasticsearch for you via their plugin mechanism16:46
*** doug-fish has joined #openstack-infra16:46
lakshmiSno it doesn't rely on devstack16:47
fungiargh! why are we getting NOT_REGISTERED on gate-openstackci-beaker-devstack-centos7, gate-openstackci-beaker-ubuntu-trusty and gate-infra-puppet-apply-devstack-centos7 now?16:48
clarkbfungi: zuul updated before jenkins is my guess16:48
fungii'm diving into zuul logs/gearman queue16:49
fungiyeah, but updated what? did we switch nodes for those?16:49
clarkbfungi: apparently my change merged this morning yes16:49
fungilike in the past few minutes?16:49
*** mriedem1 has joined #openstack-infra16:49
clarkbya 16:41:38 pabelanger | clarkb: fungi: Ya, JJB running right now for puppet switch16:49
*** jlanoux has quit IRC16:49
*** mriedem has quit IRC16:49
fungithere it is. 293719 merged at 162216:50
lakshmiSclarkb: can you suggest a project where func tests are using devstack. maybe i can model it based on that?16:50
*** esikachev has quit IRC16:50
clarkblakshmiS: is what the non devstack tests do, they install packages listed by bindep in other-requirements.txt16:50
*** electrofelix has quit IRC16:51
fungipabelanger: so the names of those jobs changed?16:51
clarkblakshmiS: unfortunately elasticsearch isn't in a lot of distros last I checked so your install stuff will have to be smarter16:51
clarkbfungi: yes because the old job setup was well bad16:51
clarkbfungi: they relied on "ostype" and assumed that that could be appended to "devstack"16:51
pabelangerwhat clarkb said16:52
fungiokay. looks like the devstack-centos7 ones haven't registered yet but the ubuntu-trusty one has now16:52
clarkbfungi: which meant the switch away from devstack-trusty required changing all sorts of stuff16:52
clarkbbecause that was a bad built in assumption that no longer holds16:52
*** ddieterly is now known as ddieterly[away]16:52
fungii'll see if i can urge nodepool to built more devstack-centos7 quickly16:52
*** smarcet has quit IRC16:52
fungiahh, there they go16:52
*** nadya has joined #openstack-infra16:52
fungiokay, all set to recheck things now16:52
*** esikachev has joined #openstack-infra16:52
anteayaso it was a race? not an issue with the code?16:53
clarkba couple jobs still use "ostype" for the internal beaker config, we will need to update all the projects to use less implicit node info16:53
*** nadya has quit IRC16:53
clarkbanteaya: yes I think so. If zuul updates before jenkins does then this happens16:53
*** ddieterly[away] is now known as ddieterly16:53
lakshmiSclarkb: thanks for the pointers.16:54
anteayadoes ansible track the jenkins zuul update order at all?16:54
clarkbanteaya: we don't force zuul after jenkins but zuul by virtue of starting with a z seems to go last consistently16:54
*** vincentll has quit IRC16:55
*** aeng has quit IRC16:55
anteayawonder what happened this time16:55
clarkbanteaya: the larger issue is when doing a lot of job updates puppet times out bfeore all the things update then zuul updates16:55
pabelangerfungi: Ya, jjb still making the rounds.16:55
*** placko_ has joined #openstack-infra16:55
anteayaz still happens after j for my alphabet16:55
*** placko_ has quit IRC16:55
anteayaah the timeout16:55
clarkbanteaya: it does but due to the timeouts we end up stopping the update on jenkins then running zuuls update16:55
anteayayup okay16:55
anteayathanks for the explanation16:56
nibalizerfungi: it finished!16:58
pabelanger15mins in and JJB still running on jenkins05 (plus others)16:59
funginibalizer: note that second run (it will be quicker) to generate the stable maint electorate until i or someone gets a chance to figure out a better way to make that happen as part of the same run16:59
*** madhu_ak has joined #openstack-infra17:00
nibalizerfungi: o717:00
anteayanibalizer: oh you are creating the electoral rolls17:01
pabelangerokay, looks like JJB failed on jenkins17:01
anteayanibalizer: nice17:01
pabelangerso, what is the best way to manually run it without having puppet collide17:01
*** armax has quit IRC17:02
fungiwho can help me interpret this new lovely failure?
*** ddieterly is now known as ddieterly[away]17:02
clarkbpabelanger: not sure I understand the question17:02
nibalizeranteaya: yip17:03
fungi"Error: Could not find a suitable provider for cron"17:03
clarkbfungi: wow ubuntu-minimal doesn't have cron17:03
*** tphummel_ has joined #openstack-infra17:03
clarkbmaybe we should revert everythin and go back to precomputer society17:03
anteayanibalizer: thank you17:03
anteayaclarkb: I like rocks and sticks17:03
bkero"Error: Could not find a suitable provider for cron"17:03
bkerofungi: ^17:03
fungii'm going to go ahead and force merge 294131 for now so we can get on with the zanata/openstackid troubleshooting while people work through the job change failures17:04
*** yamamoto has quit IRC17:04
anteayafungi: ack17:04
pabelangerclarkb: I want to manually run JJB.  Do I need to stop anything on puppetmaster?17:04
pabelangerfor jenkins.o.o for example17:04
clarkbpabelanger: no you should be fine to just run it17:04
*** tphummel has quit IRC17:04
*** tphummel_ is now known as tphummel17:04
fungibkero: yep, i think we switched to images that don't have cron installed by default and the manifest doesn't install it17:04
*** Sukhdev has joined #openstack-infra17:04
anteayaclarkb: so what do you think the best solution is to addressing the no cron issue?17:04
*** Daisy has joined #openstack-infra17:04
pabelangerclarkb: sudo jenkins-jobs update .. is fine?17:05
bkerofungi: sounds like something should install cron :)17:05
*** mriedem1 is now known as mriedem17:05
clarkbpabelanger: yup, except sudo likely to have cache problems, I tend to su to not worry about the env17:05
clarkbanteaya: the "best" solution imo is to make ubuntu-minimal stop being so minimal17:05
fungior sudo -H17:05
anteayaclarkb: revert? fixing the image and building new images will take a while17:05
clarkbbut everytime I suggest that I get told that ins't minimal17:05
clarkbanteaya: yes that may be the pragmatic solution for now17:06
anteayaclarkb: ah, I didn't realize there was some prior conversation here17:06
anteayaclarkb: do you know which patch needs reverting?17:06
clarkbanteaya: well previously we hav easked for sshd to be installed17:06
pabelangerdoing jenkins05 atm17:06
*** jamesmcarthur has quit IRC17:06
clarkbanteaya: is the change17:06
anteayaclarkb: do you want to offer the revert?17:07
pabelangerand jenkins.o.o17:07
*** vgridnev has joined #openstack-infra17:07
anteayaclarkb: wondered if there was anything in addition to your change17:07
openstackgerritMerged openstack-infra/system-config: Fix openstack-id location for translation-dev
clarkbanteaya: mostly trying to figure out if there is a sane alternative17:07
anteayaclarkb: yup, standing by17:07
clarkbcould add cron to sdagues change and rebuild really quick17:08
anteayaclarkb: if you think that is worth a try17:08
anteayawhich change of sdague's?17:08
*** Daisy has quit IRC17:09
clarkbya thats the one17:09
*** madhu_ak has quit IRC17:10
*** Sukhdev has quit IRC17:10
anteayasdague: you about? we seem to be missing cron on ubuntu-minimal, clarkb is wondering if we can add install cron on your patch 29394217:10
anteayahe might be offline for a bit, he said he was working on something earlier17:12
anteayaclarkb: you want to wait?17:12
*** gsilvis has joined #openstack-infra17:12
pabelangeror adding it to bindep-fallback.txt for the moment too17:12
pabelangerbut that needs a rebuild too17:13
fungipabelanger: that won't help. it's not used for those jobs anyway17:13
*** jamesmcarthur has joined #openstack-infra17:13
fungibindep-fallback.txt is basically providing a default package set for jobs which were previously running on bare-trusty17:14
anteayaright now I'm in the revert camp, if anyone is counting17:14
fungithese jobs were running on devstack-trusty17:14
fungiand yeah, i think that job update needs to be undone in the short term17:14
pabelangeranteaya: revert which?17:14
anteayapabelanger fungi thanks I didn't know that17:14
anteayaas that is the patch that moved the jobs17:15
pabelangerOh, that's the one I am monitoring ATM17:15
pabelangerso, jenkins.o.o and jenkins05 are currently updating17:15
anteayaand as fungi pointed out, we lack cron17:15
*** nmagnezi has joined #openstack-infra17:15
pabelangerodd it took this long to fine it17:15
*** esikachev has quit IRC17:16
pabelangerso, discussion is to add it to ubuntu-minimal in diskimage-builder ATM right?17:16
fungipabelanger: yeah, apparently puppet-openstackci sets up cron jobs but doesn't ensure cron is installed17:16
pabelangeror we update nodepool element in system-config17:16
anteayaso right now the options are revert or, add cron and rebuild images17:16
openstackgerritSam Betts proposed openstack-infra/glean: Enable multiple VLANs per physical interface
openstackgerritSam Betts proposed openstack-infra/glean: [WIP] Enable support for configuring bonded interfaces
*** fawadkhaliq has joined #openstack-infra17:16
*** Sukhdev has joined #openstack-infra17:16
fungiit's possible we could add a cron package resource to that manifest17:16
anteayafungi: that won't require updating images, correct?17:17
pabelangerOr have puppet team update their manifests to ensure crontab is installed17:17
*** erlon has quit IRC17:17
pabelangerlike we did for locale issue17:17
clarkbfungi: I don't know if puppet will work on a first pass if it installs cron17:17
fungiwhich would also ensure that it works for people who apply the openstackci module on a server which didn't have cron preinstalled17:17
*** Sukhdev has quit IRC17:17
clarkbthat will not work for pip iirc17:17
pabelangerfungi: right, I think that is the better solution17:17
clarkbor package manager resources17:17
*** nadya has joined #openstack-infra17:18
bkeroYou could always if (not defined(Package['cron'])) { package { "cron": ensure => installed } }17:18
*** sc68cal has joined #openstack-infra17:18
pabelangerokay, holding off on other jenkins masters until we decide to revert or not17:18
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add python-db jobs - for nova initially
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Switch nova to db jobs
clarkbbkero: right but does that require two puppet runs? I think it does17:18
anteayapabelanger: thanks17:18
pabelangerbkero: which job is failing?17:18
bkeroclarkb: nope17:18
clarkbbkero: are you sure? I know that we couldn't bootstrap pip that way which is why pip is installed out of band17:19
*** sputnik13 has joined #openstack-infra17:19
bkeroclarkb: you put a requirement/ordering in it17:19
openstackgerritEric Harney proposed openstack-infra/project-config: Add NFS devstack plugin
bkeroSo you have cron { "Mycronthing": requires => Package['cron'] } in there too17:19
*** fernnest has joined #openstack-infra17:19
fungioh, right, the cron module probably needs to be the one to make sure cron is installed17:19
AJaegeranteaya: updated as you suggested, could you review again, please?17:19
pabelangerright, I think that is the correct fix. Especially since we are talking about puppet failing17:20
*** esikachev has joined #openstack-infra17:20
pabelangerwe basically exposed a bug in the manifest17:20
anteayaAJaeger: done, thank you17:20
AJaegercould I get some review love on , please? This publishes barbican api guide. I did the last few iterations of it, thus only a +1 from me.17:20
*** fawadkhaliq has quit IRC17:20
bkeropabelanger: some might say we exposed a bug in ubuntu's image :)17:21
clarkbbkero: its our image17:21
*** fawadkhaliq has joined #openstack-infra17:21
clarkbbkero: we explicitly don't use ubuntu's image beacuse we don't want to use glean17:21
fungiclarkb: the pip issue may have been because it's a package provider and we may not have been able to say install system package before installing package resources requiring the pip provider17:21
AJaegerpabelanger: 293767 LGTM17:21
pabelangerbkero: right, but fungi makes a good point. If you run the puppet module on another image without cron, you have the same failure.  It seems sane to add ensure => package[cron], like you listed above17:22
EmilienMI think puppet beaker jobs are currently broken17:22
*** nadya has quit IRC17:22
bkeropabelanger: that's requiring a Package resource named 'cron'. So if there is none (like if cron is already installed), it will break17:22
bkeroSo you're inverting the broken :)17:22
anteayaEmilienM: yes, we are discussing the issue17:23
pabelangerbkero: yup17:23
anteayaEmilienM: merging this patch exposed that cron is not in the image when built17:23
EmilienManteaya: thanks I did not follow it17:23
anteayaEmilienM: and openstackci jobs don't install cron though use it17:23
anteayaEmilienM: no worries, good eye17:24
fungianteaya: EmilienM's error is on devstack-centos7 though17:24
anteayaoh sorry, I didn't look at the log17:24
EmilienMI'm not sure we're talking about the same bug17:24
bkero`parse_hosts_file': Host file '/home/jenkins/workspace/gate-puppet-ceph-puppet-beaker-rspec-devstack-centos7/spec/acceptance/nodesets/nodepool-[centos7].yml' does not exist! (ArgumentError)17:24
bkero^ That's different than cron17:24
anteayaEmilienM: fungi is correct, we don't seem to be, my apologies17:24
EmilienManteaya: no worries at all17:24
anteayaso two breaks on puppet beaker jobs17:25
clarkbbkero: EmilienM that may acually be a jjb bug17:25
anteayaboth from I wonder?17:25
clarkbbkero: EmilienM it should be 'centos7' but because its a list we get [centos7] ?17:25
clarkbthats a fun one17:25
clarkbanteaya: yes17:25
fungiyeah, i think that's to do with jjb embedding an explicit list construct17:25
*** chem has joined #openstack-infra17:25
*** asettle has joined #openstack-infra17:25
EmilienMfungi, clarkb: should we just wait jjb for updated? or?17:25
bkeroLooks like one of the classic 2 problems of computer science17:25
clarkbEmilienM: we are going to reverthe change I think17:26
anteayaclarkb: I'm still in the revert camp17:26
EmilienMI hate reverts :(17:26
fungiso at this point, rolling back that job update sounds best, we have more than one issue to work through17:26
anteayaclarkb: do you want to offer?17:26
*** jamesmcarthur has quit IRC17:26
clarkbEmilienM: but that is a bug in jjb, when you give it multiple list entries it uses the strings but apparently a singleton list gives you a list17:26
anteayafungi: agreed17:26
EmilienMclarkb: revert makes sense here, so many blockers17:26
fungiyeah, there's been an ongoing infra ml thread about that with jjb17:26
fungithe literal list issue i mean17:27
*** degorenko is now known as _degorenko|afk17:27
*** jistr has quit IRC17:27
bkeroShouldn't it be a simple naming fix in17:27
*** e0ne has quit IRC17:27
clarkbbkero: sort of, its jjbs parsing of the yaml that is broken17:27
clarkbbkero: we can probably game that parsing to get something that works, not sure17:27
clarkbbkero: foo:\n  - bar vs foo:\n  bar17:28
clarkbwe used the first, the second should work but they should be equivalent17:28
*** mrmartin has joined #openstack-infra17:28
*** esikachev has quit IRC17:28
fungithread starts at
bkeroSounds like hotfix to add '- ', then actually fix the jjb parsing some other time17:28
*** moravec has joined #openstack-infra17:29
fungiif you want to read up on the issue17:29
*** Zhongjun_ is now known as Zhongjun17:29
clarkbbkero: eaiser to just revert due to the cron thing17:29
clarkbaccording to cron is part of posix17:29
bkeroAh, I didn't know they were the same ocmmit17:29
clarkbI think it is reasonable to have our base images provide a base system that attempts to provide posix utilities17:29
fungiand propose a revert of the revert noting need to confirm correct parsing of the job (by looking at the config check output) and also making sure we address the missing cron somehow before reapprovinf17:30
clarkbso in addition to the revert we should work on image updates to provide that base set of utilities (note uuidgen is not)17:30
JayFuuidgen ships with, of all things, e2fsutils, at least on some distros17:30
anteayafungi: makes sense17:30
JayFe2fsprogs, I mean17:30
*** SumitNaiksatam_ has quit IRC17:30
*** asettle has quit IRC17:31
*** vgridnev has quit IRC17:31
bkeroclarkb: that also includes qmail stuff, uucp, fort77, c99, and m417:31
fungii think the argument for not including cron was an attempt to avoid shipping persistent daemons by default. though i think that's container-loving hippie mentality leaking into the implementation for our virtual machine images ;)17:31
*** krtaylor has quit IRC17:31
anteayahippie's love containers by definition?17:32
* anteaya updates her notes17:32
fungiyes. it's a well-known fact17:32
clarkbbkero: ya it may not be perfect, but tools like df, du, xargs, cron, cp, getopts, etc are all very very reasonable to expect17:32
anteayafungi: thanks for sharing17:32
clarkbkill locale printf sleep sort17:32
pabelangerclarkb: bkero don't forget ssh17:32
bkeroclarkb: So what you're saying is... 'which subset of this list of posix utilities should we select'17:32
clarkbbkero: no, I am saying install the whole thing17:33
fungibkero: i'm not sure where "qmail stuff" is coming from though. djbsoft is not anywhere in the posix spec17:33
anteayacan we get the revert in the process and then discuss the image fix?17:33
bkeroincluding the fortran compiler? O_o17:33
clarkbbkero: ignore the ones that don't make a lot of sense because we get a bunch that do17:33
*** akaszuba has quit IRC17:33
bkerofungi: Sorry, there were a bunch of 'q' commands17:33
anteayaI think we have agreement revert is our happy path for the moment, don't we?17:33
clarkbhwhere ignore is install them and don't worry about it, not don't install them17:33
pabelangerokay, so has anybody proposed the revert? if not I can do that now17:33
fungiplease do. i'll stop writing mine17:34
bkeroclarkb: so 'install the subset of this list of posix utilities we select' and think about it as 'we want to install the whole thing, but we're only willing to go this far'17:34
anteayapabelanger: I was waiting on clarkb, but no there isn't a revert up17:34
clarkbbkero: no!17:34
clarkbbkero: install the whole thing17:34
clarkbbkero: don't worry about the extra17:34
clarkbin that it will be there taking up a few megabytes of disk17:34
*** yamamoto has joined #openstack-infra17:34
clarkbcompared to 17GB I don't think anyone should care17:34
bkeroI think there's a significant portion of this list that aren't packaged too17:35
bkeroMaybe they exist in some package17:35
clarkbya so it may have to be a best effort17:35
bkeroLike asa17:35
clarkbbut we don't need to make arbitrary decisions. ubuntu does provide fort77 and its less than a megabyte according to aptitude17:36
clarkbah its a wrapper so maybe larger depending on size of the thing wrapped17:36
bkeroprobably a virtual17:36
*** Guest57182 has quit IRC17:36
fungithough i still think it's reasonable for puppet modules, job definitions, et cetera to be explicit about the packages they expect rather than telling people to go read the posix spec and come back when they've installed it all17:37
bkero'eh, still not that bad: After this operation, 2,894 kB of additional disk space will be used.17:37
*** cargonza has joined #openstack-infra17:37
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Move puppet jobs to ubuntu-trusty"
pabelangerchange up17:37
anteayafungi: and I think you make a very good point about openstackci especially, since it was created for external consumption17:37
bkerofungi: You're right, but this is how modules get bloated17:37
*** vgridnev has joined #openstack-infra17:37
clarkbfungi: yup, on the flip side though it seems reaonsable that every test job or shell script need not install cp or xargs or df or etc17:37
*** jamesmcarthur has joined #openstack-infra17:37
clarkbbeacuse there is a reasonable expectation that a stnadard set of utilities exist on a machine17:38
*** Sukhdev has joined #openstack-infra17:38
*** dprince has quit IRC17:38
anteayapabelanger: did you not just push the revert button on
anteayapabelanger: there is no comment there saying the patch is reverted17:38
pabelangeranteaya: I used git revert from commandline17:39
fungiyeah, if you use git revert it's helpful to go to the reverted change and leave a comment pointing to the rever17:39
pabelangerYup, leaving comment now17:39
bkeroSO if we are to include cron, which module should it live in, or should it live somewhere else as not a Puppet resource?17:39
anteayapabelanger: ah okay, thanks17:40
*** yamamoto has quit IRC17:40
clarkbmy preference would be that we have a dib element that we call ubuntu-useable or similar to install the things17:40
pabelangeranteaya: original patch updated17:40
clarkbsshd and uuidgen are two examples of things that should go in a "useable" image on top of what we might expect from the more minimal images17:40
anteayapabelanger: thank you17:41
fungipabelanger: if you can also make a revert of the revert, that would be awesome. set it as wip and add a comment about not approving until the missing cron package and jjb nested list issue are addressed17:41
fungiclarkb: i think greghaynes has already proposed that (called {distro}-packageinstall) though it's still got some -1's to work through17:42
pabelangerfungi: doing that now17:42
*** smarcet has joined #openstack-infra17:42
*** krtaylor has joined #openstack-infra17:43
fungismarcet: jamesmcarthur: logins are now going to openstackid-dev from what i can tell, sorry for the delay17:43
clarkbfungi: i will have to dig that up and see what the concerns are17:44
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Revert "Move puppet jobs to ubuntu-trusty""
jamesmcarthurfungi: thanks for the update. we’ll start digging in17:44
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again
anteayapabelanger: how many jenkins need to be re-updated after your revert merges?17:45
*** Sukhdev has quit IRC17:45
clarkbfungi: that one?17:45
*** fawadkhaliq has quit IRC17:46
pabelangeranteaya: at least jenkins.o.o and jenkins05 for manual updates17:46
pabelangerthat's the only 2 I did17:46
zigofungi: clarkb: I tested my scripts here, and they seem to work:
fungiclarkb: that's the one i saw, i think, yeah17:46
anteayapabelanger: cool17:46
pabelangerbut we'll have to confirm with others17:46
zigoOne question though, should I edit the .gitrc with gitconfig, or it's done already by default?17:46
*** tzn has quit IRC17:46
anteayapabelanger: and zuul? puppet will take care of that, yeah?17:46
pabelangeranteaya: hopefully17:47
*** jamesmcarthur has quit IRC17:47
zigosorry, .gitconfig, I mean.17:47
*** kzaitsev_mb has quit IRC17:47
*** tzn has joined #openstack-infra17:47
greghaynesclarkb: fungi Ya - so the concern is mostly 'what defines usable'. If there is something we can codify (like posix utilities) then thats easy - make an element which installs those things17:48
*** fawadkhaliq has joined #openstack-infra17:48
*** matrohon has quit IRC17:48
pabelangerzigo: I'd consider debian-jessie experimental atm, until we land a few infra JJB packages.  That shouldn't stop you from your JJB changes, mostly a heads up17:49
*** knikolla has joined #openstack-infra17:49
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages
zigopabelanger: Though my build scripts are tested and working...17:49
krotscheckSo, it appears that we can no longer successfully capture a firefox instance in the gate.17:50
* zigo goes out to fetch a pizza...17:50
krotscheckWhich is breaking every npm-based build that uses firefox.17:50
krotscheckOr wants to test against firefox.17:50
*** dims has joined #openstack-infra17:50
*** nadya has joined #openstack-infra17:50
*** mrmartin has quit IRC17:50
clarkbgreghaynes: it looks like even openssh-server and ping are contentious17:50
greghaynesclarkb: yes17:51
clarkbgreghaynes: how are people logging into their VMs?17:51
anteayakrotscheck: what do you mean capture a firefox instance?17:51
fungiclarkb: as i said, i think they're all making containers now17:51
fungivirtual machines are so last month17:51
clarkbI mean of all the tools including posix's funny list openssh-server seems like a straightforward thing to have17:51
pabelangerzigo: right, we just haven't launched a jenkins job yet17:52
krotscheckanteaya: To run browser tests, the gate spins up an instance of firefox in a virtual frame buffer, tells it to run tests, and then shut it down. The process of connecting the running firefox process with the testing harness is called capturing.17:52
greghaynesclarkb: yea.... I think the issue is mostly that folks dont want dib to become curator of a distro-like-thing which is an ubuntu package list for 'useful'17:52
clarkbgreghaynes: But thats exactly what dibs prupose is...17:52
anteayakrotscheck: ah thank you17:52
greghaynesif theres an upstream thing we could point at thatd be awesome17:52
clarkb"make me an image" this requires opinions17:52
*** tzn has quit IRC17:52
anteayakrotscheck: could we ever capture firefox in the gate?17:52
greghaynesYes, ideally dib isnt the one with those opinions though17:52
clarkbgreghaynes: well all the cloud images built by distros I have used include sshd17:52
krotscheckanteaya: We could as of this morning17:52
anteayakrotscheck: is not being able to do so a new behaviour?17:52
*** notnownikki has quit IRC17:52
krotscheckIt is a new behavior17:52
greghaynesclarkb: Yep. So my thinking was to make -useful be cloud-image feature compatible17:53
greghaynesclarkb: and that would be an upstream opinion, not ours17:53
clarkbeh I think its fine to have that opinion17:53
clarkbwe seem to be unable to have opinions in openstack17:53
clarkbdespite having lots of exposure to things that make good opinions17:53
clarkblike qcow2 > vhd/raw for image uploads17:54
anteayakrotscheck: that is an example of a successful capture?17:54
clarkband config drive > metadata server17:54
greghaynesclarkb: sure, so how would you codify the standard of what goes in and doesnt17:54
smarcetfungi: i was able to reproduce bug between translate dev and openstackid dev17:54
greghaynesIf we can come up with something concise-ish then I would feel a lot better abou it17:54
smarcetfixing it :)17:54
fungismarcet: that's excellent news!17:54
fungithank you!!!17:54
clarkbgreghaynes: I think you look at basic user needs. Users need remote access to instances. This requires an ssh daemon and a cloud init like utility17:54
fungizigo: for our jobs which propose changes we call git config rather than writing to .git/config directly. similar in the git-review functional testsuite17:55
krotscheckanteaya: That's an example of an unscucessful capture17:55
clarkbgreghaynes: now, of the options make an opinion on which is the most generic set. dropbear loses, openssh wins. cloud init probably wins, etc17:55
krotscheckanteaya: I have a confirmed successful run at 4AM this morning17:55
greghaynesclarkb: then then on top of that, all the posix tools?17:55
krotscheckAlso running on firefox 45.0.0, so it's not the browser version.17:55
clarkbgreghaynes: I like the idea of a separate posix element and that can be opt in or a dep17:55
*** esikachev has joined #openstack-infra17:55
*** rcernin has quit IRC17:55
fungikrotscheck: if those job logs include system and pip package lists we can diff them to see what has maybe updated17:56
greghaynesclarkb: Ya, I think that makes sense17:56
*** tzn has joined #openstack-infra17:56
greghaynesclarkb: So my thinking for the ssh/cloud-init like thing is that we can make element-deps a little bit smarter and facilitate that17:56
*** tqtran-afk has joined #openstack-infra17:56
krotscheckfungi: There shouldn't be any pip. Lemme see about system.17:56
greghaynesclarkb: where you can have a dependency that has a default, but if some other element provides that then dont use the default17:57
krotscheckfungi: is a successful run.17:57
anteayakrotscheck: ah okay thanks, sorry I couldn't interpret you log messages17:57
clarkbgreghaynes: that makes sense. Then you can opt into glean over cloud init or dropbear over openssh if you choose17:57
*** tzn has quit IRC17:57
*** SumitNaiksatam has joined #openstack-infra17:58
openstackgerritNikolas Hermanns proposed openstack-infra/project-config: Add tempest-dsvm-networking-bgpvpn jobs
clarkbkrotscheck: that says chromium not firefox17:58
*** SumitNaiksatam has quit IRC17:58
clarkboh its two lines one not part of the error, ignore me17:58
*** tzn has joined #openstack-infra17:59
krotscheckclarkb, fungi: looks like xvfb updated.17:59
krotscheckWhich could cause the issue, if firefox doesn't know how to deal with that.17:59
krotscheckWait, I lied18:00
*** sigmavirus24 is now known as sigmavirus24_awa18:00
*** doug-fish has quit IRC18:00
fungikrotscheck: there's no explicit installed package list there, but we can extrapolate a bunch of it from the "Unpacking" lines18:00
gsilvisI'm looking to make a CI job for K2K federation.  It'll need to spin up two devstacks, and do several operations to each of them.  Where should I start looking for documentation on how to do this?18:00
fungikrotscheck: grep those out, sort them, then diff them18:00
*** SumitNaiksatam has joined #openstack-infra18:00
*** doug-fish has joined #openstack-infra18:00
*** baoli_ has quit IRC18:01
*** hashar has joined #openstack-infra18:01
*** jordanP has quit IRC18:01
fungigsilvis: the openstack-infra/devstack-gate project has an ansible-based implementation for setting up devstack on two job workers with a tunneled network between them18:01
*** jamesmcarthur has joined #openstack-infra18:02
fungigsilvis: there's also an ascii network diagram in there showing how they're connected18:02
*** ddieterly[away] has quit IRC18:02
gsilvisfungi: cool, I'm looking at that now18:03
*** notnownikki has joined #openstack-infra18:03
*** HeOS has quit IRC18:03
fungigsilvis: presumably you want something like the current multi-node devstack jobs (used to test nova live migration, neutron dvr) but have it set up a keystone in each devstack with suitable configuration for what you want to model18:03
clarkbas an alternative you could probably just run two keystons18:04
clarkbnot sure which of the two options are simpler18:04
*** HeOS has joined #openstack-infra18:04
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests
*** weshay has quit IRC18:04
fungioh, good point. may not actually need multi-node18:05
clarkbwith multinode you don't have to juggle port configs on a single host, but you do have to juggle the networking18:05
fungitwo keystone deployments on one server18:05
clarkbso there are tradeoffs for each option18:05
*** doug-fish has quit IRC18:05
*** yamahata has joined #openstack-infra18:05
*** hashar has quit IRC18:06
*** jamesmcarthur has quit IRC18:06
gsilvisclarkb: I think it'd be good to test actually doing openstack API calls across the two deployments, so I think I'll want to have more than just keystone18:06
*** ddieterly has joined #openstack-infra18:07
*** gordc has quit IRC18:07
*** MarkAtwood has quit IRC18:08
*** tzn has quit IRC18:08
krotscheckfungi: No version differences.18:09
*** jamesmcarthur has joined #openstack-infra18:10
*** moravec has quit IRC18:10
*** ddieterly has quit IRC18:11
gsilvisfungi: Could I have a link to one of the tests that is using this?  I'm a little lost18:12
*** piet has quit IRC18:13
*** tqtran-afk is now known as tqtran18:13
*** moravec has joined #openstack-infra18:14
pabelangerclarkb: so, do you want to start with adding cron to our nodepool element first or wait until we get the changes you were talking about into diskimage-builder?18:14
fungigsilvis: the {pipeline}-tempest-dsvm-multinode-full{job-suffix} job-template18:14
gsilvisfungi: thanks, I'll take a look18:14
*** jaosorior_away has quit IRC18:14
clarkbpabelanger: I am waffling on that, I think our best bet for now is to make a small element in project-config/nodepool/elements that we put ssh server, uuidgen, cron, and any other things we can think of for the short term18:14
*** nadya has quit IRC18:15
*** rbrndt has quit IRC18:15
clarkbpabelanger: then when we can get a general solution in dib ew can switch elements and delete ours18:15
fungigsilvis: it sets DEVSTACK_GATE_TOPOLOGY="multinode" but most of the logic in resides in devstack-gate and devstack18:15
*** jaosorior_away has joined #openstack-infra18:15
openstackgerritMerged openstack-infra/project-config: Revert "Move puppet jobs to ubuntu-trusty"
clarkbpabelanger: I can work on that element now if you aren't already18:16
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Bug Fixing
gsilvisfungi: yup, that makes sense18:17
clarkbpabelanger: the hard part is just building the pkg-map18:17
*** angdraug has quit IRC18:17
*** pcaruana has quit IRC18:18
*** e0ne has joined #openstack-infra18:18
*** claudiub|2 has joined #openstack-infra18:19
greghaynesclarkb: You could even make the posix thing in project-config and then copy it to dib if you like18:19
openstackgerritZara proposed openstack-infra/storyboard-webclient: Make it easier to find 'add task' button
greghaynesclarkb: dib handles two elements with the same name in path just fine18:20
fungithat used to break it though, right?18:20
greghaynesat least for as long as I remember18:20
fungino, wait, it was two elements with the same filename in them18:20
greghaynesYea, that shouldnt break, one overrides the other though18:20
greghaynesWe did have a case where a downstream tool expected that to never happen, though18:21
greghaynesin tripleo-ci18:21
fungibecause it tried to copy them all into the same location before running or something18:21
AJaeger293767 needs to be rebased now again ;( Sorry ;(18:21
*** mrmartin has joined #openstack-infra18:21
*** claudiub has joined #openstack-infra18:22
pabelangerclarkb: okay, I'll defer to you18:22
greghaynesYou might be remembering some foot-gunning caused by that - if you name two files the same thing only one will ever get copied in (it gets copied over the other one)18:22
AJaegerpabelanger: let me rebase for you...18:22
pabelangerAJaeger: Ya, I think were holding off on JJB stuff so we can land 29420318:22
pabelangerbut that's fungi's call18:23
pabelangerI'm game either way18:23
AJaegerpabelanger: ok, then let's wait with rebase...18:23
*** claudiub|2 has quit IRC18:24
*** hashar has joined #openstack-infra18:24
fungipabelanger: AJaeger: i think we ought to sort out the diskimage-builder and jenkins-job-builder pieces18:25
*** moravec has quit IRC18:25
dmsimardAre you guys aware of this error?  Looks weird
*** jamesmcarthur has quit IRC18:26
*** jaosorior_away has quit IRC18:26
clarkbgreghaynes: if you don't have a package in pkg-map then the name as given is used. What if I have a package in pkg-map and the current distro doesn't match any rules in pkg-map does it return nil or does it use the package name as given?18:26
dmsimardOh wait, I totally misunderstood that error, not related to you guys18:26
anteayadmsimard: yes, we have reverted that patch18:27
anteayaI'm not a guy18:27
dmsimardoh, thanks then anteaya18:27
anteayait is related to a patch we merged a few hours ago18:27
anteayawe have reverted the patch18:27
anteayapabelanger: is working on propigating the update18:27
dmsimardanteaya: didn't mean to discriminate with the "guys" :(18:27
anteayahopefully we should be in a position where you can recheck in a few hours18:27
dmsimardanteaya: I know very well who you are! :p18:28
anteayanoone ever means to18:28
anteayathank you18:28
anteayabut that is the current status on that issue in your logs18:28
openstackgerritMerged openstack-infra/openstackid: Bug Fixing
*** sigmavirus24_awa is now known as sigmavirus2418:28
dmsimardanteaya: appreaciate it, thanks18:28
pabelangerya, JJB should be starting updates now18:28
AJaegerfungi, ok.18:28
openstackgerritLakshmi N Sampath proposed openstack-infra/project-config: Install Elasticsearch for searchlight
anteayadmsimard: welcome18:28
anteayapabelanger: awesome18:28
pabelangerso, what did we decide the fix to dmsimard log was?  we need to update our beaker settings?18:29
*** moravec has joined #openstack-infra18:29
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Bug Fixing
openstackgerritLakshmi N Sampath proposed openstack-infra/project-config: Install Elasticsearch for searchlight
*** pcaruana has joined #openstack-infra18:30
*** hashar has quit IRC18:30
clarkbhrm doesn't appear to support groupinstalls either18:30
fungipabelanger: or fix the yaml parser in jjb to not turn singleton lists into a literal list when embedding18:30
lakshmiSclarkb: Added a patch for installing elasticsearch. i am definetly missing a lot but thought have to start somewhere and get some inputs on it18:31
openstackgerritMerged openstack-infra/openstackid-resources: Bug Fixing
*** sridhar_ram has quit IRC18:31
nibalizerfungi: how critical is it that i send a gpg signed email if i've signed the tarball?18:31
funginibalizer: signing the tarball is probably file18:31
nibalizerok swee18:32
nibalizerthen we're done in 10 seconds18:32
lakshmiSclarkb: let me know if i am completely on a wrong path or needs correction18:32
funginibalizer: i usually just attach the tarball and then use multi-part pgp/mime because that will sign the attachments too, but mutt makes that a lot easier than some muas18:32
pabelangerfungi: once JJB has been reverted for puppet jobs, would restarting zuul be on the table to pick up min-ready 1 and clear surplus nodes for devstack-trusty?18:32
pabelangerfungi: has anybody committed to that?18:32
pabelangeror EmilienM do you have a preference?18:33
anteayanibalizer: nice work18:33
fungipabelanger: well, it already picked up the min-ready update. it's just that we need a restart to clear out the old job registrations in gearman18:33
pabelangerfungi: Ah, okay. So, what sort things do we do when zuul is restarted?  Any prep needed for it?18:34
pabelangeror just hold off until the queue is slow18:34
*** piet has joined #openstack-infra18:34
nibalizerfungi: yea i've just generated a detached signature18:35
fungipabelanger: yeah, we export the check and gate pipelines immediately before restarting zuul, and then reenqueue them all. it can be a bit disruptive since it results in us dropping anything queued in other pipelines and also leaves a very brief window where zuul might miss some gerrit events too18:36
*** dtantsur is now known as dtantsur|afk18:36
nibalizerok sent out18:36
fungipabelanger: so doing it when zuul is less busy tends to be a little more comfortable18:36
nibalizercc'd the infra list for funsies18:36
funginibalizer: thanks a bunch18:36
nibalizerya no problem18:36
nibalizersorry the turnaround was slow18:36
fungii'll take a look and compare to my results and let you know if anything seems amiss18:37
nibalizerawesome ty18:37
*** eharney has quit IRC18:37
nibalizerim in EU timezone so I probably won't be too responsive from here on out18:37
fungithis still leaves the election officials ~5 hours to prep for sending ballots, so it's fine turnaround18:37
nibalizeruntil tomorrow I mean18:37
openstackgerritLakshmi N Sampath proposed openstack-infra/project-config: Install Elasticsearch for searchlight
*** yamamoto has joined #openstack-infra18:38
*** Zhongjun has quit IRC18:38
*** rbrndt has joined #openstack-infra18:40
*** eharney has joined #openstack-infra18:41
greghaynesclarkb: unsure about the missing distro case. I would hope it falls through18:41
pabelangerfungi: right, that makes sense.  Do we have an ansible playbook to do that yet, or just shell commands currently?18:41
greghaynesclarkb: or, thats what id expect18:41
*** ociuhandu has quit IRC18:41
*** tongli has quit IRC18:41
fungipabelanger: it's just some scripts run manually for now. we restart zuul pretty infrequently (or did up until the recent mass node changes which have left us needing a way to wipe out stale gearman job registrations)18:42
*** jamesmcarthur has joined #openstack-infra18:42
anteayanibalizer: yeah thanks for that18:43
pabelangerI had a patch up to nodepool for a while, that did it automagically, but never did finish it off.18:43
*** adrian_otto has quit IRC18:44
*** yamamoto has quit IRC18:44
clarkbgreghaynes: I am having a hard time running it locally since it apperas to infer element path magically?18:44
*** gordc has joined #openstack-infra18:44
*** doug-fish has joined #openstack-infra18:45
*** doug-fish has quit IRC18:45
*** sambetts is now known as sambetts|afk18:45
* clarkb reads the source18:45
*** doug-fish has joined #openstack-infra18:45
*** baoli has joined #openstack-infra18:46
clarkboh it writes it out to /usr/share stuff18:47
*** fitoduarte has joined #openstack-infra18:48
EmilienMpabelanger: no pref18:48
greghaynesclarkb: unsure what 'it' is18:49
*** doug-fish has quit IRC18:50
greghaynesclarkb: oh, pkg-map? I'd just add a bash right after where it gets run and do a build18:50
*** doug-fish has joined #openstack-infra18:51
clarkbgreghaynes: ya it being pkg-map18:52
clarkbits fine I managed to test what I need I think18:52
openstackgerritClark Boylan proposed openstack-infra/project-config: Add element to handle generic package installs
clarkbgreghaynes: fungi pabelanger ^ thats a rough start18:54
clarkbI removed the stuff from the devstack element because we are using those nodes for so much more than devstack now18:55
clarkband I think I did enough rpm -qf ing to figure out what is what on fedora/centos18:55
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build Debian packages
*** asettle has joined #openstack-infra18:57
*** jamesmcarthur has quit IRC18:57
pabelangerclarkb: sure, I can test it out in a bit18:57
*** jamesmcarthur has joined #openstack-infra18:57
clarkbthta covers cron, uuidgen, rough build essential, and python things18:58
anteayaoops was going to review the elections specs and only 1 minute left18:59
*** angdraug has joined #openstack-infra18:59
anteayafungi: can I convince you to wait for me?18:59
anteayaI won't let myself get distracted18:59
anteayaI've had them open for about 2 hours18:59
*** sridhar_ram has joined #openstack-infra18:59
*** baoli has quit IRC19:00
*** asettle has quit IRC19:01
pabelangerflushing the cache for JJB on jenkins.o.o, looks bad atm19:02
*** baoli has joined #openstack-infra19:02
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Enable gate jobs for Packstack on the stable/mitaka branch
*** pradk has quit IRC19:02
*** pvaneck has quit IRC19:03
*** esikachev has quit IRC19:03
clarkbgreghaynes: reading the code line 10 of should work19:03
*** jcoufal has quit IRC19:03
clarkbgreghaynes: I tried to find a groupinstall option but couldnt find one19:03
anteayafungi: done19:05
anteaya5 after19:05
fungianteaya: that's fine, i hadn't even started yet. still juggling a bunch of things19:06
anteayayup figured19:06
bkeroclarkb: so here's a package list.
anteayajust didn't want to miss, as I had lined them up long ago19:06
*** vgridnev has quit IRC19:08
bkeroThere's a bunch of dumb SCCS (Oracle's old version control bits) in there19:08
*** krtaylor has quit IRC19:08
greghaynesclarkb: yea, I don't think group install exists. Should be easy to add19:09
*** abregman has quit IRC19:09
*** automagically has joined #openstack-infra19:11
clarkbbkero: is that list for rhel or debian?19:11
bkeroclarkb: ubuntu19:11
*** eil397 has joined #openstack-infra19:11
*** kaisers_ has quit IRC19:11
*** ihrachys has quit IRC19:11
bkeroMost of it is shell built-ins or coreutils19:12
smarcetfungi: is working fine19:12
smarcetfungi: i will prepare a new release for openstackid :)19:12
fungismarcet: sounds great--thanks again. we'll get upgraded as quickly as possible and then can see whether that also takes care of refstack and the translation check site19:13
*** fawadkhaliq has quit IRC19:13
bkeroclarkb: so I think the list is 'at bc binutils bison bsdmainutils bsdutils cflow coreutils cron cxref diffutils ed emacs-24-bin-common file findutils flex fort77 gawk gcc grep gzip libc-bin libc-dev-bin login m4 make man-db ncompress ncurses-bin patch procps sed sharutils sysvinit-utils talk util-linux uucp uustat vim'19:14
clarkbbkero: exuberant-ctags is what provides ctags according to my search19:15
bkeroclarkb: that also works. I just did a dpkg -S $(which ctags) to find what the default has been19:16
*** salv-orl_ has joined #openstack-infra19:16
bkeroThey're both done through /etc/alternatives19:16
*** vgridnev has joined #openstack-infra19:16
clarkbbkero: did you want to propose a "posix-utils" element to dib?19:16
clarkbI can work on it if you don't have time19:16
clarkbbkero: gotcha19:16
bkeroYeah I can do that19:17
greghayneshrm, that list seems like wed lose some valuable test coverage19:17
greghaynesgcc being a good example19:17
*** pradk has joined #openstack-infra19:17
pabelangerinfra-root, looks like jenkins05 was the only JJB failure, looking into it now19:18
clarkbgreghaynes: coverage of installs?19:18
anteayapabelanger: ack19:18
greghaynesI know I get nodes without gcc all the time19:18
fungipabelanger: it probably is in need of a restart19:18
openstackgerritsebastian marcet proposed openstack-infra/system-config: OpenStackID 1.0.15 Release
*** nmagnezi has quit IRC19:18
clarkbgreghaynes: infra already has build-essential listed19:18
clarkbgreghaynes: so for us at least it doesn't make a difference19:19
*** salv-orlando has quit IRC19:19
* bkero imagines gcc is part of build-essential, but isn't certain19:19
greghaynesclarkb: I mean from the standpoint of 'devstack should work on a typical node'19:19
smarcetfungi: thx :)19:19
clarkbgreghaynes: ya we already preinstall it19:19
greghaynesYou do, I mean !infra users19:19
clarkbgreghaynes: but isn't this an opt in?19:19
clarkbI guess I don't understand under what scenario we are losing test coverage19:20
*** krtaylor has joined #openstack-infra19:20
greghaynesI just mean, a user boots a cloud image and runs devstack. We dont verify that devstack installs build essential. That node will not have build-essential in it already19:20
fungismarcet: yep, just finished reading the changelog. i'm going to update that commit message to link to the tag on rather than on, and then i'll approve it19:20
*** Daisy has joined #openstack-infra19:20
*** baoli has quit IRC19:20
*** jamesmcarthur has quit IRC19:20
*** amitgandhinz has quit IRC19:20
clarkbgreghaynes: but we hvaen't ever covered that I don't think19:20
*** baoli has joined #openstack-infra19:21
clarkbso its not lost coverage19:21
greghaynesah, ok. So then I would say - were better off if devstack installs that dep19:21
*** hashar has joined #openstack-infra19:21
openstackgerritJeremy Stanley proposed openstack-infra/system-config: OpenStackID 1.0.15 Release
*** moravec has quit IRC19:22
*** jamesmcarthur has joined #openstack-infra19:22
clarkbI want to say we have needed it for our own bootstrapping19:22
bkeroinstall gcc, bootstrap, remove gcc?19:23
clarkbnot sure if that is still the case19:23
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Enable gate jobs for Packstack on the stable/mitaka branch
dmsimardAJaeger: ^ couldn't have been long ago because until fairly recently we had stable branches going back to essex :p19:24
*** e0ne has quit IRC19:24
dmsimardwe did a little spring cleaning19:24
*** Daisy has quit IRC19:24
bswartzdoes anyone know why started requiring capcha solving to make changes?19:24
dmsimardI didn't want to explicitely exclude every branch :p19:25
clarkbbswartz: because it came under a massive spam attack19:25
dmsimardbswartz: to prevent spam19:25
bswartzthat sucks19:25
anteayabswartz: indeed19:25
fungibswartz: we also have new account creation temporarily disabled on it still19:25
*** fverbs has quit IRC19:25
anteayawish folks had enough to entertain them that spamming would be eshewed19:25
fungibswartz: because a lot of the spam was coming from valid launchpad openid accounts19:26
*** yamahata has quit IRC19:26
bswartzI would have assumed that openid would use captchas to prevent bogus accounts and offending accounts could simply be banned19:26
pabelangerfungi: you might be right about the restart of jenkins05, how do I check or proceed with that?19:26
anteayabswartz: we assumed security that wasn't there19:26
openstackgerritClark Boylan proposed openstack-infra/project-config: Add element to handle generic package installs
anteayaand hasn't been implimented19:27
clarkbnow with what should be valid json19:27
anteayaor I did, anyway19:27
fungibswartz: yeah, the offending new accounts were coming in so fast that to get them all banned in a timely fashion we had to prevent new ones19:27
anteayanot sure if anyone else assumed anything about openid19:27
greghaynesianw: looks like dib gate is broken19:27
clarkbgreghaynes: ya I think I got it fixed19:27
greghaynesclarkb: what was it?19:28
clarkbgreghaynes: try rerun and see if it works. ianw mostly fixed it but didn't source the venv activate script and its not executable so failed19:28
bswartzwasn't openstack going to roll its own ID at some point? is that canceled? deprioritized?19:28
anteayabswartz: in progress19:28
clarkbgreghaynes: pyyaml and uuidgen not installed on ubuntu-trusty19:28
*** baoli has quit IRC19:28
clarkbgreghaynes: the fix was use a virtualenv to install dib but then that wasn't sourced properly19:28
*** baoli has joined #openstack-infra19:28
fungibswartz: it's still a work in progress. we were considering switching the wiki to use it instead of launchpad, but someone needs to come up with a good user migration plan19:29
clarkbgreghaynes: oh exceept now its fialing on something new?19:29
greghaynesclarkb: yea, this is an odd one19:29
clarkbPS1 unbound variable19:29
clarkbin the activate script19:29
*** andymaier has quit IRC19:29
greghaynesoh, were running with set -u19:30
greghaynesis probably the issue19:30
clarkbwoo leaky bash options19:30
clarkbsince its source ya19:30
fungibswartz: it's actually already in use for several community systems (groups.o.o, refstack.o.o, translate.o.o, the translations check deployment)19:30
*** mrmartin has quit IRC19:31
*** geekinutah has joined #openstack-infra19:31
openstackgerritgreghaynes proposed openstack-infra/project-config: Disable set -u when sourcing dib venv
*** kencjohnston_ has joined #openstack-infra19:32
*** apoorvad has quit IRC19:32
*** moravec has joined #openstack-infra19:32
*** Jeffrey4l_ has joined #openstack-infra19:33
*** Jeffrey4l has quit IRC19:34
*** cody-somerville has quit IRC19:35
*** shardy has quit IRC19:36
bswartzI don't know about openid well enough to suggest a migration plan. Ideally systems like gerrit could allow both systems for a period of a few months and then you cut off the old one. publicize the hell out of it and pick a deadline that won't cause anyone to scream19:36
*** davidlenwell has quit IRC19:36
*** apoorvad_ has joined #openstack-infra19:37
pabelangerokay, jenkins05 seems to be the only master that isn't 100%.  it will fail with 502 proxy error after some time19:37
pabelangerI am not sure how to fix that19:37
openstackgerritMerged openstack-infra/system-config: OpenStackID 1.0.15 Release
pabelangerall other masters appear to be reverted for puppet devstack-trusty19:37
clarkbbswartz: I think what we would attempt to do is update the gerrit db (or other services db) with openid urls for both systems like that, then remove the old urls after some time19:37
clarkbbswartz: the bigger issue is likely just figuring out how to do the mapping in the first place and keep it up to date19:38
*** doug-fish has quit IRC19:38
*** ociuhandu has joined #openstack-infra19:38
*** vgridnev has quit IRC19:38
clarkbpabelanger: go to and look at the thread count, if it is >1k ish then it is likely thread leaking19:38
*** jamesmcarthur has quit IRC19:38
fungipabelanger: most likely suffering from the usual jenkins thread leak. there is a restart playbook but it tends to not work well once the server is misbehaving. first order of business is to get it into prepare for shutdown19:38
*** nadya has joined #openstack-infra19:39
fungionce you've confirmed via the method clarkb mentioned19:39
openstackgerritFlavio Percoco proposed openstack-infra/project-config: Add experimental job for MongoDB
flaper87IIRC, there's a way to run just 1 of the experimental jobs in a project. Did I dream this?19:39
flaper87Like: check experimental $something19:39
clarkbflaper87: it was a dream19:40
anteayatristanC tonyb what is happening with mordred's candidacy patch?
fungiclarkb: bswartz: right, what we're lacking is a direct map between launchpad accounts and foundation member accounts. we'll need to come up with a best-effort map which takes into account that probably the only thing we can try to match between them is e-mail addresses, but they're not guaranteed to be the same for a given individual as people may have more than one e-mail address, and launchpad also19:40
flaper87clarkb: ah, oh :(19:40
fungidefaults to making e-mail addresses private19:40
flaper87clarkb: it was a nice dream, though19:40
*** jamesmcarthur has joined #openstack-infra19:41
pabelangerclarkb: fungi: ya, jenkins05 not happy, currently still loading webui19:41
anteayapabelanger: :(19:41
tristanCanteaya: well, if mordred doesn't abandon his candidacy, we'll have to approve it since it's valid19:42
pabelangerclarkb: okay, if reading this correct it is at 10821 threads?19:42
*** davidlenwell has joined #openstack-infra19:42
fungipabelanger: it can take a few minutes and multiple retries sometimes when it gets this bad. if it gets so bad that doesn't pan out, the next possibility is that we ungracefully kill the service and deal with the aftermath19:42
clarkbpabelanger: nice!19:42
*** peterlisak has quit IRC19:42
clarkbpabelanger: definitely a leaking server so ya put it in shutdown mode (via the manage jenkins page), let the jobs finish then we clean it up and restart19:42
anteayatristanC: I don't know of a provision in the rules of elections asking candidates to abandon their candidacy once posted19:42
fungi11k is certainly > 1k19:43
*** baoli has quit IRC19:43
pabelangerokay, doing that now19:43
tristanCanteaya: if mordred abandon his candidacy, then packaging-deb will be ptl-less and it'll be up to the TC to appoint zigo (since we can't validate his candidacy)19:43
fungianteaya: tristanC: the result can still be appealed to the tc though, and they can rule to invalidate afterward i think?19:43
anteayawell seems to be that mordred has expressed the desire to stand as candidate19:43
*** onovy has quit IRC19:43
anteayaif a candidate posts a candidacy and it is valid19:43
pabelangerclarkb: prepare for shutdown right?19:43
*** baoli has joined #openstack-infra19:43
clarkbpabelanger: yes19:44
tristanCfungi: anteaya: well yes, either it's a tease/joke, then it could be abandone, otherwise we have to approve it19:44
clarkbpabelanger: what that does is stop any new jobs from starting but allows running jobs to finish without interruption19:44
fungii can't recall whether the tc has the power to depose a ptl who is unwilling to abdicate, or if their only recourse is to make the program unofficial19:44
anteayais mordred a valid candidatae or not19:44
fungi(we're changing terminology too often!)19:44
tristanCanteaya: mordred is a valid candidate by the book19:44
anteayaI dont' think the election officials have the place of decidig whether a candidacy is a joke19:45
anteayajust if it is valid19:45
clarkb passes testing and is ready now if we want to try getting that in and rebuild some ubuntu-trusty images19:45
clarkbfungi: greghaynes pabelanger bkero ^ your feedback in particular would be valuable19:45
anteayawell my experience is that election officials follow the book19:45
anteayawhich is my understanding of the role19:45
zigoanteaya: Please don't waste your time with useless bureaucracy, nobody including myself cares about this debate, or the PLT badge, we only care for the project to be live and running.19:45
zigoanteaya: Likely, Monty himself doesn't care much.19:46
anteayahow do you know what monty cares about19:46
anteayaand I decide what I spend my time on19:46
AJaegerfungi, stackalytics has some mappings of email addresses already...19:46
fungiAJaeger: yes, it's one potential source we can draw from19:46
anteayaand there is a valid candidate that has announced his candidacy19:46
zigoanteaya: Ok, so let me say it in a less nice way, since you aren't being nice here: don't waste the time of others...19:46
anteayanice doen'st enter into it19:47
anteayaa valid candidate has announced his candidacy19:47
bkeroclarkb: cron might get grumpy if there's no sensible-editor installed19:47
*** cody-somerville has joined #openstack-infra19:47
clarkbbkero: let me see if the package deps handle that19:47
bkeroI just tried in a chrooted one, and it got grumpy19:47
pabelangerclarkb: reviewed19:47
lifelessoh tox. tox tox tox19:47
pabelangerclarkb: so, is it now a hands off process for the jenkins restart or do we still need to manually trigger that?19:48
lifelessclarkb: mordred: you probably already know this, but tox presumes dependencies never change after a venv is made :(19:48
zigoanteaya: I don't see Monty in candidates/newton/Packaging-Deb19:48
clarkbpabelanger: you still need to do that when the jobs are all done19:48
fungipabelanger: manually put it in prepare for shutdown, the ansible playbook isn't likely to have a lot of luck with it19:48
clarkbpabelanger: can walk you through that when it reaches that state19:48
lifeless <-19:48
bswartzclarkb fungi: what about a trasitional system that requires you to log in with your LP account to obtain a secret and then log in with your foundation account and provide that secret to prove you are the same person -- the system could then link the 2 IDs19:48
*** piet has quit IRC19:48
anteaya zigo19:48
clarkblifeless: yes that is why we override the install commadn to pip install -u19:49
anteayaa vaild candidate has announced his candidacy19:49
pabelangerclarkb: fungi: okay, stepping away for a few minutes19:49
fungibswartz: for some system, like the wiki, i think that will work though we need to do broad announcements and schedule the start and end of that transition19:49
lifelessclarkb: that doesn't fix it19:49
lifelessclarkb: it doesn't undo '--no-deps'19:49
*** flepied has quit IRC19:49
fungibswartz: for others which don't allow the user to add multiple openid urls themselves, like gerrit, we'll need some tooling to handle that for us19:49
*** flepied has joined #openstack-infra19:49
clarkbbkero: could add vim-tiny19:49
clarkblet me see what the options are on centos19:50
bswartzgerrit keeps finding ways to screw us....19:50
bswartzoh wait did I type that out loud?19:50
fungibswartz: we can make it happen by adding them to the database but it will take admin intervention19:50
clarkblifeless: overriding the install command should override the entire install command19:50
clarkblifeless: so not sure where no-deps would be sneaking in from19:50
clarkblooks like fedora/centos is just "vim" as an option19:51
*** baoli has quit IRC19:51
*** MarkAtwood has joined #openstack-infra19:52
lifelessclarkb: from where I linked19:52
lifelessclarkb: see installpkg(...)19:52
openstackgerritMerged openstack-infra/project-config: Disable set -u when sourcing dib venv
tristanCzigo: anteaya: alright, I hoped this packaging-deb PTL matter would get solved through discussion. Since zigo acknoledged the situation and mordred kept his candidacy open I'll approve the review now19:53
*** doug-fish has joined #openstack-infra19:53
anteayatristanC: thank you19:53
anteayaI don't give a hoot who the ptl is19:53
clarkblifeless: right I think we bypass all of that with our tox.inis19:53
anteayabut if a candidacy is valid it has to be approved19:53
clarkblifeless: since it has to install the local package and we provide the entire command for it to do so19:54
lifelessclarkb: we don't - I just spent time with allison debugging a weird failure19:54
lifelessclarkb: with one of our stock tox.inis19:54
clarkblifeless: which one?19:54
tristanCwell at least, it seems like this debat bring some interrest back to supporting debian and package production through openstack-infra so that's a win!19:54
anteayathere is always a benefit to discussions, I agree19:54
anteayathank you19:54
lifelessopenstack-spec-tracker - I'll get you a url19:54
*** _amrith_ is now known as amrith19:55
*** jamesmcarthur has quit IRC19:56
clarkbah I see its modifying opts with extraopts19:56
clarkbthats fail19:56
lifelessclarkb: yes19:56
clarkbso you could drop {opts}19:56
clarkbbut that might break other things19:57
lifelessclarkb: right19:57
pabelangerand back19:57
lifelessclarkb: I'd like to get to the point of dropping -U19:57
anteayapabelanger: welcome back19:57
lifelessclarkb: and I think I've gotten closer to the actual root cause of our headaches with this insight19:57
lifelessclarkb: wdyt?19:57
clarkbyes this would certianly explain some of the headache19:58
*** Jeffrey4l_ has quit IRC19:58
lifelessI'll add this to the trello I use for packaging important-things19:58
lifelessand make sure there is a bug19:58
*** cp16net has left #openstack-infra19:59
*** jamesmcarthur has joined #openstack-infra20:00
*** Sukhdev has joined #openstack-infra20:01
*** pvaneck has joined #openstack-infra20:02
openstackgerritTim Buckley proposed openstack-infra/system-config: Fix tab name indices in status template
pabelangerSo, before I start diving down a rabbit hole, is there any reason that JJB does not use the C parser for PyYaml?  In an effort to speed up things20:05
anteayatristanC: in the Newton dir of the election repo there are two directories for refstack (Refstack and refstack) I'm sure that will keep happening, how do you want to address that when data is published according to the new spec?20:05
clarkblifeless: seems like we might be able to get away with dropping {opts} if that helps20:06
anteayapabelanger: sorry what are you currently addressing?20:06
anteayapabelanger: I've lost track20:06
clarkbbut I haven't tested that20:06
lifelessclarkb: I'd rather fix this, bump the tox minimunm and drop install_command20:06
pabelangeranteaya: was looking to see how to speed up JJB, since it takes a while to parse all our yaml files20:06
*** fitoduarte has quit IRC20:06
tristanCanteaya: that new directory creation slept through review process, I'm fixing it now20:06
clarkbpabelanger: probably only because the libyaml-dev headers are not present when installing pyyaml?20:06
anteayatristanC: ah awesome, thanks20:07
*** fitoduarte has joined #openstack-infra20:07
anteayapabelanger: oh okay so tuning not fixing a bug20:07
anteayapabelanger: cool, sounds like a good idea20:07
anteayapabelanger: and I don't have the answer to your question, sorry20:07
pabelangerclarkb: okay, so we've never attempted to use it I guess20:07
*** doug-fish has quit IRC20:07
pabelangermight hack on it tonight and see if there is a difference in time20:08
fungismarcet: jamesmcarthur: it looks like slot1 (current active deployment) is running 1.0.15 and i'm still able to log into translate.o.o and refstack.o.o so i think we're in the clear for now!20:08
*** pradk has quit IRC20:08
jamesmcarthurfungi: great news!20:09
fungithanks for working through this with us, and apologies for any delays this has caused on the summit apps end20:09
clarkbwas the underlying issue a bug in openstackid or zanata? maybe both?20:09
jamesmcarthurfungi: perhaps we can discuss root access for that server for smarcet when things are a little less immediate :) I really think it would be a benefit for everyone as we could act more quickly to assist w/ all things openstackID.20:10
clarkblifeless: ya I just have little hope for any contributions to tox as my attempts have in the past been largely ignored20:10
openstackgerritRoman Podoliaka proposed openstack-infra/project-config: oslo.db: don't run tests with SQLAlchemy 0.9.x
jamesmcarthurclarkb: i believe it was a combo, but I’ve been frying other fish today. smarcet: can tell you for sure20:11
lifelessclarkb: Nakato has been having some success20:11
ianwclarkb / greghaynes: ah, yeah ... so is dib testing working now?20:12
clarkbianw: maybe? we just had one more update to fix the set -u which hpefully is the last thing20:12
jamesmcarthurand fungi: no apologies necessary on the app front. we’re going to have to figure out a way to work through this one way or the other. Obviously we don’t want to disrupt active services for openstack users. At the same time, need to be able to do rapid development. I think having a better idea of dependencies is helpful for us, so we can be on the lookout and make sure we’re testing refstack and20:12
jamesmcarthurtranslate as well as www.o.o and groups.o.o20:12
fungiclarkb: the patch on the openstackid side was pretty subtle, fwiw. looks like mostly a logic error?
greghaynesianw: unsure, just kicked off another recheck20:13
*** doug-fish has joined #openstack-infra20:13
fungijamesmcarthur: yes, we had been under the impression we should be moving more services to openstackid as soon as time permits, but we could certainly do a better job of letting you know which ones they are (that is, when _we_ know about them anyway)20:13
*** fitoduarte has quit IRC20:13
prometheanfireclarkb: when was that merged?20:14
clarkbprometheanfire: 25 minute sago20:14
ianwgreghaynes / clarkb : ok, cool, just coming online.  change was untested but i think right direction.  as long as we keep using "#!/usr/bin/env python" anyway ...20:14
*** _ari_ has quit IRC20:14
prometheanfirewill push recheck button20:14
jamesmcarthursure - i mean it’s an opensource openID project… so it’s great it’s getting used.20:14
jamesmcarthurwe just didn’t realize it had been adopted by anyone but groups20:14
jamesmcarthurSo I’m actually really excited about that part :)20:14
prometheanfireianw: that should be python2 unless we know we are py3 compat20:15
fungijamesmcarthur: also after the summit crunch i want do double down on getting the distro upgrades for openstackid-dev.o.o and done, and the openstackid-resources.o.o server deployed. it seems like some of the impact would be taking place over there in the future and would be less likely to disrupt authentication as a result, though i may be reading too much into the partitioning of20:16
fungithose apis20:16
clarkbprometheanfire: eh python says python == python220:16
clarkbprometheanfire: arch can continue to be broken if they choose20:16
bkeroThat's going to change in xenial too AIUI20:16
greghaynesbkero: nope20:17
bkeroSo ubuntu will be broken as well20:17
*** pcaruana has quit IRC20:17
greghaynesbkero: just python2 isnt there by default AIUI20:17
*** asettle has joined #openstack-infra20:17
clarkband no python20:17
clarkbonly python320:17
ianwprometheanfire: yeah, at some point we were thinking of having tricks to handle that images might only ship python320:17
fungibkero: i certainly hope no sane distros make /usr/bin/python run py3k20:17
fungithat's counter to what the python community recommends20:17
ianwi don't want to think about that today20:17
prometheanfireclarkb: gentoo does that too :P20:17
fungiarchlinux was making a poor judgement call when they did that back in the 3.2 days20:18
prometheanfireclarkb: fedora defaults python to py3 now too iirc20:18
clarkbprometheanfire: well then gentoo can continue to be broken if they choose :P20:18
prometheanfireand fedora?20:18
clarkbprometheanfire: yes but fedora won't have a `python`20:18
bkeroprometheanfire: but gentoo also has a sane system of alternatives20:18
clarkbonly a `python3`20:18
ianwpabelanger: did you have a chance to see if centos-7 was uploading ok, but just timing out because we're taking too long to bring up network?20:18
clarkbdefault does not mean that the executable is called `python` it means this is what is installed by default20:18
prometheanfireoh ya, forgot fedora was doing it that way20:18
clarkbprometheanfire: that is the way python says you have to do it20:18
prometheanfirebkero: true20:18
ianwpabelanger: i have a vhd build with changed dependencies in glean that i will upload shortly and test out20:18
prometheanfireeselect python set python2 or something20:19
pabelangerianw: no, I can check now20:19
*** fawadkhaliq has joined #openstack-infra20:19
fungithe python upstream recommendation is that you treat python and python3 as separate languages with separate interpreters. making your python3 interpreter answer to requests for python is all sorts of wrong20:19
fungithat's like deciding that your /bin/ksh should be a symlink to /bin/bash20:20
clarkbfungi: ah I see now looks like the function default is true so old code would've been fine with an else too, though new code is more readable so yay20:20
bkerofungi: or deciding /bin/vi should be a symlink to /bin/vim ;)20:20
bkeroAIUI vim has a vi-emulation mode20:20
clarkbat least vim knows whta to do in that case20:20
clarkbbkero: ya it checks $020:21
prometheanfirehow often do people go though upper-requirements to update?20:21
ianwfungi: it's a bit of a bummer though when you're trying to write something portable.  istr there was a PEP, or maybe just talk, about an #! wrapper for python that autoselected20:21
*** eharney has quit IRC20:21
prometheanfireare we in lockdown now?20:21
*** e0ne has joined #openstack-infra20:21
fungivim has a "vi" mode, python3 does not have a python (2.x) mode that it will engage when you don't call it as python320:21
clarkbprometheanfire: I believe requirements is locked down, but not sure if they are extending that to upper-constrinats20:21
pabelangerianw: they look to be queued for upload20:21
bkeroAt this point is #!/usr/bin/env python more portable? or #!/usr/bin/env python2?20:21
prometheanfiremy change is only to liberty20:21
clarkbbkero: they are supposed to be equivalent20:21
*** kzaitsev_mb has joined #openstack-infra20:21
pabelangerianw: they've been inline for 2hours now20:21
bkeroclarkb: AIUI some distros don't have a python2 binary/symlink20:22
prometheanfirebeen sitting for a week
*** asettle has quit IRC20:22
clarkbbkero: right thats the other tricky thing20:22
clarkbbkero: and the other factor is some scripts iwll work in 2 and 320:22
pabelangerianw: currently 26 images that need to be uploaded still20:22
mgagnefungi: token is now 4h at internap. lets see if it changes something20:22
clarkbbkero: so for myself I have tried to stick to what python says is correct20:22
clarkbbkero: because at least then you can't point people that get it wrong at that20:22
anteayaI think I am going to head out for a walk in the sunshine20:23
clarkbfungi: seems to be mostly happy20:23
ianwpabelanger: ok, well what about the centos-7 that has been uploaded?  did sshing to that timeout?20:23
fungimgagne: thank you! we're still debating what sane workarounds in shade would be for clouds with similarly short token lifetimes and no trusts support20:23
clarkbfungi: I can make a new ubuntu-trusty and fedora-23 if we can get that in20:23
clarkbmgagne: mordred seemd fine with gettin a new token before each upload too20:24
*** kgiusti has left #openstack-infra20:24
*** piet has joined #openstack-infra20:25
fungiclarkb: i would have liked to see comments in package-installs.yaml warning about letting it become a dumping ground and contaminating our coverage of jobs installing the less common things they actually need, but that can always come in a later patch20:26
*** jamesmcarthur has quit IRC20:26
pabelangerianw: no, they are still uploading20:26
clarkbbkero: re already installed things I figured it was good to overspecify that way if minimal becomes less minimal we are ok20:26
fungi(or more minimal!)20:26
clarkbbkero: and cron doesn't depend on an editor and I think puppet should work?20:26
clarkbfungi: er thats what I mean20:26
AJaegerprometheanfire: ask on #openstack-release for requirements20:27
fungiyeah, cron will work fine without an editor. crontab -e will not20:27
clarkbwhy don't we start there and if we need the editor can add that in too20:27
ianwpabelanger: but what about the image you shared with me yesterday?20:27
*** aeng has joined #openstack-infra20:28
ianw(btw you can probably unshare that, so it can be removed)20:28
fungisimilarly, zigo did manage to download the debian-jessie qcow2 image so it can be deleted from /var/log/nodepool/images/ too20:28
lifelessNakato: clarkb: <- please fix any mistakes I made :)20:29
*** pcaruana has joined #openstack-infra20:30
pabelangerianw: I haven't done anything with it. Is it missing?20:31
pabelangerokay, I can unshare20:31
pabelangerand also remove debian-jessie20:31
ianwpabelanger: no ... i'm wondering why jobs still fail as NOT_REGISTERED if it's there, and it boots, and it comes online (eventually)20:31
clarkblifeless: that sounds about right particularly the just let pip do its job line20:31
ianwpabelanger: all i can assume is that sshing to it times out before the network finishes timing out on dhcp and we *really* bring it up20:32
ianwbut i'd like to confirm that a bit better before i go off trying to fix a problem that doesn't exist :)20:32
*** jamesmcarthur has joined #openstack-infra20:33
clarkbfungi: it may be worth restarting zuul today even without puppet nodes updated. THat will remove the vast number of registrations for devstack-trusty20:33
fungiclarkb: yes i agree, was just waiting to see activity (particularly in post) fall due to the backup we had earlier from repeatedly offlining the proposal slave20:34
mordredclarkb, fungi, mgagne: I'm going to try to hack on both the image import url and the re-get token patches on the plane this evneing20:34
mordredsorry I've been slammed in face-to-face meetings this week20:34
pabelangerianw: right, new new images from yesterday. So, once they upload to rax, we can see if they reregister20:34
pabelangerclarkb: fungi: from what I see, JJB revert to puppet jobs is good20:35
pabelangerminus jenkins05, but we are fixing that now20:35
fungipabelanger: thanks20:36
pabelangerokay, debian-jessie image removed from nodepool.o.o directory20:37
fungipabelanger: the other thing i usually do while waiting for a jenkins master to go idle is delete any nodes from nodepool which are assigned to it and in a ready state, since they'll falsely fulfill demand even though they're not going to pick up any jobs20:37
pabelangerfungi: okay, I can start that now20:37
fungithat has a tendency to cause jobs to pile up waiting on new workers while nodepool doesn't realize it needs to build more20:37
smarcetfungi: openstackid 1.0.15 and both sites, refstack and zanata are working :)20:38
openstackgerritMerged openstack-infra/project-config: Add element to handle generic package installs
fungismarcet: yes, i agree! i checked on it earlier, and also gave the translation team a heads up that we'd upgraded again but that the regression should be gone now20:39
*** matrohon has joined #openstack-infra20:39
fungismarcet: thanks for finding that and fixing, and for your patience while we did it through the dev environment20:39
*** hamzy has quit IRC20:40
bookwarhi, all, how to find jenkins log for documentation job? here for example, i'm trying to figure out why diagrams are not generated20:40
bkeroclarkb: cron doesn't depend on an editor, but it tries to use sensible-editor when configuring20:40
bkeroand since there's no editor installed, it fails20:40
fungismarcet: from the fix, it looks like the underlying issue was a logic error introduced through an earlier refactor?20:41
clarkbbkero: when configuring the package install?20:41
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use --overcloud-update in the upgrades job
clarkbbkero: wow, but shouldn't that be a dependency then?20:41
bkeroclarkb: good question20:41
clarkbseems like a broken package :/ but we can add vim-tiny/vim to make it happy20:41
smarcetfungi: well issue was a log left from debugging , sorry , an stupid error20:41
bkeroclarkb: here's me trying to install it:
*** jamesmcarthur has quit IRC20:41
smarcetseems that log line output to response and everything goes to hell20:42
clarkbbkero: does that return non zero20:42
bkeroclarkb: ugh, nevermind. I thought sensible-editor was being used in package configure. Re-reading my pastebin it's just being used in crontab -e like one would expect20:42
fungismarcet: oh, it was the log call you removed that had caused the authentication issue? that's even weirder20:42
bkerothe failure was due to my chroot20:43
bkeroand not having a proper /dev tree20:43
AJaegerbookwar: your docs gate has this URL - replace docs-draft with logs and remove the /doc/build/html to get the log file20:44
*** jamesmcarthur has joined #openstack-infra20:44
bookwarAJaeger: cool, thank you20:45
pabelangerfungi: xargs to the rescue20:45
AJaegerbookwar: "build succeeded" - but I guess you did not generate content in the expected place which is "doc/build/html"20:45
fungipabelanger: does nodepool delete take multiple node ids? i always assumed it would only use one20:45
pabelangerfungi: Ya, 1 at at time20:46
AJaegerbookwar: run "tox -e venv python build_sphinx" locally to verify20:46
pabelangerso did the following20:46
pabelangersudo -H -u nodepool nodepool list | grep jenkins05 | grep ready | cut -d'|' -f 2 | xargs -n1 sudo -H -u nodepool nodepool delete20:46
pabelangerugly, but worked20:46
fungioh, -n120:46
fungithat's fancier than my ugly for loop, fwiw20:46
pabelangerI wish you could wildcard delete20:46
smarcetfungi: yes there were a log call like r_print(Input::all())20:46
smarcetthat was the culprit20:46
fungismarcet: yep, saw it. that's a devious bug--good find20:47
fungipabelanger: well, you can. it's called an sql query20:47
fungibut that gets tricky with the subnode table20:47
pabelangerfungi: I too like to live dangerously20:48
bookwarAJaeger: i choose the wrong example, the path issue is already solved by another patch, but my problem is this one -  the ".. uml:" is not  parsed into diagram, as it is expected
bookwaris plantuml for docs supported in gate somehow?20:49
*** weshay has joined #openstack-infra20:50
pabelangerfungi: clarkb: I see 3 jobs in the build queue on jenkins05, do I manually cancel them in the UI?20:51
clarkbpabelanger: no you can ignore those20:51
fungipabelanger: i usually just ignore them. i'm not sure how they get there or what happens to them honestly20:51
*** jamesmcarthur has quit IRC20:51
clarkbits a gearman plugin thing I think20:51
pabelangerYa, one queued for 15 days20:51
AJaegerbookwar: platuml at least installs fine in the job according to the log file20:52
*** ZZelle_ has joined #openstack-infra20:52
*** cj has quit IRC20:53
AJaegerbookwar: but I've never used it, so no idea what dependencies it has20:53
bookwarAJaeger: i see, i'll try to dig in a bit..20:53
openstackgerritIan Cordasco proposed openstack-dev/pbr: Include wsgi_scripts in generated wheels
AJaegerbookwar: if you figure out what's missing, you can add thoese locally for your repo.20:53
AJaegerAn other-requirements.txt file can be used but that's still a bit experimental. Best talk with others here since I'm signing off for now...20:54
* AJaeger waes good-bye20:54
*** cj has joined #openstack-infra20:55
*** aysyd has quit IRC20:55
*** MarkAtwood has quit IRC20:56
*** ihrachys has joined #openstack-infra20:58
*** jamesmcarthur has joined #openstack-infra20:58
*** fawadkhaliq has quit IRC21:00
*** kaisers_ has joined #openstack-infra21:00
*** moravec has quit IRC21:00
*** moravec has joined #openstack-infra21:02
*** jamesmcarthur has quit IRC21:02
*** akshai_ has joined #openstack-infra21:03
*** rossella_s has quit IRC21:03
*** rossella_s has joined #openstack-infra21:03
pabelangerclarkb: okay, jobs done on jenkins05, but still a few nodes listed as offline21:04
*** e0ne has quit IRC21:04
funginibalizer: tristanC: i've finished looking for the electorate tarball e-mail and since determined that mailman probably rejected the post. hence only tristanC's reply went to the ml21:04
*** akshai has quit IRC21:05
*** kaisers_ has quit IRC21:05
clarkbpabelanger: you can manually delete them in the ui or if its terribly slow between stopping and starting jenkins service edit the config.xml to remove all the slaves and set nodepool to delete the ones on 05 it knows about21:05
*** |-paul-| has joined #openstack-infra21:06
*** rfolco has quit IRC21:06
*** sdague has quit IRC21:06
prometheanfireclarkb: dib still fails21:06
*** asettle has joined #openstack-infra21:06
prometheanfire2016-03-17 20:33:12.867 | ./env/bin/activate: line 13: _OLD_VIRTUAL_PYTHONHOME: unbound variable21:06
prometheanfirecentos one21:06
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove some extra mitaka translation jobs
clarkbprometheanfire: I think that just means jjb has't updated the jobs yet21:08
*** ihrachys has quit IRC21:08
prometheanfireso wait til then I guess21:08
*** kencjohnston_ has quit IRC21:08
ianwprometheanfire: yep ->
pabelangerclarkb: okay, slaves deleted via the UI21:09
openstackgerrityolanda.robla proposed openstack-infra/project-config: Add non voting job to test k8s with dib atomic image
*** ihrachys has joined #openstack-infra21:09
prometheanfirewell, recheck in 30 min21:10
*** asettle has quit IRC21:11
*** Sukhdev has quit IRC21:12
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Changing gate on devstack identity v3 only voting
*** ldnunes has quit IRC21:12
*** kencjohnston_ has joined #openstack-infra21:13
*** jamesmcarthur has joined #openstack-infra21:13
*** moravec has quit IRC21:14
fungipabelanger: clarkb: any more i've just gotten used to editing the config.xml and deleting all lines between <slaves> and </slaves> before starting the service back up21:15
*** nadya has quit IRC21:15
*** regXboi has quit IRC21:15
*** fawadkhaliq has joined #openstack-infra21:15
fungithanks for the resend, nibalizer!21:16
pabelangerfungi: clarkb: So, I should have jenkins stopped now and do that step?21:16
*** jamesmcarthur has quit IRC21:16
openstackgerritMerged openstack/requirements: uc: openstackdocstheme==1.3.0
clarkbpabelanger: yup21:17
clarkbpabelanger: just remove everything between the two slaves tags21:17
fungipabelanger: once all jobs have finished, you need to `sudo service jenkins stop` and then keep an eye on the process table for the java processes to die off21:17
clarkboh ya it may not die and need kill -9ing21:17
clarkb(which is in the playbook fwiw)21:17
fungiand sometimes they don't, and you need to get increasingly aggressive with kill signals21:17
crinklefungi: clarkb: jeblair: yolanda: rcarrillocruz: greghaynes: SpamapS: Venu and wendar would like to set up a call to review our infra-cloud networking request and discuss a network diagram, would some of you wish to attend? (also posted to the infra ml)21:17
fungiright that21:17
fungicrinkle: saw the e-mail, thanks!21:18
fungicrinkle: what timeframe are we looking at for the call? just sometime next week (tbd)?21:18
rcarrillocruzcrinkle: i'll check the time, sure21:18
*** moravec has joined #openstack-infra21:18
fungicrinkle: nevermind, i see you said Tuesday at 1800 UTC'21:19
crinklefungi: sometime next week is the time frame, I'm hoping we can come up with a couple of proposals to find something that works for us and Venu and wendar21:19
rcarrillocruztuesday works for me ( i think )21:19
fungicrinkle: the proposed time wfm21:19
rcarrillocruzi'll respond in mailing list21:19
fungiit's right before the infra team meeting, but as long as it's under an hour should be no problem21:19
*** knikolla has quit IRC21:20
funginibalizer: tristanC: so far the only differences i see between the electorate provided and the version i ran yesterday is a bunch of people who updated their preferred e-mail addresses in gerrit at the last possible moment, but that's not unusual. looks good!21:23
fungitonyb: ^21:23
pabelangerokay, cleared up and starting jenkins05 again21:23
*** baoli has joined #openstack-infra21:23
*** jamesmcarthur has joined #openstack-infra21:23
tonybfungi: Thanks.  good to know.21:23
openstackgerritJames E. Blair proposed openstack-infra/project-config: Grafana: Add Tempest check runs graph
fungicounts and user ids didn't change, which is the important bit21:24
pabelangerand looks to be launching jobs21:25
pabelangerclarkb: I'll trying the playbook next time21:25
*** gordc has quit IRC21:25
*** ihrachys_ has joined #openstack-infra21:26
*** denisra has joined #openstack-infra21:26
fungitristanC: tonyb: nibalizer: looks like the extra step of copying the stable branch electorate in from the other run didn't happen, but that's probably fine for now since nobody stepped up to be ptl for them?21:27
*** matrohon has quit IRC21:27
tonybfungi: that's my understanding.  I think it will fall on the TC to make a decision.21:28
fungiokay, then you should have what you need to move forward with the contested positions21:28
*** ihrachys has quit IRC21:28
fungibut let us know if you need anything21:28
*** sdake has quit IRC21:28
jeblaircrinkle: that time sounds great21:29
*** amotoki has quit IRC21:29
tonyblots of actual elections this time round.21:29
*** geekinutah has quit IRC21:30
*** smarcet has quit IRC21:30
*** sdake has joined #openstack-infra21:30
*** claudiub has quit IRC21:31
*** jamesmcarthur has quit IRC21:31
tonybfungi: on a tangental note what happen now with the electiom infra-specs?21:31
fungitonyb: makes me glad we're not on the old script that required hand-assembling the team electorates out of per-repo lists, adding extra-atcs in by hand, manually searching for duplicate accounts...21:31
*** matrohon has joined #openstack-infra21:31
fungitonyb: i am now 2.5 hours past due for approving the two patches for that21:31
fungithanks for the reminder ;)21:31
tonybfungi: it wasn't a nagg it's just my first time throught tht process21:32
fungitonyb: after the day we've had in here, can use all the reminder help i can get21:32
*** jamesmcarthur has joined #openstack-infra21:32
*** vgridnev has joined #openstack-infra21:33
tonybfungi: As I'm just gettign started I don't knwo what happend but still I has a sad if y'all had a rough time21:33
*** jamesmcarthur has quit IRC21:33
*** lakshmiS has quit IRC21:34
*** vgridnev has quit IRC21:34
fungitonyb: it's just been a busy day. omelettes and broken eggs, you know21:34
*** doug-fish has quit IRC21:35
*** jamesmcarthur has joined #openstack-infra21:36
*** Daisy has joined #openstack-infra21:36
anteayatonyb: so basically after fungi hits go you work on the bits you said you would21:36
anteayaand then tell us so we can do the things we said we would do, like review21:37
fungigo has already been hit, it'll just be a few minutes until these merge21:37
tonybanteaya: jhesketh has done them all for me :)21:37
anteayaisn't he wonderful21:37
fungiand yeah, jhesketh wrote a preliminary implementation21:37
tonybanteaya: yes he is21:37
anteayaeveryone should have a jhesketh in their life21:37
fungipredicting approval of the current spec changes21:37
fungiaccurately predicting, i might add21:37
anteayatonyb: then you bug us til they merge21:37
*** moravec has quit IRC21:38
anteayaha ha ha21:38
anteayacrystal ball he has21:38
openstackgerritMerged openstack-infra/infra-specs: Create
openstackgerritMerged openstack-infra/infra-specs: Mark public hiera as completed
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Reduced log level from warning to error on global config
*** maratoid has joined #openstack-infra21:38
*** ihrachys_ has quit IRC21:39
maratoidHello. Is this the right place to ask questions about jenkins job builder?21:39
*** ihrachys has joined #openstack-infra21:39
openstackgerritMerged openstack-infra/openstackid-resources: Reduced log level from warning to error on global config
anteayawish they all were that easy21:40
maratoidI can't quite get past this error: "Duplicate entry point found for component type: 'property', 'property',name: 'build-discarder'"21:40
anteayamaratoid: we like stacktraces21:40
*** doug-fis_ has joined #openstack-infra21:40
maratoidi see.21:40
anteayaposted to a paste service is our favourite21:40
*** Daisy has quit IRC21:40
*** jamesmcarthur has quit IRC21:40 unless you prefer something else21:41
*** mriedem has quit IRC21:41
anteayaquick you are21:42
fungimaratoid: is this a regression from a recent jenkins-job-builder upgrade, or new configuration you're trying to work through?21:42
*** thorst has quit IRC21:42
maratoidnew configuration21:42
anteayamorning jhesketh21:43
*** thorst has joined #openstack-infra21:43
fungieasiest way to debug these things is to distill your config down to a minimal version which reproduces the error21:43
*** jamesmcarthur has joined #openstack-infra21:43
fungiand then provide a paste of that minimal configuration so that we can more easily spot the issue in it21:44
fungioften times i find the answer myself while i'm trying to come up with a minimal reproduction21:44
*** sdake has quit IRC21:45
*** doug-fis_ has quit IRC21:45
maratoidseems like this comes up for me whenever i have more than 1 job template in my project, using build-discarder21:45
*** doug-fish has joined #openstack-infra21:45
*** jpr has quit IRC21:45
maratoidperhaps i misunderstood how job templates work. do i need to make sure they are in a job group, or anything like that?21:46
*** fawadkhaliq has quit IRC21:46
openstackgerritMerged openstack-infra/tripleo-ci: Boot from volume in ping test
*** thorst has quit IRC21:47
*** xyang1 has quit IRC21:47
JayFI'm trying to track down a gate failure building IPA ( ) -- is there any way to get onto one of these machines when it fails before it's deleted?21:48
*** tonytan4ever has quit IRC21:48
anteayamaratoid: can you paste what you have for templates?21:48
JayFOr a script I can run in a VM / cloud server to reproduce the build exactly like it happens in the devstack gate21:48
*** baoli has quit IRC21:49
*** doug-fish has quit IRC21:50
*** baoli has joined #openstack-infra21:50
clarkbJayF: there should be a script in the logs/ dir21:50
clarkbJayF: now that requires you provide your own image/OS but its a start21:51
JayFJust a trusty VM, right?21:51
maratoidanteaya: +
clarkbif you can't reproduce with that then we can hold an instance for you21:51
clarkbJayF: yes21:51
JayFCool, I have a cluster of big beefy servers just waiting to run this :P21:51
maratoidanteaya: I'll try to distill to a smaller example21:51
JayFNobodyCam: ^ going to start that science, but I'm out Fri/Mon so hopefully 2 hours is enough :P21:51
NobodyCamJayF: please let me know how far you get and what you had to do :)21:53
NobodyCamand Morning / afternoon / evening Infra peeps :)21:53
JayFNobodyCam: I'm doing this on an OnMetal(tm) v2 server, so hopefully it won't take long :P21:54
NobodyCamnice :)21:54
clarkbJayF: based on my reading of the log looks like there is no docker socket to connect to, maybe docker isn't autostarting?21:55
clarkbor permissions have hcanged/21:55
JayFclarkb: docker is starting, I think perms are likely21:55
*** asettle has joined #openstack-infra21:55
anteayamaratoid: yeah it is going to take me some time to figure out how your files are structured21:55
JayFclarkb: initial thought was the socket wasn't getting fully created, so we added a busyloop to check for the socket, that passes, and it still fails21:55
anteayamaratoid: I think simplified would save both of us time21:55
JayFclarkb: so I can keep guessing and eating up gate resources or I can repro and find which magic fix it is :P21:55
JayFclarkb: being our first stab at it21:56
*** doug-fish has joined #openstack-infra21:56
JayFclarkb: (you won't see the failure there, because we don't run ipa -src jobs against Ironic, but we have it testing against another commit in IPA with depends-on)21:56
anteayaclarkb: should firefox be forcing me to download from when I click on it?21:57
*** kencjohnston_ has quit IRC21:57
clarkbanteaya: it isn't treated as a text file due to the suffix so yes it will want you to download21:57
*** fawadkhaliq has joined #openstack-infra21:57
anteayachrome doesn't21:57
clarkbwhich makes sense, you have to execute it, viewing it in the browser isn't helpful21:57
clarkbanteaya: chrome may be smart then21:57
anteayaseems to be21:58
anteayabut thanks, didn't know about reproduce.sh21:58
clarkbfyi I have filed and have left my test instance and volume up in vexxhost so that it can be poked at more as part of bug triage if necessary21:58
openstackLaunchpad bug 1558807 in OpenStack Compute (nova) "Volume attached at different path than reported by nova/cinder when using config drive" [Undecided,New]21:58
anteayaclarkb: ack21:59
*** jamesmcarthur has quit IRC21:59
*** IlyaG has quit IRC21:59
JayFclarkb: are the prereqs for the reproduce script docuemnted anywhere?21:59
*** amotoki has joined #openstack-infra22:00
*** flwang has joined #openstack-infra22:00
clarkbJayF: uh just what devstack-gate need22:00
clarkbit should mostly run22:00
flwangclarkb: sorry for bugging you again22:00
JayF./ line 69: virtualenv: command not found22:00
*** asettle has quit IRC22:00
flwangclarkb: can you have work out the zaqar-ui-core issue for me? thanks22:00
JayFclarkb: that actually appears to be all it needed; virtualenv22:01
anteayaJayF: awesome, was going to give it a spin myself too22:01
*** dims has quit IRC22:01
clarkbflwang: you need group membership?22:01
*** dims_ has joined #openstack-infra22:01
JayFwell it also needs a c compiler now22:01
clarkbflwang: what is the gerrit user to add?22:01
clarkbJayF: aha! we do need gcc for boostrapping (that was a question earlier today)22:01
flwangclarkb: thanks22:01
clarkbgreghaynes: ^ there you go22:01
clarkbJayF: we thought we might not actually need it, you have discovered otehrwise22:02
JayFclarkb: ./ line 79: /usr/zuul-env/bin/zuul-cloner: No such file or directory22:02
*** tphummel has quit IRC22:02
JayFclarkb: that is one I'm going to have a bit hrder of a time finding22:02
clarkbJayF: it definitely attempts to make that one22:02
JayFhmm, so if it failed on c compliler22:02
clarkbJayF: you probably need to sudo the script22:02
JayFmaybe remove the dir22:02
clarkbah ya22:03
clarkbits does a -d22:03
JayFa -e, in fact :P22:03
clarkbeven better22:03
clarkbwe should probably have the script self document these things in comments22:03
JayFthat seems to have made it happier; so far only build-essential and virtualenv22:03
JayFand apparently even more stuff now22:03
clarkb# run as root, have build-essential and python-dev installed and python-virtualenv22:03
clarkbJayF: ya thats required to pip install many things22:03
anteayaclarkb: that would help22:03
JayFoh god for a sec I thought you were saying that was already in the file22:04
JayFalmost felt stupid there22:04
clarkbJayF: no I think it isn't but it should be!22:04
*** fawadkhaliq has quit IRC22:04
anteayaI agree22:04
*** jamesmcarthur has joined #openstack-infra22:04
clarkbbut one thing at a time before I go crazy, zaqar-ui22:04
*** tphummel has joined #openstack-infra22:04
JayFneeds git too22:04
*** mylu has joined #openstack-infra22:04
JayFclarkb: point me at the repo, I'll make this not-suck when I'm done22:04
docaedoanyone have a minute to log on to the app-catalog server and check to see if a file exists for me?22:04
*** fawadkhaliq has joined #openstack-infra22:04
clarkbJayF: its in openstack-infra/devstack-gate22:05
JayFgit, python-dev, build-essential and pip install virtualenv so far :)22:05
JayFclarkb: cool, I'll put something up there if I get to the bottom22:05
clarkbJayF: I think that should be it22:05
anteayaJayF: I'll wait for you and then review your patch after you figure it out22:05
clarkbJayF: and dont forget to mention root22:05
clarkbflwang: ok done22:05
flwangclarkb: thank you so much :)22:05
*** zul has joined #openstack-infra22:06
clarkbkrotscheck: did you get to the bottom of the firefox issues?22:08
JayFanteaya: clarkb: assuming I don't find more prereqs, something like this look sane?
clarkbJayF: yup22:09
clarkbwe can worry about the fact that this also runs on centos and fedora later :)22:09
JayFOh really, it does?22:09
JayFbut like, in the gate, it's always trusty, right?22:09
jrollJayF: I wonder if "must have proper internets" is obvious22:09
*** baoli has quit IRC22:10
anteayajroll: ha ha ha22:10
clarkbJayF: no it can be centos7 or fedora as well22:10
clarkbbut most jobs are trusty22:10
anteayaJayF: thanks22:10
JayFclarkb: okay, cool, just making sure my assumption that the ipa -src job ran always in trusty was sane22:10
jrollanteaya: I'm somewhat serious, I've seen people rabbit hole behind a proxy etc22:10
anteayaoh yeah, that22:10
anteayagood point22:10
anteayaI'm so far removed from that myself22:11
clarkbJayF: for that it is. We don't pick them randomly22:11
anteayaI consider it fairytale22:11
clarkbjust that devstack-gate does run on all 322:11
JayFcool, makes sense22:11
jrollmaybe "assumes computer has proxy-less access to the internet on every port" would be a valid addition22:11
*** jamesmcarthur has quit IRC22:11
clarkbubunt-trusty is being rebuilt with the new infra packages list element22:11
openstackgerritMerged openstack/diskimage-builder: Reorder developer quickstart docs
clarkbjroll: it should honor http_proxy22:11
openstackgerritJay Faulkner proposed openstack-infra/devstack-gate: Document prereqs/usage for reproduce script
clarkbbut that might be getting too complicated for a short header comment22:11
jrollclarkb: not if it's unset22:11
JayFclarkb: assuming if I'm to "running devstack" there's no other prereqs ^22:12
clarkbJayF: yup22:12
jrollidk, just thinking of other failure cases22:12
clarkbthe problem with firewalls and proxies is every implementation is different22:12
JayFI mean, any documentation >> no documentation22:12
clarkbparticularly when you start talking about not http22:12
JayFas long as it's correct, even  if it's not comprehensive22:12
JayFif you feel strongly about it tell me on gerrit :P22:12
*** akshai_ has quit IRC22:13
*** yamahata has joined #openstack-infra22:14
*** yamahata has quit IRC22:15
*** yamahata has joined #openstack-infra22:15
*** piet has quit IRC22:16
*** piet has joined #openstack-infra22:16
*** amotoki has quit IRC22:17
anteayaJayF: rebuilding a node to test22:17
*** jamesmcarthur has joined #openstack-infra22:18
*** zul has quit IRC22:18
*** amrith is now known as _amrith_22:19
*** woodster_ has joined #openstack-infra22:20
*** asettle has joined #openstack-infra22:20
*** thorst has joined #openstack-infra22:21
*** smarcet has joined #openstack-infra22:21
smarcetfungi: seems that there is some error at redis instancec22:22
smarcetits throwing errors22:22
smarcetcould u check?22:22
fungismarcet: looking now22:22
*** |-paul-| has quit IRC22:22
*** dims has joined #openstack-infra22:23
smarcetok i must take off but exception is clear22:23
smarcet exception22:23
smarcet'Predis\Connection\ConnectionException' with message 'Error while22:23
smarcetreading line from the server [tcp://]'22:23
smarcetthx :)22:23
*** baoli has joined #openstack-infra22:23
fungismarcet: i'm not seeing any errors in the /var/log/redis/redis-server.log but will keep looking22:23
*** ihrachys has quit IRC22:24
fungiit's definitely still listening on 6378/tcp22:24
smarcetcould we restart it ?22:25
*** ihrachys has joined #openstack-infra22:25
*** asettle has quit IRC22:25
*** thiagop has quit IRC22:25
fungiis it possible the data being read from it was malformed?22:25
*** dims_ has quit IRC22:26
fungithe log is pretty much just status info about the number of clients connected, number of bytes in use22:26
*** baoli has quit IRC22:26
clarkbfungi: how much free memory do we have?22:26
fungia few messages about closing idle connections22:26
*** baoli has joined #openstack-infra22:27
clarkbredis is a memory hog and can get crnaky when memory is low22:27
fungiusing ~75% of 2gb ram, and a third of that is buffers/cache22:27
clarkbok thats not bd22:27
*** mylu has quit IRC22:27
fungihasn't touched the 2gb of swap at all22:27
smarcetjust started rite now22:27
ianwJayF: to really reproduce, i think you want to run -> first22:27
fungisystem load is negligible22:28
fungino indication of any oom killer activity in dmesg22:28
smarcetwell seems that is working again22:28
smarcetperhaps a peek22:28
clarkbgah I didn't restart the builder so it isn't using the new elements list22:28
clarkbwhen this build completes I will restart the builder and rebuild22:29
maratoidanteaya: smaller example - properties macro: project: , job template:
maratoidstack trace:
clarkbactually just going to restart it now so we don't use this build22:29
fungismarcet: yeah, i'm checking cacti now to see if we spotted any spikes in activity22:29
clarkbnope nevermind there are a bunch of image builds22:29
anteayamaratoid: looking22:29
fungismarcet: clarkb: memory use has picked up in the last 15 minutes or so:
maratoidanteaya: sorry, that's 'parameter' macro. not property22:30
*** bpokorny_ has joined #openstack-infra22:31
*** MarkAtwood has joined #openstack-infra22:31
fungismarcet: clarkb: that looks anomalous compared to the last weeks/months22:31
smarcetyeah i am not able to login in on www.openstack.org22:32
smarcetaltought ref stack and translate works ok22:32
clarkbI manually killed the dib process to prevent it from making an image that isn't correct22:32
smarcetdev works22:32
smarcetso its something on the server22:32
fungismarcet: clarkb: tcp connections have picked up too
clarkbfungi: possibly more use from translaton team?22:32
clarkbbut I agree its picked up22:33
funginot sure if those are piling up because they're hung waiting on something22:33
smarcetand redis errors still comming down from log22:33
smarcethow much ram has the node ? 2GB ?22:33
fungii can try as redis restart to see if that makes a difference, but have a feeling it might be something else22:33
smarcetwell thing is22:34
fungiyeah, 2gb but it's only got around 50% allocated and another 25% in use for buffers/cache at the moment22:34
smarcetif we restart all sessions will lost22:34
smarcetbc openstackid use redis to store sessions22:34
smarcetnot big deal22:34
*** bpokorny has quit IRC22:34
fungii'm going to take a look at the apache access logs to see if there might be a lot of connections from one or a handful of sources22:35
*** camunoz has joined #openstack-infra22:35
anteayamaratoid: so you must have a file that includes the contents -kraken-destroy or something22:35
anteayamaratoid: do you?22:35
maratoideach one of those is in a separate yaml files22:35
smarcetfungi got it22:36
maratoidjob template, project and parameter macro22:36
fungismarcet: a bunch of hits from 2001:4800:7820:103:be76:4eff:fe05:b574 in rackspace which has no reverse dns record22:36
clarkbsmarcet: is the redis not configured to write to disk? if that happens a restart shouldn't kill any sessions22:36
*** jamesmcarthur has quit IRC22:36
JayFclarkb: Not only did that not reproduce the error, it doesn't even seem to have hit the error codepath at all22:38
smarcetclarkb: not sure had to review config settings on puppet22:39
fungii have a feeling that ipv6 address is something to do with the site22:39
anteayamaratoid: have you run other job templates successfully?22:39
anteayamaratoid: like this one:
fungialso a bunch this hour from 2001:4800:7821:101:be76:4eff:fe06:858b which is also in rackspace and also has no reverse dns22:40
*** sigmavirus24 is now known as sigmavirus24_awa22:40
JayFclarkb: anteaya: Yeah, I can't even see evidence it even tried to clone in the IPA repos22:40
fungismarcet: clarkb whatever the spike in established tcp connections was, it's dying back off again22:41
fungias is the memory spike22:41
maratoidanteaya: you know, I just tried updating to the latest version (i was at 1.4.0, installed through pip) and the minimal example actually worked.22:41
maratoidsorry, this whole thing is created by automation, and i think the part that's supposed to install from source is not quite working22:42
clarkbJayF: hrm that may mean the reproduce script is buggy22:42
clarkbJayF: like maybe it isn't writing out hook functions22:43
*** matrohon has quit IRC22:43
*** jamesmcarthur has joined #openstack-infra22:43
JayFidk, I'm going to use the vm to try and repro manually now22:43
*** dingyichen has joined #openstack-infra22:43
JayFit did install docker though, and it is working /o\22:44
JayFso my vm isn't having the issue at all22:44
JayFI'm pretty sure it's some kind of race, so running it on a beefy server might make it disappear22:44
JayFbut it didn't even run automatically so I don't even know that22:45
smarcetfungi: well not sure this start to happen 30 minutes or so ago22:46
anteayaJayF: :(22:46
smarcetthe result is that login is not working22:46
anteayamaratoid: yay22:46
smarcetfungi: not sure what to do22:46
anteayamaratoid: sounds like you are fixed for the templates bit?22:46
*** mylu has joined #openstack-infra22:47
fungismarcet: some web searching on that error message indicates the application is running into a timeout reading from the redis socket22:47
smarcetyeah i saw it22:47
fungismarcet: i'll go ahead and restart redis and we'll see if it recovers for very long22:47
smarcetthey said increment time out22:47
smarcetyeah its the only alternative22:48
maratoidanteaya: i think so. seems like maybe they weren't actually broken :/ thanks!22:48
smarceti am afraid22:48
fungiokay, it's restarted22:48
fungii'm watching the application log for any more redis read errors22:49
fungiyep, there it is22:49
fungistill happening after the restart22:49
pabelangerianw: still uploading images :(22:49
fungia lot of "DelayCounterMeasure: attempt from banned ip" messages too22:49
anteayamaratoid: nice to meet you, happy trails22:49
smarcetstill not working :/22:49
*** kaisers_ has joined #openstack-infra22:49
fungismarcet: 2001:4800:7820:103:be76:4eff:fe05:b574 is the one i saw hitting apache a bunch (in rackspace, no reverse dns) and it's what's being blocked by openstackid22:50
ianwpabelanger: alright, i'm uploading my test images too.22:50
smarceti know the issue22:50
smarcetwas www.openstack.org22:50
smarcetgot banned22:50
smarcetremoved from banned ips22:50
fungithat's a fun failure mode22:51
*** jamesmcarthur has quit IRC22:51
pabelangerianw: I don't know the whole story, but jeblair and fungi did some tests to at mid-cycle to increase the amount of image uploads I believe.  I think 8 or so?22:51
ianwpabelanger: i still think we should have logs from the previous upload?22:52
smarcetfungi: yeah sorry should look on the banned table first22:52
smarceti will add www ip on the white list table22:52
clarkbpabelanger: ianw its still 4 upload threads, during the sprint we tested 16 and thats about what it took to saturate our link22:53
smarcetfungi: tnx for all your help today and sorry for false alarm22:53
*** maratoid has quit IRC22:53
fungismarcet: glad it wasn't something more serious22:53
*** smarcet has quit IRC22:53
pabelangerclarkb: is there anything stopping us from getting a patch up today to bump it close to 16?22:54
*** hashar has quit IRC22:54
fungismarcet: also, getting reverse dns entries added for those systems would be a huge help. we set them on all of our servers, so i was for example able to see that the #3 source of traffic to was, but #1 and #2 were just "random hosts in rackspace)22:54
*** tqtran is now known as tqtran-afk22:54
clarkbgreghaynes: looks like ubuntu-minimal inclues gcc too fwiw22:54
*** kaisers_ has quit IRC22:54
fungioh well, he's gone22:54
*** ihrachys has quit IRC22:54
clarkbpabelanger: I don't think so, just someone needs to write the chang eand get it merged22:54
clarkbwe also need to make the config reload on changes22:54
*** mriedem has joined #openstack-infra22:54
*** Sukhdev has joined #openstack-infra22:54
pabelangerokay, I can see what is needed22:54
*** Keedya has joined #openstack-infra22:55
*** ihrachys has joined #openstack-infra22:55
greghaynesclarkb: wah22:55
greghaynesclarkb: I wonder what pulls it in22:55
*** dimtruck is now known as zz_dimtruck22:56
*** Keedya has quit IRC22:57
greghaynesclarkb: it shouldnt be debootstrap, we use minbase there... Also, build-essential is more than gcc22:58
fungiif smarcet comes back, 2001:4800:7820:103:be76:4eff:fe05:b574 is still being blocked by openstackid.org22:58
fungior started getting blocked again, anyway22:58
*** rbrndt has quit IRC22:58
*** baoli has quit IRC22:59
clarkbgreghaynes: it looks an awful lot like debootstrap22:59
clarkbits one of the first things that happens22:59
clarkbit hasn't gotten to my new package install stuff yet22:59
*** baoli has joined #openstack-infra23:00
*** kaisers_ has joined #openstack-infra23:00
*** piet has quit IRC23:00
*** HeOS has quit IRC23:00
*** piet has joined #openstack-infra23:01
*** sridhar_ram1 has joined #openstack-infra23:01
greghaynesclarkb: hrm, maybe minbase does include gcc but not build-essential23:01
*** tqtran-afk has quit IRC23:01
greghaynesclarkb: which is funny because I think cloud images do not include gcc23:01
fungithat sounds right23:01
*** sridhar_ram has quit IRC23:01
fungigcc is considered essential on debian iirc23:01
*** jamesmcarthur has joined #openstack-infra23:01
Clintfungi: no, build-essential23:02
fungibuild-essential is a metapackage of things you can install if you intend to compile typical c software, but having gcc is useful for more than just general build work23:02
fungiClint: it's just in build-essential?23:02
fungidouble checking23:02
fungithen does debootstrap special-case gcc?23:02
*** HeOS has joined #openstack-infra23:03
fungiright you are, Priority: optional23:03
*** cody-somerville has quit IRC23:03
greghaynesoh hey, minbase does include build-essential, I was way off23:03
*** cody-somerville has joined #openstack-infra23:04
fungiokay, so we're already ending up with all of build-essential on the minimal images?23:04
clarkb there it goes23:04
greghaynesmaybe someone else can help parse this debootstrap manpage23:05
clarkbI think this is going to work fine, I will do my best to restart nodepool-builder when the current uploads finish23:05
*** baoli has quit IRC23:05
greghaynesI think it is saying that build-essential is in minbase, but everytime I parse it I get more confused23:05
*** cfarquhar has joined #openstack-infra23:05
*** cfarquhar has quit IRC23:05
*** cfarquhar has joined #openstack-infra23:05
*** baoli has joined #openstack-infra23:05
*** yamahata has quit IRC23:06
greghaynesok, I think it is saying buildd includes build-essential, minbase does not23:06
clarkbfungi: well we always explicitly added build-essential (we need it for zuul cloner install)23:06
*** kaisers_ has quit IRC23:06
clarkbfungi: but this shows we odn't need to because ubuntu-minimal adds it in anways23:06
fungigreghaynes: mine says you need --variant=buildd to get build-essential23:06
greghaynesfungi: gotcha23:06
greghaynesso ya, youall must be adding that somehow23:07
clarkbwe don't do anything other than use ubuntu-minimal23:07
fungiand that --variant=minbase "only includes essential packages and apt"23:07
fungioh, though i'm looking on debian23:07
* greghaynes kicks off a build to grep logs on23:07
fungimaybe ubuntu makes their own rules for debootstrap variants23:07
*** asettle has joined #openstack-infra23:07
*** baoli has quit IRC23:08
*** dims has quit IRC23:08
*** baoli has joined #openstack-infra23:08
greghaynesExtracting gcc-4.9-base...23:10
greghaynesfrom the debootstrap output23:10
clarkb22 images to upload23:11
clarkbso slow23:11
*** jamesmcarthur has quit IRC23:11
*** moravec has joined #openstack-infra23:12
*** ihrachys has quit IRC23:13
openstackgerritSteve Martinelli proposed openstack-infra/project-config: add more clients when checking for openstackclient command overlap
*** ihrachys has joined #openstack-infra23:14
*** piet has quit IRC23:14
*** pvaneck has quit IRC23:15
jamielennoxAJaeger, ianw: i simplified would you be able to give it a look so i can move on to the dependant patches?23:15
*** IlyaG has joined #openstack-infra23:15
clarkbgreghaynes: I don't think its a huge deal but wanted to point out its not as simple as us not installing it23:15
cody-somervilleLatest gcc-x.y-base is priority required due to the dependency chain libc6 → libgcc1 → gcc-x.y-base23:16
greghaynesclarkb: Ya, I mostly am trying to make sure that -minimal doesnt ever have packages which don't exist in the cloud images23:16
*** IlyaG has quit IRC23:17
*** lucasagomes has quit IRC23:18
*** geekinutah has joined #openstack-infra23:20
*** geekinutah has quit IRC23:21
*** jamesmcarthur has joined #openstack-infra23:22
*** IlyaG has joined #openstack-infra23:22
*** mylu has quit IRC23:22
*** adrian_otto has joined #openstack-infra23:22
ianwjamielennox: ok ... 12 revisions make me think it's something more complicated but it's really just for adding quick nv tests?23:23
jamielennoxianw: there have been so many merge conflicts23:23
jamielennoxianw: the goal is to run non voting jobs with keystone v2 disabled23:23
jamielennoxagainst other projects23:23
*** sdake has joined #openstack-infra23:24
ianwcool, just makes it look more complex than it is :)23:24
*** mylu has joined #openstack-infra23:24
*** ihrachys has quit IRC23:24
*** Sukhdev has quit IRC23:24
jamielennoxianw: this review also used to contain installing it into the other services, which was going to require +1s from everyone23:25
jamielennoxnow it's split up so we can hopefuly move it forward23:25
*** lucasagomes has joined #openstack-infra23:25
*** Qiming has joined #openstack-infra23:25
*** ihrachys has joined #openstack-infra23:26
*** dims has joined #openstack-infra23:28
*** IlyaG has quit IRC23:30
*** IlyaG has joined #openstack-infra23:30
*** fawadkhaliq has quit IRC23:31
*** bpokorny_ has quit IRC23:31
*** fawadk has joined #openstack-infra23:31
*** bpokorny has joined #openstack-infra23:32
*** bpokorny has quit IRC23:32
*** dims has quit IRC23:32
*** bpokorny has joined #openstack-infra23:33
*** tphummel has quit IRC23:33
*** Sukhdev has joined #openstack-infra23:34
*** tphummel has joined #openstack-infra23:34
*** IlyaG has quit IRC23:35
*** Sukhdev has quit IRC23:36
*** jamesmcarthur has quit IRC23:36
fungisince our job volume has died down, i'm going to temporarily delete the accumulated devstack-trusty nodes. hopefully once the post pipeline backlog catches up we can find a quiet moment to restart zuul and clear the gearman registrations23:36
clarkbmnaser: looks like vexxhost image uploads may be doing that broken caching thing again23:36
*** baoli has quit IRC23:37
mnaserclarkb: openstack-infra nagios red :p23:38
pabelangerclarkb: Ya, been like that for a few hours now23:38
*** ihrachys has quit IRC23:39
*** SumitNaiksatam has quit IRC23:39
pabelangerianw: centos7 uploaded to rax-ord now23:39
*** jamesmcarthur has joined #openstack-infra23:40
anteayafungi: yay23:40
*** ihrachys has joined #openstack-infra23:40
*** alex_xu has quit IRC23:41
fungii missed the broken caching thing23:43
*** eliqiao has quit IRC23:43
clarkbfungi: where nginx caches in front of glance and clients get confused when all the bits are acceptd but no 200 is received for another 15 minutes23:44
fungioh, right, that23:44
fungitimeout between the proxy and the backend23:44
*** baoli has joined #openstack-infra23:44
*** moravec has quit IRC23:45
fungiafter it's all buffered23:45
*** eliqiao has joined #openstack-infra23:45
ianwpabelanger: ok, is it registered?23:46
*** tpsilva has quit IRC23:47
pabelangerianw: maybe?23:47
pabelangerI see with deleted state23:47
pabelangerand another building23:47
pabelangerlet me monitor23:47
pabelangerso, it got an IP23:48
dmsimardnginx buffering uploads, I've had to deal with that before .. had to settle with an nginx fork (tengine) by the alibaba guys, they have a feature to disable body buffering23:48
*** fverbs has joined #openstack-infra23:48
pabelangerianw: looks like SSH timeout23:48
clarkbdmsimard: nginx can do it too under certain circumstainces23:48
pabelanger2016-03-17 23:45:05,838 DEBUG nodepool.NodeLauncher: Node id: 8847509 is running, ipv4:, ipv6: 2001:4802:7800:1:be76:4eff:fe21:3d0f23:49
pabelanger2016-03-17 23:45:05,838 DEBUG nodepool.NodeLauncher: Node id: 8847509 testing ssh at ip:
dmsimardinteresting, things might have changed since I last looked at this23:49
pabelangerianw: ^23:49
*** baoli has quit IRC23:49
ianwpabelanger: alright, that's pretty much matching what i expected from my testing yesterday23:49
pabelangeralso, sorry for pastebomb23:49
ianwmy upload with glean changes is still churning it's way through23:49
ianwin short, i think we're taking about 2 minutes to timeout on dhcp before we bring the interface up23:50
*** alex_xu has joined #openstack-infra23:50
*** baoli has joined #openstack-infra23:52
jrollinfra-root, could someone hold ubuntu-trusty-ovh-bhs1-8847643 for me please?23:52
clarkbjroll: done23:52
jrollssh key:
jrollthank you clarkb23:53
*** davideagnello has quit IRC23:53
*** fedexo has joined #openstack-infra23:53
clarkbjroll: ssh jenkins@
jrollty sir23:53
*** davideagnello has joined #openstack-infra23:54
*** mylu has quit IRC23:54
*** markvoelker has quit IRC23:54
*** baoli has quit IRC23:54
*** angdraug has quit IRC23:55
*** baoli has joined #openstack-infra23:55
*** jamesmcarthur has quit IRC23:57
*** MarkAtwood has quit IRC23:58
*** dalgaaf has quit IRC23:58
*** davideagnello has quit IRC23:59
*** ajmiller has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!