Friday, 2016-04-08

fungii've pinged rosmaita over in #openstack-glance asking about the upstreamness of their particular task-import implementation00:00
crinklenibalizer: fungi slightly less gross now
mordrednotmyname: so, I can't really see any way to get SwiftService to upload as an SLO00:01
mordrednotmyname: should I assume I need to do the REST calls myself to do that?00:01
mordrednotmyname: (the docs make it seem not too terrible)00:01
notmynamemordred: nope. SwiftService should be able to do it (you're right it's not terrible). let me find the right docs00:02
mordrednotmyname: thanks! I appreciate the help00:02
*** bpokorny has quit IRC00:02
*** david-lyle has quit IRC00:02
notmynamemordred: as an aside, we're just about done with a major rewrite to the swiftclient docs. well, rewrite isn't right. that implies we actually had docs to start with. but the point is that they'll be better00:02
mordrednotmyname: woot!00:03
*** bpokorny has joined #openstack-infra00:03
*** david-lyle has joined #openstack-infra00:03
mordrednotmyname: use_slo00:03
mordredin options00:03
mordredseems to be the key00:03
fungicrinkle: i don't think what you had before was especially gross either, but pip list --outdated does seem to simplify the approach00:03
notmynamemordred: yup00:04
notmynamemordred: actually, I'll just link you the new, in-progress docs.
*** mylu has quit IRC00:04
notmynamemordred: FWIW, that page (service-api.html) is complete. there's other stuff in the patch set that will be fleshed out more fully before it lands00:04
openstackgerritMerged openstack/requirements: bump keystoneauth to 2.5.0
openstackgerritMerged openstack-infra/puppet-infracloud: Order of the classes parameters is refactored
*** sridhar_ram has quit IRC00:08
*** denisra_ has quit IRC00:08
*** sridhar_ram has joined #openstack-infra00:08
*** denisra_ has joined #openstack-infra00:08
*** ayoung has joined #openstack-infra00:08
fungiafter 30 minutes of waiting i finally gave up and deleted the corresponding dlo segments. pretty sure it means the rackspace dashboard swift management interface isn't doing anything fancy behind the scenes to clean up unneeded dlo segments when you delete the manifest that way either00:09
*** mylu has joined #openstack-infra00:09
*** jamesmcarthur has joined #openstack-infra00:10
*** xarses has joined #openstack-infra00:11
tristanC#success 54 PTL and 7 TC members elected for Newton00:11
openstackstatustristanC: Added success to Success page00:11
tristanCand thank you infra folks for your help on that venture :)00:12
*** sridhar_ram has quit IRC00:12
mordredtristanC: WOOT!00:13
fungitristanC: thanks for your efforts!00:13
nibalizercrinkle: neat00:13
mordredtristanC: also, wow - I think that's the biggest change in TC membership in a single election we've had since the beginning of the project00:13
fungipretty close numbers too00:14
*** mylu has quit IRC00:14
pleia2tristanC: thanks for your work00:14
fungitonyb: thanks to you also for being an election official!00:15
tristanCheh to be fair, tonyb did most of the work ;)00:15
pleia2yay, thanks tonyb too!00:15
*** kzaitsev_mb has joined #openstack-infra00:15
tonybtristanC: I was going to say the same of you ;P00:15
*** akshai_ has quit IRC00:15
pleia2mordred: it's nice to see00:15
tonybya'll are welcome00:15
*** tphummel has quit IRC00:15
fungialso thanks to nibalizer for muddling through my questionable instructions to generate the electoral rolls!00:15
pleia2I was happy in general for the number of new faces on the list00:16
mordredpleia2: ++00:16
nibalizerfungi: and thanks to the delta wifi for making it possible00:16
fungiyep, i loved that so many people ran this time00:16
funginibalizer: heh, all praise to the delta wifi00:17
*** Swami has quit IRC00:17
fungimay it bless us with its radioactive goodness always00:17
*** thorst has joined #openstack-infra00:18
*** cdent has quit IRC00:21
*** jamesmcarthur has quit IRC00:21
anteayatristanC tonyb nibalizer nice work on the election00:21
anteayacongrats to the new tc members00:22
* anteaya creates a t-shirt with UNEEDED on it for the summit00:22
fungi652 ballots cast out of 3285 sent, so 19.8% voter turn-out. any idea how that compares to previous tc elections?00:24
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Remove the job for networking-ofagent master branch
anteayagoing down00:24
openstackgerritMerged openstack/requirements: Temporarily cap fixtures<2.0
fungipercentage going down. total votes still rising?00:25
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements
anteayaI haven't done the math in a bit but I do believe we were around 30% a while back00:25
mtreinishfungi: I think there were more in the last one00:25
openstackgerritOpenStack Proposal Bot proposed openstack-dev/pbr: Updated from global requirements
mtreinishpercentage wise00:25
anteayapercentage going down00:25
anteayaall the links are here for prior elections:
fungiif so, that may mean we're accumulating new contributors who aren't voting for the tc, but not necessarily losing interest00:25
*** mylu has joined #openstack-infra00:25
fungiaha, thanks anteaya!00:25
*** salv-orlando has joined #openstack-infra00:26
mtreinishfungi: in oct 619 voted out of 2759 authorized voters00:26
fungiso the number of voters and electorate grew00:26
fungibut the electorate growth outpaced voter growth00:27
*** jamesmcarthur has joined #openstack-infra00:28
anteayawhich is the trend00:28
anteayawhich is why are are now below 20% voter turnout00:28
fungii sort of wonder if people weren't getting free conference passes, if the number of voters would continue to rise but the size of electorate would fall00:28
anteayafungi: good question00:29
fungimaybe we'll have an opportunity to find out in future cycles00:29
anteayaonce we stop giving out free conf passes I'm curious to see what the election stats show00:29
anteayaI do hope so00:29
*** LaT0rtue_ has joined #openstack-infra00:30
*** salv-orlando has quit IRC00:30
*** EricGonczer_ has quit IRC00:31
fungior if we change how we issue free passes to something else (like attendees to a prior ops summit or dev summit get free/reduced registration rate for the conferenence) or something00:31
anteayaor something00:31
*** LaT0rtue has quit IRC00:32
*** SumitNaiksatam has joined #openstack-infra00:33
openstackgerritMonty Taylor proposed openstack-infra/shade: Upload large objects as SLOs
mordredfungi: ^^00:36
mordrednotmyname: you too, in case you have an opinion00:36
*** jamesmcarthur has quit IRC00:37
tristanCvoter count between L->M: +71, M->N: +3300:37
tristanCat least that's still growing, but not as much as expected00:37
*** jamesmcarthur has joined #openstack-infra00:37
tristanCthough electorate grew through many new projects that may not be "sensibilized" to the TC mission (yet)00:39
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: [WIP] Terminate sessions before dropping DBs in PostgreSQL
openstackgerritMonty Taylor proposed openstack-infra/shade: Add option to control whether SLO or DLO is used
*** tphummel has joined #openstack-infra00:39
*** salv-orlando has joined #openstack-infra00:40
*** tphummel has quit IRC00:40
tristanCalso these 33 new voters represents 5.33% of the voter count00:41
*** moravec has quit IRC00:42
*** thorst has quit IRC00:42
*** jamesmcarthur has quit IRC00:42
*** moravec has joined #openstack-infra00:42
*** jamesmcarthur has joined #openstack-infra00:43
*** thorst has joined #openstack-infra00:43
tristanCimo, what matters is that more and more people vote each cycles00:43
*** salv-orlando has quit IRC00:44
*** mylu has quit IRC00:46
*** tphummel has joined #openstack-infra00:47
*** ddieterly[away] has quit IRC00:48
*** fitoduarte has joined #openstack-infra00:48
openstackgerritColleen Murphy proposed openstack-infra/puppet-pip: Add new pip provider to use pypi mirrors
*** thorst has quit IRC00:51
*** mylu has joined #openstack-infra00:51
fungimordred: clarkb: so apparently it's not necessary to keep the files we upload to swift after the import task succeeds and the image is active00:52
clarkbfungi: yay, we should have shade delete them then00:52
fungimordred: clarkb: if we want, we can retry any failed imports without reuploading (as long as the upload completed), and if we're doing that then should wait to delete from swift until that's done, obviously00:52
fungialso if we wanted to use those uploads for any other glance tasks, then don't delete them before doing so00:53
*** hichihara has joined #openstack-infra00:54
fungithe next question is whether shade will recreate those containers for us when they don't exist00:54
mordredyes it will00:54
*** rfolco has joined #openstack-infra00:54
fungiif so, we can just delete the containers while no image builds are underway. blank slate00:54
mordredand we can certainly attempt to delete the object once we have a successful import00:54
mordredfungi: OR - we could maybe add in a option to delete_image to try to clean up objects00:55
mordredfungi: that way we delete them whenever nodepool thinks we should00:55
*** moravec has quit IRC00:55
mordredwhich would also account for images that totally fail that we give up on but that uploaded swift objects00:55
*** Kiall has quit IRC00:55
fungiright, we should avoid leaking images on failure for sure00:55
*** amotoki has quit IRC00:56
*** bpokorny has quit IRC00:56
fungithe delete and recreate container solution is mostly attractive as a one-time cleanup to deal with the many thousands of files and several terabytes we have right now in nested directory trees00:56
*** jamesmcarthur has quit IRC00:57
*** Kiall has joined #openstack-infra00:57
openstackgerritMerged openstack-infra/puppet-simpleproxy: Indentation of the class parameter is refactored
openstackgerritMerged openstack-infra/puppet-statusbot: Order of the class parameters is refactored
*** sdake has joined #openstack-infra01:02
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Remove the job for networking-ofagent master branch
*** LaT0rtue_ has quit IRC01:03
*** rossella_s has quit IRC01:03
*** rossella_s has joined #openstack-infra01:03
*** tiswanso has joined #openstack-infra01:08
*** ddieterly has joined #openstack-infra01:09
*** tiswanso has quit IRC01:10
*** tiswanso has joined #openstack-infra01:11
*** sdake has quit IRC01:14
mgagnemordred: what's your take on Glance image properties namespaceing?
*** apoorvad has quit IRC01:17
mgagnemordred: I see shade is using the owner_specified_ prefix. Any reason?01:17
mordredmgagne: I will now read your link01:17
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete uploaded swift objects on image delete
mordredmgagne: last time I looked at properties it was a recommended prefix to put in front of user uploaded properties01:18
mgagnemordred: lets say some clouds enforce this recommendation with policies. is that a good thing? I never figured what's the REAL recommendation/law that should be used/applied.01:18
*** sdake has joined #openstack-infra01:19
mordredmgagne: yeah - I'm not sure ... most clouds I've found do not let untrusted users mark their images as public01:19
mgagnemordred: of course =)01:19
mordredso I have no experience with the spoofing problem the wiki is trying to prevent01:20
mgagneI also don't want users to create legacy I would have to deal with in the future01:20
mordredfungi: ^^ see patch above01:20
mordredmgagne: just don't prevent me from using owner_specified. and we'll remain friends :)01:20
fungimordred: ooh, i see a patch01:20
fungidelete all the things!01:20
* mgagne proceeds to protect owner_specified01:21
mordredmgagne: :)01:21
*** aeng has quit IRC01:21
mordredmgagne: of course, the problem with protecting things like this is the same with all other policy things ... it's not discoverable by the user01:21
mordredso finding out what prefix a cloud operator has protected would be hard01:21
*** amotoki has joined #openstack-infra01:22
mordredand if people started being 'clever' I'd have to add another field to clouds.yaml for vendors01:22
*** sdake_ has joined #openstack-infra01:22
*** mylu has quit IRC01:23
*** zhurong has joined #openstack-infra01:24
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Use yaml2ical from git rather than pypi
mgagnealright, thanks for the opinion =)01:25
*** sdake has quit IRC01:25
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: [WIP] Terminate sessions before dropping DBs in PostgreSQL
*** davidlenwell has quit IRC01:28
*** amotoki has quit IRC01:29
tonybanteaya: Please take a quick look at and let me know if you have any concerns01:30
*** masahito has joined #openstack-infra01:30
*** calebb has quit IRC01:31
*** mriedem has quit IRC01:31
*** davidlenwell has joined #openstack-infra01:32
*** baoli has joined #openstack-infra01:32
masahitohi, I have a question about automatic merge tag commit.01:33
clarkbmasahito: ask away01:33
masahitoGerrit says there is a merge conflict.01:34
clarkbwhere do you see the merge conflict?01:34
clarkb(I don't see any conflict on that review)01:35
*** calebb has joined #openstack-infra01:35
masahitofor instance, I saw merge conflict label in glancev2_driver.py01:35
clarkbah right01:36
clarkbthe way gerrit shows you merge commits is to put the conflicts in the diff01:36
clarkbon the left hand side you get full conflict listing and on the right hand side you get the resolution01:36
clarkbin this case its just what was in master01:36
*** mylu has joined #openstack-infra01:37
clarkbmaster is HEAD and stable mitaka is the BRANCH01:37
clarkbso in those diffs you can see that master was preserved01:38
*** baoli has quit IRC01:38
*** edmondsw has quit IRC01:38
masahitoshould we manually resolve it to fix the conflict?01:38
clarkbno, there is no conflict01:38
clarkbthey have already been resolved, which is what the diff there is showing you01:38
clarkbon the left hand side is the conficts, the right hand side is what is prposed as the fixes. In this special case just preserving master, no code from mitaka is pulled in01:39
*** sridhar_ram has joined #openstack-infra01:39
clarkbthis change is necessary so that master knows mitaka is in its history01:39
masahitoit means gerrit just show us the difference between Master and BRANCH?01:39
mordredyes. but it will not actually change the code in master when you merge it01:39
clarkbit is showing you the conflicts between them01:39
clarkbif you fetch the change and look at it locally there are not conflict tags in the code01:40
*** kzaitsev_mb has quit IRC01:40
masahitook, thanks.01:40
clarkbwhat this does is stitch history together so that master knows mitaka came before it, but no code in master is changed01:41
clarkbits purely git DAG management01:41
masahitoI was worried about there is some conflicts :)01:41
*** denisra_ has quit IRC01:43
*** sridhar_ram has quit IRC01:43
*** denisra_ has joined #openstack-infra01:43
*** moravec has joined #openstack-infra01:44
*** gildub has quit IRC01:45
*** baoli has joined #openstack-infra01:47
*** thorst has joined #openstack-infra01:49
*** zhurong has quit IRC01:49
*** zhurong has joined #openstack-infra01:50
*** BillCr5 has quit IRC01:52
*** baoli has quit IRC01:52
*** nwkarsten has joined #openstack-infra01:53
*** thorst has quit IRC01:56
*** nwkarsten has quit IRC01:57
*** denisra_ has quit IRC02:00
*** amrith is now known as _amrith_02:00
*** denisra_ has joined #openstack-infra02:00
*** baoli has joined #openstack-infra02:00
clarkbjenkins01, 02, 04, and 07 all appear to have gone off the deep end of thread leaking.02:02
clarkbI can't address that now, its dinner then sleep time hopefully but post as fyi here and maybe someone else can02:02
*** moravec has quit IRC02:03
clarkbits possible this is related to the nodepool restart and lack of cleaning up jenkins slaves02:03
clarkbI was hoping the weekly restarts would clean that out for us but it may not be soon enough or effective02:03
*** shashank_hegde has quit IRC02:03
clarkbjenkins07 has 14,205 threads02:04
clarkband no there isn't an extra digit in there like some price is right game02:04
*** mylu has quit IRC02:04
*** yamahata has quit IRC02:04
*** lyz has quit IRC02:06
*** jcooley_ has joined #openstack-infra02:07
*** lyz has joined #openstack-infra02:07
*** baoli has quit IRC02:07
anteayatonyb: sorry was trying out my new mop, mmmmmm, clean floor, looking now02:07
tonybanteaya: that's the first time I've heard that as an excuse ;P02:08
*** baoli has joined #openstack-infra02:08
*** shu-mutou has joined #openstack-infra02:08
anteayaha ha ha02:09
openstackgerritEli Qiao proposed openstack-infra/project-config: Magnum: Skip tesing not related COE changes
anteayaI don't think I've ever seen this bit before: @master#egg=yaml2ical02:09
anteayadoes whatever parses test-requirements understand what that means?02:09
anteayamy browser doesn't apparently02:10
*** mylu has joined #openstack-infra02:10
*** Qiming has quit IRC02:13
*** jcooley_ has quit IRC02:13
*** davideag_ has joined #openstack-infra02:13
*** mtanino__ has joined #openstack-infra02:15
*** darvon has quit IRC02:15
*** Sukhdev_ has quit IRC02:15
*** davideagnello has quit IRC02:15
*** jtomasek has quit IRC02:15
*** mtanino has quit IRC02:15
*** nihilifer has quit IRC02:16
*** nwkarsten has joined #openstack-infra02:16
*** cburgess has quit IRC02:16
*** baoli has quit IRC02:16
*** rlandy has quit IRC02:16
*** darvon has joined #openstack-infra02:16
*** gildub has joined #openstack-infra02:16
*** rlandy has joined #openstack-infra02:17
*** fitoduarte has quit IRC02:17
*** nihilifer has joined #openstack-infra02:17
*** baoli has joined #openstack-infra02:17
*** amotoki has joined #openstack-infra02:17
openstackgerritMerged openstack-dev/pbr: Updated from global requirements
*** cburgess has joined #openstack-infra02:19
*** _ari_ has joined #openstack-infra02:19
tonybanteaya: Yes pip understands that02:19
*** _ari_ has quit IRC02:19
*** openstackstatus has quit IRC02:19
*** kengo_sakai has joined #openstack-infra02:19
*** fitoduarte has joined #openstack-infra02:20
tonybIt seems the 'Project Bootstrappers' either doesn't exist or is a group I can't see ... is that the way it's supposed to be?02:20
*** masahito has left #openstack-infra02:25
cloudnullcongrats on being eleceted to the TC thingee ttx02:25
*** gongysh_ has joined #openstack-infra02:25
anteayatonyb: project bootstrappers has no members at the moment02:26
fungitonyb: it's probably marked as hidden to avoid confusing people, but it may confuse more people by hiding it and having it appear in acls02:27
tonybanteaya: okay thanks.02:27
*** woodster_ has quit IRC02:27
anteayait is a group that gerrit admins can add themselves do perform certain actions02:27
*** baoli has quit IRC02:27
fungithere is actually one member, a bot account used for project imports02:27
anteayatonyb: then they remove themselves02:28
anteayaah sorry, yes the bot02:28
fungii've gone ahead and made it visible02:28
anteayatonyb: then if pip understands that then I'm good with that patch02:28
fungino harm in exposing it02:28
anteayafungi: yay!02:29
anteayatonyb: one typo in the commit message if you care02:29
tonybfungi: Thanks :)02:29
tonybanteaya: I'll fix it02:29
anteayatonyb: welcome :)02:30
*** dimtruck is now known as zz_dimtruck02:30
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Use yaml2ical from git rather than pypi
tonybfungi, anteaya: the reason I noticed is I was tryign to work out who could approve a change I was looking at02:32
tonybfungi, anteaya: turns out it's very few people.  So now I need to decide if I do ;P02:32
*** adduarte has joined #openstack-infra02:33
*** yamamoto has joined #openstack-infra02:33
*** sdake_ is now known as sdake02:33
anteayawho can approve it?02:36
*** fitoduarte has quit IRC02:36
*** gongysh_ has quit IRC02:37
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: [WIP] Terminate sessions before dropping DBs in PostgreSQL
*** yuanying has quit IRC02:46
*** FallenPegasus has joined #openstack-infra02:47
*** binbincong has quit IRC02:50
*** thorst has joined #openstack-infra02:54
*** ramishra has joined #openstack-infra02:57
*** esikachev has joined #openstack-infra02:58
*** ddieterly is now known as ddieterly[away]03:00
*** thorst has quit IRC03:01
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Parameterize libvirt virt type
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Reconcile compute and controller classes
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests
*** rfolco has quit IRC03:03
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete uploaded swift objects on image delete
openstackgerritMonty Taylor proposed openstack-infra/shade: Add option to control whether SLO or DLO is used
openstackgerritMonty Taylor proposed openstack-infra/shade: Upload large objects as SLOs
*** esikachev has quit IRC03:03
*** Goneri has joined #openstack-infra03:04
*** Qiming has joined #openstack-infra03:07
*** kengo_sa_ has joined #openstack-infra03:09
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Update parameters for puppet-apply test dance
*** aeng has joined #openstack-infra03:10
*** changzhi has joined #openstack-infra03:10
clarkbnibalizer: by any chance are you able to work on fixing the puppet apply test?03:10
nibalizerfor better hiera usage? no03:11
clarkbwell so that we don't overwrite the default hiera dile03:11
clarkbshould we revert that file so we don't create confusion?03:12
clarkbor maybe thats a workroom thing in austin related to my fix ansible item03:13
* clarkb adds03:13
*** kengo_sakai has quit IRC03:13
nibalizerclarkb: what's the issue?03:13
nibalizerI don't remember a problem related to the hiera.yaml if that's what you mean03:13
clarkbthe problem is that puppte apply test does not use the data in the in repo hiera file(s). This means that you cannot make changes that use in repo hiera file if those entries are validated in puppet03:14
*** knikolla has joined #openstack-infra03:14
clarkban example of this is gpg public key data which gets verified by the gpg resources03:14
*** wei-wang has joined #openstack-infra03:14
clarkbthe difficult thing in fixing this is that we have all these hiera file groupings and the tests will need ot make sure there is no cross contamination between them to prevent tests from passing when they actually miss keys03:15
nibalizerah right03:15
*** shashank_hegde has joined #openstack-infra03:16
mordredclarkb: I look forward to the fix ansible workroom03:16
clarkbin other news fantasia is avialable on netflix and is playing on my tv03:18
nibalizerclarkb: so what about just annotating the ones we expect to work with a comment?03:19
clarkbnibalizer: I expect all of them to work :)03:19
clarkbits something we missed when putting in repo hiera data we juts need to address it. Unfortunately my bash foo was too weak when I attempted, probably need to consider rewriting that all in python03:19
clarkbits not too bad when you have actual table data structures, but bash arrays are sadness03:20
*** mylu has quit IRC03:20
mordredI would like to seriously discuss our relationship with hiera and ansible variables and also with ansible/puppet03:20
mordredbecause I worry about franensteining03:21
clarkbmordred: right now its not too frankensteined, you make hiera files that end up on esrvers and are used via normal hieraing03:21
*** yamamoto has quit IRC03:21
clarkbthe issue with the tests is that we remove all hiera lookups and replace them with dummy data03:22
clarkbmaking the in repo hiera data useless during testing and leading to hard to debug test fails03:22
nibalizerclarkb: i only expect the ones with public data keys to work, i expect all the private ones to not work03:22
nibalizerclarkb: it does look up the in tree data, for instance the hiera_array() ones all work03:22
mordredjust saying I think a short chat potentially at the summit where we think about the whole thing holistically might be nice03:22
clarkbnibalizer: it does not03:22
clarkbnibalizer: the job does variable replacement for those iirc03:23
clarkbwell for everything03:23
clarkbits s/hiera*\(key,(default)\)/\1/ type deal03:24
openstackgerritMonty Taylor proposed openstack-infra/shade: Rely on skip_identical to skip identical objects
clarkbmordred: can has shade patch to delete the glance image on uploda fail? I supposed I can poke at that later03:26
nibalizerclarkb: it doesn't match hiera_array03:26
clarkboh so we implicitly require all hiera_array lookups to be in public hiera?03:27
clarkbthat too seems not ideal03:27
clarkband will cause hard to debug fialures03:27
mordredclarkb: yah - I was taking care of the swift uploads first03:27
nibalizerclarkb: the only time it's failed for me was a good failure because I broke the plumbing that causes hiera lookups from public data to work in the apply test03:28
nibalizerI think that's what happened anyways03:28
Clintmmm... good failure03:29
clarkbnibalizer: ya thats fine, what isn't is trying to use that with private hiera03:29
clarkbit will fail and you won't know why03:29
*** rlandy has quit IRC03:29
nibalizereh? you can hiera_array() on a key that is in private hiera03:30
nibalizerwe just don't03:30
nibalizeroh but that would fail the test03:30
openstackgerritTony Breeds proposed openstack/requirements: Blacklist Routes 2.3.0
jesusaurnibalizer: a while back, didn't you mention a hiera backend for random data? could something like that be used for testing instead of replacing hiera lookups in the tests?03:33
clarkbjesusaur: the issue is a lot of the data needs to have specific content to pass validation03:34
*** netsin has quit IRC03:34
clarkbor at least that was the issue I had, ended up not using hiera just to avoid the problem03:34
*** Sukhdev has joined #openstack-infra03:34
*** shashank_hegde has quit IRC03:34
*** shashank_hegde has joined #openstack-infra03:35
*** netsin has joined #openstack-infra03:35
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete image objects after failed upload
mordredclarkb: I have did make you present03:35
*** Goneri has quit IRC03:37
mordredianw: btw - thank you for writing
*** mylu has joined #openstack-infra03:40
*** binbincong has joined #openstack-infra03:41
mordredclarkb: oh - also, all of the functional shade tests are going to fail until lands03:42
mordredclarkb: so if you see that, it's not real failures03:43
clarkblooks like its on its way though03:43
clarkbmordred: a more defensive thing would be to have your test job define its own shade flavors03:43
clarkbthen they won't change branch to branch03:43
mordredclarkb: yah03:44
mordredclarkb: although I'd like to move it the other direction since I want to be able to run the shade functional tests against all of the live cloud accounts I have03:44
mordredit can't do that now03:44
mordredbut I think it should be possible03:45
mordredthere are a few places where we look for devstack things rather than looking for cloud resources with a wider net - image names is another good example03:46
*** Sukhdev has quit IRC03:47
*** yuanying has joined #openstack-infra03:49
mordredclarkb: btw - thnaks for the swift/glance tasks in shade - I'm currently avoiding debugging the network metadata patches because they're hard03:50
*** knikolla has quit IRC03:50
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: [WIP] Terminate sessions before dropping DBs in PostgreSQL
openstackgerritxiangxinyong proposed openstack/requirements: Add python-smaugclient into requirements
clarkbmordred I wouldve written them but need a bteak after 3 days of dib fun03:51
clarkbwell the imsge delete on fail one03:51
mordredthanks for dealing with the dib fun - sorry I was as afk as I was :(03:51
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Remove the job for networking-ofagent master branch
mordredI promise I wasn't having MORE fun03:52
*** links has joined #openstack-infra03:53
openstackgerritSpencer Krum proposed openstack-infra/system-config: Use the garbage backend in the apply test
*** knikolla has joined #openstack-infra03:54
*** ramishra has quit IRC03:55
*** knikolla has quit IRC03:56
mordredpatch bomb03:56
openstackgerritMonty Taylor proposed openstack-infra/shade: Don't use singleton dicts unwittingly
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete uploaded swift objects on image delete
openstackgerritMonty Taylor proposed openstack-infra/shade: Rely on skip_identical to skip identical objects
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete image objects after failed upload
openstackgerritMonty Taylor proposed openstack-infra/shade: Add option to control whether SLO or DLO is used
openstackgerritMonty Taylor proposed openstack-infra/shade: Upload large objects as SLOs
mordred(rebased on the functional test fix)03:57
*** ramishra has joined #openstack-infra03:57
*** tiswanso has quit IRC03:58
*** adduarte has quit IRC03:59
*** adduarte has joined #openstack-infra03:59
*** thorst has joined #openstack-infra03:59
*** Jeffrey4l_ has joined #openstack-infra04:00
*** ddieterly[away] has quit IRC04:00
*** Nikolay_St has quit IRC04:02
*** yamamoto_ has joined #openstack-infra04:03
*** thorst has quit IRC04:06
openstackgerritSpencer Krum proposed openstack-infra/system-config: Use the garbage backend in the apply test
*** shashank_hegde has quit IRC04:08
*** harlowja_at_home has joined #openstack-infra04:13
*** Sukhdev has joined #openstack-infra04:16
*** keedya has quit IRC04:16
*** sdake_ has joined #openstack-infra04:19
*** sdake has quit IRC04:19
openstackgerritRodrigo Duarte proposed openstack-infra/project-config: Enable non-voting keystone tempest plugin tests
*** FallenPegasus has quit IRC04:32
*** mylu has quit IRC04:33
*** mylu has joined #openstack-infra04:37
*** harlowja_at_home has quit IRC04:39
*** nwkarsten has quit IRC04:42
*** nwkarsten has joined #openstack-infra04:43
*** nwkarsten has quit IRC04:47
*** mtanino__ has quit IRC04:48
*** shashank_hegde has joined #openstack-infra04:50
*** wei-wang has quit IRC04:52
*** salv-orlando has joined #openstack-infra04:52
openstackgerritJamie Lennox proposed openstack-infra/project-config: Change service user domain for identity v3 only test
*** rcernin has joined #openstack-infra04:55
*** alex_xu has quit IRC04:56
*** shu-mutou is now known as shu-mutou-AFK04:57
*** fedexo has joined #openstack-infra04:57
*** LaT0rtue has joined #openstack-infra04:58
*** esikachev has joined #openstack-infra04:59
*** stevebaker has joined #openstack-infra05:00
*** alex_xu has joined #openstack-infra05:01
*** rossella_s has quit IRC05:03
*** sdake has joined #openstack-infra05:03
*** rossella_s has joined #openstack-infra05:03
*** thorst has joined #openstack-infra05:04
*** esikachev has quit IRC05:04
*** salv-orlando has quit IRC05:04
*** mylu has quit IRC05:06
openstackgerritMasayuki Igawa proposed openstack/os-testr: Decode bytes to str for Python 3
*** sdake_ has quit IRC05:06
*** imcsk8 has quit IRC05:07
*** imcsk8 has joined #openstack-infra05:07
*** links has quit IRC05:07
*** mylu has joined #openstack-infra05:10
*** hichihara has quit IRC05:11
*** thorst has quit IRC05:11
*** sdake_ has joined #openstack-infra05:13
*** sdake has quit IRC05:17
yamamoto_why do we have coverage jobs usually in post, rather than eg. check?05:23
*** links has joined #openstack-infra05:24
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests
tobiashjhesketh: wow, that was fast, the test looks good to me :)05:26
jheskethcool, hopefully I'll write the fix this evening05:26
tobiashI did a fix for our local deployment (which may break foreign projects which we don't use)05:27
jheskethtobiash: did it break the unit tests?05:28
tobiashdidn't try yet, just a sec05:29
*** hdd has joined #openstack-infra05:29
*** markvoelker has quit IRC05:30
tobiashjust reverted that line in getChangeQueue05:31
tobiash2 tests fail05:31
clarkbyamamoto_: because the coverage tool makes the tests run longer05:31
clarkbyamamoto_: and for a long time the neutron unittests took longer than tempest05:31
clarkbyamamoto_: you can run coverage upfront if you wish to, some prpjects do this05:32
*** mylu has quit IRC05:32
*** salv-orlando has joined #openstack-infra05:32
*** infra-red has joined #openstack-infra05:33
yamamoto_clarkb: thank you.  my motivation is to increase visibility of coverage job results.  is there a convenient way to find a url for post job results?05:34
clarkbyamamoto_: rbradfor had an index somehwere you can also go to for any change iirc05:36
*** tphummel has quit IRC05:37
yamamoto_clarkb: yea, it's what i'm doing but i'm not sure what the last part of url is05:40
yamamoto_eg. e34e66f of
*** cschwede has joined #openstack-infra05:40
tobiashjhesketh: now I have a fix which doesn't break the existing tests and also doesn't break your new test05:48
jheskethtobiash: you mean fixes my current test?05:48
tobiashjhesketh: yupp05:48
jhesketh(as it currently - and should - fail)05:48
jheskethtobiash: cool, nice stuff :-)05:48
tobiashjhesketh: but I first have to renew my permission to contribute before I'm allowed to push this into review05:50
*** Jeffrey4l_ has quit IRC05:50
tobiashthis will take probably 1-2 weeks05:50
tobiashbut if you want to do the fix on your own, I just added a new optional argument to getChangeQueue for preventing dynamic creation of the queue which is used then in addChange05:51
jheskethokay I'll see how I go later on05:52
*** Sukhdev has quit IRC05:54
*** LaT0rtue has quit IRC05:54
*** shashank_hegde has quit IRC05:55
*** sdake has joined #openstack-infra05:55
*** LaT0rtue has joined #openstack-infra05:57
*** sdake_ has quit IRC05:57
clarkbyamamoto_: thats a uuid prefix for the job so that you can rerun tests against the same commit multiple times05:58
yamamoto_clarkb: thank you.  how can i know which run is which uuid?05:59
*** mikelk has joined #openstack-infra06:00
*** aeng has quit IRC06:01
*** shashank_hegde has joined #openstack-infra06:02
clarkbnot sure I understand the question, each uuid is unique06:02
*** Jeffrey4l_ has joined #openstack-infra06:03
*** links has quit IRC06:03
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add a best-effort sudo safety check
*** changzhi has quit IRC06:04
yamamoto_my question is, if you have two runs, how can you say which uuid is the first one?  you need to look at the logs under the uuid directory?06:05
clarkblook at the timestamps06:05
ianwmordred / clarkb : ^ it's something ... still thinking about how to better handle dib's root requirements06:06
*** amotoki has quit IRC06:06
*** infra-red has quit IRC06:07
yamamoto_clarkb: got it, thank you06:08
*** thorst has joined #openstack-infra06:09
*** FallenPegasus has joined #openstack-infra06:09
*** sdake_ has joined #openstack-infra06:09
*** adduarte has quit IRC06:10
*** fitoduarte has joined #openstack-infra06:10
*** sdake has quit IRC06:11
*** fedexo has quit IRC06:14
*** Nikolay_St has joined #openstack-infra06:16
*** thorst has quit IRC06:16
yolandaAJaeger, apologies, I finally was all day out yesterday06:17
*** links has joined #openstack-infra06:19
*** FallenPegasus has quit IRC06:21
*** Jeffrey4l__ has joined #openstack-infra06:21
*** Jeffrey4l__ is now known as Jeffrey4l06:25
*** Jeffrey4l_ has quit IRC06:25
*** scheuran has joined #openstack-infra06:28
*** amotoki has joined #openstack-infra06:29
*** markvoelker has joined #openstack-infra06:30
*** vgridnev has joined #openstack-infra06:32
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Move coverage job from post to check
*** yaume has joined #openstack-infra06:32
*** LaT0rtue has quit IRC06:33
*** amotoki has quit IRC06:33
*** amotoki has joined #openstack-infra06:33
*** markvoelker has quit IRC06:36
*** Guest71611 has quit IRC06:37
*** nadya has joined #openstack-infra06:47
*** moravec has joined #openstack-infra06:47
*** shashank_hegde has quit IRC06:49
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** shashank_hegde has joined #openstack-infra06:49
*** FallenPegasus has joined #openstack-infra06:50
*** moravec has quit IRC06:51
*** amotoki has quit IRC06:52
*** infra-red has joined #openstack-infra06:53
*** infra-red has quit IRC06:53
*** salv-orlando has quit IRC06:54
*** abregman has joined #openstack-infra06:54
*** akaszuba has joined #openstack-infra06:54
*** infra-red has joined #openstack-infra06:57
*** yuanying has quit IRC06:57
*** e0ne has joined #openstack-infra07:01
*** shashank_hegde has quit IRC07:02
*** MarkAtwood has quit IRC07:02
AJaegermorning, yolanda. No worries...07:03
*** FallenPegasus has quit IRC07:03
*** jtomasek_ has joined #openstack-infra07:03
*** abregman has quit IRC07:03
*** jaosorior has joined #openstack-infra07:04
*** jtomasek_ has quit IRC07:04
*** infra-red has quit IRC07:04
*** amotoki has joined #openstack-infra07:05
*** jed56 has joined #openstack-infra07:06
*** infra-red has joined #openstack-infra07:07
openstackgerritDong Ma proposed openstack-infra/subunit2sql: Add more db api tests part 2
*** vgridnev has quit IRC07:09
openstackgerritMerged openstack-infra/project-config: Do not cat npm-shrinkwrap.json
*** arxcruz has joined #openstack-infra07:10
*** dtantsur|afk is now known as dtantsur07:12
*** shardy has joined #openstack-infra07:13
*** vgridnev has joined #openstack-infra07:14
yolandaAJaeger, are we ok to land ?07:14
yolandai can babysit today07:14
*** thorst has joined #openstack-infra07:14
openstackgerritMerged openstack-infra/project-config: Magnum: Skip tesing not related COE changes
*** esikachev has joined #openstack-infra07:15
*** zeih has joined #openstack-infra07:17
openstackgerritMerged openstack-infra/subunit2sql: Add more db api tests
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Add more db api tests part 2
AJaegeryolanda: yes, 303253 should go in if possible.07:20
openstackgerritMerged openstack-infra/project-config: Make gate-neutron-dsvm-api job non-voting
*** ramishra has quit IRC07:20
yolandaok i will finish with the review queue, then land it07:21
*** thorst has quit IRC07:22
*** placko_ has joined #openstack-infra07:23
AJaegeryolanda: read in backscoll around 04:02 clarkb's comments on jenkins...07:26
*** ramishra has joined #openstack-infra07:26
openstackgerritMerged openstack-infra/project-config: Add openstack/monasca-transform project
*** flepied has joined #openstack-infra07:26
*** flongo has joined #openstack-infra07:27
*** jlanoux has joined #openstack-infra07:27
openstackgerritMerged openstack-infra/project-config: [sahara] setup additional jobs for sahara-dashboard
AJaegeryolanda: clarkb was suggesting to restart some jenkins there - do you want to do that as part of the manual JJB run?07:29
*** mrmartin has joined #openstack-infra07:29
*** camunoz has quit IRC07:32
*** alex_xu_ has joined #openstack-infra07:32
yolandaok let me read it and i can do it07:32
*** vincentll has joined #openstack-infra07:33
*** alex_xu has quit IRC07:34
openstackgerritMerged openstack-infra/project-config: Add experimental bindep JJB for kolla
*** jtomasek has joined #openstack-infra07:38
*** ifarkas has joined #openstack-infra07:38
*** kushal has joined #openstack-infra07:41
*** gildub has quit IRC07:42
openstackgerritJamie Lennox proposed openstack/requirements: Allow maintaining extras from project
*** flepied has quit IRC07:46
*** jaosorior has quit IRC07:47
*** kushal has quit IRC07:47
*** jaosorior has joined #openstack-infra07:47
*** pcaruana has joined #openstack-infra07:49
*** lezbar has quit IRC07:49
*** bauzas is now known as bauwser07:52
*** Hal has joined #openstack-infra07:52
*** Hal is now known as Guest487207:53
*** jamielennox is now known as jamielennox|away07:53
*** lezbar has joined #openstack-infra07:53
openstackgerritMerged openstack-infra/project-config: zuul: switch system-config to modern puppet-syntax jobs
*** dizquierdo has joined #openstack-infra07:56
*** fitoduarte has quit IRC07:59
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] remove icehouse branch from regexes
*** fitoduarte has joined #openstack-infra07:59
*** igormarnat has quit IRC08:00
*** igormarnat has joined #openstack-infra08:01
*** rakhmerov has quit IRC08:01
*** rakhmerov has joined #openstack-infra08:01
*** chem has joined #openstack-infra08:02
*** zeih has quit IRC08:03
AJaegeryolanda: what do you think about ?08:04
*** tojuvone_ has joined #openstack-infra08:05
yolandalooks good, i'll approve08:05
AJaegerand one more review, please:
*** bartash has quit IRC08:06
*** tojuvone has quit IRC08:06
*** kushal has joined #openstack-infra08:07
openstackgerritMerged openstack-infra/shade: Update func tests for latest devstack flavors
*** chem has quit IRC08:08
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: clean up sahara-extra release jobs
AJaegerargh, runs into a timeout ;(08:10
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add new jobs to openstack/designate-tempest-plugin
*** jordanP has joined #openstack-infra08:12
*** fitoduarte has quit IRC08:12
*** fitoduarte has joined #openstack-infra08:13
* AJaeger kicked the job out and requeued it in ;)08:14
openstackgerritMerged openstack-infra/shade: Don't use singleton dicts unwittingly
*** kushal has quit IRC08:17
*** dizquierdo has quit IRC08:18
*** flepied has joined #openstack-infra08:18
*** thorst has joined #openstack-infra08:19
openstackgerritMerged openstack-infra/project-config: Migrate openstackid jobs to ubuntu-trusty
*** hashar has joined #openstack-infra08:21
*** nadya has quit IRC08:23
openstackgerritMerged openstack-infra/project-config: [i18n] update reno i18n filenames for consistency
openstackgerritMerged openstack-infra/project-config: Enable release notes translation for python projects
openstackgerritMerged openstack-infra/project-config: Enable src and non-src tinyipa jobs against Ironic
openstackgerritMerged openstack-infra/project-config: Make sure we run bindep for all puppet-jobs
openstackgerritMerged openstack-infra/project-config: upload both sdist and wheel on pre-release
openstackgerritMerged openstack-infra/project-config: Add python-appcatalogclient repository
*** thorst has quit IRC08:26
*** jistr has joined #openstack-infra08:28
*** derekh has joined #openstack-infra08:28
openstackgerritVincent Fran├žoise proposed openstack-infra/project-config: Mutlinode fixup - let Watcher plugin install CLI
*** dtantsur is now known as dtantsur|brb08:30
*** kushal has joined #openstack-infra08:31
jtomasekkrotscheck: jobs passed with your patch merged. thanks!08:31
*** markvoelker has joined #openstack-infra08:31
*** e0ne has quit IRC08:32
*** openstackgerrit has quit IRC08:33
*** openstackgerrit has joined #openstack-infra08:33
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Refactor, moved mysql part out of init.pp
*** markvoelker has quit IRC08:36
*** kzaitsev_mb has joined #openstack-infra08:38
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Test a mongo puppet module pin
*** e0ne has joined #openstack-infra08:40
openstackgerritMerged openstack-infra/project-config: Add new jobs to openstack/designate-tempest-plugin
*** kushal has quit IRC08:41
AJaegeryolanda: FYI, everything in project-config merged now...08:41
*** ihrachys has joined #openstack-infra08:42
*** fitoduarte has quit IRC08:42
yolandayep, i started to monitor jjb08:42
*** vgridnev has quit IRC08:42
yolandano problem08:45
*** dalgaaf has joined #openstack-infra08:46
*** ramishra has quit IRC08:49
*** _amrith_ is now known as amrith08:51
*** fitoduarte has joined #openstack-infra08:56
openstackgerritJulien Danjou proposed openstack/requirements: Add pifpaf to requirements
*** kushal has joined #openstack-infra09:00
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-openstackci: Include nodepool::mysql from openstackci
*** kzaitsev_mb has quit IRC09:03
*** harlowja has quit IRC09:06
*** dizquierdo has joined #openstack-infra09:06
*** mrmartin has quit IRC09:09
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-openstackci: Include nodepool::mysql from openstackci
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Update regex for tempest-dsvm-ironic-inspector job
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-openstackci: Include nodepool::mysql from openstackci
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-openstackci: Include nodepool::mysql from openstackci
*** moravec has joined #openstack-infra09:14
*** dizquierdo has quit IRC09:16
*** sdake_ has quit IRC09:17
*** salv-orlando has joined #openstack-infra09:18
*** moravec has quit IRC09:18
*** tojuvone_ is now known as tojuvone09:19
*** dtantsur|brb is now known as dtantsur09:21
rcarrillocruzyolanda, jhesketh : heya, mind reviewing ?09:21
rcarrillocruzit's been around for a while, and i'd like to land it to resume work09:21
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-openstackci: Include nodepool::mysql from openstackci
jheskethrcarrillocruz: I'll add it to my queue09:22
*** cdent has joined #openstack-infra09:23
*** zhurong has quit IRC09:23
*** pkoniszewski has quit IRC09:23
*** thorst has joined #openstack-infra09:24
yolandarcarrillocruz, ugly red whitespaces, are removing those?09:25
*** e0ne has quit IRC09:25
*** vgridnev has joined #openstack-infra09:25
*** flongo has quit IRC09:27
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/infra-specs: Add extension to openstackci for next phase of work
*** yamamoto_ has quit IRC09:29
*** thorst has quit IRC09:32
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Moved mysql part out of init.pp
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Moved mysql part out of init.pp
electricalHiya all09:35
yolandai need to leave for a couple of hours, bbl09:36
*** ramishra has joined #openstack-infra09:37
AJaegeryolanda: do you have any status on JJB runs and Jenkins restart?09:42
yolandajjb seems to have run without errors, although i have not checked the outcome. About jenkins restart, i put jenkins01 on restart mode, but still has not finished09:43
*** arxcruz has quit IRC09:44
*** arxcruz has joined #openstack-infra09:44
*** sdague has joined #openstack-infra09:45
openstackgerritvikram.choudhary proposed openstack-infra/project-config: Adding neutron-dynamic-routing as neutron's stadium project.
*** yuanying has joined #openstack-infra09:50
*** amotoki has quit IRC09:50
*** BobBall_AWOL is now known as BobBall09:52
*** ociuhandu has quit IRC10:02
*** esikachev has quit IRC10:05
*** yamamoto has joined #openstack-infra10:08
*** claudiub has joined #openstack-infra10:12
rcarrillocruzfolks, when you need to test new jobs, do you follow ?10:13
rcarrillocruzthey seem outdated instructions, clarkb suggested to use but i'm getting failures10:14
rcarrillocruzi meant
rcarrillocruzaka 'simulating devstack-gate locally'10:15
*** _degorenko|afk is now known as degorenko10:16
*** mrmartin has joined #openstack-infra10:17
*** Qiming has quit IRC10:18
*** e0ne has joined #openstack-infra10:20
*** ramishra has quit IRC10:21
*** nadya has joined #openstack-infra10:22
*** LaT0rtue has joined #openstack-infra10:22
HeOSHello, team. Could we merge this cute request: ?  :)10:24
*** kzaitsev_mb has joined #openstack-infra10:25
AJaegerHeOS: please explain briefly what it is so that the right reviewers can have a look...10:26
*** esikachev has joined #openstack-infra10:27
HeOSAJaeger, that request is about puppet-gerrit manifests and changes of ordering.10:27
*** yamamoto has quit IRC10:27
*** tosky has joined #openstack-infra10:28
*** thorst has joined #openstack-infra10:29
*** mrmartin has quit IRC10:31
*** markvoelker has joined #openstack-infra10:32
*** ramishra has joined #openstack-infra10:32
*** thorst has quit IRC10:35
*** vponomaryov has quit IRC10:37
AJaegerThen I cannot help you - and seems the rest of the team is out right now...10:37
*** markvoelker has quit IRC10:39
openstackgerritMerged openstack-infra/tripleo-ci: Add routing info to host_info.txt
*** vponomaryov has joined #openstack-infra10:40
*** amotoki has joined #openstack-infra10:41
*** erlon has joined #openstack-infra10:41
*** thorst has joined #openstack-infra10:43
*** amotoki has quit IRC10:43
*** ociuhandu has joined #openstack-infra10:44
*** tpsilva has joined #openstack-infra10:47
*** yamamoto has joined #openstack-infra10:49
*** yamamoto has quit IRC10:49
*** abregman has joined #openstack-infra10:51
*** claudiub|2 has joined #openstack-infra10:52
*** yamamoto has joined #openstack-infra10:52
*** yamamoto has quit IRC10:52
*** thorst has quit IRC10:53
*** cdent has left #openstack-infra10:54
*** LaT0rtue has quit IRC10:55
*** amotoki has joined #openstack-infra10:55
*** claudiub has quit IRC10:55
*** yamamoto_ has joined #openstack-infra10:55
*** yamamoto_ has quit IRC10:57
*** ramishra has quit IRC11:00
*** kengo_sa_ has quit IRC11:01
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Pin the mongodb puppet module
openstackgerritMerged openstack-infra/tripleo-ci: Pin the mongodb puppet module
*** yamamoto_ has joined #openstack-infra11:08
*** yamamoto_ has quit IRC11:08
*** yamamoto_ has joined #openstack-infra11:12
*** kengo_sakai has joined #openstack-infra11:15
*** yamamoto_ has quit IRC11:17
*** amotoki has quit IRC11:17
*** yamamoto_ has joined #openstack-infra11:20
*** kengo_sakai has quit IRC11:21
*** Qiming has joined #openstack-infra11:23
*** amotoki has joined #openstack-infra11:24
*** Nikolay_St has quit IRC11:25
*** amotoki has quit IRC11:26
*** keedya has joined #openstack-infra11:28
*** yaume has quit IRC11:28
*** ddieterly has joined #openstack-infra11:28
*** ldnunes has joined #openstack-infra11:29
*** yamamoto_ has quit IRC11:29
*** piet_ has joined #openstack-infra11:30
*** amotoki has joined #openstack-infra11:30
*** kzaitsev_mb has quit IRC11:30
odyssey4meodd - has a depends-on which has merged, but after a recheck it's still not going into the gate queue regardless - any thoughts on how to resolve this?11:32
*** chem has joined #openstack-infra11:36
openstackgerritRob Cresswell proposed openstack-infra/project-config: Make the Django 1.9 tests voting on OpenStack Auth
*** dizquierdo has joined #openstack-infra11:40
openstackgerritMerged openstack/requirements: Blacklist Routes 2.3
*** amotoki has quit IRC11:40
*** moravec has joined #openstack-infra11:41
*** moravec has quit IRC11:45
*** salv-orlando has quit IRC11:46
*** dprince has joined #openstack-infra11:47
*** baoli has joined #openstack-infra11:47
*** baoli_ has joined #openstack-infra11:48
*** keedya has quit IRC11:50
*** baoli has quit IRC11:51
*** ayoung has quit IRC11:54
AJaegerodyssey4me: that's normal behaviour11:54
*** ramishra has joined #openstack-infra11:54
*** moravec has joined #openstack-infra11:54
AJaegerodyssey4me: see
*** fhubik has joined #openstack-infra11:54
AJaegerodyssey4me: better link - last para of
*** kzaitsev_mb has joined #openstack-infra11:56
odyssey4meah, I though it might be due to the backport :/11:56
AJaegerodyssey4me: the change is not ready to merge11:56
AJaegerodyssey4me: exactly: Because of the backport.11:56
AJaegerClick on the depends-On and you see that the backport is not merged, so this cannot go in.11:57
odyssey4methanks I guess that makes sense as depends-on is not branch aware11:57
AJaegerodyssey4me: as documented ;)11:57
*** derekh has quit IRC11:58
odyssey4methanks AJaeger11:59
*** fitoduarte has quit IRC11:59
*** ramishra has quit IRC11:59
*** fitoduarte has joined #openstack-infra11:59
*** zeih has joined #openstack-infra11:59
*** e0ne has quit IRC11:59
yolandai'm back, jenkins01 has not yet been restarted, still waiting for jobs to finish12:00
*** kzaitsev_mb has quit IRC12:00
*** ramishra has joined #openstack-infra12:01
*** EricGonczer_ has joined #openstack-infra12:02
AJaegeryolanda: I just checked jenkins01 - it shows 3 jobs active but looking closer, these all finished.12:05
AJaegerCan you double check, please?12:05
*** aysyd has joined #openstack-infra12:07
*** psilvad_ has joined #openstack-infra12:09
AJaegeryolanda: our jenkinses might be confused due to nodepool move - so not having an accurate state12:10
*** EricGonczer_ has quit IRC12:11
*** kzaitsev_mb has joined #openstack-infra12:11
*** ramishra has quit IRC12:12
*** ramishra has joined #openstack-infra12:12
*** mwagner_lap has joined #openstack-infra12:13
*** thorst has joined #openstack-infra12:15
*** thorst has quit IRC12:15
*** thorst has joined #openstack-infra12:15
*** Nikolay_St has joined #openstack-infra12:16
AJaegersdague: can you import nova translations again, please?
openstackgerritMerged openstack-infra/release-tools: script to create branches for a bunch of repos at once
AJaegergood morning, pabelanger !12:19
*** thorst has quit IRC12:20
sdagueAJaeger: are those version numbers right?12:20
*** amrith is now known as _amrith_12:20
sdaguethose look like mitaka version numbers for master12:20
yolandaoh, i will take a look and abort if needed, it may be waiting forever12:21
AJaeger"nova 13.0.1.dev177" ?12:21
sdaguethat's a mitaka stable version number12:21
*** markvoelker has joined #openstack-infra12:21
AJaegerlet's check what pbr says on the repo...12:22
AJaegeris there an easy way to do it?12:22
*** salv-orlando has joined #openstack-infra12:22
sdaguewhich, for content reasons, shouldn't be a huge issue, but it seems like there might be an issue with the translation pipeline12:22
AJaegersdague: looking at it uses "nova 13.0.1.dev185"12:23
AJaegerSo, number should be fine12:23
*** ildikov_ has joined #openstack-infra12:23
sdaguettx / dims version number question12:23
sdagueAJaeger: something seems wrong then12:23
AJaegersdague: good idea to ask the experts ;)12:23
*** mrmartin has joined #openstack-infra12:24
AJaegerthe tag has been merged -
*** yaume has joined #openstack-infra12:26
*** ildikov_ has quit IRC12:27
sdagueok, does that mean we'll get different results?12:27
*** ddieterly is now known as ddieterly[away]12:28
AJaegersdague: I was wondering whether we got wrong results due to the tag not merged - but it's merged ;)12:28
AJaegerSo, I'm out of ideas.12:28
*** kgiusti has joined #openstack-infra12:29
sdagueright, dims / ttx / dhellmann need to weigh in12:29
*** Goneri has joined #openstack-infra12:29
sdagueI'd like to understand it before we merge that, just to make sure there isn't a missing thing in master now12:29
* AJaeger doesn't know whether the number is correct or not - but it's consistent. So, doesn't look like translations per se a broken.12:29
AJaegersdague: sure, let's understand what's going on first...12:29
*** trown|outtypewww is now known as trown12:29
dimsAJaeger : was the output for translations before or after this got merged into nova master?
*** yamamoto has joined #openstack-infra12:30
*** Goneri has quit IRC12:30
AJaegerdims 302578 merged yesterday around 13:00 UTC, the translation import is from this morning 6:00 UTC12:30
*** mtanino has joined #openstack-infra12:30
AJaegerAnd the nova docs page has "Last updated on 'Fri Apr 8 11:57:11 2016, commit 7242cec'"12:31
AJaegerSo, 2 1/2 hours ago12:31
*** dizquierdo has quit IRC12:31
dimsAJaeger : there's details on this thread. give me a few ( to look deeper (look for null-merge in the emails)12:31
*** gordc has joined #openstack-infra12:31
odyssey4meis anyone else getting a failure to resolve ?12:31
AJaegerdims: question is : Should the version number on master now be 14.0.0 or 13.0.1?12:31
sdaguedims: ok, it's not :)12:32
sdagueso we'd like to figure out why12:32
AJaegerodyssey4me: works for me here....12:32
ttxwhy would it be 14.0.0 ?12:32
sdaguettx: master12:32
sdagueisn't that what master is supposed to be now?12:32
odyssey4meyeah, as I recall the master versioning should be 13.x.xdevx12:32
ttxsdague: sure, but how would it know that ?12:32
dims"python ./ --version" shows "13.0.1.dev187"12:32
sdaguettx: well, that's the question12:32
*** edmondsw has joined #openstack-infra12:33
ttxthat was before dhellmann dropped all preversioning12:33
AJaegerttx, will it be 14.0.0 after the first beta tag only?12:33
ttxthere was a recent thread about that12:33
dimsah. interesting12:33
sdagueok, but 13.0.1.dev70 is also a mitaka/stable version?12:33
ttxyes, that's the known drawback12:33
sdagueso we have mitaka stable and master in conflicting versions12:33
ttxlet me fetch that thread for you12:33
*** Goneri has joined #openstack-infra12:33
krotscheckjtomasek: Awesome, you're welcome!12:34
odyssey4meas I recall, we also had a cross-project discussion about it in Tokyo12:34
krotscheckSo.... why would executing `cat` make jenkins mark a job as failed? I guess we'll never know.12:34
dimsttx : this thread?
sdagueok, the issue is there are actually a bunch of artifacts like docs that push per commit12:35
sdaguewhich then end up with the wrong version on them12:35
sdagueis there any way to address that?12:35
jtomasekkrotscheck: it looked like there is some limit of lines that can be output, but that is just a wild guess12:35
ttxMy proposed solution was to push some setup.cfg preversion or a fake tag to master so that we have the bases covered until 0b112:36
*** yamamoto has quit IRC12:36
krotscheckjtomasek: That's my guess too.12:36
*** EricGonczer_ has joined #openstack-infra12:36
ttxbut Doug resisted, because the simplification of using the same thing everywhere was more important12:36
dimssdague : ttx : or do "export PBR_VERSION=14.0.0" in the specific job12:37
ttxI can see his point, it's a trade-off12:37
dimsthat env variable seems to work too12:37
AJaegersdague: so, the translations are fine and can be approved. Note that before proposing a new file we check whether strings get added/removed - so a change of only version number is ignored.12:38
sdagueAJaeger: yeh, I approved the translations12:38
ttxsdague: Agree that makes the docs look slightly uglier/confusing, though12:38
AJaegerthanks, sdague12:39
*** alaski is now known as lascii12:39
sdaguettx: it's going to be problematic when we publish api docs straight from project trees12:39
*** EricGonczer_ has quit IRC12:39
ttxJust difficult to discuss/fix until dhellmann is around since he was the most vocal proponent of that12:39
sdagueright, that's fine12:39
ttxsdague: I'd like a better long-term solution for sure12:39
sdagueit's a thing that can wait12:39
sdaguebut I think it assumed that servers only push artifacts to the world on milestones12:39
ttxwe didn't take doc versioning into account when we last weighed that tradeoff12:39
sdagueforgetting things like sample config and docs12:39
sdaguewhich are pushed on every commit12:40
ttxsdague: agreed12:40
sdagueok, so I think we at least have all the data on the table now12:40
dimsright. i missed the subtle point of (until beta)12:40
sdagueI'm sure that all the smart folks here will solve it :)12:40
*** Jeffrey4l has quit IRC12:42
*** Jeffrey4l has joined #openstack-infra12:42
yolandaAJaeger, jenkins01 has been restarted and i'm re-running jjb on it. I am also restarting jenkins04 and 0712:42
redixinhi all. any ideas how to fix .css content type on
*** yaume has quit IRC12:43
*** derekh has joined #openstack-infra12:43
AJaegerthanks, yolanda !12:44
AJaegeryolanda: did you check stale images for jenkins01?12:44
yolandastale nodes? yes12:45
AJaegernodes, yes ;) Thanks12:46
*** matt-borland has joined #openstack-infra12:46
*** piet_ has quit IRC12:47
*** ramishra has quit IRC12:48
AJaegersdague, ttx, could you review , please? That's a change to add some more jobs to requirements repo - when upper-constraints changes12:48
sdagueAJaeger: I think I don't have enough brain power for that today12:49
AJaegersdague: queue it for next week, then ;)12:50
dimsAJaeger : y, that one. i am debating why we would add one project and not another one.12:52
*** mtanino has quit IRC12:53
*** julim has joined #openstack-infra12:53
*** akaszuba has quit IRC12:54
sdagueoh, wait, this is the add all the unit tests to the req changes?12:54
sdagueno, that's just going to cause deadlock12:54
dimssdague : y. if we do this for neutron, other projects would want to do the same (even if we make the jobs non-voting)12:56
sdagueyeh, I just -2ed it12:57
sdaguemore importantly a change like that should have been a broader community discussion first12:57
sdaguedefinitely not just a project-config change fire/forget12:57
*** rlandy has joined #openstack-infra12:57
dims++ sdague12:58
*** e0ne has joined #openstack-infra12:58
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Update Summit Entity Events endpoints
*** smarcet has joined #openstack-infra13:00
openstackgerritMerged openstack-infra/openstackid-resources: Update Summit Entity Events endpoints
*** annegentle has joined #openstack-infra13:02
*** jcoufal has joined #openstack-infra13:04
*** ddieterly[away] is now known as ddieterly13:05
*** Jeffrey4l has quit IRC13:05
*** Jeffrey4l has joined #openstack-infra13:06
*** pradk has quit IRC13:07
*** pradk_ has joined #openstack-infra13:07
*** pradk_ is now known as pradk13:07
*** bkeller` has joined #openstack-infra13:09
openstackgerritKATO Tomoyuki proposed openstack-infra/project-config: [i18n] Fix release notes path
*** rfolco has joined #openstack-infra13:09
*** tiswanso has joined #openstack-infra13:09
*** mrmartin has quit IRC13:11
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Switch python-ironic-inspector-client dsvm job to iPXE as well
*** bkeller` has quit IRC13:12
*** piet has joined #openstack-infra13:12
*** mrmartin has joined #openstack-infra13:12
*** xyang1 has joined #openstack-infra13:12
*** _ari_ has joined #openstack-infra13:14
*** _amrith_ is now known as amrith13:15
openstackgerritMerged openstack/requirements: Update flake8 constraints
*** ddieterly is now known as ddieterly[away]13:16
*** mrmartin has quit IRC13:16
*** abregman has quit IRC13:18
pabelangerso, looks like we might have invalid SELinux security context on /etc/hostname for centos-7 dibs13:19
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements
*** mriedem has joined #openstack-infra13:22
rbradforyamamoto_; clarkb My demo is at part of  It's not running anymore but the last version is still present.13:22
*** rodrigods has quit IRC13:22
*** vgridnev has quit IRC13:22
*** rodrigods has joined #openstack-infra13:22
*** lucasagomes has quit IRC13:22
*** amitgandhinz has joined #openstack-infra13:25
*** ddieterly[away] is now known as ddieterly13:26
*** |-paul-| has joined #openstack-infra13:27
*** denisra_ is now known as denisra13:27
*** amitgandhinz has quit IRC13:27
*** lucasagomes has joined #openstack-infra13:28
*** jsavak has joined #openstack-infra13:29
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements
*** nwkarsten has joined #openstack-infra13:30
pc_mAnyone know how to get more info on a testr failure? I'm trying to get coverage to work in a vendor plugin repo and all I get for output is FAIL: process-returncode and returncode 1.13:31
*** yamamoto has joined #openstack-infra13:32
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add finalise script for restorecon
pabelangerclarkb: fungi: ianw: let me know what you think ^13:33
*** lucasagomes has quit IRC13:33
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on empty entity event
*** amitgandhinz has joined #openstack-infra13:35
*** aysyd has quit IRC13:35
openstackgerritMerged openstack-infra/openstackid-resources: Fix on empty entity event
*** aysyd has joined #openstack-infra13:35
*** yamamoto has quit IRC13:37
*** amitgandhinz has quit IRC13:38
*** vgridnev has joined #openstack-infra13:38
*** dims is now known as dimsum__13:41
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete uploaded swift objects on image delete
openstackgerritMonty Taylor proposed openstack-infra/shade: Rely on skip_identical to skip identical objects
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete image objects after failed upload
openstackgerritMonty Taylor proposed openstack-infra/shade: Add option to control whether SLO or DLO is used
openstackgerritMonty Taylor proposed openstack-infra/shade: Upload large objects as SLOs
*** lucasagomes has joined #openstack-infra13:42
*** mtanino has joined #openstack-infra13:42
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on date format on sql query
*** dansmith is now known as superdan13:42
*** piet has quit IRC13:42
*** mylu has joined #openstack-infra13:43
openstackgerritMerged openstack-infra/openstackid-resources: Fix on date format on sql query
ihrachysanyone to talk about Depends-On behaviour?13:44
ihrachysI think we caught some limitation for it in neutron, trying to understand whether it's supposed to work for our needs.13:45
openstackgerritKATO Tomoyuki proposed openstack-infra/project-config: [i18n] Fix release notes path
mordredihrachys: what's the issue?13:46
ihrachysmordred: there is a requirements patch that bumps a dep in upper-constraints; we also have a neutron patch that depends-on it. but we see that the neutron jobs install the older version of the constraint. which probably means that upper-constraints are not pulled from a source that includes the depends-on patch, but probably from current requirements repo HEAD.13:47
*** amitgandhinz has joined #openstack-infra13:47
*** ramishra has joined #openstack-infra13:48
*** |-paul-| has quit IRC13:48
ihrachysmordred: I wonder whether this line is supposed to consider Depends-On when pulling requirements repo:
ihrachysnot sure how zuul-cloner works.13:48
yolandaAJaeger,so jjb ran in all servers, but now i'm in the process of restarting jenkins02, 04 and 07, they are still waiting for jobs13:50
mordredihrachys: zuul-cloner will natively grok depends-on ...13:50
ihrachysyeah, that's what I thought (I guess through some envvars?)13:50
mordredihrachys: however, the requirements repo may not be in the list of projects zuul is considering for neutron unit tests13:50
ihrachysmordred: where to check?13:51
rcarrillocruzrunning devstack-gate in an lxc container is a bad idea13:51
mordredihrachys: well - we're going to do this...13:51
rcarrillocruzovs was not happy and froze13:51
rcarrillocruzgoing thru now in a vm13:51
pabelangerNeat, kolla is now passing on centos-713:51
*** links has quit IRC13:51
mordredAJaeger, pabelanger, fungi: I'm not up to speed on the latest with constraints and unit tests ... are we not getting the requirements repo into the projects list?13:51
ihrachysmordred: it's not for unit tests job that13:52
ihrachysmordred: it's for -api job we have13:52
mordredihrachys: what's an -api job?13:53
mordred(sorry, I may be behind)13:53
ihrachysmordred: it's a tempest based job that executes tempest tests maintained in neutron tree.13:53
*** BobH_ has joined #openstack-infra13:53
*** LaT0rtue has joined #openstack-infra13:53
BobH_Hello all. Wondering if anyone has seen ansible pegging cpu's during devstack-gate runs? Two CPUs at 90+%13:54
mordredihrachys: ah. neat. so, I may have run to the end of my usefulness ... hopefully we'll get smarter humans in a little bit13:54
*** esker has quit IRC13:56
*** yamamoto has joined #openstack-infra13:56
*** rbrndt_ has joined #openstack-infra13:56
*** fhubik has quit IRC13:56
*** ayoung has joined #openstack-infra13:57
*** cloudtrainme has joined #openstack-infra13:57
fungidimsum__: sdague: ttx: another solution floated was to make use of pbr's smarter-than-i-knew commit message scraping and push a simple commit with "Sem-Ver: feature" which will cause versions to jump to something like 13.1.0.dev70 instead13:58
dimsum__fungi : ack. i saw that in the email thread13:58
fungiihrachys: mordred: i think it should be z-c'ing requirements for neutron's check/gate jobs already at least. is there a link to the suspect job log?13:59
AJaegermordred: zuul-cloner does not work in the post pipeline and thus every job running in post line cannot use zuul-cloner - and thus won't get u-c14:00
ttxonly solves the conflict part of the issue, not the "should be 14.0.0" one14:00
ttxfungi: ^14:00
fungittx: do we know it should be 14.0.0 and not 13.1.0?14:00
*** tonytan4ever has joined #openstack-infra14:00
ihrachysAJaeger: what's 'post' pipeline really?14:00
mordredAJaeger: yah - I tink this ia  check/gate?14:00
AJaegerihrachys: everything *after* merge14:00
rcarrillocruzihrachys: it's a pipeline of jobs post-merge14:00
ttxfungi: for milestones-driven stuff yes, since we will soon post a
fungittx: i thought dhellmann's concern was that he wanted to get closer to actual semver and not predict that every cycle would signal an api break14:01
AJaegerlet me dig into the specific job14:01
ihrachysfungi: the affected patch:
rcarrillocruzit's typical on publish jobs14:01
ihrachysfungi: in job logs, we see tempest 10.0.0 is used:
ihrachysfungi: while dependency bumps it to 11.0.014:01
fungittx: but regardless, use "Sem-Ver: api-break" in that case
ihrachysAJaeger: so it's not our case since it's pre-pipeline14:02
AJaegerihrachys: correct, not our case14:02
AJaegerLooking at the log above, I do not see requirements setup by zuul-cloner14:02
AJaegeror should this be done at different place by devstack-gate?14:03
* AJaeger now reads full backscroll and not only last page14:03
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Added HTML cleanning for * add events * update events
fungiAJaeger: ihrachys: right, i think for you to do constraining in the context of devstack you have to make sure you use its pip_install() wrapper function14:03
*** knikolla has joined #openstack-infra14:03
fungiso could be that's the issue?14:04
ihrachysfungi: in where?14:04
fungistill digging in the log14:04
*** annegentle has quit IRC14:04
*** ayoung has quit IRC14:04
fungiihrachys: oh! this is for tempest14:04
*** psanchez has joined #openstack-infra14:04
*** annegentle has joined #openstack-infra14:04
fungiihrachys: tempest is installed into a virtualenv and not subjected to constraining14:04
*** xarses has quit IRC14:04
ihrachysfungi: not really. the api job is executed from neutron repo thru tox -e api14:05
fungiihrachys: mtreinish and others in #openstack-qa are aware thatthere's weirdness around this and may have suggestions14:05
ihrachysand the tox target is relying on UPPER_CONSTRAINTS_FILE14:05
*** mylu has quit IRC14:05
fungiihrachys: right, you're running a project tox type job under devstack-gate. they had entirely different solutions for applying constraints14:06
ihrachysmeaning the job is not really constrained for us?14:06
fungiihrachys: for tox-based jobs we use zuul-cloner to checkout the right rev of the requirements repo and copy the constraints file into your project repo so that tox/pip finds it14:06
fungiihrachys: for devstack-gate based jobs, devstack has a wrapper function that applies constraints from the appropriate repo when pip is called through an explicit wrapper function it provides14:07
ihrachysfungi: you copy but you don't set UPPER_CONSTRAINTS_FILE envvar to point to it?14:07
*** pradk_ has joined #openstack-infra14:07
fungiihrachys: you're mixing two paradigms and so the constraining that either would normally do is not applicable to the way your job is currently designed14:07
ihrachysfungi: I am not sure projects should have been exposed to the difference.14:08
*** asselin_ has joined #openstack-infra14:08
fungiihrachys: you could probably change UPPER_CONSTRAINTS_FILE to reference something like /opt/stack/new/requirements/upper-constraints.txt in that job, or have a step that copies that file to where UPPER_CONSTRAINTS_FILE is set14:08
ihrachysfungi: interesting. so we could do it in our gate hook?14:09
fungiihrachys: i don't think the people working on the constraints job implementations expected you'd set up a system with devstack-gate, but then use something other than devstack to do your python package installation14:09
ihrachysfungi: we are inventive, yes :)14:09
*** ramishra has quit IRC14:09
fungiihrachys: if those packages aren't installed until at least the gate hook, then yeah you could apply your workarounds there14:09
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate kolla jobs to centos-7 dib
ihrachysfungi: ok I think I now will try to craft a patch to fix that :)14:10
ihrachysfungi: thanks for directions!14:10
*** mylu has joined #openstack-infra14:11
fungiihrachys: you're welcome--let us know if that doesn't seem to work either14:11
pabelangerfungi: clarkb: okay, kolla seems happy on centos-7 required patches for the project are in flight14:12
*** eharney has joined #openstack-infra14:13
*** zeih has quit IRC14:15
ihrachysfungi: but we set UPPER_CONSTRAINTS_FILE in devstack jobs? or that's not available either?14:15
*** sc68cal has joined #openstack-infra14:16
*** zeih has joined #openstack-infra14:16
*** amuller has joined #openstack-infra14:16
pabelangerAJaeger: fungi: Thoughts on dropping devstack-centos7 to min-ready 0?
AJaegerpabelanger: let's merge kolla first - or anything else outstanding?14:19
fungiihrachys: i'm not sure what you're asking. devstack calls pip with its -c option to specify the constraints file
pabelangerAJaeger: just kolla, however they won't release mitaka until the 15th.  So we can wait or make the migration today14:19
*** nwkarsten has quit IRC14:19
*** jaosorior has quit IRC14:20
*** nwkarsten has joined #openstack-infra14:20
pabelangerAJaeger: but, we should land it before kolla otherwise, we'll have to jobs to use up the existing online devstack-centos7 nodes14:20
AJaegerpabelanger: I'm fine doing the migration today ;)14:21
fungipabelanger: the risk is that if we have to restart zuul, those jobs will go into an indefinite NOT_REGISTERED state because nodepool won't know to build any devstack-centos7 workers to run the jobs, and no workers means they won't be registered in zuul14:21
*** zeih has quit IRC14:21
pabelangerfungi: Ah, that is true14:21
fungii'm fine with setting it to 0 _if_ we no longer have any jobs depend on it14:22
AJaegerpabelanger: So, let's WIP your change for now.14:22
fungiotherwise we need a min-ready of at least 114:22
fungiso that we'll build nodes to make sure those jobs using them get reregistered14:22
AJaegerfungi, should we do kolla today - or after the 15th?14:22
fungiAJaeger: i don't have an opinion on timing for kolla. as long as they're on board with it, today is fine by me14:23
*** jaypipes has joined #openstack-infra14:23
redixinfungi: hello there. any ideas how to fix mime-types on
pabelangerAJaeger: ack14:23
fungiredixin: depends on what you mean by "fix"14:23
*** jaypipes is now known as leakypipes14:23
fungiredixin: better to be explicit about what you want changed/working14:23
pabelangerAJaeger: okay, lets make sure kolla PTL knows this14:24
AJaegerpabelanger: agreed, let's ask sdake when he's around14:24
redixinfungi: is showing wrong content-type for .css files14:24
*** nwkarsten has quit IRC14:24
pabelangerAJaeger: also, i missed oslo.messaging doing patch now14:24
redixinfungi: text/x-asm instead of text/css14:25
redixinfungi: that's why coverage reports are looks like this:
*** ramishra has joined #openstack-infra14:26
*** dizquierdo has joined #openstack-infra14:27
*** mylu has quit IRC14:27
*** zz_dimtruck is now known as dimtruck14:27
*** burgerk has joined #openstack-infra14:27
*** jordanP has quit IRC14:28
*** piet has joined #openstack-infra14:28
*** jordanP has joined #openstack-infra14:28
clarkbif that was uploaded by swift the mimetype is determined at upload time iirc14:28
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate oslo.messaging to centos-7 dib
clarkbvia the zuul swift uploader14:28
fungiredixin: the vhost configuration for the site can be changed at if you want to propose an addition/modification14:28
pabelangeranother devstack-centos7 migration ^14:29
redixinfungi: ill try. thanks a lot14:29
*** nwkarsten has joined #openstack-infra14:29
AJaegerpabelanger: LGTM. Seem sdake is not around yet14:31
pabelangerAJaeger: Ya, hopefully soon14:32
openstackgerrityolanda.robla proposed openstack-infra/shade: Update delete flavor test to work with betamax
*** persia has quit IRC14:34
*** daemontool has quit IRC14:35
openstackgerritMark Vanderwiel proposed openstack-infra/project-config: Enable lbaasv2 jobs into check queue
*** persia has joined #openstack-infra14:37
*** Nikolay_St has quit IRC14:37
*** trown is now known as trown|relocating14:37
*** annegentle has quit IRC14:40
odyssey4meAJaeger weird still has depends-on, but the dependent change has merged, but it still won't gate14:41
*** annegentle has joined #openstack-infra14:41
AJaegerodyssey4me: that was the other part of my answer ;) Did you read the links I gave?14:41
AJaegerodyssey4me: better link - last para of
AJaegerDoes that help, odyssey4me ?14:43
fungifyi, "recheck" doesn't get an already passing change to enqueue into the gate. when zuul "replaces" its +1 with another +1 gerrit registers that as a non-vote in its event stream14:43
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate packstack to centos-7 dib
pabelangerdmsimard: See ^ for centos-7 migration14:43
*** nwkarsten has quit IRC14:43
odyssey4meAJaeger ah, my bad - missed that last part - fixed and queued, thanks14:44
jeblairclarkb, fungi, AJaeger: hi.  sorry i wasn't around yesterday -- my day did not go as planned.  i have some time today -- anything urgent?14:44
*** mikelk has quit IRC14:44
fungijeblair: i think the urgent stuff is out of the way14:44
*** nwkarsten has joined #openstack-infra14:44
fungiwhich was mainly just clarkb rebuilding nodepool.o.o and replacing it14:45
*** sdake has joined #openstack-infra14:45
*** annegentle has quit IRC14:46
fungiwhich was also not a huge emergency, since it was still working, we just couldn't log into it or update its config any longer14:46
*** ddieterly is now known as ddieterly[away]14:46
*** piet has quit IRC14:46
openstackgerritKATO Tomoyuki proposed openstack-infra/project-config: [i18n] Fix release notes path
*** yamahata has joined #openstack-infra14:47
*** ddieterly[away] is now known as ddieterly14:48
*** xarses has joined #openstack-infra14:48
*** nwkarsten has quit IRC14:48
*** ddieterly is now known as ddieterly[away]14:48
*** sdake_ has joined #openstack-infra14:49
pabelangersdake: if you mind reading backscroll, we're talking about updating kolla jobs today:
*** psilvad_ has quit IRC14:50
*** sdake has quit IRC14:51
*** ddieterly[away] is now known as ddieterly14:51
*** annegentle has joined #openstack-infra14:52
fungijeblair: well, and there were some release hiccups but everyone who was around did an awesome job and still got it done on time14:52
*** yamamoto has quit IRC14:52
* fungi was not around. was in a car when release was happening14:53
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate infra off devstack-centos7
*** annegentle has quit IRC14:53
openstackgerritSergey Skripnick proposed openstack-infra/puppet-openstackci: Force *.css files to text/css
*** annegentle has joined #openstack-infra14:54
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Terminate sessions before dropping DBs in PostgreSQL
*** pcrews has quit IRC14:57
*** esker has joined #openstack-infra14:57
*** esker has quit IRC14:57
*** annegentle has quit IRC14:58
*** esker has joined #openstack-infra14:58
*** hashar has quit IRC15:00
*** nwkarsten has joined #openstack-infra15:00
*** mylu has joined #openstack-infra15:02
jeblairfungi: cool15:03
*** lucasagomes has quit IRC15:03
*** nwkarsten has quit IRC15:03
*** nwkarsten has joined #openstack-infra15:04
*** leakypipes has quit IRC15:06
openstackgerritPedro Alvarez Piedehierro proposed openstack-infra/storyboard: requirements: Set upper version limit for 'apscheduler'
nibalizergood morning15:09
*** pradk_ has quit IRC15:10
*** cloudtrainme has quit IRC15:10
*** dave-mccowan has joined #openstack-infra15:10
*** nwkarsten has quit IRC15:11
jeblairnibalizer: good morning!15:11
*** jcoufal has quit IRC15:12
*** lucasagomes has joined #openstack-infra15:12
*** jlibosva has joined #openstack-infra15:13
*** nadya has quit IRC15:14
*** pradk_ has joined #openstack-infra15:14
*** jlibosva has quit IRC15:15
*** imcsk8 is now known as imcsk8|afk15:15
*** ayoung has joined #openstack-infra15:16
*** tphummel has joined #openstack-infra15:16
*** bkeller` has joined #openstack-infra15:17
*** jlibosva has joined #openstack-infra15:18
*** thorst_ has joined #openstack-infra15:18
*** scheuran has quit IRC15:20
AJaegerjeblair: I'm not sure at what time jhesketh went to bed yesterday ;( His help was essential.15:21
pabelangerpleia2: docaedo: Was going to start on an outline this weekend for our talk. We should likely spend sometime next week working on it?15:22
*** thorst_ has quit IRC15:22
*** tonytan4ever has quit IRC15:22
AJaegerany project config core around to approve a fix for translation of release notes? Please queue for review - not release criticial but good to have ;)15:23
*** tonytan4ever has joined #openstack-infra15:23
AJaegerand one review request for moving us off centos-7:
AJaegerpabelanger: thanks for getting this through!15:24
pabelangerAJaeger: np!15:25
AJaegerand one more of pabelanger's changes for review:
* AJaeger signs off now and wishes everybody a great weekend!15:26
nibalizerAJaeger: same to you!15:29
*** piet has joined #openstack-infra15:29
openstackgerritMerged openstack-infra/subunit2sql: Terminate sessions before dropping DBs in PostgreSQL
openstackgerritPedro Alvarez Piedehierro proposed openstack-infra/storyboard-webclient: Add story status to the Story detail template
*** cloudtrainme has joined #openstack-infra15:30
*** trown|relocating is now known as trown15:31
*** eharney has quit IRC15:32
*** twm2016 has joined #openstack-infra15:32
*** mylu has quit IRC15:33
*** denisra has quit IRC15:34
*** mylu has joined #openstack-infra15:34
*** bkeller` has quit IRC15:35
*** amuller is now known as amuller_afk15:35
*** yamamoto has joined #openstack-infra15:36
*** smarcet has quit IRC15:36
*** smarcet has joined #openstack-infra15:37
*** infra-red has quit IRC15:37
*** yamamoto has quit IRC15:37
*** mylu has quit IRC15:38
*** mylu has joined #openstack-infra15:38
*** yamamoto has joined #openstack-infra15:38
twm2016Hey everyone, I'm having issues running tox tests. When I enter tox, i get the following error output:
*** pcrews has joined #openstack-infra15:38
clarkbtwm2016: that means the version of pip in your virtualenvs is too old. you need to upgrade virtualenv which bundles pip15:39
*** EricGonczer_ has joined #openstack-infra15:40
*** tphummel has quit IRC15:41
*** nwkarsten has joined #openstack-infra15:41
*** amrith is now known as _amrith_15:42
twm2016clarkb: Thanks for your quick response. I did see that solution but when I type sudo pip install --upgrade pip I get the following error: Cannot fetch index base URL
twm2016It may be worth mentioning I am working with a proxy. I read pip doesn't work well with those.15:42
*** flaper87 has quit IRC15:42
openstackgerritMerged openstack/requirements: Tempest 11.0.0 is released
*** wfoster has quit IRC15:43
jlibosvasdague: hi, recently we hit some race in one of Neutron jobs on gate that makes running test stuck. do you have a minute to talk about gate hooks?15:43
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Do not run npm shrinkwrap if npm-shrinkwrap.json already exists
*** sdake has joined #openstack-infra15:43
*** sdake_ has quit IRC15:44
clarkbtwm2016: you need to upgarde virtualenv which will upgrade pip in virtualenvs. If you update global pi that wont fix this. As for the proxy I cant help much with that. pip shpuld honor the http_proxy env var15:44
*** Nikolay_St has joined #openstack-infra15:44
*** flaper87 has joined #openstack-infra15:44
*** wfoster has joined #openstack-infra15:45
*** lakshmiS has joined #openstack-infra15:46
*** abregman has joined #openstack-infra15:46
*** eharney has joined #openstack-infra15:47
*** vincentll has quit IRC15:47
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add migration to move attrs and tags into columns
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Rework migration 1679b5bc102c to not use the db api
*** dtantsur is now known as dtantsur|afk15:50
*** ifarkas has quit IRC15:50
openstackgerritMerged openstack/requirements: Updated from generate-constraints
*** fedexo has joined #openstack-infra15:51
*** fitoduarte has quit IRC15:51
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements
*** jlibosva has quit IRC15:53
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test for deleting non-existing flavors
openstackgerrityolanda.robla proposed openstack-infra/shade: Update delete flavor test to work with betamax
*** tphummel has joined #openstack-infra15:55
*** bnemec has quit IRC15:56
*** migi is now known as migi_afk15:58
*** placko_ has quit IRC15:59
*** jsavak has quit IRC16:00
*** yamamoto has quit IRC16:00
*** regXboi has joined #openstack-infra16:00
*** e0ne has quit IRC16:00
*** smarcet has quit IRC16:00
*** kzaitsev_mb has quit IRC16:01
openstackgerritMerged openstack-infra/puppet-infracloud: Update parameters for puppet-apply test dance
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Add releasenotes job for sahara-tests
*** bnemec has joined #openstack-infra16:04
*** Jeffrey4l has quit IRC16:04
esikachevAJaeger: hi! please, take a look in this patch? all right?16:05
*** jlanoux has quit IRC16:05
*** piet has quit IRC16:05
clarkbthere are a large number offline jenkins slaves that were associated with the old nodepool. Today's jenkins master restarts should clean that up as part of the ansible playbook is to delete all offline slaves prior to restarting the service16:06
clarkbif we find that they slaves are still in jenkins tomorrow we can do some manual cleanup16:06
*** Guest4872 has quit IRC16:06
*** jsavak has joined #openstack-infra16:06
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Revert "Pin the mongodb puppet module"
clarkband that should be the last of the nodeool fallout16:07
pleia2pabelanger: yes re: working on slides next week16:09
clarkbfungi: image builds seem to take a reasonable amount of time on this non ssd volume. almost two hours in ubuntu-trusty is half built and xenial is the only other image queued. We will likely continue to see image uploads be the bottleneck not the volume16:10
*** amotoki has joined #openstack-infra16:10
clarkbzaro: morning16:10
openstackgerritSpencer Krum proposed openstack-infra/system-config: Use the garbage backend in the apply test
nibalizerzaro: morning16:12
openstackgerritColleen Murphy proposed openstack-infra/puppet-pip: Add new pip provider to use pypi mirrors
*** daemontool has joined #openstack-infra16:13
*** Jeffrey4l has joined #openstack-infra16:14
*** Jeffrey4l has quit IRC16:14
pabelangerclarkb: fungi: Once we're done with the fall out from nodepool.o.o getting SSH keys deleted we should possible look at standing up nodepool-builder.o.o to help protect for the future.16:14
*** Jeffrey4l has joined #openstack-infra16:14
yolandaAJaeger, jjb has run on all jenkins masters, and i've successfully restarted the ones that needed16:14
clarkbpabelanger: that won't protect us, it will just break a smaller subset of the service16:14
pabelangerclarkb: Ya, protect might not be the right word.16:15
pabelangermaybe having disposable builders is a good thing16:15
clarkbpabelanger: I no issue with doing that, but right now we are unable to due to quota16:15
pabelangerclarkb: Ya, not saying we do it today. Maybe see how to avoid it in the future16:16
clarkbonce we free the extra review.o.o next week we can start digging into it16:16
*** bpokorny has joined #openstack-infra16:17
pabelangerAs a coincident, I was pointed to virt-dib ( earlier this week, which uses libguestfs. Was planning on taking it for a spin next week and see what its deal is16:18
clarkbpabelanger: nodepool already supports such things :)16:18
clarkbsnapshot builds16:18
*** yamamoto has joined #openstack-infra16:19
clarkbwhich we have to do because we can't do nested virt on these VMs reliably (some regions do have it enabled but devstack at least seems to have a higher fail rate when we use it)16:19
*** imcsk8|afk is now known as imcsk816:19
*** mriedem is now known as mriedem_lunch16:20
pabelangerRight, that's the down side, with out nested virt it will likely be slower using libguestfs16:20
clarkbalso why wasnt that just added to dib itself ...16:20
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Parameterize libvirt virt type
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Reconcile compute and controller classes
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Add functional tests
*** yamamoto has quit IRC16:20
pabelangerthe upside, if I read right, libguestfs wouldn't need sudo permissions16:21
clarkbpabelanger: you'd be looking at week long build cycles though16:21
*** hdd has quit IRC16:21
pabelangereither way, I think now that we have have external nodepool-builders, we can work around some of the issues16:21
pabelangerclarkb: /me nods16:21
clarkband we already support this and have since the beginning of nodepool16:22
pabelangerbaremetal nodepool-builder FTW!16:22
clarkbso its weird to use yet another tool to build an image ina VM16:22
*** apoorvad has joined #openstack-infra16:22
pabelangerclarkb: Ya, not saying replace diskimage-builder I know that won't get traction16:22
*** vgridnev has quit IRC16:22
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add common bash functions to help track metrics.
toskyweird but not blowing out your system16:23
toskybut anyway, devstack-gate cores, anyone for a review (not mine)?
clarkbtosky: aroo? my point is that nodepool has for like 4 years been able to build images in VMs16:24
clarkband do so without nested virt problems16:24
clarkband we have consciously moved away from that. So seems weird to go back to it using an inferior tool16:24
*** jsavak has quit IRC16:24
*** asselin__ has joined #openstack-infra16:25
*** cloudtrainme has quit IRC16:26
*** mylu has quit IRC16:26
*** mylu has joined #openstack-infra16:26
anteayapleia2: found my kvm notes, I'll look up an xenial image to download16:27
*** asselin_ has quit IRC16:27
openstackgerrityolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax
toskyclarkb: I wouldn't automatically call inferior another tool - you didn't move away from a different way of building, if I'm not mistaken, but not specifically from that16:28
clarkbtosky: inferior because of the nested virt problem16:28
clarkbtosky: nodepool deals with that by booting instances in clouds16:28
*** Qiming has quit IRC16:29
clarkbaiui virt-dib will not, and considering its takes 5-10 minutes to just boot an ubuntu VM using qemu on these clouds an image build will take forever16:29
pleia2anteaya: great16:30
toskyclarkb: uhm, I have experience with using virt-customize on nested systems without hardware acceleration, and it's not so bad; the appliance used is really slim16:30
clarkbtosky: but compared to just using dib or using the existing nodepool VM image build stuff why bother?16:30
clarkbthats my point. It would be weird to add this in given what we already support16:31
toskyI would at least try first16:31
toskyless code to maintain could be a reason16:31
*** jsavak has joined #openstack-infra16:32
*** roxanaghe has joined #openstack-infra16:32
greghayneswhat is virt-dib?16:32
clarkbI think if the features were ported into dib we wouldn't have a problem but as is we would have to figure out install, etc etc etc16:32
pabelangerYa, I don't have much experience with nested virt, so can't say how good or bad it is.16:33
greghaynesah, link found in scrollback16:33
clarkbpabelanger: in my experience and magnums and so on its pretty terrible16:33
toskyI think the level of complication is not so high as you would expect16:33
pabelangergreghaynes: Ya, just found that this week16:33
*** bkeller` has joined #openstack-infra16:33
pabelangergreghaynes: looks to be a wrapper for libguestfs but using dib elements16:33
pabelangerclarkb: /me nods16:33
*** derekh has quit IRC16:33
pabelangerclarkb: speaking of dib elements, ran into this today with the kolla migration:
dimsum__Clint : is this you?
pabelangerlooks like our SELinux security context is not correct for some reason16:35
clarkbpabelanger: glean updates /etc/hostname16:35
clarkbpabelanger: I think its more likely it isn't settings things properly?16:36
dimsum__Clint : if it is, please please please don't do it. it's not useful to anyone16:36
pabelangerclarkb: Ah, I didn't think about glean16:37
openstackgerritEric Brown proposed openstack/requirements: Bump bandit linter to 1.0.1
pabelangerclarkb: let me look at the code16:37
pabelangerand would explain why devstack-centos7 is not affected16:37
clarkbpabelanger: but also, its probably worthwhile restorting all the selinux contexts not just for specific files (and that shoul dprobably go in the centos element?)16:37
*** salv-orlando has quit IRC16:38
pabelangerclarkb: Ya, that might be a good idea. I can bring it up with ianw for his thoughts too16:38
*** esker has quit IRC16:38
*** jamesmcarthur has joined #openstack-infra16:38
tristanCpabelanger: are we missing an /.autorelabel file ?16:39
*** cloudtrainme has joined #openstack-infra16:39
*** mylu has quit IRC16:40
*** imcsk8 is now known as imcsk8|afk16:40
pabelangertristanC: Oh, maybe. Didn't know of that file16:40
*** yamamoto has joined #openstack-infra16:40
anteayadimsum__: wow quite the line up of folks chasing the proposal bot around to boost their +1 stats:
anteayadimsum__: how disappointing16:40
tristanCpabelanger: it seems to be touched by rpm-distro element16:41
dimsum__anteaya : yes, i sent "please don't do it" to the ericsson folks16:41
anteayadimsum__: thanks for noticing and taking action16:41
tristanCpabelanger: and after a reboot, if the file is present, there is something that's going to relabel all known files16:41
anteayasuch a waste of time that could be better spent actually reviewing patches16:41
*** baoli_ has quit IRC16:42
pabelangertristanC: right, so we should see how to make the relabel happen after glean, just to be safe16:42
dimsum__indeed anteaya16:42
*** jsavak has quit IRC16:42
anteayadimsum__: thanks for drawing attention to it, hopefully this will cease16:42
anteayawhich means more actual reviews are possible16:42
tristanCpabelanger: suggests to use "fixfiles relabel"16:42
dimsum__anteaya : thanks goes to stevemar (
*** jsavak has joined #openstack-infra16:43
pabelangertristanC: Ya, looking at that now16:43
anteayastevemar: \o/16:43
*** migi_afk is now known as migi16:43
*** ddieterly is now known as ddieterly[away]16:43
*** ddieterly[away] is now known as ddieterly16:44
*** mylu has joined #openstack-infra16:44
*** harlowja has joined #openstack-infra16:46
*** nadya has joined #openstack-infra16:46
*** pcaruana has quit IRC16:46
*** ramishra has quit IRC16:47
openstackgerritElizabeth K. Joseph proposed openstack-infra/python-storyboardclient: Fix link to docs in README
*** ddieterly is now known as ddieterly[away]16:49
*** yamamoto has quit IRC16:50
*** jordanP has quit IRC16:52
*** bkeller` has quit IRC16:52
*** edmondsw has quit IRC16:52
*** sridhar_ram has joined #openstack-infra16:53
*** devananda has quit IRC16:53
*** cloudtrainme has quit IRC16:53
clarkbfungi: pabelanger greghaynes one thing we could do to speed up the cycle time on image builds and uploads is to start uploading the first image as soon as it is done building16:54
*** rcernin has quit IRC16:55
*** e0ne has joined #openstack-infra16:56
*** EricGonc_ has joined #openstack-infra16:57
*** e0ne has quit IRC16:57
*** ddieterly[away] is now known as ddieterly16:57
clarkbwe are on pace to build 6 images in ~3 hours which is not bad16:58
nibalizerclarkb: pabelanger would one of you have time to look at today?16:58
*** ddieterly is now known as ddieterly[away]16:58
nibalizermtreinish: is sad that cacti does not update16:58
nibalizeri can babyshit16:58
nibalizerer sit16:58
pleia2I loled16:59
nibalizerhaha yea sorry16:59
clarkbits ok thats half my life now17:00
anteayapleia2: Zara who is mike_g, any idea? looks like they meant to vote +1 according to their comment, but I see a -1:
*** EricGonczer_ has quit IRC17:00
Zaraanteaya: I don't know them.17:01
*** esker has joined #openstack-infra17:01
nibalizerpleia2: i would have pinged you but didn't see that you were online17:01
anteayaokay tough to get their attention then, but I do think they meant to +117:01
clarkbnibalizer: any reason to not just splat the hosts into the cron directly?17:01
jeblairclarkb said splat17:01
Zaraanteaya: yes, I can change my vote if necessary17:01
pleia2nibalizer: yeah, been lurking in #storyboard this morning to prep for our bug day next week17:01
jeblairmordred: i've been looking at and don't grok the failure17:02
Zarapleia2: hahaha, you picked the quietest day for us!17:02
clarkbnibalizer: I just came up with one reason :) comments in a sec17:02
pleia2Zara: EoD on a Friday, you're welcome ;)17:02
nibalizerclarkb: i mean its a long list of hosts17:03
*** rossella_s has quit IRC17:03
nibalizerif I was debugging it i'd want it in a file not splatted17:03
*** ihrachys has quit IRC17:03
*** rossella_s has joined #openstack-infra17:03
Zarapleia2: :D I was meant to finish at 3! Monday will be busy, anyway; SotK's back at last.17:04
clarkbnibalizer: commented17:04
jeblairclarkb, nibalizer: responded to clarkb and approved17:05
clarkbjeblair: gotcha17:06
*** flepied has quit IRC17:06
nibalizerthanks yall17:06
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate kolla jobs to centos-7 dib
mordredjeblair: so - with 30316917:06
*** mylu has quit IRC17:06
*** devananda has joined #openstack-infra17:06
mordredjeblair: oh - you mean you don't understand the test failure?17:06
nibalizerso when that goes in I will remove cacti from the emergency file and babysit17:07
*** amuller_afk is now known as amuller17:07
pabelangerAJaeger: once sdake lands the depends on we can do the cut over ^17:07
pabelangerideally later today17:07
*** roxanaghe has quit IRC17:07
jeblairmordred: yes, sorry17:07
*** baoli has joined #openstack-infra17:07
sdakepabelanger needs to make its way throug hthe gate17:07
jeblairmordred: it seemed like something that should just be handled in swiftclient, but i stopped digging17:08
pabelangersdake: ack17:08
sdakeif its a 911 i'll accelreate hte merging process17:08
*** sridhar_ram has quit IRC17:08
*** jamesmcarthur has quit IRC17:08
*** pblaho has quit IRC17:08
*** Swami has joined #openstack-infra17:08
pabelangersdake: Ya, I've already tested it with other reviews, we added an experimental job to validate it17:08
mordredjeblair: oh - so - the keystone2 tests run on old openstack (to get keystone2)17:09
*** roxanaghe has joined #openstack-infra17:09
mordredjeblair: so it seems that there is a behavior change since the other ones work17:10
jeblairmordred: ah, i wondered about that too.17:10
*** shashank_hegde has joined #openstack-infra17:10
mordredmaybe I can just update the segment size to have a floor of 104857617:10
jeblairmordred: so that's a swift server change?17:10
anteayado kvm vm names dislike the use of underscores in them?17:11
anteayaI'm believing the answer is yes17:11
jeblairanteaya: unsure, but they are not valid in the DNS17:11
*** pvaneck has joined #openstack-infra17:11
*** jiggalator has joined #openstack-infra17:12
jeblairanteaya: (and so are generally perhaps not the best idea for hostname like things)17:12
anteayathank you17:12
mordredjeblair: yes - the error is: 400 Bad Request17:12
*** xarses_ has joined #openstack-infra17:12
*** EricGonczer_ has joined #openstack-infra17:12
mordredjeblair: so, it seems that the devstack on old openstack has actually caught a compat issue for us.17:13
anteayaon my laptop, gave it a shot and couldn't ping or ssh, destroyed then recreated sans underscore and happiness results17:13
*** EricGonc_ has quit IRC17:13
*** xarses has quit IRC17:13
*** avarner has quit IRC17:15
*** netsin has quit IRC17:15
*** esikachev has quit IRC17:15
*** e0ne has joined #openstack-infra17:15
pabelangernibalizer: sure, I should have some time shortly17:15
*** fitoduarte has joined #openstack-infra17:16
nibalizerpabelanger: ah no worries, clark and jeblair pushed it through already17:17
*** lezbar has quit IRC17:17
clarkb1712 is when image builds completed so just at 3 hours17:17
*** lezbar__ has joined #openstack-infra17:17
pabelangernibalizer: great17:17
clarkband now we have 51 image uploads17:18
*** _amrith_ is now known as amrith17:18
mordrednotmyname: if you're around
mordrednotmyname: is that value a discoverable value?17:19
*** roxanaghe has quit IRC17:19
mordrednotmyname: and/or is tim burke in the IRCs? he's been super helpful with reviews17:19
clarkbtimburke mordred17:20
*** shardy has quit IRC17:20
openstackgerritMerged openstack-infra/python-storyboardclient: Fix link to docs in README
clarkboh right I was going to finish my storyboard email setup17:20
* clarkb doe sthis17:20
mordredtimburke: oh. maybe next time I'll just try tab completion :)17:21
clarkbZara: random storboard feature idea, under project groups if whe nI expand the group I could subcsribe to the group members that would be awesome17:21
notmynamemordred: actually, that limit is gone now17:22
notmynamemordred: as of the last release17:22
*** smarcet has joined #openstack-infra17:22
Shrewsmordred: that's a fun exception17:22
clarkbZara: or even a "subscribe to all project group membres magic button17:23
*** roxanaghe has joined #openstack-infra17:23
mordrednotmyname: yes ... the code works with new swift17:23
*** sputnik13 has joined #openstack-infra17:23
mordrednotmyname: but we have a job that tests shade against keystone v2 and older things (because shade should work with your cloud even if it's out of date)17:23
mordrednotmyname: so, I'm wondering on old clouds, can I find out about that limit? or is it just a hard-coded value?17:23
notmynamemordred: I think it was a documented hard-coded value, but let me see if it was in /info17:23
mordredlike if I grab capabilities, is it in there?17:24
mordredok. cool17:24
mordredI'm fine just adding it as a constant too17:24
mordredsince it' sjust for backwards compat17:24
jeblairmordred, notmyname: out of curiosity (this isn't that important) -- why did this value only creep up with using slo (rather than dlo which i believe we were using before this change)?17:25
*** akshai has joined #openstack-infra17:25
*** jistr has quit IRC17:25
*** jsavak has quit IRC17:25
mordredjeblair: I do not know the answer to that ... notmyname ?17:26
*** jsavak has joined #openstack-infra17:26
notmynamehang on. one thing at a time :-)17:26
*** fedexo has quit IRC17:26
*** BobBall is now known as BobBall_AWOL17:27
notmynamemordred: yes, it was in /info results. at least according to the 2.6.0 code17:27
dimsum__anteaya : email on -dev@17:27
notmynamemordred: ah, and git blame tells me it's been there since 1.12.017:27
*** jamesmcarthur has joined #openstack-infra17:28
anteayadimsum__: thank you!17:28
mordrednotmyname: neat!17:28
mordrednotmyname: (what's the key?)17:29
notmynamemin_segment_size under the slo key17:29
anteayafungi: who has 811 changes?17:30
notmynamejeblair: mordred: DLOs were created first. their contents are computed at the time fo the request, and they have/had a maximum segment per second limit to them. when SLOs were created, since they are defined at PUT time, we could do the check at creation time, thus simplifying the request path (ie no extra ratelimiting)17:30
anteayafungi: were the bots removed for your math?17:30
mordrednotmyname: ah - neat17:31
notmynamejeblair: mordred: but then we added the ability to refer to ranges of other objects in SLOs, so since a range might be small, we removed the 1MiB limit. so we added the rate limiting thing17:31
anteayafungi: and nice math!17:31
mordrednotmyname: is swift_client.get_capabilities() the same as GET /info ?17:32
* mordred is testing17:32
jeblairnotmyname: thanks, that's illuminating17:32
notmynamemordred: yes17:33
mordrednotmyname: woot! I've already got code getting a max size there17:33
*** nwkarsten has quit IRC17:33
*** xarses_ is now known as xarses17:33
notmynamemeeting time. stepping out. if you have more questions, ping me and I'll answer later17:33
*** nwkarsten has joined #openstack-infra17:34
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks
*** degorenko is now known as _degorenko|afk17:34
anteayadimsum__: and nice patch, let's see if behaviour can prove we don't need your patch17:34
fungianteaya: that was based on the structured data for the actual electoral rolls, so yes17:35
anteayawow I'm amazed someone can get 811 patches reviewed in one year17:35
crinklejeblair: if you feel like looking at some ruby i made a thing that might help with the "puppet doesn't use our mirrors" problem
fungidid i mention 811 patches for someone?17:36
*** nikhil has quit IRC17:36
anteayain amoungst the zeros17:36
anteayaat 811 was the first occurance I saw of 117:36
fungianteaya: oh!, i thought i only provided the first 50 lines. oops!17:36
amulleranteaya: well, there's collaborative reviews, then there's "-1 you have a spelling mistake in this comment"17:36
fungianteaya: i'll find out though ;)17:36
anteayafungi: I know you will!17:36
*** sambetts is now known as sambetts|afk17:36
clarkbfyi we appear to have accumulated a bunch of ready nodes that are old on jenkins07, this was one of the jenkinses that had price is right game show thread numbers, I am telling nodepool to delete them now17:37
anteayaamuller: right17:37
dimsum__amuller : very annoying17:37
clarkbthis explains the big green bar on the zuul status graph17:37
anteayaso yeah I am really having a hard time concieving that 811 patches merged from one author17:37
fungiamuller: it's number of commits, not number of reviews17:37
amullerfungi: ah ha :)17:37
fungiwell, number of merged gerrit changes owned by an account17:37
amullerfungi: global reqs bot? =p17:37
anteayaright, so 811 merges means how many reviews?17:38
clarkbfungi: anteaya what is plan for putting rax-ord back into srevice?17:38
anteayaamuller: he just said above the bots were removed for these stats17:38
anteayaclarkb: how are we doing on the 10 nodes we have with them?17:38
anteayaclarkb: sorry I haven't been able to keep track17:38
clarkbanteaya: I haven't heard screaming, but haven't looked at the e-r bug17:38
anteayaI am getting immune to screaming17:38
*** fawadkhaliq has joined #openstack-infra17:39
anteayaI also have not looked at anything that would give me any indication17:39
fungianteaya: "Qiming Teng (tengqm)"17:39
*** rbrndt_ has quit IRC17:40
anteayafungi: wow17:40
fungianteaya: 746 of those commits are in repos governed by the senlin team17:40
*** arxcruz has quit IRC17:41
anteayasenlin can churn out the patches17:41
nibalizerso it looks like gate-system-config-puppet-syntax-3-centos-7 and gate-system-config-puppet-syntax-4-centos-7 just never pass?17:41
nibalizer pabelanger maybe this was your deal?17:41
*** cbader has quit IRC17:42
amulleranteaya: I used to work on a project where if someone wanted to do something like change assertTrue(some_list is not []) to assertEqual(some_list, []), that would get done in one change per patch, as opposed to a single patch for the entire code base. It really depeneds on the review culture. I can easily see how some people could get that many patches in.17:43
*** cbader has joined #openstack-infra17:43
pabelangernibalizer: Hmm, maybe which patch is that for17:43
*** roxanaghe has quit IRC17:43
anteayaamuller: yes, I guess17:43
pabelangernibalizer: found it17:43
nibalizerwell for instance
anteayaamuller: I can see why you characterize your work on that project in the past tense17:43
pabelangernibalizer: Ya, we just updated puppet-jobs yesterday to centos-7 dibs17:43
*** sdake_ has joined #openstack-infra17:43
*** esikachev has joined #openstack-infra17:44
amulleranteaya: Well I learned a lot over there :)17:44
anteayaI bet17:44
*** sdake has quit IRC17:45
*** jsavak has quit IRC17:45
*** psilvad_ has joined #openstack-infra17:45
*** jsavak has joined #openstack-infra17:46
*** piet has joined #openstack-infra17:46
*** sdague has quit IRC17:46
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Add releasenotes job for sahara-tests
*** psilvad has quit IRC17:47
*** psilvad_ is now known as psilvad17:47
*** roxanaghe has joined #openstack-infra17:47
*** ybathia has joined #openstack-infra17:48
*** ociuhandu has quit IRC17:48
*** pradk_ has quit IRC17:48
*** psilvad has quit IRC17:48
*** Sukhdev has joined #openstack-infra17:48
*** jamesmcarthur has quit IRC17:48
*** mriedem has joined #openstack-infra17:49
anteayaclarkb: was this the bug?
openstackLaunchpad bug 1326813 in OpenStack-Gate "Connectivity issues between hpcloud and rax cloud causing pip install from our mirrors to fail" [Low,Confirmed]17:49
*** mriedem_lunch has quit IRC17:49
clarkbanteaya: I htink it had to do with git.o.o17:49
clarkbit appears to persist17:50
*** asselin__ has quit IRC17:50
*** psilvad has joined #openstack-infra17:50
*** bpokorny has quit IRC17:51
*** yamamoto has joined #openstack-infra17:51
*** bpokorny has joined #openstack-infra17:51
clarkband logstash says it is largely rax-ord, lets check the traceroutes17:51
*** mylu has joined #openstack-infra17:51
anteayahow do you check the traceroutes?17:52
*** psilvad_ has joined #openstack-infra17:52
clarkb is thr traceroute17:52
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again
clarkbfrom a job that failed on this according to logstash17:52
*** esikachev has quit IRC17:52
*** claudiub|2 has quit IRC17:52
clarkbfungi: is ^ the information we needed to pass along to rax?17:52
*** e0ne has quit IRC17:52
jeblaircrinkle: nice; and i like your test approach :)17:52
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "zuul: switch system-config to modern puppet-syntax jobs"
crinklejeblair: :D17:53
pabelangernibalizer: somehow that landed ^ which is not correct17:53
jeblairfungi: do you want to re-up ?17:53
*** amotoki has quit IRC17:53
*** sdague has joined #openstack-infra17:53
nibalizerpabelanger: awesome thanks17:53
pabelangerAJaeger: if you don't mind ^17:54
fungiclarkb: checking17:54
pabelangerAJaeger: we're lacking the Gemfile for system-config17:54
*** psilvad_ has quit IRC17:55
*** psilvad_ has joined #openstack-infra17:55
*** mylu has quit IRC17:55
*** psilvad has quit IRC17:55
*** lezbar__ has quit IRC17:56
*** psilvad_ has quit IRC17:56
anteayaclarkb: yay traceroute is available in the logs!17:56
*** jsavak has quit IRC17:57
*** ybathia has quit IRC17:57
clarkbhrm 07 went right back to accumulating a ton of ready nodes :/ I am going to clear out the old slaves from pre nodepool switch on 07 to make it easier to see what is actually in use there17:57
*** ybathia has joined #openstack-infra17:57
*** yamamoto has quit IRC17:57
*** achanda has joined #openstack-infra17:57
jeblairclarkb: i wonder if jenkins07 zmq is fubar17:57
*** jsavak has joined #openstack-infra17:57
clarkboh maye17:58
*** ddieterly[away] has quit IRC17:58
*** ldnunes has quit IRC17:59
*** sdake has joined #openstack-infra17:59
*** flepied has joined #openstack-infra17:59
*** ldnunes has joined #openstack-infra17:59
clarkbI also haven't heard back on the ipv6 thread that JayF started18:00
clarkbre rax networking things18:00
anteayaso here is my output from running apt-get update & install git on xenial:
anteayais the perl locale thing a known issue?18:00
jeblairclarkb: no zmq events from 07 since 15:1218:00
clarkbjeblair: that would do it18:00
clarkbjeblair: we need to restart nodepoold to reconnect iirc18:01
jeblairclarkb: oh, i thought restart of jenkins07 would do it?18:01
clarkbbecause zmq client side thinks everything is fine18:01
jeblairoh, hrm18:01
JayFclarkb: I will ping someone about that :)18:01
*** sdake_ has quit IRC18:01
clarkbthis is the big problem with zmq18:01
clarkbit hides errors from you and pretends everything is fine preventing you from taking action18:01
*** jsavak has quit IRC18:02
clarkbin this case though its likely the client never received the tcp fin when jenkins07 was restearted to fix the thread leak18:02
clarkbwe might be able to kill that connection forcefully on nodepool.o.o18:02
jeblairclarkb: ok, it's easy to try a nodepool restart18:02
*** aysyd has quit IRC18:02
jeblairclarkb: any image stuff we should wait for tho?18:02
*** jsavak has joined #openstack-infra18:03
pabelangeranteaya: is language-pack-en installed?18:03
clarkbjeblair: ya, lets let the image uploads complete and restart later today. I will put 07 in shutdown mode so it stops getting jobs18:03
*** fawadkhaliq has quit IRC18:03
pabelangeranteaya: you see those warnings when you are missing a locale18:03
clarkbI put it in shutdown mode and will continue to cleanup the old slaves on it18:03
*** fawadkhaliq has joined #openstack-infra18:03
clarkbjeblair: down to 35 image uploads (was 51 just over under an hour ago)18:04
jeblairclarkb: cool; sounds good18:05
*** jsavak has quit IRC18:05
*** jsavak has joined #openstack-infra18:05
anteayapabelanger: nothing is installed18:06
anteayapabelanger: this is my first command on a brand new image18:06
*** roxanaghe has quit IRC18:06
anteayapabelanger: so if language-pack-en is now required I think we should ensure all the puppet modules have it for xenial18:07
pabelangeranteaya: we do, it is installed with bindep18:07
*** smarcet has quit IRC18:07
anteayaI don't recall trusty throwing this error on a brand new image18:07
anteayaall the puppet modules use bindep?18:07
*** MarkAtwood has joined #openstack-infra18:07
clarkbanteaya: how did you get your image?18:07
pabelangeranteaya: no, bindep is run with the job is run from jenkins18:08
anteayauvt-simplestreams-libvirt sync release=xenial arch=amd6418:08
pabelangeranteaya: ubuntu-trusty should give you the same error now18:08
openstackgerritMerged openstack-infra/project-config: Revert "zuul: switch system-config to modern puppet-syntax jobs"
jeblairclarkb: i started a zmq-stream session on jenkins07 to see if it's emitting anything18:08
clarkbok, so it isn't something weird with our image bulds then thats xenial proper18:08
clarkbjeblair: good idea18:08
anteayapabelanger: right, I'm talking about puppet modules like the mediawiki module I am trying to get running on xenial18:08
anteayaclarkb: yeah, nothing to do with image builds18:09
anteayaclarkb: this is me trying to work on the mediawiki puppet module on my laptop using kvm18:09
pabelangeranteaya: not sure, it is possible ubuntu xenial removed locales by default18:09
pabelangerwould have to check18:09
mtreinishinfra-root: when someone gets a chance we're trying to debug some weird behavior on the subunit-worker18:10
mtreinisha bunch of periodic results aren't getting picked up. For example:
mtreinishshould have 1 hit per day18:10
*** esikachev has joined #openstack-infra18:10
anteayapabelanger: right that is my point18:10
*** jamesmcarthur has joined #openstack-infra18:11
pabelangeranteaya: so if you did uvt-simplestreams-libvirt sync release=trusty arch=amd64 you don't see the issue?18:11
anteayapabelanger: no18:11
anteayaI used uvt-simplestreams-libvirt sync release=precise arch=amd64 and I did see the issue18:11
*** achanda has quit IRC18:11
EmilienMpabelanger: sorry for breakage18:12
jeblairmtreinish: i volunteer18:12
pabelangeranteaya: I've never used uvt-simplestreams-libvirt so not sure how it works.  But, if the output from pastebin is before you run puppet apply for modules, then something changed between trusty and xenial upstream18:13
clarkbit downloads the ubutnu cloud images and boots them with libvirt18:13
pabelangereither way, we could add something into puppet to check for locales on the servers18:13
clarkband sets up config drive for them18:13
anteayapabelanger: yes the output in the paste is the first command I ran18:13
mtreinishjeblair: cool, thanks18:13
pabelangerclarkb: Ya, so ubuntu cloud image must not setup locales for xenial now18:13
*** nikhil has joined #openstack-infra18:13
mtreinishjeblair: I think the first step would be to check the worker logs on subunit-worker01.o.o and subunit-worker02.o.o18:13
pabelangerthat's how I see it18:13
openstackgerritFei Long Wang proposed openstack-infra/project-config: Add Tempest gate job for Zaqar
anteayapabelanger: didn't I make that point some time ago18:14
EmilienMnibalizer: it failed because Gemfile is missing in system-config18:14
jeblairmtreinish: we have an 02 now?!18:14
*** lucasagomes has quit IRC18:14
*** roxanaghe has joined #openstack-infra18:14
anteayaI thought I remembered making that point previously18:14
mtreinishjeblair: we do :)18:14
*** cdent has joined #openstack-infra18:14
*** ybathia has quit IRC18:14
jeblairmtreinish: did we exhaust host resources on 01 (iirc, we can run multiple workers on a machine)18:14
*** ayoung has quit IRC18:14
*** tonytan_brb has joined #openstack-infra18:15
mtreinishjeblair: yeah, when we opened it up to all jobs not just tempest there were ram exhaustion issues on a single node18:15
pabelangeranteaya: it wasn't clear to me as you said our puppet modules should install the language pack for xenial. Was trying to figure out why.18:15
jeblairmtreinish: gotcha18:15
*** jamesmcarthur has quit IRC18:16
pabelangerEmilienM: np18:16
anteayapabelanger: perhaps maybe asking me rather than taking my work and trying to put your name on it might work better in future18:16
jeblairmtreinish: the first thing i note is that it looks like both workers handled that job this morning18:16
*** smarcet has joined #openstack-infra18:17
pabelangerstepping away18:17
mtreinishjeblair: both? heh, I would have expected only 1 to have dealt with it18:17
*** yamahata has quit IRC18:18
*** tonytan4ever has quit IRC18:18
mtreinishjeblair: ah, ok worker01 was checking paths that didn't exist18:19
mtreinishand worker02 got the correct url18:19
jeblairmtreinish: oh, do we put 2 gearman jobs on the queue with different paths?18:20
*** jamesmcarthur has joined #openstack-infra18:20
jeblairmtreinish: (and thanks, i didn't spot the url diff, but i see it now)18:20
mtreinishjeblair: I wasn't ever completely clear on that, but it looks that way18:21
openstackgerritMerged openstack-infra/puppet-subunit2sql: Fix log_url parsing for new job types
jeblairmtreinish: ok, not necessarily a problem :)18:21
jeblairfocusing on worker02 then18:21
*** ybathia has joined #openstack-infra18:22
*** e0ne has joined #openstack-infra18:22
jeblairmtreinish: that's all i see related to that job18:22
jeblairno exceptions or anything18:22
jeblairmtreinish: (i updated the etherpad with all relevant log lines)18:22
*** lucasagomes has joined #openstack-infra18:22
mtreinishjeblair: hmm, ok. So it got the subunit stream decompressed it and put it on the queue18:23
*** roxanaghe has quit IRC18:23
*** netsin has joined #openstack-infra18:24
mtreinishjeblair: do you see any log lines with: 0x7f0f15bc8be0 ?18:24
*** nwkarsten has quit IRC18:24
mtreinishwhich is the addr for the stringio object for the stream18:24
mtreinishwe should log when it tries to write that to the DB18:24
*** aysyd has joined #openstack-infra18:24
jeblairmtreinish: ah, yeah, next log file; grabbing now18:24
jeblair(crossed a rotation boundary)18:25
*** e0ne has quit IRC18:25
jeblairmtreinish: do we have interleaved threads logging here?18:25
mtreinishjeblair: yeah, the http/gearman stuff is one thread18:26
mtreinishand then there is another thread todo the db population18:26
*** e0ne has joined #openstack-infra18:26
jeblairmtreinish: ah... this is sounding vaguely familiar -- have i said that we should make that one thread before?18:26
*** e0ne has quit IRC18:26
mtreinishand I agree, it's just a time thing18:26
jeblairmtreinish: k.  at least i'm consistent.18:26
*** jamesmcarthur has quit IRC18:27
jeblaircause that's where our ram problems are coming from18:27
jeblairthe uncompressed data sat around for an hour before the other thread got to it18:27
jeblairmuch cheaper to leave the job on the gearman queue18:27
mtreinishit'll also help with consistency to not say work completed until it's actually in the db18:27
jeblairmtreinish: i'm pasting into etherpad, but give me a bit, it will take some time to untangle the interleaved threads18:28
anteayahere is a paste on a brand new xenial image (I destroyed the prior vm):
anteayafirst command is adust mtu so apt-get doesnt' hang18:28
anteayaseems to work after installing the language pack18:29
anteayapleia2: any idea why xenial doesn't come with the language pack installed by default on xenial?18:29
anteayaremove one xenial18:29
*** jamesmcarthur has joined #openstack-infra18:29
*** ybathia has quit IRC18:30
mtreinishjeblair: oh, there was a db connection failure18:30
*** rbrndt has joined #openstack-infra18:31
jeblairmtreinish: yeah, but right before that blob... i'm looking for what's next18:31
*** jamesmcarthur has quit IRC18:31
pleia2anteaya: if you do the installation without internet access, there's a long standing bug about languages, you shouldn't need the pack though (english is default)18:31
pleia2anteaya: also, xenial is not yet released, so there are likely bugs18:32
anteayaokay cool18:32
pleia2afaik the language pack is if you want something other than US English18:32
fungiare they actually setting default to en_US.UTF-8 still, haven't switched to the built-in C.UTF-8?18:32
anteayashould I file this as a bug do you think?18:32
pleia2anteaya: if you used a daily image, it may be valuable to go through the reporting process (and report it on the ISO tracker)18:33
fungii suppose that could also be a difference between the ubuntu installer and the ubuntu cloud images (sounds like you're using the latter?)18:33
anteayafungi: line 55 - 60 of
*** jsavak has quit IRC18:33
openstackgerritMerged openstack/requirements: Updated from global requirements
pleia2anteaya: is this server or desktop?18:33
*** jsavak has joined #openstack-infra18:34
openstackgerritFei Long Wang proposed openstack-infra/project-config: Add Tempest gate job for Zaqar
anteayapleia2: I don't know what uvt-simplestreams-libvirt sync release=xenial arch=amd64 picks up18:34
anteayaonce this command finishes, I'll ask the vm18:34
*** shashank_hegde has quit IRC18:35
pleia2anteaya: when you booted it, did you get a graphical interface, or just a command prompt?18:35
anteayaI'm in a vm18:35
anteayaso a command prompt18:35
anteayaI'm using kvm18:35
anteayaI ssh'd in18:35
pleia2VMs can run desktops, and you can see it through virt-machine-manager :)18:35
clarkbjeblair: did you get any zmq messages on your connection to jenkins07?18:35
anteayaI don't think I have virt-machine-manager installed18:36
clarkbI don't think these images have desktops installed, they are the ubuntu cloud images18:36
pleia2anteaya: ah, ok18:36
jeblairclarkb: no, and the 3 sentinel jobs i was looking at have stopped, so i think there may be a problem on 07 (was it restarted recently?)18:37
jeblairmtreinish: etherpad updated18:37
clarkbjeblair: yes I think yolanda restarted the 4 masters earlier today that had thread leaks. jenkins07 has over 14k threads18:37
jeblairmtreinish: and yeah, it does look like that job, and *many* others were hit by the connection errors18:37
anteayauname -a gives me Linux xenial 4.4.0-15-generic #31-Ubuntu SMP Fri Mar 18 19:08:31 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux18:37
clarkbjeblair: so maybe we restart it again, see if nodepool is happy. Then if nodepool is not happy restart nodepoold too18:37
jeblairclarkb: ++18:37
mtreinishjeblair: hmm, ok. :( I guess that's not something I can fix on my end18:38
mtreinishI probably should prioritize the single threaded work18:38
*** dizquierdo has quit IRC18:38
mtreinishbecause we would have kept it on the gearman queue in this case18:38
jeblairmtreinish: hrm, it shouldn't be trying to make a local connection18:38
jeblairmtreinish: was something missed in setting up worker02?18:38
jeblairDBConnectionError: (_mysql_exceptions.OperationalError) (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")18:39
*** lezbar has joined #openstack-infra18:39
mtreinishjeblair: it should be trying to talk to the subunit2sql trove node directly18:39
jeblairthe config file looks good18:39
*** ihrachys has joined #openstack-infra18:40
*** roxanaghe has joined #openstack-infra18:40
jeblairmtreinish: every job on worker02 fails with this18:40
jeblairmtreinish: i'm going to start by restarting the worker on 02 under the assumption that it raced puppet for its initial config18:41
anteayapleia2: so if this image is from march 18th I'm going with this isn't a daily build18:41
jeblairthere is no init script18:41
pleia2anteaya: ah, ok, that's from just before the beta218:42
mtreinishjeblair: hmm, there should be, puppet should make it18:42
jeblair(we *did* have a policy of using init scripts so that every system wasn't a snowflake)18:42
*** sdake has quit IRC18:42
jeblairoh it starts with jenkins18:42
jeblairthat's counter-intuitive :)18:43
mtreinishyeah, the module was a lot of copy and paste from the logstash stuff18:43
mtreinishand a lot of the naming is still inconsistent18:43
jeblairokay it's restarted18:43
jeblairwow they are both idle18:44
jeblairthere's a job18:44
jeblairstill failing sql connections on 0218:44
anteayapleia2: oh my, prior to beta2 insn't very useful, I wonder why uvt-simplestreams-libvirt picked that18:44
fungiyeah, i never can remember the name for that initscript18:44
anteayapleia2: I'll finish off this test and then see if I can get a more recent image18:45
clarkbjeblair: 30 image uploads now so steady progress18:45
mtreinishjeblair: hmm18:45
jeblairfound it18:45
*** akshai has quit IRC18:45
jeblairmtreinish: see etherpad18:47
fungiclarkb: looking at kibana for hits on these git remote failure patterns, even with rax-ord on minimal use it's already accounting for the vast majority again18:47
jeblairmtreinish: but short version is /etc/subunit2sql/jenkins-subunit-worker-A.yaml refers to the conf file with the wrong path18:47
jeblairmtreinish: it points to /etc/logstash/subunit2sql.conf rather than /etc/subunit2sql/subunit2sql.conf18:47
mtreinishah, ok we missed that when we patched it to switch logstash to subunit2sql18:47
jeblairmtreinish: both files exist; the logstash version has a bad mysql url18:48
clarkbfungi: yup18:48
jeblairon 01, both files have the right mysql url18:48
clarkbfungi: maybe we should set rax-ord back to 0 and give rax a snapshot of the data we hvae collected?18:48
jeblairmtreinish: so i'm guessing there was some puppet sequencing during the transition that caused it to happen to work on 0118:48
fungiclarkb: yeah, i agree that's wise (though dfw has plenty of hits too)18:49
fungii'll write up the turn ord back down again change now18:49
jeblairmtreinish: i bet you can fix that in puppet; i'm going to grab some lunch now and will attempt to write the threading change for subunit2sql on a plane later18:50
anteayajeblair: you are still on planes?18:50
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Update subunit2sql conf path
mtreinishjeblair: ^^^18:50
mtreinishthat's the config change, we missed it because the yaml is in system config not puppet-subunit2sql18:50
jeblairanteaya: yes, i'm mordreding this week.  it is, as he would say, "fun".18:51
*** sc68cal has quit IRC18:51
jeblairmtreinish: +218:51
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Revert "Turn rax-ord on slowly, max to 10"
* jeblair lunches18:52
fungiclarkb: ^18:52
*** sdake has joined #openstack-infra18:52
*** nadya has quit IRC18:52
*** ihrachys has quit IRC18:52
jeblairclarkb, fungi:
anteayafungi: +218:52
anteayafungi: it is this bug correct?
openstackLaunchpad bug 1298006 in OpenStack-Gate "network sometimes unreachable accessing our git server" [Medium,Triaged]18:53
fungianteaya: yep, that's the one i'm linking from on the elastic-recheck page18:53
*** yamamoto has joined #openstack-infra18:53
anteayaI put the link to the bug in a comment on your revert18:54
anteayato help me track when we talk about turing it oon again18:54
*** akshai has joined #openstack-infra18:55
*** twm2016 has quit IRC18:55
*** dprince has quit IRC18:56
*** vgridnev has joined #openstack-infra18:57
pabelangerclarkb: stepping out to run some errands, will check back later and see if we want to continue with today. Otherwise, I'll follow up on Monday for it18:58
clarkbpabelanger: I will take a look at those shortly, trying to finish up the jenkins07 cleanup18:59
*** piet has quit IRC18:59
*** yamamoto has quit IRC18:59
pabelangerclarkb: sounds good18:59
*** ddieterly has joined #openstack-infra19:00
clarkbI am setting all non tripleo nodes on jenkins07 to delete now beucase the only jobs running there are tripleo jobs19:01
*** salv-orlando has joined #openstack-infra19:01
*** roxanaghe has quit IRC19:02
*** jsavak has quit IRC19:02
clarkbpabelanger: kolla said they were happy to switch today?19:02
*** jsavak has joined #openstack-infra19:03
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Make subunit2sql population single threaded
*** jed56 has quit IRC19:03
mtreinishjeblair: at a high level I think all we need is this: ^^^19:03
anteayamtreinish: ha ha ha19:03
mtreinishbut of course it's completely untested :)19:03
anteayaI love the way you phrased that19:04
*** cloudtrainme has joined #openstack-infra19:04
cloudnullwhen using diskimage-builder is there a good way to specify the packages I need without munging things in /usr/local/share/diskimage-builder/elements ?19:04
cloudnullive been searching about and i cant find what im looking for, though i may be dense19:04
clarkbcloudnull: we have an infra-package-needs element19:05
clarkbcloudnull: you can put those elements in an arbitrary dir and set your element path19:05
mtreinishanteaya: heh, thanks. I just meant I moved things around to show how I would go about doing it19:05
clarkbcloudnull: I would not edit the system install for that19:05
mtreinishbut I'm too lazy to spin up the machinery to actually test it19:05
*** hdd has joined #openstack-infra19:05
mtreinishso who knows whether it'll work or not19:05
* mtreinish should have backed up his hp cloud image that had things already setup19:05
anteayamtreinish: ha ha ha19:06
anteayamtreinish: :(19:06
anteayamtreinish: and good to know about the lazy19:06
anteayamore as confirmation than news19:06
*** BobH_ has quit IRC19:06
*** smarcet has quit IRC19:07
anteayahere is a paste of my first attempt to set up the puppet-mediawiki module on xenial (having installed the language pack first) the command was sudo puppet apply local.pp --modulepath=/etc/puppet/modules:system-config/modules -l /tmp/puppet-mediawiki_01.log19:07
cloudnullclarkb:  is that part of the package-install element ?19:07
clarkbcloudnull: its the one we have written for our specific list of packages19:08
clarkbcloudnull: you will likely need your own19:08
*** ayoung has joined #openstack-infra19:09
cloudnullyea i need a newer kernel and a few other packages for the hosts im working with.19:09
anteayathe biggest thing I see is that it isn't complaining about ffmpeg, so yay!19:10
anteayabut it does seem unhappy with how it is being asked to set up mysql19:10
*** flepied has quit IRC19:10
openstackgerritMerged openstack-infra/project-config: Migrate oslo.messaging to centos-7 dib
clarkbanteaya: puppet-mysql may not understand xenial yet19:12
*** salv-orlando has quit IRC19:12
cloudnullok thanks clarkb i think im on the right track now.19:12
*** roxanaghe has joined #openstack-infra19:12
*** flepied has joined #openstack-infra19:14
*** piet has joined #openstack-infra19:15
*** dave-mccowan has quit IRC19:15
openstackgerritMerged openstack-infra/project-config: Migrate packstack to centos-7 dib
openstackgerritMerged openstack-infra/project-config: Revert "Turn rax-ord on slowly, max to 10"
anteayaclarkb: ah, hmmmmm19:16
openstackgerritMerged openstack-infra/system-config: Update subunit2sql conf path
anteayathe module dependency thing19:16
*** imcsk8|afk is now known as imcsk819:17
anteayaso many things use the puppet-mysql module, no trivial thing to xenial it while ensure it still works on other operating systems19:17
clarkbanteaya: I believe there may be a way to override the distro defeaults and specify the specific packages to install regardless of distr19:18
clarkbthat might be an option in your testing19:18
anteayaoh, good to know19:19
anteayado you know where I might start looking for an example of how to do this?19:19
clarkbanteaya: maybe the puppet-mysql readme? we don't do it anywhere that I know of19:19
anteayagood point19:19
anteayamissing the obvious as usual19:19
* anteaya goes to look at the readme19:20
clarkbtripleo jobs are about 45 minutes away from completion then I can restart jenkins0719:21
anteayaclarkb: thanks the readme looks helpful19:21
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete uploaded swift objects on image delete
openstackgerritMonty Taylor proposed openstack-infra/shade: Rely on skip_identical to skip identical objects
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete image objects after failed upload
openstackgerritMonty Taylor proposed openstack-infra/shade: Add option to control whether SLO or DLO is used
openstackgerritMonty Taylor proposed openstack-infra/shade: Upload large objects as SLOs
openstackgerritMonty Taylor proposed openstack-infra/shade: Set min_segment_size from the swift capabilities
mordredsorry for patch bomb19:22
*** roxanaghe has quit IRC19:23
mordrednotmyname: 303589 ^^ ?19:23
clarkbmordred: re 303201 I think its v2 as well19:23
clarkboh ya the code handles both, but the commit message only says v119:23
mordredyah. want me to update the commit mesage?19:23
*** shashank_hegde has joined #openstack-infra19:24
clarkbmordred: would be nice for completeness if its not too much trboule19:24
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete image objects after failed upload
mordredclarkb: there ya go19:25
clarkbmordred: you are too fast, I think you also need to do that same thing in the rax case19:25
clarkbcomments posted19:27
*** thiagop has quit IRC19:27
mordredclarkb: rax case is a bit trickier19:28
mordredclarkb: oh, wait, no, we don't need it in rax19:30
clarkbwe should its  the same rocess there isn't it19:30
mordredbecause the task-import either creates or doesn't create the image19:30
clarkbcreate then do some step that copies the bits19:30
mordredit's not19:30
*** ddieterly is now known as ddieterly[away]19:30
clarkbcan we fix this api alredy?19:30
mordredclarkb: flaper87 is working on it19:30
clarkbI will update my vote19:30
clarkbthe idea that client libs have to maintain state for a service is bonghits19:31
*** Jeffrey4l_ has joined #openstack-infra19:31
clarkbmordred: look at line 240219:32
clarkbthat makes the task to create the image19:32
clarkbbut then we do it again? is that hte handle the rax fails sometimes for no reason?19:32
clarkb(making sure I understand the control flow before I change my vote)19:33
*** daemontool has quit IRC19:33
*** ddieterly[away] is now known as ddieterly19:33
clarkbya I get it now19:34
*** ddieterly is now known as ddieterly[away]19:34
*** Jeffrey4l has quit IRC19:34
*** avarner has joined #openstack-infra19:36
*** daemontool has joined #openstack-infra19:37
mordredand you can never delete the tasks :)19:39
*** mylu has joined #openstack-infra19:39
mriedemclarkb: random question, do you know if the ceph package mirror update stuff is done? i'm assuming you know what i'm talking about. :)19:40
clarkbmriedem: it isn't that has been a very low priority compared to fixing all the things recently :(19:41
mriedemclarkb: ok, np, was just checking on it, i had forgotten about it19:41
*** esikachev has quit IRC19:42
clarkbmriedem: the code is up, I need to address some comments and make an afs volume but then nodepool imploded and have been working on that19:42
mriedemclarkb: ok, thanks19:42
clarkbthere is light at the end of the tunnel, now I am just reviewing changes to make the cleanup things less painful19:42
anteayalooks like the mysql error I hit with xenial is a mysql bug:
openstackLaunchpad bug 1566406 in mysql-5.7 (Ubuntu) "postinst fails on mysql_upgrade if database is already upgraded" [High,Fix released] - Assigned to Robie Basak (racb)19:43
mordredclarkb: could use love too19:44
anteayaapparently the fix was posted yesterday19:44
*** nadya has joined #openstack-infra19:45
anteayapleia2: when are the xenial night builds built? would I have to wait until tonight's builds to get the fix for the above bug in the image?19:45
clarkbmordred: I have another nodepool bug fix that eyes on would be good
clarkbmordred: I am reviewing the shade things19:46
pleia2anteaya: it should have landed in the repo, just an apt-get update && apt-get upgrade should get you there19:46
mordredclarkb: nodepool patch makes sense to me19:46
anteayapleia2: I apt-get updated before I started but didn't apt-get upgrade, will upgrade and go again, thanks19:47
pleia2anteaya: upgrade is what actually downloads and installs the packages pulled from the refreshed sources (update)19:48
*** mylu has quit IRC19:48
*** mylu has joined #openstack-infra19:48
anteayapleia2: great the mysql-server package was the only upgrade19:48
anteayathank you19:48
anteayaI'll run puppet again19:48
*** mylu has quit IRC19:49
clarkbmordred: see comment on and decide if that is worth worrying about19:49
clarkbmordred: turns out clouds are hard19:49
*** nikhil has quit IRC19:50
*** nikhil has joined #openstack-infra19:51
clarkbmordred: and comment on
*** eharney has quit IRC19:52
clarkbzaro: do you think upstream gerrit would accept a feature request to disable the ^F search overlay19:54
clarkbzaro: maybe make it a per user setting, then I can get back to using my browser search?19:54
anteayapleia2: great that solved that error, thank you19:55
*** yamamoto has joined #openstack-infra19:55
pleia2anteaya: great :)19:55
mordredclarkb: I actually don't think that's an issue ...19:56
mordredclarkb: that is, deleting the swift objects that were the source of another imported image19:56
*** bigjools has quit IRC19:56
clarkbmordred: ya the glance image will still work19:56
clarkbyou may not have the swift object anymore though which should be fine19:57
mordredthe only reason I didn't go with "delete the swift object on successful task-import" was purely to keep things symmetrical19:57
*** bodepd has quit IRC19:57
clarkbmordred: oh you know where this might break though is when you delete the swift object that isn't there19:57
clarkbmordred: if you handle that (maybe its already handled) then it will be fine19:57
fungiclarkb: what i'd love is to figure out how to turn off all gerrit keyboard shortcuts. since i use a keyboard-driven browsing plugin they tend to pretty much all conflict with a variety of my navigation keys19:57
*** SpamapS has quit IRC19:58
*** zeus has quit IRC19:58
*** bradm has quit IRC19:58
*** lezbar has quit IRC19:58
*** bookwar has quit IRC19:58
clarkbmordred: +2 on but comment that would be nice to address in a followup19:59
mordredclarkb: yah. I agree - we should handle the object-doesn't-exist19:59
mordredclarkb: nevermind - that's handled19:59
clarkbmordred: ok I will swing back around and update my review there20:00
mordredclarkb: or, it would be if I didn't suck20:00
mordredclarkb: no, let me update it20:00
mordredclarkb: I should have used self.delete_object there20:00
*** yamamoto has quit IRC20:00
*** cloudtrainme has quit IRC20:01
*** redixin has quit IRC20:02
*** daemontool has quit IRC20:02
*** zeus has joined #openstack-infra20:03
*** zeus is now known as Guest8001920:03
*** salv-orlando has joined #openstack-infra20:03
*** salv-orlando has quit IRC20:03
*** salv-orlando has joined #openstack-infra20:04
*** roxanaghe has joined #openstack-infra20:04
mordredclarkb: ++ on the comment on 30172120:05
mordredclarkb: Shrews and I were just talking about making the docstrings better20:05
*** bodepd has joined #openstack-infra20:06
*** bigjools has joined #openstack-infra20:06
*** bigjools has quit IRC20:06
*** bigjools has joined #openstack-infra20:06
*** infra-red has joined #openstack-infra20:07
*** bradm has joined #openstack-infra20:07
*** bookwar has joined #openstack-infra20:07
*** akshai has quit IRC20:08
morganmordred: oooh better docstrings ++20:08
*** akshai_ has joined #openstack-infra20:08
Shrewsthx for the reminder. meant to review that earlier20:08
*** sridhar_ram has joined #openstack-infra20:11
*** fawadkhaliq has quit IRC20:11
anteayathe lovely big blobs that shade provides, can I format them at all?20:11
clarkbanteaya: look up json pretty printing20:11
anteayaI am20:11
clarkbanteaya: you should have a few options built into python depending on what you need20:11
*** fawadkhaliq has joined #openstack-infra20:11
anteayaah thank you20:11
Shrewsanteaya: since it's a Munch that is returned, it can help to call toDict() on the return value to get an actual dict that can be formatted20:12
anteayaShrews: I will do that, thank you20:13
anteayaI also will look up Munch20:14
mordredanteaya: (you might need to "import pprint" first)20:14
anteayamordred: thank you20:14
anteayaI will do that too20:14
*** SpamapS has joined #openstack-infra20:15
*** cloudtrainme has joined #openstack-infra20:16
anteayaenjoy lunchtime20:16
pabelangerclarkb: ya, we need kolla to merge patch first, I added a depends on.  Once that lands, we can confirm again with them20:17
*** cloudtrainme has quit IRC20:20
*** sridhar_ram has quit IRC20:22
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Remove the job for networking-ofagent master branch
*** sridhar_ram has joined #openstack-infra20:23
krotscheckmordred, jeblair: In the line "k5start -t -f /etc/bandersnatch.keytab service/bandersnatch -- timeout -k 2m 30m run-bandersnatch", what does service/bandersnatch mean?20:25
clarkbkrotscheck: I believe it is the principal id, like a username20:25
*** ddieterly[away] is now known as ddieterly20:25
mordredthat is correct20:25
*** sdake_ has joined #openstack-infra20:26
krotscheckclarkb, mordred: Cool, thanks.20:26
* krotscheck is hot on the heels of an npm mirror.20:26
mordred"run this as service/bandersnatch using the /etc/bandersnatch.keytab to auth"20:26
mordredkrotscheck: WOOR20:26
mordredkrotscheck: WOOT20:26
*** daemontool has joined #openstack-infra20:26
EmilienMpabelanger: what is the status on centos7-dib issues we had yesterday? should I recheck something to see if it works? can I help?20:27
*** annegentle has joined #openstack-infra20:27
*** sdake has quit IRC20:27
*** annegentle is now known as Guest3456820:28
pabelangerEmilienM: I got bogged down with kolla.  Let me jump into puppetlabs channel and start asking20:28
*** Guest34568 is now known as agentle20:28
openstackgerritMerged openstack/requirements: Update upper-constraints for python-troveclient v2.2.0
krotscheckmordred: Does bandersnatch run as root?20:29
mordredkrotscheck: hrm20:32
*** bpokorny has quit IRC20:32
mordredkrotscheck: I think it has to becaues it has to be able to read /etc/bandersnatch.keytab20:32
*** nikhil has quit IRC20:32
*** bpokorny has joined #openstack-infra20:32
clarkbpleia2: you had mentioned that the git and gerrit lunch stuff is well staffed?20:32
* clarkb is starting to mess with schedules20:33
*** woodster_ has joined #openstack-infra20:33
*** eharney has joined #openstack-infra20:33
clarkbdown to one tripleo job left20:34
*** infra-re_ has joined #openstack-infra20:34
krotscheckmordred: Got it. One more question- would this project live under infra?
*** vgridnev has quit IRC20:36
*** infra-red has quit IRC20:37
*** esikachev has joined #openstack-infra20:37
clarkbkrotscheck: it probably could though none of us are likely to touch js :)20:38
clarkbI only write js when under duress (eg etherpad broken during summit)20:38
dimsum__lol clarkb :)20:38
krotscheckclarkb: Ok, so, in the event that something were to happen to my ability to contribute to OpenStack, who'd be able to step in and maintain these pieces?20:39
clarkbkrotscheck: probably the folks interested in horizon/storyboard that are otherwise using the tooling20:39
pleia2clarkb: I'll be there and there are a few others, they really want some folks who know how to set up a dev environment on Windows :\20:40
*** vgridnev has joined #openstack-infra20:40
pleia2getting off streetcar now20:41
clarkbpleia2: unfortunately I do not have that skillset20:41
anteayaI don't know if it can be done20:41
clarkbI will gladly help people reinstall with linux >_>20:41
*** esikachev has quit IRC20:42
crinkleand suddenly openstack is developed on tumbleweed20:42
*** sdake_ has quit IRC20:42
*** vgridnev has quit IRC20:42
clarkbspeaking of maybe that is my fun for tomorrow20:42
clarkbfinally get my laptop running tumbleweed20:43
*** edmondsw has joined #openstack-infra20:43
mordredwhat's tumbleweed?20:43
clarkbmordred: its SUSE/gregkh's rolling release distro20:43
clarkbmordred: the great thing about it is they use gated CI20:43
clarkbso they basically do for rolling distros what we do for openstack. Its pretty neat20:43
clarkbunlike say arch which involves a lot more yolo20:43
mordredclarkb: neat20:44
*** EricGonczer_ has quit IRC20:44
mordredclarkb: too bad they don't zuul - but awesome!20:44
clarkbmordred: ya their gating is still less automated, aiui they have an automated thing that posts results20:44
clarkbmordred: then the things that all pass are bundled together into new images periodically and those are tested togethre20:45
clarkbthen a human comes along and does the actual finally push into "release"20:45
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: WIP Add CI summary stats
clarkbso you get new packages about weekly in a big group20:45
*** EricGonczer_ has joined #openstack-infra20:45
*** baoli_ has joined #openstack-infra20:46
mordredkrotscheck: shouldn't this: be ?20:46
*** baoli has quit IRC20:47
krotscheckmordred: now it's istanbul.20:47
krotschecknot constantinople.20:48
krotscheckYou've been a long time gone.20:48
krotscheckfrom constantinople.20:48
clarkbjenkins07 is done running jobs I will be restarting it shortly20:49
mordredkrotscheck: am I living in istanbul?20:49
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete uploaded swift objects on image delete
openstackgerritMonty Taylor proposed openstack-infra/shade: Rely on skip_identical to skip identical objects
openstackgerritMonty Taylor proposed openstack-infra/shade: Delete image objects after failed upload
mordredclarkb: I think I fixeded all of your things20:50
clarkbmordred: yay20:50
* krotscheck blames the turks20:51
*** piet has quit IRC20:51
*** bdemers_ has quit IRC20:51
*** bdemers has joined #openstack-infra20:51
*** thorst_ has joined #openstack-infra20:53
clarkbmordred: on 303176 if the object doens't exist the get metadata will fail and we will return False from delete_object ya?20:53
*** roxanaghe has quit IRC20:53
mordreddeleteing a thing in shade is never an error if that thing does not exist20:53
anteayapleia2: do you have any background on why the font packages seem to be renamed every release?20:54
anteayapleia2: precise seemed to be ttf-*, trusty seemed to be fonts-* and I haven't figure out what xenial calls them yet20:55
fungianteaya: another option is to pipe it through and then you can look at the yaml equivalent instead20:55
anteayafungi: thank you20:55
*** sridhar_ram has quit IRC20:55
*** ddieterly is now known as ddieterly[away]20:55
anteayaso far shade-inventory --yaml --list seems to be the winner, but it is nice to know about flamel20:56
anteayathanks for writing that SpamapS20:56
*** yamamoto has joined #openstack-infra20:57
clarkbjenkins07 has been restarted20:57
SpamapSanteaya: I use it every day. :-) you're very welcome20:57
anteayahave you an example?20:57
anteayaof its use?20:58
SpamapSI do a lot of flamel file.yml | jq .key20:58
*** roxanaghe has joined #openstack-infra20:58
SpamapSI kind of just want to add yaml support to jq20:58
SpamapSAlso I use it to validate yaml syntax20:58
*** rbrndt has quit IRC20:59
*** ddieterly[away] is now known as ddieterly20:59
jeblairmtreinish: cool, i'll look at that one the plane then :)21:00
*** esikachev has joined #openstack-infra21:00
clarkbjeblair: maybe you want to restart your zmq listener (if it was stopped)21:00
*** Sukhdev has quit IRC21:00
mtreinishjeblair: heh, maybe test it too :)21:00
SpamapSYeah, you can just apt install jq in Debian/Ubuntu21:00
jeblairmtreinish: hey we have 2 workers ;)21:01
anteayaSpamapS: interesting, thank you21:01
jeblairclarkb: yeah, i stopped it in my airport translocation21:01
jeblairclarkb: i see events21:01
jeblairso many21:01
SpamapSanteaya: it's quite useful for doing JSON in scripts21:01
SpamapSAlso it makes pretty colors21:02
*** yamamoto has quit IRC21:02
*** regXboi has quit IRC21:02
jeblairclarkb: nodepool sees events from 07 as well21:03
anteayaohhh pretty colours21:03
*** amuller has quit IRC21:04
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Remove DB existence check from open session termination
*** daemontool has quit IRC21:05
*** kgiusti has left #openstack-infra21:05
*** thorst_ has quit IRC21:05
*** trown is now known as trown|outtypewww21:05
*** jsavak has quit IRC21:06
anteayaI spruced up the sprints page:
anteayaso far storyboard is the lone sprint on it21:06
clarkbmonday is gerrit things right?21:07
clarkbnow that jenkins/nodepool seem happy is there any last minute stuff for prep we should do? I think its mostly done and just a matter of cutting the service over21:07
anteayaclarkb: yes, monday is gerrit things21:08
clarkbnext week is going to be fun. btw I will likely be busy much of tuesday and wednesday. Tuesday is house buying things and Wednesday is allergist for baby21:10
*** piet has joined #openstack-infra21:10
anteayabusy week21:10
anteayaand I hope all the things go well21:11
anteayahappy house21:11
anteayahealthy family21:11
*** gordc has quit IRC21:12
*** ldnunes has quit IRC21:15
*** amitgandhinz has quit IRC21:16
*** nadya has quit IRC21:21
*** mylu has joined #openstack-infra21:21
*** tonytan_brb has quit IRC21:23
*** mylu has quit IRC21:25
*** hdd has quit IRC21:26
*** mylu has joined #openstack-infra21:26
*** vgridnev has joined #openstack-infra21:27
*** _ari_ has quit IRC21:28
*** mylu has quit IRC21:28
*** lakshmiS has quit IRC21:28
*** jiggalator has quit IRC21:29
krotscheckAnyone know why mod_rewrite might magically decide to use the full on-server directory path as the 'uri'?21:29
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Revert "Make gate-neutron-dsvm-api job non-voting"
jeblairkrotscheck: not offhand, but in the past i have found the mod_rewrite debug log (separately enabled by special mod_rewrite options) helpful with such questions21:31
krotscheckjeblair: Yeah, that's what I'm looking at.21:31
krotscheckjeblair: It's odd. If I put the rewrite rules in the root virtualhost block, it works fine.21:31
*** vgridnev has quit IRC21:32
krotscheckjeblair: If I put them into a <location> or <directory> sub section, I get this... applying pattern '^([^/])([^/]*)(/.*)?$' to uri '/var/www/npm/ClearSilver/'21:32
*** rfolco has quit IRC21:32
*** tonytan4ever has joined #openstack-infra21:32
*** cfarquhar_ has quit IRC21:33
*** tonytan4ever has quit IRC21:33
krotscheckOh. Hrm. Found docs that explain what's going on.21:33
*** cfarquhar has joined #openstack-infra21:35
*** cfarquhar has quit IRC21:35
*** cfarquhar has joined #openstack-infra21:35
*** mriedem has quit IRC21:35
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Note that MySQL-python is more than just GPL.
*** asselin__ has joined #openstack-infra21:36
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Move gate-tempest-dsvm-neutron-dvr to gate queue
*** pradk has quit IRC21:36
*** amrith is now known as _amrith_21:36
*** dimtruck is now known as zz_dimtruck21:36
*** matt-borland has quit IRC21:37
*** roxanaghe has quit IRC21:38
*** ddieterly is now known as ddieterly[away]21:38
*** roxanaghe has joined #openstack-infra21:39
*** bdemers_ has joined #openstack-infra21:39
*** roxanaghe has quit IRC21:40
*** roxanaghe has joined #openstack-infra21:40
*** roxanaghe has quit IRC21:40
*** cdent has quit IRC21:41
*** bdemers has quit IRC21:41
*** aysyd has quit IRC21:44
*** piet has quit IRC21:45
*** tiswanso has quit IRC21:45
pabelangernibalizer: do you have an POC at puppetlabs to help debug facter issues? We're trying to run it on centos-7 but having problems21:45
pabelangernibalizer: this is the C version21:46
*** piet has joined #openstack-infra21:46
*** Nikolay_St has quit IRC21:46
nibalizerpabelanger: lets go to #puppet21:47
pabelangernibalizer: ack21:47
*** hdd has joined #openstack-infra21:47
*** nikhil has joined #openstack-infra21:47
*** ddieterly[away] is now known as ddieterly21:48
*** dalgaaf has quit IRC21:48
*** ddieterly is now known as ddieterly[away]21:50
zaroclarkb: i think upstream gerrit would be open to that.  i never noticed it because on the MAC command-F does the brower search.21:51
*** yamahata has joined #openstack-infra21:54
*** fernnest has quit IRC21:54
*** hdd has quit IRC21:54
*** rlandy has quit IRC21:56
*** esikachev has quit IRC21:56
anteayazaro: have you had a chance to look over the plan to migrate gerrit yet?
fungikrotscheck: yeah, it'll behave differently in a <location> vs <directory> context, and also in the apache config vs .htaccess file21:58
*** yamamoto has joined #openstack-infra21:58
anteayazaro: it happens on Monday:
krotscheckfungi: Yeah, I managed to figure that out.21:59
krotscheckfungi: You don't happen to know if there's a good way for me to do a mod_substitute-like thing with the current request's hostname?21:59
krotscheckfungi: The json index files created by the mirror all have the tarballs referenced with absolute URL's, and the npm client doesn't know how to get relative urls. So if we're going to have a mirror, I need to figure out how to replace the hostname in the tarball's absolute URL with the name of the current virtualhost.22:00
*** piet has quit IRC22:00
*** EricGonc_ has joined #openstack-infra22:01
*** EricGonczer_ has quit IRC22:01
krotschecki.e. When the mirror script runs, the json file it generates points to 'http://localhost/npm/things`... and once it's replicated to our AFS slaves it needs to return as `http://<afs.mirror.hostname>/npm/things`22:01
*** lascii is now known as alaski22:02
*** yamamoto has quit IRC22:04
*** xyang1 has quit IRC22:05
clarkbkrotscheck: I think you want ProxyPass?22:07
clarkbthat should rewrite urls and headers on the data going back to the client iirc22:07
clarkbbut not the content22:07
krotscheckclarkb: I want the content22:07
clarkbI do not think there is an easy option for that22:08
clarkbthe problem its the data going to the client not the data from the client that you need to rewrite22:08
krotscheckclarkb: - see the tarball sectino at the bototm.22:09
*** ajmiller has quit IRC22:09
clarkbya I understand the issue22:09
clarkbbasically your client says give me package foo, mirror says package foo is at localhost, client talks to localhost and says nerp no server here22:09
clarkbor if there is a server 404 not found22:09
krotscheckclarkb: Yeah. mod_substitute was _almost_ the magic bullet, but it turns out I can't access request environment variables.22:09
*** piet has joined #openstack-infra22:09
*** kragniz has quit IRC22:09
clarkbseems like an issue in the mirror imo22:10
*** kragniz has joined #openstack-infra22:10
clarkbbut maybe npm doesn't support relative urls?22:10
*** burgerk_ has joined #openstack-infra22:10
krotscheckclarkb: It doesn't. I mentioned that.22:10
krotscheck(Maybe not clearly)22:10
clarkbsomething something cloud native something something22:11
zaroanteaya: yep, it looks ok to me.  just not sure how the new machine was prepared.22:11
pabelangerclarkb: fungi: so, looks like we are having some locale issues on centos-7 dib:
*** ddieterly[away] is now known as ddieterly22:12
zaroanteaya: i'm assuming that it is just a copy of old machine's filesystem22:12
pabelangerclarkb: fungi: according to the web, reinstalling glibc-common is said fix22:12
pabelangerbut no reason to why they are missing22:12
anteayazaro: I can't make the same assumption22:12
anteayazaro: yolanda created the node and put gerrit on it22:12
clarkbpabelanger: you can add it to the infra-package-needs or centos-minimal maybe22:13
anteayazaro: perhaps you can add a question to the etherpad so she can reply to it once she is online again on Monday?22:13
clarkbkrotscheck: you could have puppet write out the right url prefix22:13
*** burgerk has quit IRC22:13
clarkbkrotscheck: so every mirror would get a different vhost with the correct hostname in the url22:13
clarkbthen use mod_substitute22:13
anteayapleia2: any idea why apt-cache policy can't find a font file that xenial says is avaiable?
pabelangerclarkb: I need to see what is installing it now.  Because it is installed just locales seem not to be generated22:13
pabelangeror included22:13
clarkband for completeness you should use the proxypassreverse stuff22:13
anteayazaro: it would be good to ensure any questions about filesystems are addressed before the migration22:14
zaroanteaya: done22:14
anteayazaro: thank you22:14
*** markvoelker has quit IRC22:16
*** ddieterly is now known as ddieterly[away]22:16
krotscheckclarkb: That's likely the solution.22:16
pleia2anteaya: I don't know re: fonts names22:17
*** tpsilva has quit IRC22:17
anteayapleia2: yeah, I'm stumped as well22:17
anteayaon trusty it just downloaded the packages22:17
*** baoli_ has quit IRC22:18
pleia2anteaya: and not sure about why it's not showing in policy22:18
anteayayeah, that is a first for me22:18
clarkb5 image builds left all to rax-iad22:19
clarkbso total image build + upload cycle seems about what it was in the past22:20
clarkbhowever I do think we should move up the build time to closer to 1000UTC so that stuff starts to come online around the 1414UTC current time22:20
krotscheckclarkb: Would you be able to check out one of our mirrors and see what the vhost says right now? I see a default setting of $vhost_name = $::fqdn, but I don't know what that maps to22:20
clarkbprobably after we stop using snapshot builds entirely22:20
clarkbkrotscheck: ServerName mirror.dfw.rax.openstack.org22:21
*** nadya has joined #openstack-infra22:21
krotscheckclarkb: Cool, so I can use @srvname22:21
*** infra-re_ has quit IRC22:21
krotscheckThat worked.22:22
clarkbkrotscheck: don't forget the proxypassreverse just in case npm adds some headers orsomething22:23
clarkbprobably not needed on simple static site but good for sanity22:23
krotscheckYeah, I don' tthink that's necessary22:23
* krotscheck has all the pieces of an npm mirror working. Sweeeeet.22:24
krotscheckinfra-core: Can I ask anyone with afs credentials to go and clear out the mirror/npm directory?22:25
* krotscheck lost his.22:25
*** fawadkhaliq has quit IRC22:25
krotscheckOh, and if you can get rid of my access while you're at it, that'd be neat.22:25
*** fawadkhaliq has joined #openstack-infra22:25
*** nadya has quit IRC22:26
*** burgerk_ has quit IRC22:26
*** kragniz has quit IRC22:27
*** kragniz has joined #openstack-infra22:28
*** avarner has quit IRC22:29
*** davidlenwell has quit IRC22:32
*** Guest80019 is now known as zeus22:34
*** zeus has joined #openstack-infra22:34
*** |-paul-| has joined #openstack-infra22:37
*** marcusvrn_ has quit IRC22:38
*** EricGonc_ has quit IRC22:38
*** davidlenwell has joined #openstack-infra22:40
fungizaro: anteaya: new machine was prepared by just applying puppet on it. the mysql database is remote (in trove) and the gerrit2 homedir is remote (on a cinder volume) so that should be all the persistent data we care about?22:40
*** apoorvad has quit IRC22:42
*** knikolla has quit IRC22:43
anteayafungi: thank you22:43
zarofungi: seems like a sound way to do it.  just wondering about local replication location?22:43
*** sdague has quit IRC22:44
fungizaro: it'll re-replicate fairly quickly from scratch22:47
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added NPM Mirror to mirror_update
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add NPM mirror hosting to unified mirrors
fungisince it's local file activity on the server22:47
krotscheckHoookay let's see what people think :)22:47
fungiat least that's my expectation22:47
zarofungi: just wondering if local replication location needs to be setup or is that done by puppet?22:47
*** kragniz has quit IRC22:49
fungii can check the puppet module and find out22:49
*** kragniz has joined #openstack-infra22:49
*** claudiub|2 has joined #openstack-infra22:50
openstackgerritMerged openstack-infra/system-config: Run cacti's node generation from cron
fungiso yes, puppet is creating the replication directory22:52
fungiand making sure it's owned by the gerrit2 account22:52
anteayaI added that link to the etherpad22:53
anteayaso we can find it again if anyone else asks the question22:53
*** BigWillie has joined #openstack-infra22:56
*** pabelanger has left #openstack-infra22:59
*** pabelanger has joined #openstack-infra22:59
*** rfolco has joined #openstack-infra22:59
*** BigWillie has quit IRC23:00
*** piet has quit IRC23:00
*** yamamoto has joined #openstack-infra23:00
*** yamamoto has quit IRC23:06
*** ddieterly[away] is now known as ddieterly23:07
*** ddieterly is now known as ddieterly[away]23:13
clarkbtodays image uploads are done, most succeeded. ovh (both regions) appear to have failed across the board23:14
*** ddieterly[away] is now known as ddieterly23:14
clarkbthen a failed image or two in each of the raxes and internap23:14
anteayasad face on ovh image failures23:14
anteayamore sad face23:15
clarkboveral lthough thats pretty well23:15
anteayayay again23:15
*** ddieterly is now known as ddieterly[away]23:15
clarkbso ya if we mov ethings back about 4 hours we should be done about noon PST23:16
*** markvoelker has joined #openstack-infra23:16
clarkbI discovered something new just now that might be interesting to others, libreoffice draw can edit pdfs23:18
clarkbits like magic23:19
*** lakshmiS has joined #openstack-infra23:20
anteayathat is interesting23:20
anteayathat's not just a random thing, unless you often open pdfs with libreoffice draw23:20
anteayado you?23:20
* anteaya goes looking for a pdf to open23:21
clarkbI needed to edit a pdf because someone got my address wrong23:21
clarkband the internets told me to use draw23:21
clarkbit wasn't 100% useable due to font issues but it was neat in that I could edit it23:21
*** markvoelker has quit IRC23:21
anteayayay internets, something useful23:22
anteayalike winning the lottery23:22
*** wolsen has quit IRC23:24
pleia2sometimes evince lets you edit things, I usually use xournal if not23:24
anteayait worked23:24
anteayaI too can edit a pdf23:24
anteayathanks clarkb23:24
*** sputnik13 has quit IRC23:24
*** wolsen has joined #openstack-infra23:25
clarkbpleia2: evince lets you do that on form pdfs23:25
clarkbpleia2: eg pdfs that were built to be filled out. draw seems to let you edit any text in a pdf23:25
anteayaI edited text in a pdf that was clearly not meant to be edited23:25
anteayaand was able to export it as a new pdf and open it in a pdf doc viewer23:26
anteayaah my doc viewer is evince23:26
*** EricGonczer_ has joined #openstack-infra23:26
*** sridhar_ram has joined #openstack-infra23:28
anteayaI didn't know about xournal23:29
anteayamind you I don't have a stylus23:29
clarkbanteaya: you will enjoy this, the last time I booted windows was canada and pdf related23:29
clarkbanteaya: my mother needs a visa to visit canada and my parents wanted to go for their $bignumber wedding aniversary23:30
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's download protocols
clarkbanteaya: so I helped her with the application process and I could not get evince or anything else to edit the pdf they have you use23:30
*** tosky has quit IRC23:30
clarkbfinally gave up, disconnected cat5 cable, rebooted into windows and edited pdf using adobe reader23:30
anteayaha ha ha23:30
*** cburgess has quit IRC23:30
anteayaI'm so sorry23:30
clarkbthen read it back off of my ntfs drive in linux to upload it23:31
anteayaour Canadian visa process needs to do better23:31
anteayait can't keep forcing folks to use windows23:31
*** rfolco has quit IRC23:31
anteayaha ha ha23:31
anteayaand did they get their visa?23:31
clarkbanyways she got the visa everything was fine, then my dad got sick day before trip so they didn't go :(23:31
anteayaoh no23:31
anteayaoh how disappointing23:31
anteayaI'm sorry23:32
*** cburgess has joined #openstack-infra23:32
anteayatell your parents Canada says hi23:32
anteayaand we are all looking forward to them coming to visit real soon23:32
clarkball in all its pretty nice that that is the only thing in about the last 6 years that I have needed windows for23:33
anteayathank goodness23:33
anteayasorry you have such a terrible Canada story23:33
anteayathe country where you _have_ to use windows23:33
anteayawhat a reputation23:33
clarkbwell I don't, my passport lets me straight in23:34
anteayathank goodness23:34
clarkbanteaya: if it makes you feel better I am sure the MURICAN process is worse (but I have never gone throgh that so don't know)23:37
anteayaha ha ha23:37
anteayathe small compensation Canadians comfort each other with regularly23:38
*** xarses has quit IRC23:39
*** salv-orl_ has joined #openstack-infra23:40
*** sridhar_ram has quit IRC23:42
*** salv-orlando has quit IRC23:42
*** lakshmiS has quit IRC23:44
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add NPM mirror hosting to unified mirrors
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added NPM Mirror to mirror_update
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add NPM mirror hosting to unified mirrors
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added NPM Mirror to mirror_update
*** fawadkhaliq has quit IRC23:48
*** pvaneck has quit IRC23:48
*** fawadkhaliq has joined #openstack-infra23:48
openstackgerritMerged openstack/requirements: Note that MySQL-python is more than just GPL.
*** claudiub|2 has quit IRC23:50
*** Swami_ has joined #openstack-infra23:51
*** markvoelker has joined #openstack-infra23:52
*** Jeffrey4l_ has quit IRC23:53
*** Swami has quit IRC23:54
*** piet has joined #openstack-infra23:57
*** Swami_ has quit IRC23:57
*** MarkAtwood has quit IRC23:58

Generated by 2.14.0 by Marius Gedminas - find it at!