1043728 - Turn on blobber upload for Gaia Unit Tests

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Chris Manchester (limited bugmail, email directly)]

We'll need this to upload the raw structured logs from these runs.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Looks like BlobUploadMixin will need to be added to the GaiaUnitTest definition in mozharness/scripts/gaia_unit.py, plus associated plumbing. Plus this in buildbot-configs:

diff --git a/mozilla-tests/b2g_config.py b/mozilla-tests/b2g_config.py
--- a/mozilla-tests/b2g_config.py
+++ b/mozilla-tests/b2g_config.py
@@ -661,8 +661,9 @@ GAIA_UNITTESTS = [(
     'gaia-unit', {
         'suite': 'gaia-unit',
         'use_mozharness': True,
         'script_path': 'scripts/gaia_unit.py',
+        'blob_upload': True,
     },
 )]

Comment 2

[Chris Manchester (limited bugmail, email directly)]

Attachment: Add blob upload ability to gaia unit test jobs.

This is plausible, but requires the buildbot config change above. What's the procedure for modifying those?

Comment 3

[Chris Manchester (limited bugmail, email directly)]

Also, perhaps you could review this patch or help me find someone who could? Thank you!

Comment 4

[Jonathan Griffin (:jgriffin)]

buildbot-configs is at http://hg.mozilla.org/build/buildbot-configs/; you can submit the patch and after it lands it won't go into production until the next buildbot reconfig (typically a couple of times a week).

Comment 5

[Chris Manchester (limited bugmail, email directly)]

Attachment: Turn on blobber uploads for gaia unit tests.

Buildbot config patch.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Chris, I'm not overly familiar with mozharness, so someone like jlund would be a better person to review. 

Has the mozharness patch had any local testing ? It may be helpful for someone in RelEng to run it through our staging systems. Are any other changes needed to enable structured logging ?

Comment 7

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8467213 [details] [diff] [review]
Add blob upload ability to gaia unit test jobs.

Review of attachment 8467213 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, let's get jlund's review

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8467239 [details] [diff] [review]
Turn on blobber uploads for gaia unit tests.

I'll pass this over too, so jlund see's the whole picture.

Comment 9

[Jordan Lund (:jlund)]

Comment on attachment 8467213 [details] [diff] [review]
Add blob upload ability to gaia unit test jobs.

Review of attachment 8467213 [details] [diff] [review]:
-----------------------------------------------------------------

neat. this will be nice to have. r+ once we try this on ash (see below)

it lgtm. Could we land this on ash-mozharness and trigger a few related jobs? Ash is a project branch that uses its own mozharness repo: ash-mozharness. This is the testing playground. More details here: https://wiki.mozilla.org/ReleaseEngineering/SpecialBranches#Ash

What I would be hoping for is hitting this for gaia-unit jobs http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/blob_upload.py#95 since the required https://bug1043728.bugzilla.mozilla.org/attachment.cgi?id=8467239 isn't live yet in production.

Then, to complete the test, we could land a patch in ash-mozharness that would force blob_upload_branch so we could see what happens when https://bug1043728.bugzilla.mozilla.org/attachment.cgi?id=8467239 does land. By force I mean adding something like this to gaia_unit_production_config.py:

 "default_blob_upload_servers": ["https://blobupload.elasticbeanstalk.com",],
 "blob_uploader_auth_file": os.path.join(os.getcwd(), "oauth.txt"),
+"blob_upload_branch": "ash",  # XXX hack, simulating --blob-upload-branch {branch} for gaia-unit on ash.

Once all looks well, we can land on our main mozharness repo on default. Does this make sense?

::: configs/b2g/gaia_unit_production_config.py
@@ +42,5 @@
>      ],
> +    "default_blob_upload_servers": [
> +        "https://blobupload.elasticbeanstalk.com",
> +    ],
> +    "blob_uploader_auth_file": os.path.join(os.getcwd(), "oauth.txt"),

oauth.txt is created from buildbot when use_credentials_file is True against the ScriptFactory that runs this mozharness script.

For test jobs we pretty much default this to True plus I double checked it for this specific script and the tl;dr here is I think this is fine to do

::: scripts/gaia_unit.py
@@ +51,5 @@
> +        upload_dir = self.query_abs_dirs()['abs_blob_upload_dir']
> +        if not os.path.isdir(upload_dir):
> +            self.mkdir_p(upload_dir)
> +
> +        env = self.query_env()

I think this is fine. I don't see self.config['env'] in any config file used by this script but as this is a once off and we only care about os.environment + this MOZ_UPLOAD_DIR, it might be better to use partial_env like in this example:

http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/testing/talos.py#629

@@ +52,5 @@
> +        if not os.path.isdir(upload_dir):
> +            self.mkdir_p(upload_dir)
> +
> +        env = self.query_env()
> +        env['MOZ_UPLOAD_DIR'] = upload_dir

For other blobber integrated scripts, it looks like we sometimes use a variation of pointing MOZ_UPLOAD_DIR and MINIDUMP_SAVE_PATH to abs_blob_upload_dir.

I am not sure how for this script (gaia_unit) they are swallowed by gaia_unit_test/main.py but I'll assume you know what is needed here and why.

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8467239 [details] [diff] [review]
Turn on blobber uploads for gaia unit tests.

Review of attachment 8467239 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. let's hold off landing this until the mozharness patch has been tested on ash.

Comment 11

[Chris Manchester (limited bugmail, email directly)]

Thanks jlund, sounds like a plan. Here's the push to ash-mozharness:

http://hg.mozilla.org/build/ash-mozharness/rev/5813dc999aae

I retriggered the Gu jobs on this ash push:

https://tbpl.mozilla.org/?tree=Ash&rev=2c97f5469703

I'll check back in the morning.

Comment 12

[Chris Manchester (limited bugmail, email directly)]

We hit the expected "Blob upload gear skipped. Missing cmdline options." in this log:

https://tbpl.mozilla.org/php/getParsedLog.php?id=45302979&full=1&branch=ash

I pushed the "blob_upload_branch" -> "ash" change to ash-mozharness and retriggered again on that same push.

Comment 13

[Chris Manchester (limited bugmail, email directly)]

The log here:

https://tbpl.mozilla.org/php/getParsedLog.php?id=45337804&tree=Ash&full=1

shows this code path is now hit:

http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/blob_upload.py#43

Comment 14

[Jordan Lund (:jlund)]

(In reply to Chris Manchester [:chmanchester] from comment #13)
> The log here:
> 
> https://tbpl.mozilla.org/php/getParsedLog.php?id=45337804&tree=Ash&full=1
> 
> shows this code path is now hit:
> 
> http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/
> blob_upload.py#43

OK, so at least this does not seem to break anything. :)

It looks like nothing was put in abs_blob_upload_dir as we returned early: http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/blob_upload.py#59

Is this expected? Does that mean UPLOAD_DIR did nothing? I'm not too familiar with blobber so that is why I ask.

Comment 15

[Jonathan Griffin (:jgriffin)]

I don't see any code that actually puts a file in that directory.  For instance, the Marionette script does this:

http://hg.mozilla.org/build/mozharness/file/03a994d2ec21/scripts/marionette.py#l415

Comment 16

[Chris Manchester (limited bugmail, email directly)]

As I have it the harness is putting the file in that directory, so without my changes this doesn't result in any uploads. I'll push to ash pointing to my gaia branch to test this out.

Comment 17

[Chris Manchester (limited bugmail, email directly)]

Ok, this resulted in an upload:

https://tbpl.mozilla.org/?tree=Ash&rev=233c0150fc51
https://tbpl.mozilla.org/php/getParsedLog.php?id=45371659&tree=Ash&full=1

Is this good to go? The mozharness changes should land ahead of the config change, right?

Comment 18

[Jordan Lund (:jlund)]

yes lgtm. tx for doing the rounds of testing before deploying.

we can land both at the same time at their respective default branch. default buildbot is used by nothing. mozharness default is used by Cedar and Cypress (we should watch those branches to ensure everything is still green). mozharness default merges to production usually when we merge buildbot default -> production but even if mozharness merges to prod long before, this shouldn't turn any jobs orange or red right? I'm basing that off your comment 11-13 just showing logs that blobber step didn't happen but the overall job was still a success.

buildbot default merge to production should never happen without a mozharness merge

Comment 19

[Chris Manchester (limited bugmail, email directly)]

Thanks. Right, I wouldn't expect any issue from mozharness hitting production first. I'll back this out if anything looks weird on Cypress tonight.

mozharness:
https://hg.mozilla.org/build/mozharness/rev/bba9efc62b21

buildbot:
https://hg.mozilla.org/build/buildbot-configs/rev/900f21cfaf08

Comment 20

[Chris AtLee [:catlee]]

Comment on attachment 8467213 [details] [diff] [review]
Add blob upload ability to gaia unit test jobs.

in production