Project Introduction

Table of Contents

This an introduction to my work projects since all the projects are closed source. And the main goal is to prove that I have abilities to propose solutions to meet the complex requirements and am well qualified for developing about Gradle.

Gradle Plugin Based Project

I write more than 20 close-sourced Gradle plugins when working at APUS Group. Following are some of them.

Development Kit for Extending Android Gradle Plugin

I’m working on this project to refactor and prepare for open source.

Our company has dozens of custom build flows for android application. In practice, we find:

  1. so much feature cannot be implemented without depending on the internal source of the android plugin, which produces many compatibility issues

  2. when modifying the Android extension and variants, every feature should take care of the execution order of the apply, afterEvaluate action relative to the android’s action

  3. the code implementing the business are always separate among apply, afterEvaluate, TaskGraph.whenReady, which produces unreadable and fragile code

  4. in integrating tests, it’s so hard to generate android projects and check the artifacts produced by the build

So I create a development kit to solve these issues and speed up the development and extending the Android build process.

Components

compatible layer

A compatible layer for some android plugin internal or hiding API, and also some other plugin, such as providing universal APIs to register annotation processor and arguments no matter whether users use the kapt plugin.

The kit is responsible for keeping and testing the compatibilities with different versions of the Android Gradle Plugin.

annotation based plugin entry and dependency inject

The kit provides annotations to mark a method as entry and will ensure the execution order relative to Android. You can also offer a human-friendly description by annotations.

There is a sample.

@Description("A Foo Feature Providing do fool things.")
public class FooFeature implements AndroidFeature {

    @ApplyAction(Timing.BeforeAndroid)
    @Description("foo action to be execute before android plugin applying")
    public void foo(Project project) {
        assert project != null
        assert null = project.getExtensions().getByName("android")
    }

    @ApplyAction(Timing.AfterAndroid)
    @Description("example action to be execute after android plugin applying, and the BaseExtension will never be null")
    public void provideDefaultAppIdForAndroidExtension(BaseExtension android) {
        android.defaultConfig.applicationId = "default.application.id"
    }

    @EvaluatedAction(Timing.BeforeAndroid)
    @Description("example action to be execute before android create tasks")
    public void appendAdditionTextToAppId(BaseExtension android) {
        android.defaultConfig.applicationId += ".suffix"
    }

    @EvaluatedAction(Timing.AfterAndroid)
    @Description("do something when assembleAlpha")
    public void modifyAndroidTask(Project project) {
        final Task assembleAlpha = project.getTasks().getByName("assembleAlpha")
        assert assembleAlpha != null
        assembleAlpha.doLast(...)
    }
}

Note that there are some annotations to describing when the action executes relative to Android plugin. It’s important to make your modification visible for the Android plugin.

To extending the Android build process, we need to retrieve many objects from the android plugin, such as extension, task, variant and variant data where developers have to read many sources of android plugin to find out how to access.

So the kit provides dependencies injection to reduce repetitive and error-prone codes.

For example, if you want to retrieve the ApplicationVariant collections and do something. Without the kit, you have to do like this:

project.getPlugins().withId("com.android.application", new Action<Plugin>() {
        @Override
        public void execute(Plugin plugin) {
            DomainObjectSet<ApplicationVariant> variants = project.getExtensions().getByType(AppExtension.class)
                                        .getApplicationVariants();
            // ... do somethings with variants
        }
    });

And for the LibraryVariant collections in an android library project, you should make some changes:

project.getPlugins().withId("com.android.library", new Action<Plugin>() {
        @Override
        public void execute(Plugin plugin) {
            DomainObjectSet<LibraryVariant> variants = project.getExtensions().getByType(LibraryExtension.class)
                                        .getLibraryVariants();
            // ... do somethings with variants
        }
    });

There, you must use withId, or whenPluginAdded, because you cannot assume the android plugin was applied in advance. And as for some features which need BaseVariant, you have to register the code into both of the cases of the different plugins.

And you still have trouble with how to throw an exception if there is neither com.android.application nor com.android.library plugin. In the above code, your code will only silently be ignored. You may waste time to figure out why your plugin does not work just because users forget to apply the android plugin.

Powered by our development kit, you can simply:

@EvaluatedAction
public void doSomethingWithBaseVariant(DomainObjectSet<BaseVariant> variants) {
    varinats.all(variant -> {
        // do somethigns with variant
    })
}

When BaseVariant cannot be found in the project(aka. not an android project), the code will fail for unable to resolve all required arguments.

The injected objects include:

  • what is commonly used by extending the android plugin,

  • everything provided by Gradle’s ServiceRegistry

  • other universal services offered by this development kit

In early versions, the dependencies injection is provided by a simple JSR-330 implementation written by myself from scratch. At now, I migrate it to Google Guice.

some common services

  • runtime requirement check, such as Gradle version, AGP version

  • plugin update check

  • some build script check, such as the order of plugin applied

  • common build exception and performance issue diagnostics

  • document/guide generating for feature and property/extension

  • human-readable feature description and status reports

  • …​

a set of tests utils and DSL

Gradle officially only provides TestKit and ProjectBuilder to boot a Gradle build. I create many useful DSL for writing tests and craft some android specific project generators and assertions which make writing tests easier.

There is a sample for writing test with the kit.

class ComponentsObfuscateFeatureSpec extends AbstractSingleFeatureSpec { (1)
    def "obfuscate android activity in external module"() {
        given:
        maven { (2)
            androidModule('org.example', 'test-lib', '3.2.1') {
                classes { (3)
                    newClass('org.example.lib', 'TestActivity') { (4)
                        sourceFor('TestActivity')
                    }
                }

                manifest { (5)
                    packageName = "org.example.lib"
                    activity {
                        name = "org.example.lib.TestActivity"
                        export = true
                        intent-filter {
                            // ...
                        }
                    }
                }
            }
        }

        buildFile """
${defaultMaven()}
dependencies {
    implementation 'org.example:test-lib:3.2.1' (6)
}

apply plugin: "android-componments-obfuscaton"
"""
        when:
        run 'assembleRelease'

        then:
        noExceptionThrown()

        and:
        flatOutputApk { (7)
            dexes { (8)
                assertThat(containes("Lorg.example.lib/aab; extends Landroid/app/Activity;"))
            }
        }
    }
}
1 abstract parent spec will generate example application, including an empty activity, necessary resource, AndroidManifest and build.gradle file.
2 DSL to declare maven repository in tests. The maven library produced by the closure will be published in the maven repo
3 DSL to create classes.jar for Android library. The sources generated by the closure will be compiled and bundled as classes.jar
4 DSL to generate sources files from string
5 DSL to generate AndroidManifest file
6 you can declare a dependency on the maven library that generated by the above code
7 DSL to verify flatten apk archives. The code in the closure will run against the decompressed apk file
8 DSL to verify dex file. The code in the closure will run against the parsed DexFile

Thanks to the kit, all features of plugins I written are covered by unit and integration tests which will be run against the range of supported version of the Android Plugin on Jenkins CI.

Multiple Modules Annotations Process Solution

Given the example project structure:

CompileClasspath
- project A
|--- project B
- project B

In project B, there is a class:

@Entry
public class B1Entry implements Runnable {
    public void run(){}
}

In project A, it needs to generate:

// generated code
public class MainEntry {
    public void runAllEntry() {
        new BEntry().run();
    }
}

It is pretty simple by annotation process if all classes are in the same module.

But annotations are processed when compiling. When module A compiling, the B has compiled over. The A consumes the compiled class jar rather than source codes.

To solve this problem, we have to store the annotation information to the artifacts of the B module. It can be the origin annotation name and parameters or processed necessary words.

We wrote a special annotation processor to run when compiling the B module. The processor generates a file to record the annotations in B module. The file format can be JSON, XML, properties, or any others.

And then, put it into the produced artifacts. As for android archive bundle(.aar), we add a new file into the archives.

files in the produced .aar file
- classes.jar
- resources.zip
- AndroidManifest.xml
...
- extra_annotations.json (1)
1 added by our plugin

In module A, we create a Task to collect all the extra_annotations.json by artifactView and attribution API, and then, merge the information and pass to annotation process by arguments.

This project is just like the annotations.zip file in AAR which Google has done the same things.

Universal Library Integrating Solution

When developing an android app, there are many dependent libraries to integrate, such as Google Firebase, Facebook Audience Network SDK. Our company provides more than 30 private libraries for android application. To integrate those android libraries, you have to read the document of every library and write code to configure or initialize each. It may take about one week to start from scratch.

The project gives the ability to library developers to define how to integrate and initialize code just in their library and distribute the specification alongside the maven dependencies. Developers can change how and when the library initialize without breaking the consumer’s code during version iteration. This project may be one of the best practices of the Inversion of Control (IoC).

As for application developers, when integrating libraries, this project reduces the parameters or configs to a minimum and hides almost everything by code generating. In most cases, the codes in android.app.application is cut down from more than one thousand lines to several lines.

My Duty

Two people write this project. I’m the project leader, proposer, and primary developer.

Architecture

The solution divides libraries and the application into three layers.

Layer 1: Service Registry

Provides some standard service interfaces needed by SDK and a stub implementation.

Layer 2: SDKs

These SDK can implement the interfaces in ServiceRegistry and use annotations to mark:

  • the factory method to create SDK instance.

  • the entry should be called at android application lifecycle

  • arguments it needs

A sample for one SDK is below:

public class DemoCrashCollectSDK implements ICrashCollectAPI {
    DemoCrashCollectSDK(ServiceProvider<UserTracker> userTracker,
            ServiceProvider<AppInfo> appInfo) {
        ...
    }

    @SDKFactory
    public static ICrashCollectAPI create(ServiceRegistry registry) {
        return new DemoCrashCollectSDK(
            registry.getUserTrackerProvider(),
            registry.getAppInfoProvider());
    }

    @OnApplicationAttachBaseContext
    public void init(Context context) {
        registerUncaughtExceptionHandler();
        collectionBootInfo();
    }

    @OnApplicationAttachBaseContext
    public void init(Context context, boolean debug) {
        registerUncaughtExceptionHandler();
        collectionBootInfo();
    }
}
Layer 3: Android Application

For application developers, the only thing to do is calling the setup method of the whole platform. The code generator will take care of delivering events to every SDK.

Below a sample for application:

public class MyApplication extends android.app.Application {

    @Override
    public void attachBaseContext(Context context) {
        PlatformService.setAppInfo(
            "App Name",
            "1.0",
            R.drawable.icon)
        PlatformService.applicationAttachBaseContextBuilder(context)
            .setDebug(true) // optional
            .invoke();
    }

    @Override
    public void onCreate() {
        PlatformService.applicationOnCreate().invoke();
    }
}

The code of behind PlatformService.applicationAttachBaseContextBuilder(context) is generated when building, and it will call every SDK only if the SDK is in the CompileClasspath.

How the annotations in the library are handed over between modules has been introduced in the previous project.

Distributed Smoke Test Solution

Our team does not just provide libraries for other department and also takes responsibility for the wrong usage, missing configuration, and any other possibility that break down our libraries functions. We cannot assume other teams will write code for the library (Exactly, I’m the only one in our company who write tests). 
This framework gives developers the ability to distribute smoke tests along with the maven libraries.
The tests are regressive and protect the most vital function from wrong configurations or bad usages.

My Duty

Two people write this project. I’m the project leader, proposer, and primary developer.

Features

  1. Library developers can write android tests case which will run against the produced android app

  2. No conflicts with origin unit/integrate tests of the library

  3. Tests can be delivered and distributed along with maven library. No need for addition dependency declaration in consume

Components

Gradle plugin for android library build (for library developers)

Any android library can declare verifiedBy dependency to record the smoke test dependencies.

dependencies {
  verifiedBy project(':verifiesTest')
}

And when publishing, an extra dependency will be added into maven pom as test dependency.

Gradle plugin for smoke test module(for library developer)

The module will be compiled and bundled as an android AAR library.

Gradle plugin for android application build (for app developer)

The plugin will:

  • create additional androidTest (which is only be created for debug build type) variant for release build type as special verifyTest variant

  • extract the test dependencies from maven dependencies in compileClasspath, and add them to verifyTestImplementation

  • create tasks to run the smoke tests all together or separately, and collect the result

a handy test library

to accomplish complicated user operations (based on Espresso/UIAutomator)

a Jenkins plugin

to demonstrate the result and a Jenkins workflow to run the smoke tests for every build

Gradle Plugin for Android Components/Views Obfuscation

By default, Android Plugin for Gradle will never obfuscate the Android components (activity, service, content-provider, and receiver) Class name and their members. This Plugin hack this and allows Proguard/R8 to obfuscate them.

Implementation Scratch

In the Android build process, we concern about these tasks:

:processReleaseManifest (1)
:mergeReleaseResources
:processReleaseResources (2)
:compileReleaseJavaWithJavac
:transformResourcesWithMergeJavaResForRelease
:transformClassesAndResourcesWithProguardForRelease (3)
1 generate AndroidManifest.xml file
2 compile android resource to resource.ap_, and generate additional ProGuard rules
3 use ProGuard/R8 to process java source and java resource(not android resource), and generate mapping.txt

The ProGuard rules generated by processReleaseResources will keep all the components and views that are used in layout resources.

# Referenced at /Users/yiyazhou/Projects/Works/PowerPlusLauncher/app/build/intermediates/merged_manifests/release/AndroidManifest.xml:356
-keep class com.dckaz.DazGuideActivity { <init>(); }

# Referenced at /Users/yiyazhou/Projects/Works/PowerPlusLauncher/app/src/main/res/layout/workspace_search_bar.xml:2
-keep class org.uma.graphics.view.EnhancedLinearLayout { <init>(android.content.Context, android.util.AttributeSet); }

# Referenced at /Users/yiyazhou/Projects/Works/PowerPlusLauncher/modules/velorum-guide/build/intermediates/packaged_res/release/layout/permission_guide_view.xml:21
-keep class org.velorum.guide.PermissionAnimatorView { <init>(android.content.Context, android.util.AttributeSet); }

If you block these rules(may just delete these rules), these class will be obfuscated but you need also modify the layout resource and AndroidManifest.xml according to the ProGuard mapping.

Because task transformClassesAndResourcesWithProguard depends on processManifest and processResources, so you have to alter these files by append new task after ProGuard/R8 transformation task.

But in this way, we have to:

  1. find out a way to replace the next bundleRelease task inputs with what you modified.

  2. process the resource.ap_ file: a zip with the compiled resource. The resource has been compiled to .flat file. It’s dirty to decompile them.

Final Solutions

Only if the obfuscating mapping were generated by ProGuard, I would not merely append action to android resource/manifest process task to alter resource file because a cycle isn’t allowed in task graph.

So I decided to manipulate the mapping by myself, the apply it to ProGuards.

And the final solution is like this:

:processReleaseManifest (1)
:mergeReleaseResources
:processReleaseResources (2)(3)
:compileReleaseJavaWithJavac
:appendInMapToProguardRulesForRelease (4)
:transformResourcesWithMergeJavaResForRelease
:transformClassesAndResourcesWithProguardForRelease
1 replace all components in AndroidManifest.xml and record to self-managed mapping.
2 replace all custom views in layout.xml and record to self-managed mapping.
3 new task to modify android generated ProGuard rules to keep just necessary constructors.
4 generate mapping.txt by self-managed mapping, and apply to ProGuard Transform

The self-managed mapping is a dictionary guaranteed to produce consistent value for input during the one-time build and also takes care of the method and class visibility after code obfuscation. Its input is origin class name, and the output is the corresponding obfuscated class name.

In the processReleaseResources task, we should process all the layout XML resources and alter all reference to origin class to obfuscated class. It’s impossible to modify the task inputs because android computes and directly reads the project sources sets(We shouldn’t change the user’s sources code, either). And it’s also tough to alter the task output since the output is binary .flat file.

After deep diving into the Android Plugin source, I found it’s simple to write an DataBindingProcessor to archive our goal. During the processReleaseResources, Android will call DataBindingProcessor to preprocess the layout XML resources to remove data-binding expressions. It’s relatively easy to replace the data-binding processor by reflection. With the delegation pattern, there is no stumbling block to coexistence with data-binding, either.

Gradle Plugin for Variant-Aware Android Library Publish

In our company, every library should publish a release version and debug version.

For Gradle 3.x, AGP(Android Gradle Plugin) 2.x

There is no native variant-aware support in Gradle. For the project module dependency, android publishes every variant to a different configuration. When publishing to maven, we separate it by version name suffix in the uploadArchives task.

dependencies {
    debugCompile project(':lib', configuration: 'debug')
    releaseCompile project(':lib', configuration: 'release')

    debugCompile 'org.example:lib:1.0.0-debug'
    releaseCompile 'org.example:lib:1.0.0-release'
}

But the variant selection may be messed up because of version resolution.

For example, when resolving ''debugCompileClasspath'', there are: - org.example:lib:1.0.0-debug - org.example:lib:2.0.0-release // some bad guys introduces by compile 'org.example:lib:2.0.0-release'

Gradle will resolve to org.example:lib:2.0.0-release.

As for debugCompileClasspath, I expect every library should be the variant of debug. The variant selection should not be affected by version.

Gradle 4.x, AGP(Android Gradle Plugin) 3.x

Native support for the variant-aware build is introduced in Gradle. So I try to migrate to Artifact Transformation API. I imagine bundling multi-variants into one archive file:

archive file contents
library-debug.aar
library-release.aar

We can name the format as .bundle archives. When consuming, we can use ArtifactTransform to separate the variants from the bundle file. As a result, we can write dependencies more simply.

dependencies {
   implementation 'org.example:lib:1.0.0' // no debug/release suffix any more
}

I also encountered a Gradle bug prevents me: https://github.com/gradle/gradle/issues/7061. I fixed it finally.

Gradle 5.3

Cheers! Gradle Module Metadata 1.0 is released.

I have not migrated libraries in my company to Gradle metadata format for scheduling reason, but I know it should be the final solution to publish variants libraries.

Others

As an Android engineer in the past company, my other projects are mainly android libraries and applications. So I omit most details.

Intellij IDEA Plugin

An IDEA/Studio plugin implements:

  • provide a GUI browsing panel for company internal SDK and quickly open documentation sites

  • one-click dependency version update for company internal maven library for Gradle project

  • GUI editor for a custom binary format based on Google’s FlatBuffer (this format commonly used in data transmission of our company internal SDK)

  • grammar highlighter for flat buffer schema

Android Polymerized Ad SDK

A polymerized ad SDK which provides a universal API for dozens third-party advertisement SDK and has a very complex loading and retry logic to get the best price ad among all those third-party advertising sources.

Another team initially maintains it. But they are trapped with the terrible code style and chaotic architecture and ask me for help. I rewrite it from scratch, including unit and functional tests. I use RxJava to implement the most complex concurrent loading and retry logic.

Android Anti-cheating SDK

A native android library. It tracks user behavior and uploads to the server for robot recognition and commercial fraud identification.

I write the verbose client SDK, which boringly collects data, builds up into flat buffer and uploads. No more interesting details here.

Android Crash Collector SDK

To collect Android Java and Native Crash, upload and provide extensional API for other developers.

Android Linked Wakening SDK

A simple SDK to wake process by Android Binder Mechanism.

Android App: One Moment Video Shot

A video shot Application when I was in college. I wrote it in Kotlin before Kotlin got prevalent. It can shot video and filter. It also has one SNS society in the app. There is a total of 100,000 users before I quit.