Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major refactor of functions deploy #3132

Merged
merged 37 commits into from
Feb 16, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
06b62a1
Adding onPoll option to operation-poller (#3046)
joehan Jan 19, 2021
6258dd5
Typescriptify functionsDeployHelper (#3059)
joehan Jan 20, 2021
ec7d079
Typescriptifying gcp.cloudfunctions (#3060)
joehan Jan 20, 2021
f583ef6
Typescriptifying functionsConfig (#3063)
joehan Jan 21, 2021
08b9d56
Typescriptifying deploymentTool (#3061)
joehan Jan 21, 2021
b4944a4
Refactoring prepare stage of functions deploy (#3067)
joehan Jan 21, 2021
3be0dca
refactoring release step of functions deploy to use typescript
joehan Jan 21, 2021
e0e703e
Adding logic to build regional deployments
joehan Jan 24, 2021
046c7d7
Implementing createDeploymentPlan
joehan Jan 26, 2021
b876523
First round of PR feedback, removing most usages of lodash
joehan Jan 28, 2021
9e0e6e9
moving function prompts into their own file
joehan Jan 28, 2021
2a3b547
seperating out a bunch of code from functionsDeployHelper
joehan Jan 28, 2021
51f2395
Resolves merge conflicts
joehan Jan 28, 2021
30cc0e9
refactoring release step of functions deploy to use typescript (#3071)
joehan Feb 1, 2021
6916000
Implements core logic of running function deploys
joehan Feb 1, 2021
3c8d4a0
Typescriptifying prepareFunctionsUpload (#3064)
joehan Feb 1, 2021
11956fa
Implementing createDeploymentPlan (#3081)
joehan Feb 1, 2021
85d0afe
adding timing and logs for deployments
joehan Feb 2, 2021
00b1989
cleaning up unused code
joehan Feb 2, 2021
397d7c4
Fixing some things that were broken while merging
joehan Feb 3, 2021
21f4906
Fixing up the order of wait and close to ensure that queue promsies a…
joehan Feb 4, 2021
3b3edbd
Format and clean up typos
joehan Feb 4, 2021
e428bcb
refactoring error handling to be cleaner
joehan Feb 5, 2021
4c8e2fb
cleaning up extera newlines
joehan Feb 8, 2021
7f48130
first round of pr fixes
joehan Feb 9, 2021
39a7e86
Readding some changes that I accidenttally wiped out during a merge
joehan Feb 9, 2021
1366955
Switching name to id where appropriate
joehan Feb 9, 2021
7513229
fixing another bug caused by functionName vs Id
joehan Feb 9, 2021
8d3d82d
Merge pull request #3107 from firebase/jh-execute-deployment-plans
joehan Feb 9, 2021
6d2260e
Refactor functions-delete (#3110)
joehan Feb 9, 2021
42e6c15
Cleaning up error reporting
joehan Feb 10, 2021
e4ce126
Merge remote-tracking branch 'public/master' into jh-functions-refactor
joehan Feb 10, 2021
12a48ea
Merge remote-tracking branch 'public/master' into jh-functions-refactor
joehan Feb 11, 2021
7cfe9d9
Implement validation for changing trigger types, and fixes from bug b…
joehan Feb 12, 2021
5eb08bd
Merge branch 'master' into jh-functions-refactor
joehan Feb 12, 2021
5ca6bbf
Merge branch 'master' into jh-functions-refactor
joehan Feb 16, 2021
344b674
fixes package.json
joehan Feb 16, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Next Next commit
cleaning up unused code
  • Loading branch information
joehan committed Feb 2, 2021
commit 00b198912f9c2932cd3a2bc6ad77a9c3dfbaad60
38 changes: 23 additions & 15 deletions src/deploy/functions/errorHandler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@ import * as clc from "cli-color";
import * as logger from "../../logger";
import { FirebaseError } from "../../error";

type OperationType = "create" |
"update" |
"delete" |
"upsert schedule" |
"delete schedule" |
"make public";
type OperationType =
| "create"
| "update"
| "delete"
| "upsert schedule"
| "delete schedule"
| "make public";

type Level = "error" | "warning";

Expand All @@ -27,15 +28,14 @@ export class ErrorHandler {
functionName,
operationType,
message,
}
};
if (level === "error") {
this.errors.push(info);
} else if (level === "warning") {
this.warnings.push(info);
}
}

// TODO: Make this print prettier.
printErrors() {
if (this.errors.length === 0) {
return;
Expand All @@ -50,36 +50,44 @@ export class ErrorHandler {
clc.bold("firebase deploy --only ") +
clc.bold('"') +
clc.bold(
this.errors.map((failedDep) => `functions:${failedDep.functionName.replace(/-/g, ".")}`).join(",")
this.errors
.map((failedDep) => `functions:${failedDep.functionName.replace(/-/g, ".")}`)
.join(",")
) +
clc.bold('"')
);
logger.info("\n\nTo continue deploying other features (such as database), run:");
logger.info(" " + clc.bold("firebase deploy --except functions"));
// Print all the original messages at debug level.
for (const failedDep of this.errors) {
logger.debug(`\tError during ${failedDep.operationType} for ${failedDep.functionName}: ${failedDep.message}`);
logger.debug(
`\tError during ${failedDep.operationType} for ${failedDep.functionName}: ${failedDep.message}`
);
}
throw new FirebaseError("Functions did not deploy properly.");
}

printWarnings() {
if (this.warnings.length === 0 ) {
if (this.warnings.length === 0) {
return;
}
const failedIamCalls = this.warnings.filter(e => e.operationType === "make public");
const failedIamCalls = this.warnings.filter((e) => e.operationType === "make public");
if (failedIamCalls.length) {
logger.info("\n\nUnable to set publicly accessible IAM policy on the following functions:");
for (const failedDep of failedIamCalls) {
logger.info(`\t${failedDep.functionName}`);
}
logger.info("\n\nUnauithorized users will not be able access this function.");
logger.info("\n\nThis may be caused by an organization policy that restrict Network Access on your project.");
logger.info(
"\n\nThis may be caused by an organization policy that restrict Network Access on your project."
);
}

// Print all the original messages at debug level.
for (const failedDep of this.warnings) {
logger.debug(`\tWarning during${failedDep.operationType} for ${failedDep.functionName}: ${failedDep.message}`);
logger.debug(
`\tWarning during${failedDep.operationType} for ${failedDep.functionName}: ${failedDep.message}`
);
}
}
}
}
8 changes: 5 additions & 3 deletions src/deploy/functions/prepare.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,11 @@ export async function prepare(context: any, options: any, payload: any): Promise
check(projectId, "runtimeconfig.googleapis.com", "runtimeconfig", true),
checkRuntimeDependencies(projectId, context.runtimeChoice),
]);
context["runtimeConfigEnabled"] = checkAPIsEnabled[1];
context.runtimeConfigEnabled = checkAPIsEnabled[1];

// Get the Firebase Config, and set it on each function in the deployment.
const firebaseConfig = await functionsConfig.getFirebaseConfig(options);
context["firebaseConfig"] = firebaseConfig;
context.firebaseConfig = firebaseConfig;

// Prepare the functions directory for upload, and set context.triggers.
logBullet(
Expand All @@ -47,7 +47,7 @@ export async function prepare(context: any, options: any, payload: any): Promise
" directory for uploading..."
);
const source = await prepareFunctionsUpload(context, options);
context["functionsSource"] = source;
context.functionsSource = source;

// Get a list of CloudFunctionTriggers, and set default environemnt variables on each.
const defaultEnvVariables = {
Expand All @@ -69,6 +69,8 @@ export async function prepare(context: any, options: any, payload: any): Promise

// Build a regionMap, and duplicate functions for each region they are being deployed to.
payload.functions = {};
// TODO: Make byRegion an implementation detail of deploymentPlanner
// and only store a flat array of Functions in payload.
payload.functions.byRegion = functionsByRegion(projectId, functions);
payload.functions.triggers = allFunctions(payload.functions.byRegion);

Expand Down
8 changes: 7 additions & 1 deletion src/deploy/functions/prompts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { FirebaseError } from "../../error";
import { promptOnce } from "../../prompt";
import * as utils from "../../utils";
import * as logger from "../../logger";

/**
* Checks if a deployment will create any functions with a failure policy.
* If there are any, prompts the user to acknowledge the retry behavior.
Expand Down Expand Up @@ -53,7 +54,12 @@ export async function promptForFailurePolicies(
throw new FirebaseError("Deployment canceled.", { exit: 1 });
}
}

/**
* Checks if a deployment will delete any functions.
* If there are any, prompts the user if they should be deleted or not.
* @param options
* @param functions A list of functions to be deleted.
*/
export async function promptForFunctionDeletion(
functionsToDelete: string[],
force: boolean,
Expand Down
18 changes: 8 additions & 10 deletions src/deploy/functions/release.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import * as clc from "cli-color";

import * as utils from "../../utils";
import * as helper from "../../functionsDeployHelper";
import { createDeploymentPlan } from "./deploymentPlanner";
import { createDeploymentPlan } from "./deploymentPlanner";
import * as tasks from "./tasks";
import { getAppEngineLocation } from "../../functionsConfig";
import { promptForFunctionDeletion } from "./prompts";
Expand Down Expand Up @@ -52,7 +52,7 @@ export async function release(context: any, options: any, payload: any) {
cloudFunctionsQueue
.run(tasks.deleteFunctionTask(retryFuncParams, fnName))
.then(() => {
helper.printSuccess(fnName, 'delete');
helper.printSuccess(fnName, "delete");
})
.catch((err) => {
errorHandler.record("error", fnName, "delete", err.message || "");
Expand All @@ -71,19 +71,17 @@ export async function release(context: any, options: any, payload: any) {

for (const regionalDeployment of fullDeployment.regionalDeployments) {
// Run the create and update function calls for the region.
regionPromises.push(tasks.runRegionalDeployment(retryFuncParams, regionalDeployment, cloudFunctionsQueue));
regionPromises.push(
tasks.runRegionalFunctionDeployment(retryFuncParams, regionalDeployment, cloudFunctionsQueue)
);

// Add scheduler creates and updates to their queue.
for (const fn of regionalDeployment.schedulesToUpsert) {
const task = tasks.upsertScheduleTask(
retryFuncParams,
fn,
appEngineLocation
);
const task = tasks.upsertScheduleTask(retryFuncParams, fn, appEngineLocation);
schedulerQueue
.run(task)
.then(() => {
helper.printSuccess(fn.name, 'upsert schedule');
helper.printSuccess(fn.name, "upsert schedule");
})
.catch((err) => {
errorHandler.record("error", fn.name, "upsert schedule", err.message || "");
Expand All @@ -97,7 +95,7 @@ export async function release(context: any, options: any, payload: any) {
schedulerQueue
.run(tasks.deleteScheduleTask(fnName, appEngineLocation))
.then(() => {
helper.printSuccess(fnName, 'delete schedule');
helper.printSuccess(fnName, "delete schedule");
})
.catch((err) => {
errorHandler.record("error", fnName, "delete schedule", err.message || "");
Expand Down
80 changes: 36 additions & 44 deletions src/deploy/functions/tasks.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import * as _ from "lodash";
import * as clc from "cli-color";

import * as logger from "../../logger";
Expand Down Expand Up @@ -55,7 +54,7 @@ export function createFunctionTask(
functionName: helper.getFunctionName(fn.name),
entryPoint: fn.entryPoint,
trigger: helper.getFunctionTrigger(fn),
labels: _.assign({}, deploymentTool.labels(), fn.labels),
labels: Object.assign({}, deploymentTool.labels(), fn.labels),
sourceUploadUrl: params.sourceUrl,
sourceToken: params.sourceToken,
runtime: params.runtime,
Expand All @@ -67,7 +66,7 @@ export function createFunctionTask(
vpcConnectorEgressSettings: fn.vpcConnectorEgressSettings,
serviceAccountEmail: fn.serviceAccountEmail,
});
const pollerOptions: OperationPollerOptions = _.assign(
const pollerOptions: OperationPollerOptions = Object.assign(
{
pollerName: `create-${fn.name}`,
operationResourceName: createRes.name,
Expand All @@ -85,8 +84,6 @@ export function createFunctionTask(
policy: cloudfunctions.DEFAULT_PUBLIC_POLICY,
});
} catch (err) {
logger.debug(err);
// TODO: Better warning language when we can't set IAM policy to make functions public?
params.errorHandler.record("warning", fn.name, "make public", err.message);
}
}
Expand Down Expand Up @@ -118,7 +115,7 @@ export function updateFunctionTask(
functionName: helper.getFunctionName(fn.name),
entryPoint: fn.entryPoint,
trigger: helper.getFunctionTrigger(fn),
labels: _.assign({}, deploymentTool.labels(), fn.labels),
labels: Object.assign({}, deploymentTool.labels(), fn.labels),
sourceUploadUrl: params.sourceUrl,
sourceToken: params.sourceToken,
runtime: params.runtime,
Expand All @@ -130,7 +127,7 @@ export function updateFunctionTask(
vpcConnectorEgressSettings: fn.vpcConnectorEgressSettings,
serviceAccountEmail: fn.serviceAccountEmail,
});
const pollerOptions: OperationPollerOptions = _.assign(
const pollerOptions: OperationPollerOptions = Object.assign(
{
pollerName: `update-${fn.name}`,
operationResourceName: updateRes.name,
Expand All @@ -156,7 +153,7 @@ export function deleteFunctionTask(params: RetryFunctionParams, fnName: string)
const deleteRes = await cloudfunctions.deleteFunction({
functionName: fnName,
});
const pollerOptions: OperationPollerOptions = _.assign(
const pollerOptions: OperationPollerOptions = Object.assign(
{
pollerName: `delete-${fnName}`,
operationResourceName: deleteRes.name,
Expand Down Expand Up @@ -189,14 +186,18 @@ export function deleteScheduleTask(fnName: string, appEngineLocation: string) {
};
}


export function runRegionalDeployment(
/**
*
* @param params
* @param regionalDeployment
* @param queue
*/
export function runRegionalFunctionDeployment(
params: RetryFunctionParams,
regionalDeployment: RegionalDeployment,
queue: Queue<any, any>
): Promise<any> {

// Build an onPoll function to check for sourceToken and queue up the rest of the deployment.
// Build an onPoll function to check for sourceToken and queue up the rest of the deployment.
const onPollFn = (op: any) => {
// We should run the rest of the regional deployment if we either:
// - Have a sourceToken to use.
Expand All @@ -209,69 +210,60 @@ export function runRegionalDeployment(
`Got sourceToken ${op.metadata.sourceToken} for region ${regionalDeployment.region}`
);
regionalDeployment.sourceToken = op.metadata.sourceToken;
finishRegionalDeployment(
params,
regionalDeployment,
queue,
);
finishRegionalFunctionDeployment(params, regionalDeployment, queue);
}
}
};
// Choose a first function to deploy.
if (regionalDeployment.functionsToCreate.length) {
const firstFn = regionalDeployment.functionsToCreate.shift()!;
const task = createFunctionTask(
params,
firstFn!,
onPollFn
);
return queue.run(task)
const task = createFunctionTask(params, firstFn!, onPollFn);
return queue
.run(task)
.then(() => {
helper.printSuccess(firstFn.name, 'create');
helper.printSuccess(firstFn.name, "create");
})
.catch((err) => {
params.errorHandler.record("error", firstFn.name, "create", err.message || "")
params.errorHandler.record("error", firstFn.name, "create", err.message || "");
});
} else if (regionalDeployment.functionsToUpdate.length) {
const firstFn = regionalDeployment.functionsToUpdate.shift()!;
const task = updateFunctionTask(
params,
firstFn!,
onPollFn
);
return queue.run(task)
const task = updateFunctionTask(params, firstFn!, onPollFn);
return queue
.run(task)
.then(() => {
helper.printSuccess(firstFn.name, 'update');
helper.printSuccess(firstFn.name, "update");
})
.catch((err) => {
params.errorHandler.record("error", firstFn.name, "update", err.message || "")
params.errorHandler.record("error", firstFn.name, "update", err.message || "");
});
}
// If there are no functions to create or update in this region, no need to do anything.
return Promise.resolve();
}

export function finishRegionalDeployment(
function finishRegionalFunctionDeployment(
params: RetryFunctionParams,
regionalDeployment: RegionalDeployment,
queue: Queue<any, any>
) {
for (const fn of regionalDeployment.functionsToCreate) {
queue.run(createFunctionTask(params, fn))
.then(() => {
helper.printSuccess(fn.name, 'create');
})
.catch((err) => {
params.errorHandler.record("error", fn.name, "create", err.message || "")
});
queue
.run(createFunctionTask(params, fn))
.then(() => {
helper.printSuccess(fn.name, "create");
})
.catch((err) => {
params.errorHandler.record("error", fn.name, "create", err.message || "");
});
}
for (const fn of regionalDeployment.functionsToUpdate) {
queue
.run(updateFunctionTask(params, fn))
.then(() => {
helper.printSuccess(fn.name, 'update');
helper.printSuccess(fn.name, "update");
})
.catch((err) => {
params.errorHandler.record("error", fn.name, "update", err.message || "")
params.errorHandler.record("error", fn.name, "update", err.message || "");
});
}
}
3 changes: 2 additions & 1 deletion src/functionsDelete.js
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ var printSuccess = function (op) {
var printFail = function (op) {
failedDeployments += 1;
utils.logWarning(
clc.bold.yellow("functions[" + helper.getFunctionLabel(op.funcName) + "]: ") + "Deployment error."
clc.bold.yellow("functions[" + helper.getFunctionLabel(op.funcName) + "]: ") +
"Deployment error."
);
if (op.error.code === 8) {
logger.debug(op.error.message);
Expand Down