Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major refactor of functions deploy #3132

Merged
merged 37 commits into from
Feb 16, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
06b62a1
Adding onPoll option to operation-poller (#3046)
joehan Jan 19, 2021
6258dd5
Typescriptify functionsDeployHelper (#3059)
joehan Jan 20, 2021
ec7d079
Typescriptifying gcp.cloudfunctions (#3060)
joehan Jan 20, 2021
f583ef6
Typescriptifying functionsConfig (#3063)
joehan Jan 21, 2021
08b9d56
Typescriptifying deploymentTool (#3061)
joehan Jan 21, 2021
b4944a4
Refactoring prepare stage of functions deploy (#3067)
joehan Jan 21, 2021
3be0dca
refactoring release step of functions deploy to use typescript
joehan Jan 21, 2021
e0e703e
Adding logic to build regional deployments
joehan Jan 24, 2021
046c7d7
Implementing createDeploymentPlan
joehan Jan 26, 2021
b876523
First round of PR feedback, removing most usages of lodash
joehan Jan 28, 2021
9e0e6e9
moving function prompts into their own file
joehan Jan 28, 2021
2a3b547
seperating out a bunch of code from functionsDeployHelper
joehan Jan 28, 2021
51f2395
Resolves merge conflicts
joehan Jan 28, 2021
30cc0e9
refactoring release step of functions deploy to use typescript (#3071)
joehan Feb 1, 2021
6916000
Implements core logic of running function deploys
joehan Feb 1, 2021
3c8d4a0
Typescriptifying prepareFunctionsUpload (#3064)
joehan Feb 1, 2021
11956fa
Implementing createDeploymentPlan (#3081)
joehan Feb 1, 2021
85d0afe
adding timing and logs for deployments
joehan Feb 2, 2021
00b1989
cleaning up unused code
joehan Feb 2, 2021
397d7c4
Fixing some things that were broken while merging
joehan Feb 3, 2021
21f4906
Fixing up the order of wait and close to ensure that queue promsies a…
joehan Feb 4, 2021
3b3edbd
Format and clean up typos
joehan Feb 4, 2021
e428bcb
refactoring error handling to be cleaner
joehan Feb 5, 2021
4c8e2fb
cleaning up extera newlines
joehan Feb 8, 2021
7f48130
first round of pr fixes
joehan Feb 9, 2021
39a7e86
Readding some changes that I accidenttally wiped out during a merge
joehan Feb 9, 2021
1366955
Switching name to id where appropriate
joehan Feb 9, 2021
7513229
fixing another bug caused by functionName vs Id
joehan Feb 9, 2021
8d3d82d
Merge pull request #3107 from firebase/jh-execute-deployment-plans
joehan Feb 9, 2021
6d2260e
Refactor functions-delete (#3110)
joehan Feb 9, 2021
42e6c15
Cleaning up error reporting
joehan Feb 10, 2021
e4ce126
Merge remote-tracking branch 'public/master' into jh-functions-refactor
joehan Feb 10, 2021
12a48ea
Merge remote-tracking branch 'public/master' into jh-functions-refactor
joehan Feb 11, 2021
7cfe9d9
Implement validation for changing trigger types, and fixes from bug b…
joehan Feb 12, 2021
5eb08bd
Merge branch 'master' into jh-functions-refactor
joehan Feb 12, 2021
5ca6bbf
Merge branch 'master' into jh-functions-refactor
joehan Feb 16, 2021
344b674
fixes package.json
joehan Feb 16, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Next Next commit
Format and clean up typos
  • Loading branch information
joehan committed Feb 4, 2021
commit 3b3edbd103347c73e58d3a830738554b6a6abbb1
4 changes: 2 additions & 2 deletions src/deploy/functions/errorHandler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,9 @@ export class ErrorHandler {
for (const failedDep of failedIamCalls) {
logger.info(`\t${failedDep.functionName}`);
}
logger.info("\n\nUnauithorized users will not be able access this function.");
logger.info("\n\nUnauthorized users will not be able access this function.");
logger.info(
"\n\nThis may be caused by an organization policy that restrict Network Access on your project."
"\n\nThis may be caused by an organization policy that restricts Network Access on your project."
);
}

Expand Down
21 changes: 8 additions & 13 deletions src/deploy/functions/release.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ export async function release(context: any, options: any, payload: any) {
context.existingFunctions,
context.filters
);
const cloudFunctionsQueue = new Queue<() => Promise<CloudFunctionTrigger|void>, void>({});
const cloudFunctionsQueue = new Queue<() => Promise<CloudFunctionTrigger | void>, void>({});
const schedulerQueue = new Queue<() => Promise<any>, void>({});
const regionPromises = [];

const retryFuncParams: tasks.RetryFunctionParams = {
const taskParams: tasks.TaskParams = {
projectId,
sourceUrl,
runtime: context.runtimeChoice,
Expand All @@ -50,7 +50,7 @@ export async function release(context: any, options: any, payload: any) {
if (shouldDeleteFunctions) {
for (const fnName of fullDeployment.functionsToDelete) {
cloudFunctionsQueue
.run(tasks.deleteFunctionTask(retryFuncParams, fnName))
.run(tasks.deleteFunctionTask(taskParams, fnName))
.then(() => {
helper.printSuccess(fnName, "delete");
})
Expand All @@ -72,12 +72,12 @@ export async function release(context: any, options: any, payload: any) {
for (const regionalDeployment of fullDeployment.regionalDeployments) {
// Run the create and update function calls for the region.
regionPromises.push(
tasks.runRegionalFunctionDeployment(retryFuncParams, regionalDeployment, cloudFunctionsQueue)
tasks.runRegionalFunctionDeployment(taskParams, regionalDeployment, cloudFunctionsQueue)
);

// Add scheduler creates and updates to their queue.
for (const fn of regionalDeployment.schedulesToUpsert) {
const task = tasks.upsertScheduleTask(retryFuncParams, fn, appEngineLocation);
const task = tasks.upsertScheduleTask(taskParams, fn, appEngineLocation);
schedulerQueue
.run(task)
.then(() => {
Expand All @@ -100,25 +100,20 @@ export async function release(context: any, options: any, payload: any) {
}

// Once everything has been added to queues, starting processing.
const queuePromises = [cloudFunctionsQueue.wait(), schedulerQueue.wait()];
// Note: We need to set up these wait before calling process and close.
const queuePromises = [cloudFunctionsQueue.wait(), schedulerQueue.wait()];
cloudFunctionsQueue.process();
schedulerQueue.process();
schedulerQueue.close();

// Wait until the second round of creates/updates are added to the queue before closing it.
await Promise.all(regionPromises)
await Promise.all(regionPromises);
cloudFunctionsQueue.close();

// Wait for the first function in each region to be deployed, and all the other calls to be queued,
// then close the queue.
// Wait for all of the deployments to complete.
await Promise.all(queuePromises);
// TODO: We should also await the scheduler queue. However, for reasons I don't understand,
// awaiting 2 queues makes it so none of the code below this executes.
// If I remove either queue, it works correctly.
// Not sure if this is a bug with queue or if I'm doing something subtly incorrect, but functions deploys are ~2 orders of magnitude longer
// than schedule deployments, and there are no deployments that contain only calls to scheduler, so this works for now.
// await schedulerQueue.wait();
helper.logAndTrackDeployStats(cloudFunctionsQueue);
errorHandler.printWarnings();
errorHandler.printErrors();
Expand Down
16 changes: 8 additions & 8 deletions src/deploy/functions/tasks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ const defaultPollerOptions = {
masterTimeout: 150000,
};

export interface RetryFunctionParams {
export interface TaskParams {
projectId: string;
runtime: string;
sourceUrl: string;
Expand All @@ -32,7 +32,7 @@ export interface RetryFunctionParams {
}

export function createFunctionTask(
params: RetryFunctionParams,
params: TaskParams,
fn: CloudFunctionTrigger,
onPoll?: (op: any) => any
): () => Promise<CloudFunctionTrigger> {
Expand Down Expand Up @@ -93,7 +93,7 @@ export function createFunctionTask(
}

export function updateFunctionTask(
params: RetryFunctionParams,
params: TaskParams,
fn: CloudFunctionTrigger,
onPoll?: (op: any) => any
): () => Promise<CloudFunctionTrigger> {
Expand Down Expand Up @@ -141,7 +141,7 @@ export function updateFunctionTask(
};
}

export function deleteFunctionTask(params: RetryFunctionParams, fnName: string) {
export function deleteFunctionTask(params: TaskParams, fnName: string) {
return async () => {
utils.logBullet(
clc.bold.cyan("functions: ") +
Expand All @@ -167,7 +167,7 @@ export function deleteFunctionTask(params: RetryFunctionParams, fnName: string)
}

export function upsertScheduleTask(
params: RetryFunctionParams,
params: TaskParams,
fn: CloudFunctionTrigger,
appEngineLocation: string
): () => Promise<any> {
Expand Down Expand Up @@ -209,7 +209,7 @@ export function deleteScheduleTask(fnName: string, appEngineLocation: string): (
* @param queue
*/
export function runRegionalFunctionDeployment(
params: RetryFunctionParams,
params: TaskParams,
regionalDeployment: RegionalDeployment,
queue: Queue<() => Promise<any>, void>
): Promise<void> {
Expand Down Expand Up @@ -258,9 +258,9 @@ export function runRegionalFunctionDeployment(
}

function finishRegionalFunctionDeployment(
params: RetryFunctionParams,
params: TaskParams,
regionalDeployment: RegionalDeployment,
queue: Queue<()=>Promise<any>, void>
queue: Queue<() => Promise<any>, void>
): void {
for (const fn of regionalDeployment.functionsToCreate) {
queue
Expand Down