Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Onboard New Fec dataset #486

Merged
merged 23 commits into from
Oct 14, 2022
Merged

feat: Onboard New Fec dataset #486

merged 23 commits into from
Oct 14, 2022

Conversation

gkodukula
Copy link
Contributor

@gkodukula gkodukula commented Sep 29, 2022 •
edited

Description

Dataset: FEC

  • FEC Committee Contributions
  • 2016 FEC Candidate Committee
  • 2016 FEC Candidate
  • 2016 FEC Opex
  • 2016 Other Committee Tx
  • 2016 FEC Committee
  • 2016 FEC Individuals
  • 2018 FEC Committee
  • 2018 FEC Opex
  • 2018 FEC Committee Contributions
  • 2018 FEC Candidate Committee
  • 2018 FEC Other Committee Tx
  • 2018 FEC Candidate
  • 2020 FEC Candidate Committee
  • 2020 FEC Candidate
  • 2020 FEC Other Committee Tx
  • 2020 FEC Committee
  • 2020 FEC Committee Contributions
  • 2020 FEC Opex
  • 2020 FEC Individuals - Step 3: Ingest

Checklist

  • (Required) This pull request is appropriately labeled
  • Please merge this pull request after it's approved

Use the sections below based on what's applicable to your PR and delete the rest:

Feature

  • I'm adding or editing a feature
  • I have updated the README accordingly
  • I have added/revised tests for the feature

Data Onboarding

  • I'm adding or editing a dataset
  • The Google Cloud Datasets team is aware of the proposed dataset
  • I put all my code inside datasets/fec and nothing outside of that directory

gkodukula and others added 19 commits August 12, 2022 20:26
@gkodukula
fix: lint issue, production ready
6702bd5
@gkodukula
Merge remote-tracking branch 'upstream/main' into uniref50
00a81ab
@gkodukula
fix: removing terraform files
37e741f
@gkodukula
feat: onboarding fec dataset
9e31ec5
@gkodukula
fix: fixing csv transform
4a0b5a3
@gkodukula
fix: fixed csv transform file
2001597
@gkodukula
fix: fixed write truncate issue
13d82f2
@gkodukula
fix: fixed individuals ingest pipeline
d77815e
@gkodukula
fix: fixing code for individuals ingest pipeline
91c2cf4
@gkodukula
fix: changes in pipeline.yaml
dc509c0
@gkodukula
fix: fixing individuals pipeline
6fcf1b7
@gkodukula
feat: onboardinf fec dataset , production ready
874238d
@gkodukula
Merge remote-tracking branch 'upstream/main' into new_fec
c426fdd
@gkodukula
feat: onboarding fec dataset , production ready
d137e8b
@gkodukula
feat: onboarding fec dataset , production ready
ecdf8dd
@gkodukula
fix: fixed failing pipelines
f3ad568
@gkodukula
Merge branch 'main' into new_fec
f703086
@gkodukula
Update committee_contributions_2020_dag.py
f2dd5af
@gkodukula
Update individuals_2016_dag.py
8573826
@nlarge-google
Copy link
Collaborator

nlarge-google commented Sep 29, 2022 •
edited

Gowtham, there are multiple issues with the transform_csv python file:

(1) Please remove ALL blank lines between code. There should be no inter-function blank lines.
(2) process_dataframe_chunk has a list of columns hard coded into it. These should be in the pipeline.yaml file, not in the python file.
(3) Ensure ALL functions have a return type. If there is no return type specify "None"
(4) fill_null_values has one line of code in it. simply replace the calling function entry with the code in the function and remove the function itself. Same for rename_headers.

nlarge-google
nlarge-google requested changes Sep 29, 2022
View reviewed changes
Copy link
Collaborator

@nlarge-google nlarge-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As specified in my comment

nlarge-google
nlarge-google approved these changes Oct 14, 2022
View reviewed changes
Copy link
Collaborator

@nlarge-google nlarge-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Gowtham.

@nlarge-google nlarge-google merged commit 6ee1fa3 into GoogleCloudPlatform:main Oct 14, 2022
@release-please release-please bot mentioned this pull request Oct 14, 2022
Merged
@nlarge-google
Copy link
Collaborator

@gkodukula This is now merged and deployed to production. Please ensure execution cycle completes successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nlarge-google nlarge-google nlarge-google approved these changes

Assignees

@gkodukula gkodukula

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@gkodukula @nlarge-google