Iago F.Data Engineer
Skills
Data Engineer specializing in the construction and maintenance of data processing architectures utilizing cloud services. Proficient in developing data ingestion and processing algorithms with Python and PySpark, and skilled in the design and modeling of Data Lakes and Data Warehouses to facilitate data storage and analysis.
Data Engineer
8/1/2022 - 11/1/2023
Constructed an integrated data ingestion architecture from various APIs, achieving a 600% cost reduction. Tasks included sizing virtual machines, developing API data ingestion scripts, modeling Data Lake for raw data storage, and designing Data Warehouse to handle refined data layers. Security policies for data access control were implemented, with credential management facilitated through Secret Manager. Enabled instant data consumption using Datastream and orchestrated data ingestion flows via Cloud Composer. Utilized technologies include Python, SQL, GCP Compute Engine, GCP Cloud Storage, GCP Big Query, Cloud Composer, Secret Manager, Datastream, IAM, Billing, Git, and Airflow.
Migrated an on-premises database to an AWS-based Data Warehouse and Data Lake. Responsibilities included building data ingestion routines with Pentaho, storing raw data in the Data Lake, constructing ingestion layers for refined data, and modeling the Data Warehouse for refined data delivery. Credential management was performed via Secret Manager, while monitoring and orchestration of ingestion flows were managed with Airflow. Key technologies used were Pyspark, SQL, Databricks, AWS S3, AWS RDS, AWS Secret Manager, Git, Airflow, Pentaho, and Oracle Database.Technology Intern
5/1/2021 - 12/1/2021
Assisted in managing teams responsible for the migration of infrastructure from an on-premise environment to the Cloud. Supported Agile development of teams, ensuring effective communication through the management of various communication channels. Contributed to the construction of metric dashboards to track progress and performance.ETL Developer
6/1/2020 - 7/1/2022
Developed proficiency in the creation of ETL flows to consume APIs related to user behavior on videos from an investment bank and socioeconomic factors from DHIS2. Gained extensive experience in developing Python algorithms for data ingestion and processing, employing Pandas for data modeling and analysis. Integrated processed data with AWS S3 for storage in CSV format and constructed robust dashboards using Power BI for data visualization. Demonstrated competence in utilizing SQL Server for relational database integration and storage. Key technologies and tools include Python, Pandas, AWS S3, SQL Server, and Power BI.
Electrical Engineering at Federal University of Itajubá
2016 - 2023
Iago is available for hire
Hire Iago F.
All Howdy Candidates are vetted for skills and english proficiency.
