Nilson J.Data Engineer
Skills
Data Engineer with over three years of experience in Big Data, specializing in the creation and maintenance of Data Lakes, Data Lakehouses, Data Marts, and Pipelines. Proficient in executing ETL/ELT processes across various sources, including SQL, NoSQL, and APIs, and optimizing pipelines to enhance performance and reduce costs. Experience includes a focus on the logistics and finance sectors. Demonstrates expertise in employing data to deliver value, emphasizing collaboration with teams and interfaces for achieving results.
Holds certifications such as AWS Certified Solutions Architect – Associate and AWS Cloud Practitioner. Possesses a comprehensive skill set that includes proficiency in AWS services (S3, EC2, EMR, Glue, Lambda, DynamoDB, Athena), programming languages (Python, PySpark), database management systems (SQL Server, PostgreSQL, MongoDB), and workflow orchestration tools (Airflow). Experienced in using Infrastructure as Code tools (Terraform), version control systems (Git), and containerization technologies (Docker and Kubernetes management tools).
Data Engineer
11/1/2023 - Present
Led a customer database project for a major financial institution, handling billions of records. Leveraged expertise in ELT and ETL processes using PySpark within AWS Glue, optimizing PySpark scripts to enhance performance. Conducted comprehensive data modeling and managed cloud services via Terraform. Played a crucial role in creating and managing data architectures including Data Mesh, Data Lake, and Data Lakehouse, ensuring seamless data availability through AWS Athena. This initiative resulted in significant performance improvements, time optimization, and cost reductions for the organization.Data Engineer
4/1/2022 - 10/1/2023
Worked on a project focused on obtaining and analyzing information from a credit product to assess its continuity in the product portfolio. Executed and created One Big Table to enhance data distribution, improve process efficiency, and optimize consultation time in business areas. Developed proficiency in performing ETL and ELT using PySpark, with pipeline orchestration managed via Airflow. Ingested data into the Data Lake from SQLServer, NoSQL databases, and APIs, and created Data Marts for organized data management. Ensured data availability and optimized queries using Trino, while also validating data in the Data Lake. Demonstrated skills in code versioning and collaborative development using GitHub.Data Engineer
3/1/2021 - 4/1/2022
Participated in a project in the logistics sector focused on developing processes and a mindset for data-driven decision-making from the ground up. Conducted ETL and ELT processes using PySpark on AWS EMR and AWS Glue. Established and maintained cloud infrastructure on AWS through Terraform, ensuring data accessibility for the Data Science and Analysis teams by creating tables in AWS Athena via AWS Glue Crawler. Developed AWS Lambda functions with Python, automating tasks through AWS CloudWatch. Managed application databases on AWS DynamoDB and developed applications using Streamlit within Docker environments. Additionally, created and managed permissions in AWS IAM, contributing to secure and efficient cloud resource management.
Computer Networks at Promote Colleges
2012 - 2014Bachelor's Degree in Information Systems at Promote Colleges
2016 - 2018Postgraduate Degree in Data Science and Big Data at Pontifical Catholic University of Minas Gerais
2019 - 2021
AWS Solutions Architect - Associate at Amazon Web Services Training and Certification
4/1/2024AWS Cloud Practitioner at Amazon Web Services Training and Certification
2/1/2022
Nilson is available for hire
Hire Nilson J.
All Howdy Candidates are vetted for skills and english proficiency.
