Software Engineer - Data Engineer (Scala / Streaming Data Pipelines)
Description:
Employment Type: Full-time
Position Overview
Sharp InfoTech Inc is seeking a Data Engineer to support the development and maintenance of scalable data pipelines that ingest, transform, and deliver company data to internal stakeholders, reporting teams, and external partners.
The ideal candidate will have strong experience working with streaming data platforms, distributed data processing frameworks, and AWS cloud technologies, helping to ensure reliable and efficient data flow across the organization.
Key Responsibilities
Design, develop, and maintain scalable data pipelines that ingest, transform, and distribute company data to internal and external stakeholders.
Build and maintain real-time and batch data pipelines for hourly data processing.
Implement new data exports and update business logic for multiple data feeds.
Contribute to the development of large-scale streaming data pipeline infrastructure.
Ensure data quality, integrity, and reliability throughout the ingestion and processing lifecycle.
Collaborate with internal reporting teams and external partners to support data distribution requirements.
Typical Day-to-Day
Participate in daily morning stand-up meetings (15 minutes) and developer sync meetings with the team (10–30 minutes).
Work independently on assigned tasks with minimal meetings outside of weekly team syncs.
Work in an Agile development environment with 2-week sprint cycles.
Production releases occur every Tuesday during working hours.
Participate in on-call support rotation approximately once every six weeks:
Monday–Friday, 8:00 AM – 7:00 PM MST
Required Skills
Hands-on experience with Scala programming.
Experience working with Apache Kafka for real-time message streaming.
Knowledge of Spark Structured Streaming using Scala.
Understanding of AWS cloud technologies, including:
Amazon S3
Athena
EC2
IAM
EMR
Experience with Git version control systems.
Experience developing in a Mac-based environment.
Preferred Skills
Familiarity with Terraform for infrastructure automation.
Experience with Kubernetes and containerized applications.
Experience handling large-scale datasets and distributed systems.
Comfortable using SQL for data analysis and exploration.
Education
Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.