Data Platform Engineer
BrightAI
Salary undisclosed
Need a reasonable accommodation to apply or interview? Contact us.
JobMinglr uses automated technology to recommend jobs based on profile information and job preferences. Match Score does not determine eligibility for a position, prevent a user from viewing or applying to a job, or make hiring decisions on behalf of an employer.
Description
We are a high-growth company transforming how businesses operate by integrating AI, IoT, and cloud-native services into scalable, real-time platforms. As a Platform Data Engineer, you’ll play a critical role in building and maintaining the data infrastructure that powers our products, services, and insights.
You’ll join a multidisciplinary team focused on ingesting, processing, and managing massive streams of sensor and operational data across a wide array of devices—from drones and robots to industrial systems and smart environments.
Responsibilities
- Design, build, and maintain scalable, reliable, and high-throughput data ingestion pipelines for structured and semi-structured data.
- Implement robust and secure data lake and SQL-based storage architectures optimized for performance and cost.
- Develop and maintain internal tools and frameworks for data ingestion using Python, Golang, and SQL.
- Collaborate cross-functionally with Cloud, Edge, Product, and AI teams to define data contracts, schemas, and retention policies.
- Use AWS cloud infrastructure (including Argo Workflows, S3, Lambda, Glue, Kinesis, Athena, and RDS) to support end-to-end data workflows.
- Employ Infrastructure-as-Code (IaC) practices using Terraform to manage data platform infrastructure.
- Monitor data pipelines for quality, latency, and failures using tools such as CloudWatch, SumoLogic, or DataDog.
- Continuously optimize storage, partitioning, and query performance across large-scale datasets.
- Participate in architecture reviews and ensure the platform adheres to security, compliance, and best practice standards.
Skills and Qualifications
- 5+ years of professional experience in software engineering or data engineering.
- Strong programming skills in Python and Golang.
- Deep understanding of SQL and modern data lake architectures (e.g., using Parquet, Iceberg, or Delta Lake).
- Hands-on experience with AWS services including but not limited to: S3, Lambda, Glue, Kinesis, Athena, and RDS.
- Proficiency with Terraform for automating infrastructure deployment and management.
- Experience working with real-time or batch data ingestion at scale, and designing fault-tolerant ETL/ELT pipelines.
- Familiarity with event-driven architectures and messaging systems like Kafka or Kinesis.
- Strong debugging and optimization skills across cloud, network, and application layers.
- Excellent collaboration, communication, and documentation skills.
Bonus Points
- Experience working with time-series or IoT sensor data at industrial scale.
- Familiarity with analytics tools and data warehouse integration (e.g., Redshift, Snowflake).
- Exposure to gRPC and protobuf-based data contracts.
- Experience supporting ML pipelines and feature stores.
- Working knowledge of Kubernetes concepts.
- Prior startup experience and/or comfort working in fast-paced, iterative environments.
About this role
BrightAI is building real-time data platforms that integrate AI, IoT, and cloud services at scale, and this role sits at the heart of that infrastructure. You'd own the design and maintenance of data pipelines ingesting massive streams from industrial devices, sensors, and operational systems—architecting everything from ingestion through storage to query optimization across AWS. The work spans the full stack: building Python and Golang tools, designing data lake schemas, managing Terraform infrastructure, and collaborating with cloud, edge, product, and AI teams to define data contracts and ensure the platform meets security and performance standards.
This role demands 5+ years of software or data engineering experience with strong fundamentals in Python, Golang, and SQL, plus hands-on expertise across AWS services (S3, Lambda, Glue, Kinesis, Athena, RDS) and modern data architectures like Iceberg or Delta Lake. You'll need to be comfortable designing fault-tolerant ETL/ELT pipelines, working with event-driven systems, and optimizing performance at scale—debugging across cloud, network, and application layers. Experience with Terraform, real-time or batch ingestion, and messaging systems like Kafka rounds out the core requirements.
If you've worked with IoT or time-series data at industrial scale, touched ML pipelines or feature stores, or navigated Kubernetes and gRPC, you'd be well-positioned. The role is based in Palo Alto and requires in-office presence. It suits engineers who thrive in fast-paced environments and want to own critical infrastructure that directly powers a growing platform.
How this employer is doing
solid
- Hiring Mentioned
- Recently Raised Funding
In the news
BrightAI plans to hire more than 1,000 Bay Area workers to fix aging U.S. infrastructure, The Business Journals
This role's local market on JobMinglr
Pay for this role
The employer didn't post a pay range for this role. That usually means pay is set in negotiation, which favors whoever arrives with numbers. Check ranges on comparable Data Platform Engineer postings in Palo Alto, and analyze any offer before you accept it.
Before you apply, worth reading
How JobMinglr reads this job
Every listing here is scored against your profile before you apply: skills overlap, experience level, location and work arrangement, each weighted and explained. You see the score and the reasons, not just a list. How the matching works.