ELK / ESS Engineer

Role: ELK / ESS Engineer

Location: McLean, VA / Plano, TX

JOB Description:

Key feature required include query tuning, indexing strategy, cluster monitoring, and troubleshooting, often using the ELK stack
Designing, implementing, and managing search and analytics solutions using Elasticsearch.
Responsibilities may include indexing large datasets, optimizing search queries, maintaining cluster performance, and ensuring data availability.
Single person in requirement backlog and direct interaction with client.

Key Responsibilities

Visualization Creation: Build and assemble interactive panels, charts, maps, and metrics using Kibana Lens to create comprehensive dashboards.
Data Analysis & Mapping: Design efficient time-series index mappings and data streams to ensure optimal data storage and retrieval.
Query Optimization: Utilize aggregations, date histograms, and filters (KQL) to analyze large datasets and ensure fast dashboard response times.
Alerting & Monitoring: Set up threshold-based alerts (Watcher) and monitor system health to provide actionable insights.
Dashboard Optimization: Tune dashboard panels for performance, implementing data retention policies (ILM) to maintain efficiency.
Cluster Management: Deploy, configure, and maintain Elasticsearch clusters on-premise or in cloud environments (AWS, Azure). [NICE TO HAVE]
Performance Optimization: Fine-tune query performance, index management, and shard allocation for large-scale data.  [NICE TO HAVE]
Data Integration: Develop pipelines for indexing data from various sources using Logstash or ingestion APIs.
Monitoring & Security: Monitor cluster health, maintain security protocols, and ensure data integrity.
Troubleshooting: Perform root cause analysis on performance bottlenecks and cluster failures 
Work with Development Teams: Collaborate with software engineers to implement search features and improve user experiences.
Provide Technical Support: Troubleshoot and resolve issues related to Elasticsearch performance, data integrity, and availability.
Knowledge of indexing strategies for high-volume data.
Experience in designing scalable, secure, and resilient search architectures.
Ability to work INDEPENDENTLY  in agile teams, collaborating with DevOps and Data Engineers
Connect AI assistants, agents & automations to your data w/the first managed MCP platform  [NICE TO HAVE]

Technical Skills

Expertise in ELK Stack: Proficient in Elasticsearch, Logstash, and Kibana.
Visualization Experience: Strong experience with Kibana visualization tools (Lens, Maps, Graph).
Data Modeling and Knowledge of JSON and REST APIs: Familiarity with JSON data format and RESTful API principles is crucial for interacting with Elasticsearch.
Querying: Proficiency in Kibana Query Language (KQL) and Elasticsearch aggregations.
Monitoring/Observability: Background in creating operational dashboards for log analysis or metric tracking. 
Familiarity with Elasticsearch Ecosystem: Knowledge of related tools like Kibana, Logstash, and Beats enhances the engineer’s ability to deliver complete solutions.
Basic Programming Skills: Proficiency in programming languages such as Python, Java, or Go is beneficial for automation and customization tasks.