Como você prioriza e resolve os gargalos que afetam a experiência do usuário e as metas de negócios?

Start with the end user experience, as this is the ultimate metric. Combine this with additional metrics from monitoring. Determine the key metrics that can be cross-referenced with the end user responses to determine the general area of the bottleneck. It one tier is the culprit, dive deep with additional metrics that are more granular. Just keep in mind that just because a CPU is 100% utilized does not mean that it's a bad thing - first determine what affects the user experience and go from there.

I have used application performance monitoring tools such as Dynatrace, Datadog to collect data on key performance metrics such as CPU usage, memory utilization, disk I/O, network traffic, and application response times. Monitor these metrics over time to identify patterns, trends, and anomalies that may indicate performance bottlenecks.

I have used synthetic monitoring tools like Dynatrace/Microfocus BSM to monitor the application end user behavior .Develop the Vugen scripts for UI functionality and for API calls. These APM tools measures the end users experience using consistent , repeatable end-user transactions from multiple locations and proactively identifies availability and performance issues before end user report.

To address bottlenecks impacting user experience and business goals: Identify, prioritize, collaborate, use data, iterate, monitor, and adapt.

From my experience, With 1000's of KPI's available available with the APM tools. It is easy to lose focus and may be overwhelming at times. Brendan Gregg's USE methodology which worked very well in the recent past for me . If the system is not yielding expected performance . For every Resource (CPU, I/O, Memory, N/W etc) used by the application architecture / workflow, review U - Utilization (High utilization >70% of a resource may tend to show that resource is overwhelmed and not able to keep up with demand. Probably queue building up ). S - Saturation (Slow response time. Typically more wait time than service time) E - Error Count (typically Service Unavailable errors)

Correctly Identifying the bottleneck is a crucial step. Root cause analysis needs to be done post that. This can involve collecting data from analytical tools or time series data from kafka, prometheus and monitoring performance metrics. Next would be understand from usability point of view and hence user feedback. Then Use tools like Lighthouse or WebPageTest to perform audits and identify areas for improvement (e.g., slow loading resources, unoptimized images).

Comparatively it is easier to find the root cause during a hardware contention (on CPU / I/O , N/W, Memory) as it will directly show up with 1000's of available metrics. Root Causes for S/W contention are harder . It needs proper logging, may involve profiling applications, reviewing architecture to find the root cause for S/W contentions

One cause of bottlenecks I've seen is queries that pull unexpectedly large amounts of data. For example, a Name + DOB + SSN query that pulls a test identity that had been duplicated many times. Always make sure to bound your queries and consider worst-case scenarios like these when writing application logic.

Prioritize by Impact: Evaluate how each bottleneck affects user experience (e.g., high bounce rates, low engagement) and business goals. Use a matrix to categorize issues based on their severity and frequency. Define Key Metrics: Establish KPIs related to user experience (e.g., page load time, user satisfaction scores) and business objectives. Analysis shall involve keen eye to detail. Analyze the logs as per timestamp or trace id and error or debug details. Tools like dynatrace, JvisualVm can point out the issues and underlying causes.

Once I've identified performance bottlenecks, prioritizing them is crucial to efficiently allocate resources and address the most critical issues first. Here's how I prioritize bottlenecks. Prioritize bottlenecks that directly impact the user experience, such as slow page load times, unresponsive UI elements, or frequent timeouts.

Usually I priortize in this order 1. Business critical errors / performance exceeding threshold. [Isolate / Profile / Address / Retest] 2. Bottlenecks that stops the system becoming unscalable or not meeting the expected capacity. 3. Other Slow Test Cases

Once provided with enough information, For H/W Contention - Components that couldn't keep up with the demand. For S/W Contention - Diagnosis results from Profiling, Memory analysis etc. These information should be provided to the responsible teams (Infrastructure / Dev / DBA etc) that will help to address and resolve the bottlenecks

As performance engineers we have to run iterations with the suggested optimization or fix before sharing with the stakeholders. So let's say we find that creating an index for a DB query works well for the related API, bringing down its response time significantly, we do not just share the insight for one API but we run the whole test again and monitor the overall effect. Other than that suggesting tweaks around db connection pool, xmx, xms, gc approach, infra level wise, cloud wise (aws alb v/s nlb) need to be thoroughly checked and then put forward not just for sake of recommending or as per previous experience or "industry good practices".

Retest the applied fix in isolation and in totality will help to ensure the issue is addressed and meets defined performance goals.