Work through a test incident

This exercise follows a simulated checkout latency problem from its first alert to resolution. You will create a team, trigger an incident, check its email notification, record an investigation, hand off ownership, and close the incident. You will also test the required notes for resolution and reopening.

Allow about 20 minutes. The query uses a small inline table, so you do not need to ingest data or change a running service. The exercise creates real incident records and sends real email to the people you select.

Before you start

Use a test account with a working query service, alert evaluator, and messenger service. Stay in that account throughout the exercise.

You need:

• Account admin access, or both Logship.Alert.Editor and Logship.Incident.Manager. These let you create the alert, manage the test group, and update the incident. Global admins can also perform these steps.
• At least one user who belongs to the account and can receive Logship email. Use your own user for a solo exercise.
• Outbound email configured for the deployment or account. If it is not already working, ask an administrator to set the email endpoint, username, password, and from address. See System Settings: Email defaults and Account Settings: Email overrides.
• Logship.Application.FrontendUrl set to the frontend's absolute URL, such as https://app.example.com. A global admin sets this under Administration → System Settings. See Frontend URL for incident links.

If you want to test a handoff between two people, add both to the account and give the second person incident editor access. Both need access to their email inboxes.

1. Create the response team

1. Open Groups in your account and select Create Group.
2. Set Group name to Incident tutorial responders and Description to Test group for the incident tutorial.
3. Select Create Group to save it.
4. Under Members, use Add member to select yourself, then select Add Member. Add the second responder too if you are working with one.
5. Reload the page.

Check: The group still exists and lists each person you added. Everyone in this group will receive email while the group owns the incident. Group membership does not grant incident permissions; those are assigned separately.

For membership and permission details, see Group Management.

2. Check the test query

Open the account's query page and run:

datatable (service: string, latencyMs: int64)
[
    "checkout-tutorial", 1500
]
| where latencyMs > 1000

Check: The result contains exactly one row:

The row represents a service whose latency exceeds 1,000 ms. Later, you will change 1500 to 200 to simulate recovery. The where clause supplies the alert condition: a healthy service produces no rows.

See datatable for the inline table syntax.

3. Turn the query into an incident alert

Open Alerts, select Create Alert, and turn Enabled off while you fill in the form.

Set these values:

Expand Incident Configuration if its fields are hidden. The alert form calls the group selection an assignee, but it sets the incident's owner. A group-owned incident can have a separate individual assignee; this exercise uses group ownership first and a user owner at handoff.

The unique field keeps repeated results for checkout-tutorial tied to the same incident. Latency is deliberately excluded so a different latency value does not identify a different problem.

Select Save. Reopen the saved alert from Alerts and check its query, delivery action, group, interval, and unique field. Turn Enabled on and save again.

Check: You see Alert saved, and the alert is enabled. Refresh Incidents over the next two minutes. You should find one incident titled Incident tutorial - checkout latency.

If it has not appeared after two minutes, use the troubleshooting table below before continuing. A one-minute interval is the requested evaluation cadence, not a delivery deadline.

4. Verify creation and delivery

Open the incident and copy its page URL. Keep this URL for the rest of the exercise.

Check all of the following:

• Status is OPEN and severity is SEV3.
• Owner is Incident tutorial responders.
• Description contains 1500 and checkout-tutorial, with no unexpanded {latencyMs} or {service} placeholders.
• History contains an alert-triggered creation event, ALERT_TRIGGERED_INCIDENT.
• Each test group member receives an Incident created email. Its incident ID matches this incident, and View incident opens the page you just copied.

Now leave the alert enabled for two more evaluation intervals. Refresh the incident list and open the saved URL again.

Check: There is still one incident for this alert and service, with the same ID. Repeat evaluations should not produce another incident-created email. Do not expect a new history event for every evaluation; duplicate processing can be suppressed.

5. Record the investigation

1. On the incident page, select IN PROGRESS under Status and confirm with Save Status Change.

2. Under Comments, enter the following and select Post Comment:

   Copy

   Tutorial drill: investigating checkout latency. The sample reports 1500 ms; no live service has been changed.

3. Reload the incident page.

Check: Status remains IN_PROGRESS, the comment is present, and history includes the status change and a COMMENT_ADDED event. Group members receive the comment email, including its text.

6. Hand off ownership

Add a handoff comment:

Tutorial handoff: the next owner will simulate recovery, verify the result, and resolve this incident.

Under Owner, select Reassign. Choose the second responder, or yourself for a solo exercise. Select Reassign, then confirm with Reassign Owner.

Check: The owner is now the selected user, the incident URL and ID are unchanged, and history records the ownership change. The new owner receives an update email. If you are working with a second responder, have them open the saved incident URL and perform the remaining incident actions.

Subsequent notifications go to the current owner and any individual assignee. Former group members do not continue receiving email solely because they belonged to the previous owner group. See Ownership and assignment and Notifications.

7. Simulate recovery and mitigate

Edit the same alert. Replace its query with:

datatable (service: string, latencyMs: int64)
[
    "checkout-tutorial", 200
]
| where latencyMs > 1000

Save the alert. Run this query on the query page too.

Check: The query returns zero rows. After allowing two evaluation intervals, reload the incident. It should still be IN_PROGRESS with Automatically resolve incidents when the alert clears left off (the default). Enabling that option would resolve the incident after a successful evaluation no longer matches this service.

Return to the alert, turn Enabled off, and save. Keep it disabled for the rest of the exercise.

On the incident, add this comment:

Tutorial recovery check: sample latency is now 200 ms and the alert query returns no rows. The test alert is disabled.

Select MITIGATED and confirm with Save Status Change.

Check: The incident is MITIGATED, retains its user owner, and has the recovery comment and status change in its history.

8. Resolve, then test reopening

Select RESOLVED. Leave the resolution note blank at first.

Check: Save Status Change is disabled until you enter a resolution note.

Enter:

Tutorial complete: sample latency reduced from 1500 ms to 200 ms. The query returns no rows and the test alert is disabled.

Confirm the status change, then reload the page.

Check: Status is RESOLVED, the resolution note is saved, history includes INCIDENT_RESOLVED, and the current owner receives a resolution email.

To check the reopen path, select OPEN. Leave the reopen reason blank first; the confirmation must remain disabled. Then enter:

Tutorial only: reopening to verify the required reason and notification.

Confirm the change.

Check: The same incident is now OPEN, history includes INCIDENT_REOPENED, and its owner receives a reopened email. The alert remains disabled; reopening an incident does not enable its alert.

Resolve it once more, using Tutorial finished; reopen check passed and the test alert remains disabled. as the resolution note.

For the full transition matrix, see State transition rules.

Finish and record the result

Keep the resolved incident as the exercise record. Confirm that the test alert is disabled and that the incident has these results:

For a shared test report, record the incident URL, test date, and any failed checks. Treat missing email or history as a failed check even if the incident itself saved successfully.

You can delete the test group once it owns no incidents. This exercise transfers ownership to a user before closure, so the group should be unused. Resolving an incident alone does not release its group owner. See Group deletion rules.

To repeat the exercise, restore the original failing query and re-enable the same alert. The resolved incident stays resolved; a matching evaluation creates a new incident for the service. While that new incident is unresolved, subsequent matching evaluations reuse it.

If a check fails