/sig api-machinery
/sig node
/do-not-merge work-in-progress

This PR in first commit is to give an idea what is desired output for the resolution of issue: #123176 .
The approach might change in future after more community discussion.

/cc @harche @saschagrunert

/retitle publish an event when the container is restarted

/remove-sig api-machinery

@Ritikaa96 please, squash your comments into one, thanks!

@Ritikaa96 please, squash your comments into one, thanks!

Sure @bart0sh .

Hi @SergeyKanzhelev @tallclair Our use case is based on utilising the events and users depends on it. I understand and agree events are not guaranteed to be available but they surely helps us in getting our results and observation.
Adding ContainerDied or ContainerTerminated event is essential for our issue here.

I'll be happy to work on long term solution for better container state clarity , however this event addition can solve our issue as per the use case.

Hi @SergeyKanzhelev @tallclair Our use case is based on utilising the events and users depends on it. I understand and agree events are not guaranteed to be available but they surely helps us in getting our results and observation. Adding ContainerDied or ContainerTerminated event is essential for our issue here.

I'll be happy to work on long term solution for better container state clarity , however this event addition can solve our issue as per the use case.

Any word for this? Please let me know whats your opinion.

@SergeyKanzhelev This addition will also be useful for the users to differentiate the container state as it is for us. Still waiting for further discussion from reviewers

Hi @tallclair @SergeyKanzhelev @bart0sh
To explain the need to add this event is that our requirement lies in the "timestamp and specifically state change" which we want to access from event and further details like reason can be filtered out from Logs. I understand the pod logs are more reliable but as events are real-time insights into state changes and other significant occurrences within cluster. I believe events are there for some benefit of the user and they can help in identifying issues early on atleast.

I for msyelf I'm not against adding events for something which is supposed to happen rarely like container restart. We should take care to not cause extra load to the apiserver, like always, but conceptually I don't see any blockers.

But I second the considerations about the usefulness of the events. Meaning, events are garbage collected and, if memory serves, most relaxed setting still don't retain events past few hours, so events would be little helpful in troubleshooting issues which are, say from last week. I hit a similar condition in the past few months and this signifincantly cooled down my interest in events.

There's another side though. Events are most useful when someone is actively monitoring them, but that someone doesn't need to be a human. May (and probably should) be a program. So, is the usecase we're trying to fix ultimately related to observability? do we need a signal to detect more easily that a container is restarted? can other observability mechanism be more useful?

Hi @ffromani thanks for your insight.

Events are most useful when someone is actively monitoring them, but that someone doesn't need to be a human.

You are right , I believe default event retention value is 1 hour. There are also methods to customise event retention though.

However the real time changes are also usable in case of automation based on them and for me here, i need only the real time changes.
Cluster using Automation scripts to detect pod failure can utilise such "event trigger" .The script can be more effectively triggered by discrete container state changes, such as container death.

@ffromani

Currently, k8s just show ContainerStarted --> ContainerStarted event sequence in event record when the container died for some reasons, however it should be in sequence:

• ContainerStarted --> ContainerDied --> ContainerStarted or ContainerRestarted ,if it is restarted again after it died once.

It will provide real-time insights into container deaths, enabling better root cause analysis and faster troubleshooting.
I hope I'm able to explain why i raised the need for it.
I don't find any issue in adding one event message if this also aligns with Kubernetes goal of usability, I'll leave the decision on the community though as they know best.

@ffromani

Currently, k8s just show ContainerStarted --> ContainerStarted event sequence in event record when the container died for some reasons, however it should be in sequence:

* ContainerStarted --> ContainerDied --> ContainerStarted or ContainerRestarted ,if it is restarted again after it died once.

It will provide real-time insights into container deaths, enabling better root cause analysis and faster troubleshooting. I hope I'm able to explain why i raised the need for it. I don't find any issue in adding one event message if this also aligns with Kubernetes goal of usability, I'll leave the decision on the community though as they know best.

ok, so it seems to me you need a more explicit signal when container dies. Were alternative to events be evaluated and then discarded? if so, why? I think this is overarching thought shared by few reviewers.

It will provide real-time insights into container deaths, enabling better root cause analysis and faster troubleshooting.

In our production env, we usually use metrics to detect container restarts. 🤔

Hi @googs1025 I understand , there are various ways to attain it.

In my script event proves fruitful and as I said ,not all roles can access pod logs, still adding an event message to show the right sequence of container state change should clarify kubernetes before as it is. even as a normal user i would wish that container states do not change from ContainerStarted to again ContainerStarted. If there is a record of sudden failure the user can understand the container state better which will in turn make easier for developers and operators to explain these small pod failures.

Will adding this event message cause issue or load to kubernetes performance?

Having symmetry in the reporting of container start and stop is a interesting argument to have this change (once inline review comments have been addressed)!

Granted, lack of explicit "container died" signal is irking, but dead container are supposed to restart ASAP, so one can reasonnably expect a rapid back-to-back sequence of "container dead" + "container started". Perhaps, but I'm really thinking aloud, we should coalesce right from the beginning such events in "container REstarted"? not sure we have this already. Maybe this is premature optimization? Didn't explore this line of thought too much.

However I still feel that there it was not good explanation why other alternatives are insufficient or unsuitable. I think this is where the most concern is.

Granted, lack of explicit "container died" signal is irking, but dead container are supposed to restart ASAP, so one can reasonnably expect a rapid back-to-back sequence of "container dead" + "container started". Perhaps, but I'm really thinking aloud, we should coalesce right from the beginning such events in "container REstarted"? not sure we have this already. Maybe this is premature optimization? Didn't explore this line of thought too much.

I have shared screenshot for multiple container restart and it doesnt show any cumbersome back-to-back sequence of "container dead" + "container started" here i used container restarted ( x times the restart occurred in past XX min) .
If we take only purpose to change this is making the event more readable ,consider this situation:

• having a list of events in a system with multiple pods and only one pod is failing
• Now when we look at it it show container started again and again when the pod container restart
• however there are other pods we are introducing and there containers are also starting.
• Now here What is the difference between a container just started afresh & container that is restarted?
• Wouldnt introducing a different term for both same some time when a person is self monitoring the system?

However I still feel that there it was not good explanation why other alternatives are insufficient or unsuitable. I think this is where the most concern is.
Were alternative to events be evaluated and then discarded? if so, why? I think this is overarching thought shared by few reviewers.

Hi @ffromani
The concern is not using other alternatives tools but using the other observability tools along with events to capture changes in real time . While utilising events with observability tools, most of the container state changes are captured ~almost in real time. What my motivation is to capture these changes with minimum latency which understandably occur when we use observability tools. using events alongside with these tools works best for me, having no record in event list for "containerRestart" or "containerDeath" cause the issue. hence I tried the change.

does this clarify the questions behind motivation for the change?

I still feel that there it was not good explanation why other alternatives are insufficient or unsuitable. I think this is where the most concern is.

@ffromani any thought on this. as a user point of view as well ,having a clean sequence of events( that are created to real time container state change for base level monitoring and troubleshooting) is a big motivation. Regarding back to back containerStarted+ containerDied events, i tested the code change and the event message will not have a back to back sequence, it will show the latest containerRestart or containerDeath with number of times it already happened.

If this is still unacceptable as a motivation or enough for the change , then please do let me know.
Thanks for your responses.
Regards,
Ritika

@SergeyKanzhelev @dchen1107 @tallclair any final points for this? i hope the use case was clear by my explanation.

I understand community pov about observability tools . but as there are multiple events for container state , it gets confusing to see or explain that why there is no event (even if it shows the last time container died only) for containerDeath.
k8s defines many events, e.g:

• Scheduled
• AddedInterface
• Pulling
• Pulled
• Created
• Started
• SuccessfulCreate
• Killing

In cases where observability tools and checking pod logs for this one trigger will cost high in implementation, if there is such event to mention containerDeath it will provide simpler solution . Let me know WDYT about it.

I thought adding one event message would not be difficult as it will also simplifies user experience, however does this create any apiserver load issue if we merge this PR?

If we can differenciate the "containerStarted" event when the container is newly created with "containerStarted" event when the Container actually restarted , it will resolve the issue im facing.
How about if the event message or status after the container restart shows ContainerRestarted and containerStarted is only used for first start of container. it can be done if we consider the restart count in the code file , if it is >=1 the event message recorded will be containerRestarted

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale