Problem Description
A binocular vision plug-in, the previous input resolution was 540*960, the output resolution was 540*960, now it needs to modify the output resolution to 1080*1920, after modifying the event function, the previous plug-in reported WARNING Could not send sticky Events also have result not-negotiated, mark pending events.
I didn't know much about the event and query mechanism of gstreamer before. After two days of digging through the source code of gstreamer, I have a deeper understanding of the mechanism of gstreamer. Let me record it.
Gstreamer event event definition:
Here is the source code definition of the event event directly:
typedef enum {
GST_EVENT_TYPE_UPSTREAM = 1 << 0,
GST_EVENT_TYPE_DOWNSTREAM = 1 << 1,
GST_EVENT_TYPE_SERIALIZED = 1 << 2,
GST_EVENT_TYPE_STICKY = 1 << 3,
GST_EVENT_TYPE_STICKY_MULTI = 1 << 4
} GstEventTypeFlags;
#define GST_EVENT_NUM_SHIFT (8)
#define GST_EVENT_MAKE_TYPE(num,flags) \
(((num) << GST_EVENT_NUM_SHIFT) | (flags))
#define FLAG(name) GST_EVENT_TYPE_##name
typedef enum {
GST_EVENT_UNKNOWN = GST_EVENT_MAKE_TYPE (0, 0),
/* bidirectional events */
GST_EVENT_FLUSH_START = GST_EVENT_MAKE_TYPE (10, FLAG(BOTH)),
GST_EVENT_FLUSH_STOP = GST_EVENT_MAKE_TYPE (20, FLAG(BOTH) | FLAG(SERIALIZED)),
/* downstream serialized events */
GST_EVENT_STREAM_START = GST_EVENT_MAKE_TYPE (40, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_CAPS = GST_EVENT_MAKE_TYPE (50, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_SEGMENT = GST_EVENT_MAKE_TYPE (70, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_TAG = GST_EVENT_MAKE_TYPE (80, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY) | FLAG(STICKY_MULTI)),
GST_EVENT_BUFFERSIZE = GST_EVENT_MAKE_TYPE (90, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_SINK_MESSAGE = GST_EVENT_MAKE_TYPE (100, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY) | FLAG(STICKY_MULTI)),
GST_EVENT_EOS = GST_EVENT_MAKE_TYPE (110, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_TOC = GST_EVENT_MAKE_TYPE (120, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY) | FLAG(STICKY_MULTI)),
GST_EVENT_PROTECTION = GST_EVENT_MAKE_TYPE (130, FLAG (DOWNSTREAM) | FLAG (SERIALIZED) | FLAG (STICKY) | FLAG (STICKY_MULTI)),
/* non-sticky downstream serialized */
GST_EVENT_SEGMENT_DONE = GST_EVENT_MAKE_TYPE (150, FLAG(DOWNSTREAM) | FLAG(SERIALIZED)),
GST_EVENT_GAP = GST_EVENT_MAKE_TYPE (160, FLAG(DOWNSTREAM) | FLAG(SERIALIZED)),
/* upstream events */
GST_EVENT_QOS = GST_EVENT_MAKE_TYPE (190, FLAG(UPSTREAM)),
GST_EVENT_SEEK = GST_EVENT_MAKE_TYPE (200, FLAG(UPSTREAM)),
GST_EVENT_NAVIGATION = GST_EVENT_MAKE_TYPE (210, FLAG(UPSTREAM)),
GST_EVENT_LATENCY = GST_EVENT_MAKE_TYPE (220, FLAG(UPSTREAM)),
GST_EVENT_STEP = GST_EVENT_MAKE_TYPE (230, FLAG(UPSTREAM)),
GST_EVENT_RECONFIGURE = GST_EVENT_MAKE_TYPE (240, FLAG(UPSTREAM)),
GST_EVENT_TOC_SELECT = GST_EVENT_MAKE_TYPE (250, FLAG(UPSTREAM)),
/* custom events start here */
GST_EVENT_CUSTOM_UPSTREAM = GST_EVENT_MAKE_TYPE (270, FLAG(UPSTREAM)),
GST_EVENT_CUSTOM_DOWNSTREAM = GST_EVENT_MAKE_TYPE (280, FLAG(DOWNSTREAM) | FLAG(SERIALIZED)),
GST_EVENT_CUSTOM_DOWNSTREAM_OOB = GST_EVENT_MAKE_TYPE (290, FLAG(DOWNSTREAM)),
GST_EVENT_CUSTOM_DOWNSTREAM_STICKY = GST_EVENT_MAKE_TYPE (300, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY) | FLAG(STICKY_MULTI)),
GST_EVENT_CUSTOM_BOTH = GST_EVENT_MAKE_TYPE (310, FLAG(BOTH) | FLAG(SERIALIZED)),
GST_EVENT_CUSTOM_BOTH_OOB = GST_EVENT_MAKE_TYPE (320, FLAG(BOTH))
} GstEventType;
How Gstreamer designs macros is actually very clear from here.
GST_EVENT_MAKE_TYPE In fact, the macro only does one very simple thing. It shifts the first digital parameter to the left by 8 bits, and the vacated lower 8 bits are used to store the nature of the event. This is explained in the enumeration variable, and the corresponding bit GstEventTypeFlagsis 1 or 0 represents the presence or absence of this property. This technique of designing macros in C language can also be used when we write C/C++ programs ourselves.
In addition, it can be seen that many common events are sticky events, such as CAPS and EOS, so if Gstreamer appears "Could not send sticky events", the first thing to judge is what the event is.
gst_pad_peer_query function
gboolean
gst_pad_peer_query (GstPad * pad, GstQuery * query)
{
[...]
serialized = GST_QUERY_IS_SERIALIZED (query);
GST_OBJECT_LOCK (pad);
if (GST_PAD_IS_SRC (pad) && serialized) {
/* all serialized queries on the srcpad trigger push of
* sticky events */
if (check_sticky (pad, NULL) != GST_FLOW_OK)
goto sticky_failed;
}
[...]
peerpad = GST_PAD_PEER (pad);
[...]
res = gst_pad_query (peerpad, query);
[...]
return res;
/* ERRORS */
[...]
sticky_failed:
{
GST_WARNING_OBJECT (pad, "could not send sticky events");
GST_OBJECT_UNLOCK (pad);
return FALSE;
}
[...]
}
}
The insignificant part has been deleted, gst_pad_peer_querywhich is used to send a query to the peerpad of the pad (that is, the pad that is successfully connected to the pad and bound together), and is actually calling a function for the peerpad gst_pad_query.
"could not send sticky events" is caused by the failure of the check_sticky function, so what does this function do?
Here, its main purpose is to judge whether there are any lingering events in the current pad, and if so, send them to the peerpad and clear them.
check_sticky function
/* check sticky events and push them when needed. should be called
* with pad LOCK */
static inline GstFlowReturn
check_sticky (GstPad * pad, GstEvent * event)
{
PushStickyData data = {
GST_FLOW_OK, FALSE, event };
if (G_UNLIKELY (GST_PAD_HAS_PENDING_EVENTS (pad))) {
GST_OBJECT_FLAG_UNSET (pad, GST_PAD_FLAG_PENDING_EVENTS);
GST_DEBUG_OBJECT (pad, "pushing all sticky events");
events_foreach (pad, push_sticky, &data);
/* If there's an EOS event we must push it downstream
* even if sending a previous sticky event failed.
* Otherwise the pipeline might wait forever for EOS.
*
* Only do this if pushing another event than the EOS
* event failed.
*/
if (data.ret != GST_FLOW_OK && !data.was_eos) {
PadEvent *ev = find_event_by_type (pad, GST_EVENT_EOS, 0);
if (ev && !ev->received) {
data.ret = gst_pad_push_event_unchecked (pad, gst_event_ref (ev->event),
GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM);
/* the event could have been dropped. Because this can only
* happen if the user asked for it, it's not an error */
if (data.ret == GST_FLOW_CUSTOM_SUCCESS)
data.ret = GST_FLOW_OK;
}
}
}
return data.ret;
}
The source code of check_sticky is very short and can be divided into two parts:
- Call the function
events_foreachiteratively for unhandled events viapush_sticky - If the event to be processed is EOS, even if its push fails, it must be passed to the downstream, otherwise the pipeline will not end.
GST_PAD_HAS_PENDING_EVENTS Literally, the macro is to judge whether there are unhandled events. Among the parameters passed by the previous function, datathe ones in the structure eventare actually NULL.
data.retIt is the cause of the problem, and this ret has only been push_stickymodified, events_foreachand has not been changed. It simply sends the unprocessed events of the pad to the push_stickysecond parameter in turn (see below), so the problem occurs in push_stickythe function
push_sticky function
static gboolean
push_sticky (GstPad * pad, PadEvent * ev, gpointer user_data)
{
PushStickyData *data = user_data;
GstEvent *event = ev->event;
if (ev->received) {
GST_DEBUG_OBJECT (pad, "event %s was already received",
GST_EVENT_TYPE_NAME (event));
return TRUE;
}
/* If we're called because of an sticky event, only forward
* events that would come before this new event and the
* event itself */
if (data->event && GST_EVENT_IS_STICKY (data->event) &&
GST_EVENT_TYPE (data->event) <= GST_EVENT_SEGMENT &&
GST_EVENT_TYPE (data->event) < GST_EVENT_TYPE (event)) {
data->ret = GST_FLOW_CUSTOM_SUCCESS_1;
} else {
data->ret = gst_pad_push_event_unchecked (pad, gst_event_ref (event),
GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM);
if (data->ret == GST_FLOW_CUSTOM_SUCCESS_1)
data->ret = GST_FLOW_OK;
}
switch (data->ret) {
case GST_FLOW_OK:
ev->received = TRUE;
GST_DEBUG_OBJECT (pad, "event %s marked received",
GST_EVENT_TYPE_NAME (event));
break;
case GST_FLOW_CUSTOM_SUCCESS:
/* we can't assume the event is received when it was dropped */
GST_DEBUG_OBJECT (pad, "event %s was dropped, mark pending",
GST_EVENT_TYPE_NAME (event));
GST_OBJECT_FLAG_SET (pad, GST_PAD_FLAG_PENDING_EVENTS);
data->ret = GST_FLOW_OK;
break;
case GST_FLOW_CUSTOM_SUCCESS_1:
/* event was ignored and should be sent later */
GST_DEBUG_OBJECT (pad, "event %s was ignored, mark pending",
GST_EVENT_TYPE_NAME (event));
GST_OBJECT_FLAG_SET (pad, GST_PAD_FLAG_PENDING_EVENTS);
data->ret = GST_FLOW_OK;
break;
case GST_FLOW_NOT_LINKED:
/* not linked is not a problem, we are sticky so the event will be
* rescheduled to be sent later on re-link, but only for non-EOS events */
GST_DEBUG_OBJECT (pad, "pad was not linked, mark pending");
if (GST_EVENT_TYPE (event) != GST_EVENT_EOS) {
data->ret = GST_FLOW_OK;
ev->received = TRUE;
}
break;
default:
GST_DEBUG_OBJECT (pad, "result %s, mark pending events",
gst_flow_get_name (data->ret));
GST_OBJECT_FLAG_SET (pad, GST_PAD_FLAG_PENDING_EVENTS);
break;
}
if (data->ret != GST_FLOW_OK && GST_EVENT_TYPE (event) == GST_EVENT_EOS)
data->was_eos = TRUE;
return data->ret == GST_FLOW_OK;
}
user_data is the previous data pointer. The whole code looks long, but in fact only one function is called gst_pad_push_event_unchecked , and it is impossible for codes other than this function to make the ret value not-negotiated.
If the debug level is set to DEBUG in gstreamer, you can still see result not-negotiated, mark pending eventsthe words. This not-negotiated is the value of data->ret, which is why the "Could not send sticky events" error occurred. Next enter gst_pad_push_event_unchecked the function.
Supplement: Regarding the definition of the GstFlowReturn enumeration type, the aforementioned not-negotiated is actually the macro GST_FLOW_NOT_NEGOTIATED = -4, so when debugging below, you should pay attention to where the ret variable is assigned a GST_FLOW_NOT_NEGOTIATED
typedef enum {
/* custom success starts here */
GST_FLOW_CUSTOM_SUCCESS_2 = 102,
GST_FLOW_CUSTOM_SUCCESS_1 = 101,
GST_FLOW_CUSTOM_SUCCESS = 100,
/* core predefined */
GST_FLOW_OK = 0,
/* expected failures */
GST_FLOW_NOT_LINKED = -1,
GST_FLOW_FLUSHING = -2,
/* error cases */
GST_FLOW_EOS = -3,
GST_FLOW_NOT_NEGOTIATED = -4,
GST_FLOW_ERROR = -5,
GST_FLOW_NOT_SUPPORTED = -6,
/* custom error starts here */
GST_FLOW_CUSTOM_ERROR = -100,
GST_FLOW_CUSTOM_ERROR_1 = -101,
GST_FLOW_CUSTOM_ERROR_2 = -102
} GstFlowReturn;
gst_pad_push_event_unchecked function
static GstFlowReturn
gst_pad_push_event_unchecked (GstPad * pad, GstEvent * event,
GstPadProbeType type)
{
GstFlowReturn ret;
GstPad *peerpad;
GstEventType event_type;
gint64 old_pad_offset = pad->offset;
/* pass the adjusted event on. We need to do this even if
* there is no peer pad because of the probes. */
event = apply_pad_offset (pad, event, GST_PAD_IS_SINK (pad));
[...]
/* the pad offset might've been changed by any of the probes above. It
* would've been taken into account when repushing any of the sticky events
* above but not for our current event here */
if (G_UNLIKELY (old_pad_offset != pad->offset)) {
event =
_apply_pad_offset (pad, event, GST_PAD_IS_SINK (pad),
pad->offset - old_pad_offset);
}
[...]
/* now check the peer pad */
peerpad = GST_PAD_PEER (pad);
[...]
ret = gst_pad_send_event_unchecked (peerpad, event, type);
[...]
return ret;
[...]
}
The type parameter is not very important, so some relevant code has been deleted. Here we focus on when to return GST_FLOW_NOT_NEGOTIATED, it can be seen that the main function is gst_pad_send_event_unchecked, and then continue
gst_pad_send_event_unchecked function
static GstFlowReturn
gst_pad_send_event_unchecked (GstPad * pad, GstEvent * event,
GstPadProbeType type)
{
GstFlowReturn ret;
GstEventType event_type;
gboolean serialized, need_unlock = FALSE, sticky;
GstPadEventFunction eventfunc;
GstPadEventFullFunction eventfullfunc = NULL;
GstObject *parent;
[...]
event = apply_pad_offset (pad, event, GST_PAD_IS_SRC (pad));
[...]
eventfullfunc = GST_PAD_EVENTFULLFUNC (pad);
eventfunc = GST_PAD_EVENTFUNC (pad);
if (G_UNLIKELY (eventfunc == NULL && eventfullfunc == NULL))
goto no_function;
[...]
if (eventfullfunc) {
ret = eventfullfunc (pad, parent, event);
} else if (eventfunc (pad, parent, event)) {
ret = GST_FLOW_OK;
} else {
/* something went wrong */
switch (event_type) {
case GST_EVENT_CAPS:
ret = GST_FLOW_NOT_NEGOTIATED;
break;
default:
ret = GST_FLOW_ERROR;
break;
}
}
[...]
return ret;
[...]
}
Here we still focus on when to return GST_FLOW_NOT_NEGOTIATED, pay attention to these two macros GST_PAD_EVENTFUNC and GST_PAD_EVENTFULLFUNC, in addition, note that the pad pointer points to the downstream pad at this time, that is, the peerpad of the pad at the beginning. According to the official instructions, these two macros are responsible for calling the event function of the downstream pad. This is the event function set by gst_pad_set_event_function() and gst_pad_set_event_full_function() when we wrote the plugin.
/**
* GST_PAD_EVENTFUNC:
* @pad: a #GstPad
*
* Get the #GstPadEventFunction from the given @pad, which
* is the function that handles events on the pad. You can
* use this to set your own event handling function on a pad
* after you create it. If your element derives from a base
* class, use the base class's virtual functions instead.
*/
#define GST_PAD_EVENTFUNC(pad) (GST_PAD_CAST(pad)->eventfunc)
/**
* GST_PAD_EVENTFULLFUNC:
* @pad: a #GstPad
*
* Get the #GstPadEventFullFunction from the given @pad, which
* is the function that handles events on the pad. You can
* use this to set your own event handling function on a pad
* after you create it. If your element derives from a base
* class, use the base class's virtual functions instead.
*
* Since: 1.8
*/
#define GST_PAD_EVENTFULLFUNC(pad) (GST_PAD_CAST(pad)->ABI.abi.eventfullfunc)
Then when ret is set to GST_FLOW_NOT_NEGOTIATED, it will be like this when eventfullfunc and eventfunc do not return True, so the problem comes down to when the event function I wrote returns True and when it returns False.
Summarize
In fact, after going through the whole process, if you encounter this kind of problem later, you can directly judge that the event function of the peerpad returns 0.
By the way, the gst_pad_push_event function is usually called in the event function. If you look at this function, if the event is sticky, it will call the check_sticky function, which forms a closed loop with the above. The entire gstreamer pipeline is called level by level. The function passes the event event.