From: Tomasz Figa <tfiga@chromium.org>
To: Nicolas Dufresne <nicolas@ndufresne.ca>
Cc: Stanimir Varbanov <stanimir.varbanov@linaro.org>,
Hans Verkuil <hverkuil-cisco@xs4all.nl>,
Linux Media Mailing List <linux-media@vger.kernel.org>,
Michael Tretter <m.tretter@pengutronix.de>
Subject: Re: [PATCHv3 1/5] media: docs-rst: Document memory-to-memory video encoder interface
Date: Wed, 10 Jun 2020 15:55:56 +0200
Message-ID: <CAAFQd5CHOh82m9atCZB5X-m1ZmND4O=id3pQzHiOO0Qm6m80-w@mail.gmail.com> (raw)
In-Reply-To: <a8b466eee221bda45489eca2b7b8ac52c1635bb4.camel@ndufresne.ca>
On Wed, Jun 10, 2020 at 3:08 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>
> Le mercredi 10 juin 2020 à 14:28 +0200, Tomasz Figa a écrit :
> > On Fri, Jun 5, 2020 at 11:24 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> > > Le vendredi 05 juin 2020 à 10:19 +0300, Stanimir Varbanov a écrit :
> > > > On 5/26/20 1:09 PM, Hans Verkuil wrote:
> > > > > From: Tomasz Figa <tfiga@chromium.org>
> > > > >
> > > > > Due to complexity of the video encoding process, the V4L2 drivers of
> > > > > stateful encoder hardware require specific sequences of V4L2 API calls
> > > > > to be followed. These include capability enumeration, initialization,
> > > > > encoding, encode parameters change, drain and reset.
> > > > >
> > > > > Specifics of the above have been discussed during Media Workshops at
> > > > > LinuxCon Europe 2012 in Barcelona and then later Embedded Linux
> > > > > Conference Europe 2014 in Düsseldorf. The de facto Codec API that
> > > > > originated at those events was later implemented by the drivers we already
> > > > > have merged in mainline, such as s5p-mfc or coda.
> > > > >
> > > > > The only thing missing was the real specification included as a part of
> > > > > Linux Media documentation. Fix it now and document the encoder part of
> > > > > the Codec API.
> > > > >
> > > > > Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > > > > Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> > > > > ---
> > > > > .../userspace-api/media/v4l/dev-encoder.rst | 728 ++++++++++++++++++
> > > > > .../userspace-api/media/v4l/dev-mem2mem.rst | 1 +
> > > > > .../userspace-api/media/v4l/pixfmt-v4l2.rst | 5 +
> > > > > .../userspace-api/media/v4l/v4l2.rst | 2 +
> > > > > .../media/v4l/vidioc-encoder-cmd.rst | 51 +-
> > > > > 5 files changed, 767 insertions(+), 20 deletions(-)
> > > > > create mode 100644 Documentation/userspace-api/media/v4l/dev-encoder.rst
> > > > >
> > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-encoder.rst b/Documentation/userspace-api/media/v4l/dev-encoder.rst
> > > > > new file mode 100644
> > > > > index 000000000000..aace7b812a9c
> > > > > --- /dev/null
> > > > > +++ b/Documentation/userspace-api/media/v4l/dev-encoder.rst
> > > > > @@ -0,0 +1,728 @@
> > > > > +.. SPDX-License-Identifier: GPL-2.0
> > > > > +
> > > > > +.. _encoder:
> > > > > +
> > > > > +*************************************************
> > > > > +Memory-to-Memory Stateful Video Encoder Interface
> > > > > +*************************************************
> > > > > +
> > > > > +A stateful video encoder takes raw video frames in display order and encodes
> > > > > +them into a bytestream. It generates complete chunks of the bytestream, including
> > > > > +all metadata, headers, etc. The resulting bytestream does not require any
> > > > > +further post-processing by the client.
> > > > > +
> > > > > +Performing software stream processing, header generation etc. in the driver
> > > > > +in order to support this interface is strongly discouraged. In case such
> > > > > +operations are needed, use of the Stateless Video Encoder Interface (in
> > > > > +development) is strongly advised.
> > > > > +
> > > >
> > > > <cut>
> > > >
> > > > > +Encoding Parameter Changes
> > > > > +==========================
> > > > > +
> > > > > +The client is allowed to use :c:func:`VIDIOC_S_CTRL` to change encoder
> > > > > +parameters at any time. The availability of parameters is encoder-specific
> > > > > +and the client must query the encoder to find the set of available controls.
> > > > > +
> > > > > +The ability to change each parameter during encoding is encoder-specific, as
> > > > > +per the standard semantics of the V4L2 control interface. The client may
> > > > > +attempt to set a control during encoding and if the operation fails with the
> > > > > +-EBUSY error code, the ``CAPTURE`` queue needs to be stopped for the
> > > > > +configuration change to be allowed. To do this, it may follow the `Drain`
> > > > > +sequence to avoid losing the already queued/encoded frames.
> > > > > +
> > > > > +The timing of parameter updates is encoder-specific, as per the standard
> > > > > +semantics of the V4L2 control interface. If the client needs to apply the
> > > > > +parameters exactly at specific frame, using the Request API
> > > > > +(:ref:`media-request-api`) should be considered, if supported by the encoder.
> > > >
> > > > In request-api case does the control will return EBUSY immediately when
> > > > S_CTRL is called from the client? I'm asking in the context of the
> > > > controls which are not dynamic (cannot set during streaming and Reset
> > > > sequence is needed).
> > >
> > > This is all hypothetical, as nothing of that has been implemented. But
> > > I suppose there should be instant validation to allow that, even if
> > > that means more plumbing inside the kernel. It would be better then
> > > just running the request ignoring silently that control. Ideally
> > > userspace should not have to go into trial and errors, so controls that
> > > are means for the should be marked.
> > >
> >
> > There isn't much to be implemented here, as it's the same standard
> > V4L2 behavior as existed for a while. If a device supports changing
> > the control on the fly, a call to S_CTRL succeeds, if not, it fails
> > with -EBUSY. This is regardless of whether requests are used or not.
>
> Are you against introducing flags to signal generically when a control
> will have live effect, delayed effect or will simply be rejected (-
> EBUSY) if set while streaming ? I'm particularly concern for cases
> where the same control may work live for one driver, and not for
> another. It would be very disappointing to get an EBUSY on a bitrate
> control of an encoder as an example. So if someone foresee having
> bitrate control that cannot be adapted while streaming (or that will
> only be applied if a key frame is generated), I'd rather prefer to know
> in advance, and likely not select this encoder for WebRTC purpose as an
> example. Having to do all the steps to get the driver in streaming in
> order to set the control and discover that it's not suitable for the
> purpose seems quite sub-optimal.
>
> I'm just saying, but the "standard" that existed for a while has always
> been hardcoded to the control (through control documentation, and
> that's fine with me) or the HW driver being used (which isn't fine in
> my opinion).
I was just clarifying the behavior of S_(EXT_)CTRL(S).
A way to query the behavior in advance is an orthogonal problem, which
I agree that should be solved and I guess a flag returned by
VIDIOC_QUERY_EXT_CTRLS could do.
Best regards,
Tomasz
next prev parent reply other threads:[~2020-06-10 13:56 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-26 10:09 [PATCHv3 0/5] Stateful Encoding: final bits Hans Verkuil
2020-05-26 10:09 ` [PATCHv3 1/5] media: docs-rst: Document memory-to-memory video encoder interface Hans Verkuil
2020-05-29 9:57 ` Stanimir Varbanov
2020-05-29 10:17 ` Stanimir Varbanov
2020-05-29 12:21 ` Tomasz Figa
2020-06-05 7:19 ` Stanimir Varbanov
2020-06-05 21:24 ` Nicolas Dufresne
2020-06-06 7:58 ` Stanimir Varbanov
2020-06-07 0:27 ` Nicolas Dufresne
2020-06-10 12:28 ` Tomasz Figa
2020-06-10 13:08 ` Nicolas Dufresne
2020-06-10 13:55 ` Tomasz Figa [this message]
2020-06-10 13:58 ` Tomasz Figa
2020-06-23 10:36 ` Hans Verkuil
2020-06-23 11:47 ` Tomasz Figa
2020-06-23 10:47 ` Mauro Carvalho Chehab
2020-05-26 10:09 ` [PATCHv3 2/5] vidioc-g-parm.rst: update the VIDIOC_G/S_PARM documentation Hans Verkuil
2020-05-28 7:54 ` Michael Tretter
2020-05-26 10:09 ` [PATCHv3 3/5] dev-decoder.rst: small fixes Hans Verkuil
2020-05-26 10:09 ` [PATCHv3 4/5] videodev2.h: add V4L2_FMT_FLAG_ENC_CAP_FRAME_INTERVAL flag Hans Verkuil
2020-05-26 10:09 ` [PATCHv3 5/5] dev-encoder.rst: add reference to V4L2_FMT_FLAG_ENC_CAP_FRAME_INTERVAL Hans Verkuil
2020-05-28 7:58 ` [PATCHv3 0/5] Stateful Encoding: final bits Michael Tretter
2020-06-02 9:01 ` Tomasz Figa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAAFQd5CHOh82m9atCZB5X-m1ZmND4O=id3pQzHiOO0Qm6m80-w@mail.gmail.com' \
--to=tfiga@chromium.org \
--cc=hverkuil-cisco@xs4all.nl \
--cc=linux-media@vger.kernel.org \
--cc=m.tretter@pengutronix.de \
--cc=nicolas@ndufresne.ca \
--cc=stanimir.varbanov@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Unnamed repository; edit this file 'description' to name the repository.
This inbox may be cloned and mirrored by anyone:
git clone --mirror http://archive.lwn.net:8080/linux-media/0 linux-media/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 linux-media linux-media/ http://archive.lwn.net:8080/linux-media \
linux-media@vger.kernel.org lwn-linux-media@archive.lwn.net
public-inbox-index linux-media
Example config snippet for mirrors.
Newsgroup available over NNTP:
nntp://archive.lwn.net/lwn.kernel.linux-media
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git