From: Jaroslav Kysela <perex@perex.cz>
To: Takashi Iwai <tiwai@suse.de>
Cc: "Hans Verkuil" <hverkuil@xs4all.nl>,
"Shengjiu Wang" <shengjiu.wang@gmail.com>,
"Amadeusz Sławiński" <amadeuszx.slawinski@linux.intel.com>,
"Mauro Carvalho Chehab" <mchehab@kernel.org>,
"Mark Brown" <broonie@kernel.org>,
"Sebastian Fricke" <sebastian.fricke@collabora.com>,
"Shengjiu Wang" <shengjiu.wang@nxp.com>,
sakari.ailus@iki.fi, tfiga@chromium.org,
m.szyprowski@samsung.com, linux-media@vger.kernel.org,
linux-kernel@vger.kernel.org, Xiubo.Lee@gmail.com,
festevam@gmail.com, nicoleotsuka@gmail.com, lgirdwood@gmail.com,
tiwai@suse.com, alsa-devel@alsa-project.org,
linuxppc-dev@lists.ozlabs.org
Subject: Re: [PATCH v15 00/16] Add audio support in v4l2 framework
Date: Wed, 15 May 2024 12:46:19 +0200 [thread overview]
Message-ID: <e63ec6c8-7da7-4b87-b7ff-a71ff12dcfc1@perex.cz> (raw)
In-Reply-To: <87o7975qcw.wl-tiwai@suse.de>
On 15. 05. 24 12:19, Takashi Iwai wrote:
> On Wed, 15 May 2024 11:50:52 +0200,
> Jaroslav Kysela wrote:
>>
>> On 15. 05. 24 11:17, Hans Verkuil wrote:
>>> Hi Jaroslav,
>>>
>>> On 5/13/24 13:56, Jaroslav Kysela wrote:
>>>> On 09. 05. 24 13:13, Jaroslav Kysela wrote:
>>>>> On 09. 05. 24 12:44, Shengjiu Wang wrote:
>>>>>>>> mem2mem is just like the decoder in the compress pipeline. which is
>>>>>>>> one of the components in the pipeline.
>>>>>>>
>>>>>>> I was thinking of loopback with endpoints using compress streams,
>>>>>>> without physical endpoint, something like:
>>>>>>>
>>>>>>> compress playback (to feed data from userspace) -> DSP (processing) ->
>>>>>>> compress capture (send data back to userspace)
>>>>>>>
>>>>>>> Unless I'm missing something, you should be able to process data as fast
>>>>>>> as you can feed it and consume it in such case.
>>>>>>>
>>>>>>
>>>>>> Actually in the beginning I tried this, but it did not work well.
>>>>>> ALSA needs time control for playback and capture, playback and capture
>>>>>> needs to synchronize. Usually the playback and capture pipeline is
>>>>>> independent in ALSA design, but in this case, the playback and capture
>>>>>> should synchronize, they are not independent.
>>>>>
>>>>> The core compress API core no strict timing constraints. You can eventually0
>>>>> have two half-duplex compress devices, if you like to have really independent
>>>>> mechanism. If something is missing in API, you can extend this API (like to
>>>>> inform the user space that it's a producer/consumer processing without any
>>>>> relation to the real time). I like this idea.
>>>>
>>>> I was thinking more about this. If I am right, the mentioned use in gstreamer
>>>> is supposed to run the conversion (DSP) job in "one shot" (can be handled
>>>> using one system call like blocking ioctl). The goal is just to offload the
>>>> CPU work to the DSP (co-processor). If there are no requirements for the
>>>> queuing, we can implement this ioctl in the compress ALSA API easily using the
>>>> data management through the dma-buf API. We can eventually define a new
>>>> direction (enum snd_compr_direction) like SND_COMPRESS_CONVERT or so to allow
>>>> handle this new data scheme. The API may be extended later on real demand, of
>>>> course.
>>>>
>>>> Otherwise all pieces are already in the current ALSA compress API
>>>> (capabilities, params, enumeration). The realtime controls may be created
>>>> using ALSA control API.
>>>
>>> So does this mean that Shengjiu should attempt to use this ALSA approach first?
>>
>> I've not seen any argument to use v4l2 mem2mem buffer scheme for this
>> data conversion forcefully. It looks like a simple job and ALSA APIs
>> may be extended for this simple purpose.
>>
>> Shengjiu, what are your requirements for gstreamer support? Would be a
>> new blocking ioctl enough for the initial support in the compress ALSA
>> API?
>
> If it works with compress API, it'd be great, yeah.
> So, your idea is to open compress-offload devices for read and write,
> then and let them convert a la batch jobs without timing control?
>
> For full-duplex usages, we might need some more extensions, so that
> both read and write parameters can be synchronized. (So far the
> compress stream is a unidirectional, and the runtime buffer for a
> single stream.)
>
> And the buffer management is based on the fixed size fragments. I
> hope this doesn't matter much for the intended operation?
It's a question, if the standard I/O is really required for this case. My
quick idea was to just implement a new "direction" for this job supporting
only one ioctl for the data processing which will execute the job in "one
shot" at the moment. The I/O may be handled through dma-buf API (which seems
to be standard nowadays for this purpose and allows future chaining).
So something like:
struct dsp_job {
int source_fd; /* dma-buf FD with source data - for dma_buf_get() */
int target_fd; /* dma-buf FD for target data - for dma_buf_get() */
... maybe some extra data size members here ...
... maybe some special parameters here ...
};
#define SNDRV_COMPRESS_DSPJOB _IOWR('C', 0x60, struct dsp_job)
This ioctl will be blocking (thus synced). My question is, if it's feasible
for gstreamer or not. For this particular case, if the rate conversion is
implemented in software, it will block the gstreamer data processing, too.
Jaroslav
--
Jaroslav Kysela <perex@perex.cz>
Linux Sound Maintainer; ALSA Project; Red Hat, Inc.
next prev parent reply other threads:[~2024-05-15 10:47 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-19 7:50 [PATCH v15 00/16] Add audio support in v4l2 framework Shengjiu Wang
2024-03-19 7:50 ` [PATCH v15 01/16] media: v4l2-ctrls: add support for fraction_bits Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 02/16] ASoC: fsl_asrc: define functions for memory to memory usage Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 03/16] ASoC: fsl_easrc: " Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 04/16] ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 05/16] ASoC: fsl_asrc: register m2m platform device Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 06/16] ASoC: fsl_easrc: " Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 07/16] media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 08/16] media: v4l2: Add audio capture and output support Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 09/16] media: uapi: Define audio sample format fourcc type Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 10/16] media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 11/16] media: uapi: Add audio rate controls support Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 12/16] media: uapi: Declare interface types for Audio Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 13/16] media: uapi: Add an entity type for audio resampler Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 14/16] media: vivid: add fixed point test controls Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 15/16] media: imx-asrc: Add memory to memory driver Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 16/16] media: vim2m-audio: add virtual driver for audio memory to memory Shengjiu Wang
2024-04-30 8:21 ` [PATCH v15 00/16] Add audio support in v4l2 framework Sebastian Fricke
2024-04-30 8:47 ` Hans Verkuil
2024-04-30 13:52 ` Mauro Carvalho Chehab
2024-04-30 14:46 ` Mark Brown
2024-04-30 15:03 ` Jaroslav Kysela
2024-04-30 16:27 ` Mauro Carvalho Chehab
2024-05-01 1:56 ` Mark Brown
2024-05-02 7:46 ` Takashi Iwai
2024-05-02 8:59 ` Mauro Carvalho Chehab
2024-05-02 9:26 ` Mauro Carvalho Chehab
2024-05-03 1:47 ` Mark Brown
2024-05-03 8:42 ` Mauro Carvalho Chehab
2024-05-06 8:49 ` Shengjiu Wang
2024-05-06 9:42 ` Jaroslav Kysela
2024-05-08 8:00 ` Hans Verkuil
2024-05-08 8:13 ` Amadeusz Sławiński
2024-05-09 9:36 ` Shengjiu Wang
2024-05-09 9:50 ` Amadeusz Sławiński
2024-05-09 10:12 ` Shengjiu Wang
2024-05-09 10:28 ` Amadeusz Sławiński
2024-05-09 10:44 ` Shengjiu Wang
2024-05-09 11:13 ` Jaroslav Kysela
2024-05-13 11:56 ` Jaroslav Kysela
2024-05-15 9:17 ` Hans Verkuil
2024-05-15 9:50 ` Jaroslav Kysela
2024-05-15 10:19 ` Takashi Iwai
2024-05-15 10:46 ` Jaroslav Kysela [this message]
2024-05-15 13:34 ` Shengjiu Wang
2024-05-16 14:58 ` Jaroslav Kysela
2024-05-15 20:33 ` Nicolas Dufresne
2024-05-16 14:50 ` Jaroslav Kysela
2024-05-27 7:24 ` Jaroslav Kysela
2024-05-15 14:04 ` Pierre-Louis Bossart
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e63ec6c8-7da7-4b87-b7ff-a71ff12dcfc1@perex.cz \
--to=perex@perex.cz \
--cc=Xiubo.Lee@gmail.com \
--cc=alsa-devel@alsa-project.org \
--cc=amadeuszx.slawinski@linux.intel.com \
--cc=broonie@kernel.org \
--cc=festevam@gmail.com \
--cc=hverkuil@xs4all.nl \
--cc=lgirdwood@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=m.szyprowski@samsung.com \
--cc=mchehab@kernel.org \
--cc=nicoleotsuka@gmail.com \
--cc=sakari.ailus@iki.fi \
--cc=sebastian.fricke@collabora.com \
--cc=shengjiu.wang@gmail.com \
--cc=shengjiu.wang@nxp.com \
--cc=tfiga@chromium.org \
--cc=tiwai@suse.com \
--cc=tiwai@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).