USB streaming buffers

USB streaming buffers

DVB USB allocates rather big chunks of coherent memory buffers for the USB streaming at the start and frees those on disconnect. One buffer is reserved for each used USB URB, which is around 5-7 buffers as there is that many URBs used generally. After the multi-frontend (MFE) implementation own USB streaming buffers were allocated for every frontend. Currently there is three frontends registered for the single adapter in worst case. And what is the problem: only one frontend can be used at the same time and we allocate buffers for all the three!

I resolved problem allocating biggest needed buffer at the start and changed all frontends in given adapter to use that. URB configuration is performed dynamically to fit needed stream. Quite lightweight solution. Maybe there is still room for the improvement as it allocs and frees URBs every time when streaming configuration is changed. Allocating and freeing few URBs is very cheap operation but it could be avoided unless USB endpoint type is changed (bulk or isochronous). And after all use case where frontend is changed all the time is very unusual. User just choose one frontend, which offer needed television standard, and uses that. So it must be fine :)

What is expensive in whole subject is coherent USB transfer buffer and now it is used 2/3 less than earlier. Someone would like to ask why not to alloc and frees all the memory, URBs and buffers, every time when streaming is started and stopped. There is very good reason to avoid continuous allocs and frees as memory goes fragmented and finally allocation of big coherent chunks of buffer memory will fail. That is realized few times in the recent future as some other DVB devices are stopped working. Limited memory is especially problem of embedded devices.  Believe or not, it is very common DVB sticks are used on embedded routers or STBs.

Related links

[PATCH] em28xx: pre-allocate DVB isoc transfer buffers
http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/44570

DVB USB: change USB stream settings dynamically
http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/48861

GSoC week 21.

Week 21 - planning

 

First week

First week is almost over. It is almost over since I am still going to do some USB streaming buffer tests tonight :-)

This week was scheduled as a RFC and planning for DVB USB changes. DVB USB means those general routines used by almost every digital television USB stick. I think it could be called as a sub-driver, framework, library or even DVB USB core. I still like to call it just DVB USB or DVB USB framework.

I did early start for the first week and sent two RFCs describing planned changes 1-2 weeks earlier. It was not a big surprise there was no many comments. Due to that I think changes I was introducing are not totally wrong :)

Implementing

As there was no much discussion about introduced changes I started implementing those. I copied current DVB USB implementation as a starting point, leaving old implantation still alive as a parallel. It is almost impossible to make such big changes for the existing one as there is about 30 individual DVB USB interface drivers using DVB USB core routines. It is still unknown how to make switch-over for the new implementation. I see two possibilities; 1) move driver by driver to the new 2) move all changes one by one to old implementation.

During the development I converted and tested all the time three of my drivers against new routines. Those drivers were au6610, ec168 and af9015. af9015 is one of the most complex DVB USB device drivers and full of hacks. Problems with the af9015 has been my main motivator for most of these DVB USB changes I planned.

Some improved numbers

Changes what I did this week removed many hacks from the af9015. Also those reduced binary size of the af9015 very much.

      text   data  bss    dec   hex  filename
old  14878  13232   40  28150  6df6  dvb-usb-af9015.ko
new  14575   3952    8  18535  4867  dvb-usb-af9015.ko

Space saving is coming from the new way of passing data to the DVB USB framework. Earlier all device data was put inside static struct as now it is passed using driver_info field of the struct usb_device_id.

Size of that configuration structure was always constant, no matter if there was one or 12 devices as current limit is. af9015 driver contains 3 of those big configurations structures. There is some drivers having ~30 similar structs - guess how much space is wasted?

Related links

[RFCv1] DVB-USB improvements
http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/47994

[RFCv1] DVB-USB improvements [alternative 2]
http://comments.gmane.org/gmane.linux.drivers.video-input-infrastructure/48572

My DVB USB development tree

http://git.linuxtv.org/anttip/media_tree.git/shortlog/refs/heads/dvb-usb

Blog is open!

About the Blog 

 

Google Summer of Code 2012 attendee!

I have been planning own blog for my Kernel development stuff for a while but newer got enough time nor motivation to learn things to start blogging. Things gone ahead after I was chosen for the Google Summer of Code programme as there was some pressure for the blog to report progression of the project.

My GSoC project is under the Linux Foundation umbrella and it is Kernel development for my favourite area - Linux television interface. Traditionally I have been coding only new television chip drivers. Type of those chip are; USB-interface (aka DVB USB bridge), demodulator and RF tuner. There is around 20 chip drivers currently I have done which is huge amount of drivers for the single developer.

Current problems

During the development of those drivers I have gained very good overall understanding of the whole digital television side of the Linux Kernel and also that technology in general. It is not big surprise I have met a quite lot general issues, issues that are outside of individual driver, somewhere in our core routines. Those issues and limitations are raised up during a long time as hardware has gone more complex and old routines are not flexible enough for the new hardware.

Due to those core limitations there is many ugly hacks needed for the drivers in order to operate. Sometimes those hacks are even very wrong but still mainly working with a little bit luck. More often those hacks still means "only" duplicated code, complex code and wasted resources.

Usually I am too busy to hack support for new the hardware and due to that I have no possibility to allocate enough resources for fixing known general issues. Now that GSoC project gives me possibility to fix those!

GSoC schedule

That GSoC project is workload of three months. Initially I split it to the two big parts, DVB USB enhancements and DVB core enhancements. DVB core contains mainly DVB frontend changes.

Planned schedule is here:
week 21: DVB-USB plan driver interface + RFC
week 22: DVB-USB implementation
week 23: DVB-USB implementation
week 24: DVB-USB implementation
week 25: DVB-USB implementation
week 26: DVB-USB convert few existing drivers to use that new interface
week 27: DVB-CORE learn frontend handling
week 28: DVB-CORE plan needed changes + RFC
week 29: DVB-CORE implementation
week 30: DVB-CORE implementation
week 31: DVB-CORE implementation
week 22: DVB-CORE convert few existing drivers to use that new interface
week 33: pencils down, finalize
week 34: pencils down, finalize