Badminton is the fastest sports game in the world. The shuttlecock can travel at a speed of over 400 km/h. The world record, measured in a laboratory in 2013, was 493 km/h. Because of such a high speed and human-eye perception, the line judges often cannot tell if the shuttlecock was in or out. A referee can judge if there was an out by analysing slow-motion video footage, but that process takes some time. The solution can be a الرؤية الحاسوبية نظام يتيح المراجعة الفورية واتخاذ قرارات القبول/الرفض تلقائياً.

Our specialists were involved in creating such an instant review system for one of our clients. Our team was responsible for the computer vision module and artificial neural networks. Read the case study below to find out more about the project and its outcome.

متطلبات نظام مراجعة الرؤية الحاسوبية في كرة الريشة

The instant review system for badminton works as follows. The number of cameras and their location is selected, so they can see, as a minimum, all the lines and areas around them and a flying shuttlecock approaching the ground. Video streams and visual data from cameras go to computers, where our software processes them and, by analysing the shuttle trajectory, calculates where the shuttle hits the ground by comparing this spot with the court model or previously detected court lines.

A player who disagrees with the linesman’s decision raises their hand and asks the umpire for a challenge. The umpire sends the challenge request to the system operator. They then start the procedure and receive information from the system about whether the line judge has made a good decision. The operator or a designated judge may verify the system’s result and manually correct it. After verification, the system generates an animation and presents it to the players, judges, spectators, and TV broadcasters.

عند تصميم النظام، اعتمدنا الافتراضات التالية.

  • ينبغي أن يكون النظام دقيقاً قدر الإمكان.
  • يجب ألا يستغرق إجراء التحقق بأكمله أكثر من 25 ثانية.
  • ينبغي أن يكون وقت التركيب أقصر ما يمكن، بحيث لا يتجاوز 4 ساعات لكل ملعب.
  • يجب ألا يتطلب التركيب أكثر من شخصين.
  • ينبغي أن تشغل جميع المعدات أقل قدر ممكن من المساحة والوزن.

كانت المتطلبات الرئيسية للنظام كالتالي:

  • تسجيل فيديو بمعدل 150-200 إطار في الثانية.
  • استجابة النظام في الوقت الفعلي تقريباً.
  • خوارزميات فعّالة تعمل على أجهزة كمبيوتر شخصية منخفضة التكلفة.

التحديات والحلول لتطبيقات أنظمة الرؤية الحاسوبية

خلال العمل، واجهنا العديد من التحديات: بدءاً من اختيار البنية والأدوات المناسبة، مروراً بتطوير خوارزميات فعّالة، وصولاً إلى التكامل مع نظام التسجيل المباشر.

computer vision system use case

معايرة الكاميرا لنظام الرؤية الحاسوبية

The camera’s calibration is essential for precise court model fitting when 3D tracking is involved. It is crucial that the calibration process is fast and easy to perform. Figure 1 shows an original image from the camera’s footage (upper) and the same image after applying distortion coefficients calculated during the calibration process.

Computer vision system

الشكل 1. نظام المراجعة الفورية للريشة الطائرة: نفس الصورة قبل وبعد تطبيق معاملات التشويه المحسوبة

Because the system is mobile, the calibration must be done in a venue. We have developed special software for fast calibration. For our calibration procedure, we use a calibration board with circles. The minimum reprojection error we got was 0.086px.

التعرف على المحكمة

To determine if the shuttlecock landed in or outside the court, we must first recognise it. The court recognition procedure is performed for each camera. The court’s dimensions are standardised, so it is easy to recognise the court and fit it into the model. Hough transform [1] can be applied to the Sobel filter [2] output to find court lines. Then the intersections of the lines can be detected by using the Bentley—Ottmann algorithm [3]. Another method is to use the Harris corner detector [4] to detect court corners.

Before running the court lines recognition procedure, we use a modified version of the Mask R-CNN [5] to detect and segment the court or a part of a court from the image. The masks generated by the neural network are dilated before applying them to the image to make sure that all court lines are visible. Then the Hough transform and pattern matching are applied to match the detected court with the model. We then analysed the lines and corners fitting and adjusted the model in case of inconsistencies caused by rubber court mat deformations. The accuracy of court segmentation is 97.7%.

computer vision system

الشكل 2. تقسيم الملعب – نتيجة Mask RCNN

computer vision system

الشكل 3. الدخول (بالأخضر) والخروج (بالأحمر) من منطقة الملعب بعد اكتشاف خطوط الملعب

سرعة الخوارزميات

خوارزميات تحليل الصور that require a long computation time or high computational resources cannot be used. We decided to develop a hybrid solution. It uses simple and fast detection based on differential frames and only sometimes uses a neural network on patches of single frames when the first method loses the tracked object. With the single computer equipped with 9th Gen. Intel® Core™ i7-processor and GPU: NVIDIA® GTX 1080 we can process data from two cameras with a speed up to 200 FPS (resolution 800×600).

ضوء وامض

Algorithms based on differential frames are sensitive to flickering light. The newest and most modern sports halls are equipped with flicker-free lights, but most venues where national tournaments are played have old, flickering lights. Although many methods and commercial software remove flickering, they are unsuitable for fast video-stream processing. We concentrated on the speed with still reasonable and applicable results. We developed an adaptive pixel-wise method of generating masks that compensate for the flickering effect. Our method is 250–300 times faster than DeFlicker[6] and FlickerFree[7]. The method is described in detail in our paper [8].

We use the fact that our cameras do not move, as this allows us to calculate the similarity level of each pixel to the same pixel from the previous frame. If a pixel has changed because of a local movement in a scene, then the similarity level would be low. If similarity levels are higher than the threshold, then we interpret it as the flickering effect, which is reduced by our algorithm. The image below shows in top-right corner original frame, the top-left corner diff frame before applying the flicker removal algorithm, the bottom-left diff frame after applying the flicker removal algorithm, bottom-right original frame after applying the flicker removal algorithm.

Figure 4. Diff frame before applying flicker removal algorithm - top-left corner, diff frame after applying flicker removal algorithm - bottom-left

الشكل 4. إطار الفرق قبل تطبيق خوارزمية إزالة الوميض – الزاوية العلوية اليسرى، إطار الفرق بعد تطبيق خوارزمية إزالة الوميض – الزاوية السفلية اليسرى

كشف الريشة وتتبعها وتقسيمها

To be able to track any object, firstly, it must be found within the image. Many methods can be used for finding a specific item and object detection, from the most straightforward, such as colour and shape segmentation, to methods utilising neural networks.

استخدام نهج عام وشامل للكشف عن الريشة يمثل مشكلة للأسباب التالية:

1. اعتمادًا على زاوية رؤية الكاميرا، يمكن رؤية ظل ريشة الريشة الطائرة على شكل دائرة أو مثلث، لذا فإن العثور عليها داخل الصورة أكثر تعقيدًا من العثور على كرة.

2. قد يتغير حجم الريشة بشكل ملحوظ عندما تقترب من الكاميرا.

3. تتحرك كرة الريشة بسرعات متغيرة بسرعة، وتصل في بعض الأحيان إلى قيم عالية جداً. يجب أن تعمل الخوارزمية بشكل جيد مع سرعات كرة الريشة السريعة (400 كم/ساعة) والبطيئة (24 كم/ساعة).

4. قد يكون اللون الأبيض لريشة الكرة الطائرة مشابهاً جداً للمشهد (جوارب اللاعبين، وخطوط الملعب البيضاء، والحروف على لوحات الإعلانات).

5. غالبًا ما تتحرك الريشة في سياقات معقدة: لاعبون متحركون، مضارب سريعة، وخلفيات متغيرة.

6. Because of the cone-like shape and centre of gravity located next to the cork (not in the centre), it is difficult to predict the trajectory following the physics laws described by simple formulas, especially after the hit, when the shuttlecock turns over and has an unstable trajectory.

We utilise the feature that a shuttlecock is a fast-moving object over a usually still background. Differential images generated from consecutive frames allow us to distinguish moving objects from static backgrounds. On the other hand, when the shuttlecock hits the ground, it changes direction and may even stop moving for a moment and vanish from the differential image. Our solution generates an accumulative differential frame from 7 consecutive differential images. This allows us to keep sight of the shuttlecock when it hits the ground. Figure 5 shows an accumulative differential frame (a) and the direction of moving blobs marked by lines (b). The blob with the white outline in Figure 5b is the one that was recognized as a shuttlecock. Other blobs in this figure were filtered out. The colour of a blob represents the filter that filtered out the shuttlecock candidate.

computer vision system

الشكل 5. تقسيم نظام الرؤية الحاسوبية: الإطار التفاضلي التراكمي (أ) (على اليسار). الكتل مع اتجاه الحركة المحدد بخطوط (ب) (على اليمين).

Shape segmentation makes it easy to distinguish a shuttlecock from another moving object. The whole process takes less than 2 ms on an Intel Core i7 machine. There are situations when, behind a shuttlecock, a player is moving. The blob of a player is usually much bigger than the blob of a shuttlecock, and both blobs overlap. The segmentation algorithm may mistakenly treat the shuttlecock as a fragment of a player. To solve it, we follow the algorithm:

1. باستخدام مرشح كالمان [9]، توقّع المكان الذي يجب أن تكون فيه المكوكة، بناءً على المسار من الإطارات السابقة.

2. إنشاء رقعة (104 × 104 بكسل) من الإطار الأصلي بالمركز المتوقع بواسطة مرشح كالمان.

3. تمرير تصحيح مُولَّد إلى كاشف شبكة عصبية كمدخل

Our neural network is a modified Tiny Yolo [10] trained with a set of images collected during the tournaments. The detection of a shuttlecock with a neural network takes 12 ms and causes a delay, which is, however, acceptable. When the first method finds the shuttlecock again, it eliminates the delay caused by the neural network, and the entire algorithm works in almost-real time. The neural network with a single class (shuttlecock) achieves a mean average precision (mAP@0.50بنسبة 94%، وبحد عتبة دقة 0.25، يحقق 0.96، واستدعاء 0.77، ودرجة F1 0.86.

أداء المتتبع

To evaluate the tracker performance of our computer vision system for badminton, we needed reference data. In this case, we could use data recorded in a laboratory, but we decided to use data from real tournaments. Such a method ensures that the calculated accuracy will match the accuracy of the system under real conditions. Moreover, we decided to use only recordings when the player disagrees with the line judge’s decision. Each time the player asked for verification, our system saved 30 seconds of camera footage at 150-190 fps. From recorded movies, we selected 52 sets of frames for annotation. We annotated a total of 20,788 frames.

Our tracker detects if the shuttlecock is visible with an accuracy of 94%. For the frames where the shuttlecock was visible, we measured if the detected position of a shuttlecock was correct. The detection of a shuttlecock position was marked successful if the difference between annotated position and the position saved by the detector was less than 12 pixels. 81% of the visible shuttlecocks were correctly detected and tracked.

تجزئة الكائنات لنظام الرؤية الحاسوبية

While the shuttlecock moves at high speed, it is blurred within the image. The motion blur effect causes segmentation difficulties due to ambiguous pixels between regions of the object and the background. Imprecise shuttlecock contour detection reduces the accuracy of localising the cork when it comes into contact with the ground, impairing the overall system accuracy.

computer vision system

الشكل 6. الفرق المتراكم (بالأصفر) مع الصورة الأصلية. الدائرة الحمراء = نقطة ملامسة الأرض. السهم الأزرق – اتجاه الحركة

إيجاد الإطار عند ملامسة الريشة للأرض

The standard approach for finding a ground hit is a trajectory change. It works well for tennis, where the ball is resilient and the bounce is high. A shuttlecock does not bounce very high, but when it hits the ground, it slows down (and even stops for a moment). However, sometimes it may slide, so the trajectory does not change significantly. The sudden change in speed is also evident in the change in the degree of blurring.

computer vision system

الشكل 7. كرة الريشة قبل لمس الأرض وبعده

Figure 7 shows a set of images of a shuttlecock before and after the ground touch. As can be seen, the speed of a shuttlecock changes more significantly than the trajectory. That is why, except for using trajectory-change information, we also utilise the shuttlecock blurriness measure calculated as the variance of the Laplacian.

النتائج

The computer vision-based system presents the results in the form of an artificial animation. Such a solution is attractive to the audience and prevents discussions and doubts when the original camera image is blurred, or the shuttlecock is partially obscured. The examples of our challenge animations from tournaments can be seen here:

The system delivers a correct decision in 62% of cases. Even a tiny error in one of the steps of the processing pipeline, i.e., shuttlecock detection, shuttlecock tracking, ground hit frame detection, line detection, cork segmentation, and in–out decision, may lead to the wrong system’s answer. That is why, in the system, the human operator confirms the final in/out decision.

اكتشف المزيد عن خدمات الرؤية الحاسوبية لدينا

اعرف المزيد
bird-eye view of streets and smart cars using computer vision
اعرف المزيد

مواصلة العمل على نظام رؤية حاسوبية لرياضة الريشة الطائرة

يتمثل التحدي الرئيسي في تحسين دقة النظام. يمكن زيادة الدقة باستخدام كاميرات ذات دقة أعلى أو زيادة عددها.

The capabilities of hardware available on the market change over time: The newest cameras have the same speed but much higher resolution than the ones we bought a few years ago. They are also much more expensive.

Buying better cameras is not enough because higher resolution means that our system must process much more data at the same constrained time. Increasing the computing power of computers must be combined with changes in algorithms to adapt to much larger data streams.

It should be noted that the camera’s high resolution is only needed when the shuttlecock hits the ground. A higher acquisition speed is needed to determine the moment at which the shuttlecock rebounds against the ground. A higher resolution can also be obtained by using super-resolution algorithms, which may be an element of further investigation.

نظراً لأن القرار التلقائي بشأن سقوط الريشة داخل ملعب اللعب أو خارجه يجب أن يُعرض خلال 10–25 ثانية، ينبغي اختيار المعدات والخوارزميات بحكمة.

استفد من وحدة الرؤية الحاسوبية وتقنية الشبكات العصبية الاصطناعية

Computer vision technology together with artificial intelligence algorithms can be used for many purposes. Specific object tracking and detection is just one of them. Analysing images, video frames and visual data can help with image recognition, pattern recognition and object classification in various solutions across industries.

يمكن لشركتك أو نظامك أو تطبيقك الاستفادة من هذه الحلول أيضًا. أخبرنا عن فكرتك – وسيساعدك خبراؤنا في مجال الذكاء الاصطناعي على تحقيقها. قم بزيارة عرضنا للذكاء الاصطناعي وتعلم الآلة للحصول على مزيد من التفاصيل عن خدماتنا أو تواصل معنا عبر النموذج أدناه.

الأسئلة الشائعة

The instant review system is a computer vision-based solution that helps determine the exact landing position of a shuttlecock in badminton. It assists referees in making more accurate and objective decisions during matches. System aims to eliminate human error and enhance the overall quality of officiating in the sport.

Computer vision is central to the system, enabling it to track shuttlecock movement and determine its position on the court. High-speed cameras and image processing algorithms work together to analyse footage in real-time. This technology helps in identifying whether the shuttle landed in or out with high accuracy.

The system utilises supervised learning techniques to train models on annotated shuttlecock images. It uses object detection algorithms to locate the shuttle in video frames. These models improve over time as more training data is introduced, enhancing accuracy and response time.

Key challenges include handling the high speed and small size of the shuttlecock, as well as ensuring consistent lighting and camera calibration. Another issue is processing large amounts of video data with minimal delay. Balancing accuracy and performance is also a significant concern during system development.

The system has demonstrated high accuracy in detecting shuttlecock landings, especially in controlled environments. It relies on well-calibrated hardware and optimised algorithms to maintain precision. Continued testing and data collection further improve its reliability.

Yes, the system is specifically designed for real-time application, allowing referees to request instant reviews during play. Fast data processing and clear visual outputs make it highly effective for use in tournaments. Its response time is tuned to ensure decisions are made without interrupting the game flow.

المراجع

  1. Duda, R. O. and P. E. Hart, "Use of the Hough Transformation to Detect Lines and Curves in Pictures," Comm. ACM، المجلد 15، ص 11–15 (يناير، 1972)
  2. إيروين سوبل، 2014، تاريخ وتعريف مؤثر سوبل
  3. Bentley, J.L.; Ottmann, T.A. Algorithms for reporting and counting geometric intersections. IEEE Trans. Comput. 1979, C-28, 643–647.
  4. كريس هاريس ومايك ستيفنز (1988). "كاشف الزوايا والحواف المدمج". مؤتمر Alvey للرؤية. المجلد 15.
  5. He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-CNN. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 2961–2969.
  6. DEFlicker—RE: Vision Effects. متاح على الإنترنت: https://revisionfx.com/products/deflicker/
  7. برنامج إزالة الوميض Flicker Free من Digital Anarchy. إضافة Flicker Free: إزالة الوميض من التصوير المتقطع وLED والحركة البطيئة/الإطارات العالية. متاح على الإنترنت: https://digitalanarchy.com/Flicker/main.html.
  8. Nowisz, J.; Kopania, M.; Przelaskowski, A. Realtime flicker removal for fast video streaming and detection of moving objects. Multimed. Tools Appl. 2021, 80, 14941–14960.
  9. Kalman, R.E. A New Approach to Linear Filtering and Prediction Problems. Trans. ASME—J. Basic Eng. 1960, 82, 35–45.
  10. Redmon, J.; Farhadi, A. YOLOv3: An Incremental Improvement. arXiv 2018, arXiv:1804.02767.

الملحق 1: جدير بالمتابعة

arrow_circle_right جدول المحتويات

Michal Kopania

مهندس برمجيات الرؤية الحاسوبية

مقالات ذات صلة