Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 6, No. 3, 2023 59 Research on the Application of AI Frame Compensation Technology in Videos Endong Wang Cheongju University, Korea Abstract: Booming AI (Artificial Intelligence) technology, amid advance in computer science, is assuming a vital role permeating all levels of people's lives and jostling for widespread attention. Asking for perspectiveness by the video industry for production technology labels AI technology as a means of great significance in the creation of videos, especially in frame compensation. This paper is to cotton on features and prevailing AI frame compensation, sum up its application in image creation, and concludes deficiencies and prospects therefrom, so as to chart course for the future development and research of imaging workers and scholars in the field. Keywords: AI frame compensation, Video frame compensation application research. 1. AI Frame Compensation Technology In the new era of imaging, a higher threshold is reported in the quest of impressions, as the past pursuit of contents has been shifted. Either in a film and television drama or other types of video works such as short videos and animations, viewers are exacting better resolution and fluency. The high frame rate shooting of cameras, coupled with the high refresh rate of terminal playback screens, pictures a future where high frame and ultra-smooth watching experience prevail. In this regard, AI frame compensation technology rises in response to the long-before-made works, outdated technology and low- cost low-frame videos. 1.1. Frames Frame, the minimum unit in the playback of a video work, makes a freeze-frame image; frame rate, another term usually expressed in frames per second, counts frames a dynamic image contains per unit time. As early as 1872, british photographer Edward Mulbridge, to clear the debate about whether a horse always lands on one foot during running, shot 24 photos of horses running by 24 cameras, and made a strip of photos in sequence. Later, a fortuitous pulling moved the original static works, marking the birth of film technology. Persistence of vision enables us to see the smooth motion effect on the screen, studies found. The close continuous playback of the two frames before and after shows the changing of motion. Theories have it that the more the frames played continuously per unit time, the smoother the image changes seen by the human eye, putting the industry on a path to high frames. To secure the smoothness of the audience's perception and a more realistic visual experience, mobile players, in addition to the strict frame rate standards on the traditional playback terminals, are raising the ceiling of the frame rate playback, such as 90 or 120 frames in some video websites. In the normalization of high frame rate video production, frame compensation technology is of great practical significance for scaling new heights in the impressions of low-frame-rate works. 1.2. AI frame compensation technology The established frame compensation technology mainly gets down to "repair" and "supplementation", that is, to repair damaged or missing frames and to fill in missing frames due to increased frame rate or prolonged video playback time. Research on such technology was started in recent years by two display technology companies, AMD and NVIDIA, and now grows popular in a short period of fast development. To compete in the computer display market, they keep enhancing the display effect. As a result, AI frame compensation emerged as one of approaches to improve the display and optimize the calculation algorithm of the graphics card. In January 2018, AMD Corporation bought out ATI and came up with AMD Fluid Motion thanks to the enhanced graphics card production capacity thereout. Its emerging technique smoothed and sharpened shootings with a frame rate of 24fps or lower than 60fps by magically compensating them to 60fps. Yet great restrictions that came with this inchoate technology seemed to work only for simple movements of planar objects in the screen. In other words, its witchcraft had been weakened in dealing with a picture rife with objects or complex movement. In the same year, NVIDIA took one step further. At the Computer Vision and Pattern Recognition conference held in Salt Lake City, USA, NVIDIA brought out a new frame compensation technology. According to the official saying, this new one, given a practice of processing over 11,000 videos in NVIDIA Tesla V100 GPU and cuDNN, learns to position and complement frames. That enables it to generate a 450fps slow-motion video out of a 30fps work, and slow down the original video rate by 15 times to achieve a super slow-motion effect1. In such a process, the newly inserted picture is no longer a supplementary picture obtained by simply calculating the difference between the front and back frame images. It naturalizes the motion blur effect of the picture while wiping out the untimely pause and transition, bringing highlights to audience in distinct pictures. In 2020, a new algorithm DAIN (Depth-Aware Video Frame Interpolation) introduced by Shanghai Jiao Tong University outperformed that by NVIDIA as it secured clearer processing effect and higher frame rate. With its hands, a 30fps video is further interpolated to 480fps, which has exceeded the slow-motion video frame rate of many professional photographic equipment. It covered the shortage of poor enhancement after frame interpolation for complex object motion or occlusion. Researchers also developed a 60 technique: a depth-aware optical flow projection layer to synthesize intermediate flows that sample distant objects, and then the model warps input frames, depth maps, and contextual features based on optical flow and local interpolation kernels, synthesize the output frame . The frame compensation technology is no longer a tool for live videos, but finds wider usage in any common type, covering the field of video works. DAIN algorithm is currently an ideal achievement for frame complementing and applications. 2. Application of AI Frame Compensation Technology in Videos High quality, convenient operation and open technology have popularized the ever-improving AI frame compensation in the field of video production, especially in the following aspects as of today. 2.1. Improve the fluency of dynamic images The viewing experience brought by low-frame movies is no match for that by high-frame ones, backed by theoretical outcomes. Recent years witnessed more theater works done at high frame rates, such as the two 120-frame "Billy Lynn's Long Halftime Walk" and "Gemini Man" by Ang Lee. This kind of experimental high-frame rate exposed audience to a journey like never before; in scenarios of the hearty chase, the smoky explosion, and the high-speed passing train, those sitting on chairs can even see clearly passengers in the carriage. These realistic and clear images remind people of the importance of high frame rate in film and television works. Images shot by ourselves turning a circle on our mobile phones are completely different from that seen by observing a circle at the same speed with the naked eye. A growing number of video practitioners are keen to increase the frame rate, and audiences also expect this. Under this trend, the said technology is widely used to increase the frame rate of completed video works for better viewing effects. In addition, to match the ever-increasing refresh rate of display devices such as mobile phones, monitors, and TVs, video websites and other playback terminals also need to increase the frame rate of video sources and turn to frame compensation as one of the technical solutions. 2.2. Slow-motion production People wish to appreciate fleeting moments in life clearly and slowly, but the premise of slow playback or replay is that the camera can capture these moments. For example, in a high-level diving competition, watching the athlete's beautiful moment of entering the water and the little splashes of water requires slow motion. In the post-editing work of traditional film and television, it is usually avoided to slow down or stretch the non-high frame rate material. Forcibly increasing the length of the video will cause the editing software to automatically copy the picture to fill in the missing frames due to the increase in the length of the video. Doing so will freeze the picture, seriously spoiling impressions. To get slow motion shots, creators have to pay for expensive high-speed cameras in pre-shooting. But now, AI frame compensation slow playback technology is very mature. The Super SloMo technology developed by NVIDIA can already turn any video into a "high-definition slow playback" mode. By using the deep neural network, it can predict and complete the missing pictures in the slow video. It has been implemented to slow down the video by 8 times or more, and can eliminate the pixels blocked by the foreground and other interference factors in the video frame, so as to avoid false or blurred pictures. Amid the progress in AI frame compensation technology, non-high frame rate videos can also achieve slow-motion effects, and the video playback effect has also been greatly improved. 2.3. Plane animation production In the production of graphic animation, the current mainstream production method is still hand-painted 2D animation, which combines hand-painted still frames to form an animation video. With the development of the times and technology, the scene content and actions of animation have become more and more complex. to reduce the workload of the drawing department, new technologies have gradually been used in the production of graphic animation. Animation is the same as other film and television works, of 24FPS in frame rate. But under normal circumstances, not every frame of picture is a valid, but repeated. For instance, background buildings, still characters around the protagonist also provide good conditions for the use of AI supplementary frame technology in planar animation production. Hand-drawn animation adopts different production methods according to the content of the animation. Usually, the so-called one-shot- two (one picture stays for two frames) method is used, that is, 12 still frames are hand-painted per second for animation. Some production teams pursuing the ultimate animation visual effects even choose the one-shot-one production method (one frame stays one frame)3. This has led to a surge in the painting workload and production costs of the hand- painting department. In addition to improving the smoothness of the video, the fame enhancement technology has become the focus of attention and exploration of producers to reduce the workload of hand-painting in the early stage. Japanese animation director Masaaki Yuasa in the animation work "Don't Develop Hands with Images", Some scenes tried to use automatic mid-cut technology to improve the smooth effect of hand-painted still frames with fewer pictures. After the practice of the producers, although the AI video supplementary frame technology can reduce the workload of the hand-painted pictures, but because the animation is not the actual shooting effect, there are often a lot of illogical and irregular movements or a lot of occlusion in the picture. It cause AI frame supplement calculation errors, resulting in screen tearing, "ghosting" and other phenomena. Generally, there are strict restrictions on the use of AI video supplementary frame technology in planar animation. Only in scenes with regular motion such as uniform translation, the effect comes good. Under other conditions, it is difficult to achieve a smooth effect, and the "disadvantages" brought about by the overall use of AI video frame supplementation often outweigh the "advantages". The graphic animation industry is also looking forward to the future development of AI frame complementing technology, which can improve the above disadvantages and apply more mature frame complementing technology to graphic animation creation as soon as possible. 2.4. Repair old image Old video materials shot in those days result in defects in quality for the undeveloped photography technology. These old image materials still have important commemorative significance today, and the restoration of such image 61 materials is of great significance. In recent years, the film restoration industry has gradually caught public attention, and more digitally restored versions of classic films are staged in theaters. The documentary "They Shall not Grow Old" combines 100 hours of video historical materials and 600 of audio materials. The frame rate is not the same when using different video historical materials. The production team passed AI video supplementary frame adjustment makes it present a smooth and coordinated playback effect at a regular 24FPS frame rate. The "China Film·Shensi" artificial intelligence image processing system used by the China Film Base has also restored a large number of film and television dramas, and AI video frame compensation technology has also played an important role as one of its functions. 3. Existing Problems First of all, AI frame compensation technology that bases computer algorithm is not perfect, just as any of AI technology. It still bears the risk of bad frames, which is relatively high. Once a bad frame appears in the video, it will seriously affect the video perception, and loses the value of frame compensation. Secondly, the work time and cost of AI video frame supplementation are relatively large, unbearable for ordinary producers. The job mainly occupies the cuda of the graphics card, the temperature of which will also increase greatly when running the supplementary frame work. The long-term high-temperature and high-load operation is obviously harmful to the hardware, which also leads to many producers. Although they have mastered AI Video complementing frame technology, but still are unwilling to perform frame complementing work in exchange for theoretically improving impressions. Finally, the image quality will be slightly reduced for frame compensation. Although the image quality can be guaranteed by controlling the compression bit rate, the exponentially expanded video volume is also a problem that is difficult to solve, especially for videos uploaded through video websites. To sum up, despite its growing perfection and easy operation, the existing problems still hinder the full popularization of AI video supplementary frame technology, and video producers are urging to maintain a high degree of interest and wait-and-see manner. 4. Conclusion Against the comprehensive development in various fields of AI, driven by continuous improvement in recent years, AI frame compensation technology has grown mature. The needs of producers in the video industry and the expectations of audiences continue to stimulate the market, prompting the continuous development of AI video supplementary frame technology. While enjoying the convenience brought by AI technology, we should also face up to its "repair" attribute and treat it with caution, so as to avoid the convenience of technology from creating inertia for creators, leading to "remedial work" in the creation of video works. AI's deep learning is based on human's own cognition, and the development of technology is to fuel the creation of images, rather than become a means to achieve theoretical image quality. As one of the eight major arts, the creator's expression and thinking about film and television works cannot be replaced by any AI. In the future, we will wait and see whether it will brings forth a deeper impact on the creation of film and television works. References [1] Xie Xin. Analysis on the Image Application of AI Frame Compensation Technology[J].Modern Film Technology, 2020(8):21-24. [2] Heart of Machine. Turn ordinary video into slow motion: "AI frame addition" technology is now open source [EB/OL]. 2018-12-29. https://zhuanlan.zhihu.com/p/53614509. [3] Gurren. Japanese TV Animation and Frame Compensation Technology [EB/OL]. 2021-3-20. https://www.zhihu.com/column/c_1356338548656975872.