Instruction FACTA UNIVERSITATIS Series: Electronics and Energetics Vol. 29, N o 1, March 2016, pp. 113 - 126 DOI: 10.2298/FUEE1601113V 3-D STEREOSCOPIC MODELING OF THE TESLA’S LONG ISLAND  Vladan Vučković, Sanja Spasić Faculty of Electronic Engineering, University of Niš, Serbia Abstract. This paper presents in detail the methods for realization of the basic software infrastructure for the conversion of 3-D animation of Tesla’s laboratory in Long Island to modern stereoscopic 3-D formats. Modeling of Tesla’s lab is done in cooperation with the Nikola Tesla Museum in Belgrade on a project entitled “Computer Simulation and Modeling of the Original Patents of Nikola Tesla” approved by the Ministry of Education and Science of the Republic of Serbia. In recent years, there has been a revolution in the field of 3-D technology, so it is clear that this will be the strategic direction of the progressing of television, cinema screenings and presentations in the future. Using modern technology for generating and conversion to stereoscopic 3-D format, the authors show in detail the procedure that was used in the realization of this segment of the project. The complete improved 3D developing pipeline from the original photograph to the stereoscopic 3D real-time model is also presented. The novelty in the phase of semi- automatic materialization of the wire models is also described. Key words: Computer simulation, 3-D modeling, Nikola Tesla’s Long Island Laboratory. 1. INTRODUCTION Three-dimensional (3-D) [1], [2] or stereoscopic 3-D (S3-D) is the presentation which improves the illusion of the depth of perception. It creates the impression of the third dimension. It is accompanied by 3-D technology based on all logical principles. However, the feature that makes this technology extraordinary is the very process of the picture presentation. 3-D or stereoscopic picture is connected to the ways in which eyes and brain create the impression of the existence of the third dimension. Two very similar pictures reach the brain creating a particular disparity. This disparity actually represents 3-D, the third dimension projected by our brain as it receives two different perspectives of the same thing. This very effect is the goal that modern 3-D technology tries to accomplish through distribution of different sequences towards both the left and the right human eye. This kind of technology is very fast spread and acquired by the users. Since all new contents of pictures, cartoons and animated films are being made in some of these formats, Received December 26, 2014; received in revised form September 15, 2015 Corresponding author: Vladan Vuĉković Faculty of Electrical Engineering, University of Niš, Alaksandra Medvedeva 14, 18000 Niš, Serbia (e-mail: vladanvuckovic24@gmail.com) 114 V. VUĈKOVIĆ, S. SPASIĆ it is very important for this high quality material to be accompanied by appropriate players which will completely point out, emphasize and show the quality. Some of the most commonly used media players nowadays which support and make possible the presentation of stereoscopic materials, and in most cases films, are: Stereoscopic Player, nVidea 3-D Vision Video Player, Ultra Studio 3-D, p2gStereoStage, Firmware version 3.50, etc. Since 3-D technology is very rapidly acquired, the formats as the main guide lines of this technology are being developed rapidly, too. Stereoscopic formats belong to the group of standards for coding of audio and video signals of the third dimension. These new formats are more complex and they require more powerful devices for processing, but better efficiency is achieved - better quality of video signal can be achieved for the same flow of data. One of the formats which appeared on the market and which stand out in the whole range of the formats is JPEG2000. This format can be in the easiest way converted into MXF format, where it is the very part of this format. This paper has the following structure. First, we describe standard stereoscopic formats that are in use nowadays. Our 3D application is based on these formats. Then, we describe the complex procedure of generating the 3D model of Nikola Tesla Long Island lab, from origin photographs to stereoscopic model. We introduce original algorithm for semi- automatic pattern re-changing. This technique is very helpful to improve and accelerate this segment of the project. The novelty accomplished in our work could be divided in two categories: First, we developed the original and complex 3D producing pipeline – from original picture to stereoscopic 3D model with these basic steps: 1. Manual 3D modeling based on original pictures. 2. Assembling the complete wire model. Long Island contains over 100 separate 3D models. 3. Materialization of the wire construction using our innovated procedures (Section 6). 4. Generating the JPEG2000 Stereoscopic 3-D model. Secondly, many of these steps are innovated or improved. The main developing line is hardly based on manual 3D projecting (especially in the phase of basic 3D modeling from original pictures), but many improvement could be accomplished in some phases. In this paper we presented the improvements in the phase of materialization. The complete C++ source code and application could be inspected after the contact with authors. 2. JPEG2000 FORMAT JPEG2000 [3], [4] is the new system of image coding. It is created by Joint Photographic Experts Group as a supplement to JPEG format. Its architecture is very useful for many different applications, including distribution of internet images, security systems, digital photography and medical records. The main characteristics of JPEG2000 standard are:  Losses compressions - it compresses pictures without loss, increases picture quality and decreases storage space,  Supports all resolutions, color depth, number of components and frame rate,  Provides higher subjective and objective quality of the image, especially at slower transmission speed,  Enables ROI-Region of Interest Coding,  Not sensitive to bit errors, 3-D Stereoscopic Modeling of the Tesla’s Long Island 115  Not sensitive to errors in the transmission channel,  New image format enables protection of intellectual property over the image. Apart from the above enumerated characteristics it should be mentioned that JPEG2000 provides precision of 1-127 bit/rate. The components may have variable precision or rate factors. Compression may be with or without loss. Restoration of the image may be progressive according to the criteria of quality or resolution. It provides visual improvement compared to JPEG for about 20%. It also provides better protection of the image for the same ratio of noise signal which exceeds the JPEG standard by almost two times. Users may notice advantages of JPEG2000. This effect is achieved by using wavelet compression. This compression provides many advantages compared to discrete cosine transform (DCT) used by JPEG format. This transformation divides the image into blocks of 8x8 size and stores them in special files. In this process of compression, blocks are compressed individually, without taking into consideration the neighboring blocks. This is a problem in compressing of the JPEG files because only the most important information is reported at the high-level and thus only the most important parts of the image are transferred. Due to this, there is a loss of certain parts of the picture and there is a loss in image quality. However, wavelet compression converts the image into a series of wavelets which are better stored and the transmission of blocks is done pixel by pixel. In this way images with large and smaller contrast are transferred with the same quality. To transfer an image the right transformation is first applied to the original image data. Transform coefficients are then encoded and sampled. The decoder is simply contrary to the encoder. Unlike other schemes for coding, JPEG 2000 can be both with without losses . It depends on using wavelet transform and quantization method. In JPEG 2000 compression of the image is divided into overlapping blocks (panels). These blocks are compressed independently, as if they were completely independent parts of the image. All operations: wavelet transform, quantization and entropy coding, are performed independently for all blocks separately, as shown in the figure. Fig. 1 Compression blocks. Each of these blocks is defragmented into particular number of parts by DWT (sub- bands). Application procedure of one-dimensional filters in both directions is then repeated in the block of low resolution. When using DWT in the one-dimensional blocks (sub-band) they are divided on the set of low-pass and a set of high-pass samples. The low-pass samples represent a small low-resolution version of the original. The high-pass javascript:void(0) 116 V. VUĈKOVIĆ, S. SPASIĆ samples represent a smaller version of the original. This step is needed for perfect reconstruction of the original set from the low-pass set. The question that is raised now is how to add audio track to the material recorded in the format JPEG2000 without losing the quality. For this kind of conversion it is the best to use the format MXF and there is a very acceptable way of converting for such purposes. As we already know, JPEG2000 enables only the view of video without any possibility of installing audio. Conversion into MXF makes possible for the inclusion of the four channels of uncompressed audio and video frame with a precise time code without loss in the compressed video. This kind of compression may be traced in the program Sony Vegas Pro 11.0. 3. COMPRESSION IN SONY VEGAS PRO 11.0. Sony Vegas Pro 11.0 is a professional software solution for nonlinear video installing (installing performed on the computer). The basic purpose of the program is installing (choosing of cadres, cutting and assembling) and rendering of that material. This version of Sony Vegas is very interesting because, apart from other functions, it makes possible rendering of the material in some of the stereoscopic formats. When it comes to stereoscopic video and computer graphics used in presentations, objects may be practically rendered and presented in a very high resolution. Such generating of 3-D scenes may contain image tags of the depth and can be rendered with increased quality. Apart from this, it also offers different styles of viewing for all kinds of tastes and glasses. Here will be shown the way of arranging and transferring of 3-D stereoscopic recording in Vegas Pro 11.0 regime. The process of rendering as 3-D stereoscopic record begins just after the uploading and installing of video is finished. It is very important before the uploading of any material into program, to make particular settings of the video, so that all the material can be equal concerning quality, as well as resolution. This is very important if one works with material of various resolutions and formats. This setting may be accomplished in one of the two possible ways. The first way is by direct connecting of the cards File/Properties, and the second way is by choosing the short cut from the tool bar. When all the settings have been done, the upload of the material for processing and installing can start. After finishing, a new record can be rendered as a stereoscopic video. It can be achieved by following the path File/Render As. After starting a card with a lot of settings appear and the most important settings here are formats. MFX is the container of the format for professional digital video and audio media defined by the set of SMPTE (Society of Motion Picture and Television Engineers) standards. In its container of formats, apart from other things, there is the JPEG2000 format. It offers possibility of choosing formats of various systems (PAL, NTSC and HD) with stereoscopic presentation. Fig. 2 The basic structure of MXF format. 3-D Stereoscopic Modeling of the Tesla’s Long Island 117 It automatically offers the resolution which minimally distorts the existing video because the setting was done in the card properties at the beginning. However, the parameters of this format can be changed if one wants that. The recommendation is that the user picks out any format with HD system, because, as we know, HD system with the best resolution provides the picture which is of higher quality and sharpness compared to ordinary SD picture. 4. THE STRUCTURE OF MXF FORMAT The purpose of MXF format is to put multimedia data in a file. It will make possible for users to use package transfer and efficient exchange in real time. The specification of MXF is also carefully devised to ensure efficient data saving on different media as well as to ensure reliable transport through communication connections. As it is presented MXF format consists of three parts: File Header, File Body and File Footer. The header provides information about the file as a whole. This part is designed to be the smallest in order to be more easily isolated and sent to a microprocessor for analysis. The body consists of a cover which includes: video, audio and program data, such as text. All these body parts are recorded separately in a single envelope whereby when playing all video frames are connected with full audio and data files. This arrangement is also known as speckled media files. However, the body can be based on several different types of material, including uncompressed audio and video material. The footer is a closing file and it can contain some information which is not available at the time of writing of the header and in some forms can be omitted. Important role in the footer is occupied by metadata which clearly define the end of the file. As it can be seen, great attention is paid to metadata. Fig. 3 Data transfer through MXF format. In general, metadata have many functions:  Including the contents used in media industry (broadcasting film and music),  The use which is based on applications (recording, creating and libraries),  It covers a wide range of business transactions, presenting information and labeling. Metadata can be divided into three broad categories:  The structure of metadata (set of information which defines the core of structure)  Description of metadata (set of information which describes content)  The dark side of metadata (metadata is unknown until the moment of their processing, usually relating to privacy of metadata). 118 V. VUĈKOVIĆ, S. SPASIĆ However, the television transmits ready-made streaming video and audio. This is logical because a viewer expects a scene in real time in parallel with video and audio materials. Computers then exchange data file transfer. This means that two of them go separately. Therefore it is very important to integrate these files. MXF is designed to simultaneously work with both files. Stream files – mostly are of open type without a beforehand defined start and the end, the transfer of data is usually synchronized, but there can also be an asynchronous transfer with specific maximum and minimum speed transmit data. Transfer files – are used for transferable media, using a package based on reliable transmission network data in which the transfer of data segments is itself reliable, data synchronization is performed while the file formats are often structured to allow access to core data. Figure 3 shows data transfer between the stream and transfer files through to MXF format. During the process of rendering the compressions through, two separate channels are split and recorded as two separate files with the purpose of getting better quality, and when someone starts playing video using some player, these two files are compatible and are merged into one file. This process is done to lessen the space needed because the image in JPEG2000 format is compressed and the audio signal is linear. Apart from this, this process ensures protection against unauthorized use. When rendering is finished it is advised to present this kind of video recording in some digital cinema. 5. MODELING OF TESLA’S LONG ISLAND LABORATORY The project Computer simulation and 3-D modeling of the original patents of Nikola Tesla that was later continued through the project Advanced techniques in modeling and simulation of the original patents of Nikola Tesla began in 2009 at the Faculty of Electronic Engineering. Our team was able to create software that is used in the presentation systems in the Nikola Tesla Museum, and now visitors have the opportunity to see original animations of Tesla’s inventions. The aim of the project is multiple. Basically, it is a digital and detailed 3-D modeling of the original patents of Nikola Tesla, which are the part of the Museum’s archives (Figures 4-8.) [5], [6], [7], [8]. Fig. 4 Long Island facility - original photograph. 3-D Stereoscopic Modeling of the Tesla’s Long Island 119 Fig. 5 Tesla’s tower - original blueprint. Using 3-D models, further objectives are rendering, animation, simulation and visualization in real time and space shaded by wire (3-D stereo) model (Figures 6 and 7). Characters and other models were also modeled and textured in Autodesk Maya, but exported in Motion Builder (MB). This program runs the character animation using forward-inverse kinematics. Nature environment was made in UDK with Terrain Editor and Speed Tree. We used original photographs from the Museum of Nikola Tesla, and other materials available in this regard. Here are the key elements that have been modeled in the first phase of the project:  Laboratory: (generator, transformer (power and high frequency), arrays of measuring devices on the shelves and low cupboards next to the main wall, large cabinets, small motors and generators, tables and chairs, Tesla’s desk, desk assistant, Tesla’s boat, other smaller components and devices) (Figure 7). Fig. 6 Tesla’s Long Island Laboratory - original photograph (courtesy of Museum of Nikole Tesla). 120 V. VUĈKOVIĆ, S. SPASIĆ Fig. 7 Tesla’s Long Island Laboratory workshop interior (model).  Boiler-room: (main steam generator, boiler, auxiliary generator, steam engine, regulators, gauges, details, furniture, stairs, railings, switches, levers, valves arrays).  Dynamo room: (with storage for coal).  Workshop (lathes, small engines and transmission belts, desks against the restraints, chairs, pieces of tools, materials, boxes, reels wire, other details in the workshop. Addition, modeling full galleries, stairs and railings). Fig. 8 Tesla’s Laboratory exterior (model). Until the end of 2013, we are working on the following sub-project activities: refinement of models of Long Island (railways, carriages, turbines in the boiler-room). Dynamics is already incorporated in the 3-D model of Long Island (Figure 8.). Also, we implemented some of the standard simulation techniques, to run the models and engines [7], [8]. 3-D Stereoscopic Modeling of the Tesla’s Long Island 121 6. DETAILS OF MODELING SEQUENCE Developing work on a very complex model such as Long Island requires a large team of collaborators and precisely defined development procedure. First of all, we need to posses quality original images that is the basis of modeling. Cooperation with Nikola Tesla Museum was indispensable bearing in mind that the Museum keeps the original Tesla’s legacy. The basic material from which? we have been working to develop models includes six original photos of the interior of the building on Long Island, as well as photographs of the transmitting tower also with the original plans and drawings of the tower. In the next phase, we started analyzing the structure of the building and the tower and completed modeling of these major facilities. After that, we gradually developed device models from inside the building, on the basis of the existing photographs. At this stage, permanent consultations among engineers and designers were necessary. Wire structure of 3D models is the essential that allows objects to schedule in building, to look into their proper dimensions and relationships (Figures 9a - 9c). This part of procedure is very complex and time-consuming. Each detail of the 3D model is manually defined [8]. Fig. 9a Tesla’s Long Island wire model. Fig. 9b Tesla’s Long Island wire model (structure). 122 V. VUĈKOVIĆ, S. SPASIĆ Fig. 9c Tesla’s Laboratory wire model (main room). The next phase is materialization of objects using expert knowledge and research related to data objects that are in the model. This job is mostly manual, but in the course of the project, we focused our research to semi-automatic systems for materialization using pre- defined structure of the material from our original database. In our application there is an option to replace the texture. After performing the segmentation and reaching the region merge, it is possible in certain segments to change material. The basic structure is database with textures. In the initial stage it is necessary to replenish the base and textures can later be added as needed. Textures can be added manually by choosing from the menu or automatically based on the color histogram [9], [10]. All functions working with texture are accommodated in the class: AddEditTextureForm. The histogram determination function (application is completely written in C++) performs the calculation and displays the color histogram for a given texture. This function simply counts the number of RGB pixels in give (x, y) rectangle separately. Red, Green and Blue color channel are determined with three identical procedures, each for every channel. The pixels values are in range [0.255]. Here the color structure prevailing in the texture can be noticed. Based on this procedure, later someone could choose a texture for a specific region. In the preview window, it is possible to determine the type and name of the texture. The application offers some types of basic materials. There are no practical limitations in number of textures in the database. We could use different sources for textures, as well as scanned photographs in high resolution. Some of the pre-defined basic textures that are commonly in use are: ’wood’,’stone’,’sand’,’water’,’stone’,’clouds’,’metal’,’brick’. After selecting and setting the attributes, texture surveying is done with a group of texture management functions. When the texture is defined in the database, we continue the process of changing textures. It is significant to note that when someone selects the texture, blend factor must be selected also. This factor determines how the texture is impressed or integrated with the region. Since we want to preserve the originality of the image with original lighting and shadows and keep the texture, blend factor determines how much level shadows and light from the original image is applied to the texture. 3-D Stereoscopic Modeling of the Tesla’s Long Island 123 The main idea implemented in this phase is using of two separate image sources, one based on original image, and the other based on synthetic (automatically generated) improved texture image. The perfect tuning between these sources is done manually. Finally, the complete materialization is done in semi-automatic manner as a combination of manually and automatic materialization. There is a possibility of info data for a defined region. This info card is remembered with each region. It is possible to see a histogram of the distribution of blue, red and green color information about the region, such as the number of pixels belonging to the region and the size of a region. The size of a region defines the smallest rectangle that can be described around the region. This data is expressed in pixels in height and width of a rectangle. There is a description of the region as well as the material from which is defined. The changes that have been entered are remembered with the region and will be visible the next time someone opens the InfoCard of the region. In this way, the system remembers the process and details at this stage of materialization and applies it when processing in the future. We can say that our approach has the elements of automatic (machine) learning. Fig. 10 Rendering of the main room after materialization. After materialization phase (Figure 10), we proceed to set up lighting. Virtual light sources are placed manually. Attributes of light sources, diffused color and structure are determined experimentally. This part of the project requires intensive use of computers, keeping in mind that iterations must be verified with new rendering scenes. For instance, there are 3 virtual light sources inside the main room (Figures 11a and 11b). 124 V. VUĈKOVIĆ, S. SPASIĆ Fig. 11a Light application in main room model. After the light application in the monochromatic mode, full scale rendering, with lights on is performed. Fig. 11b Rendering lights in main room. Now, the final rendering is done on a complete model of Long Island. This job takes several hours on a cluster that we use (about 30 physical Intel i5 or i7 cores). The final phase is passing through the scene. In doing so, we can turn the stereo rendering and generate a stereo view of Long Island in formats (JPEG2000) that are shown in the first sections in this paper. This part can be done only by using modern NVIDEA GTX video cards, performing rendering in hardware with a large number of parallel processors. 3-D Stereoscopic Modeling of the Tesla’s Long Island 125 Fig. 12 Complete functional 3D model in normal or stereo view Observation of stereo display 3D models of Long Island and a dynamic walk through the model using ordinary Anaglyph or special stereo glasses is the final phase of the project (Figure 12). 7. CONCLUSION This paper presents details about the stereo conversion techniques from the Tesla’s Long Island 3-D model and animation to modern stereo 3-D formats. We presented the detail software infrastructure from the original construction to the real time stereo model. There are many steps in this procedure, the main ones being described. The main pipeline is following the further steps: 1. Manual 3D modeling of the basic model directly from the original pictures (only six old photographs for Long Island model), model by model. 2. Assembling and tuning of the complete wire model. 3. Materialization of the wire construction using original semi-automatic procedures (described in Section 6). 4. Finalizing the model and generating of the JPEG2000 Stereoscopic 3-D model using the possibilities of nVidea GPUs and real-time 3D software. Finally, our Long Island model is one of the most complex and detailed models made till nowadays. Real time application is the first one of that kind, as we know. This work is a part of our work on an a interdisciplinary project III44006 realized at the Faculty of Electronic Engineering in cooperation with the Ministry of Science and Technology and the Nikola Tesla Museum in Belgrade [6], [8]. We believe that the realization of the project is very important for both institutions, especially the Nikola Tesla Museum in Belgrade that has a special status of the institution of national importance, so that the results of the project have high importance. The project 126 V. VUĈKOVIĆ, S. SPASIĆ had a number of promotions, seminars, and media coverage. To crown the success, there was a joint presentation with the Museum of Nikola Tesla at the World Exhibition in China in 2010 at the central stand of the Republic of Serbia. A 3-D movie that was especially made for this occasion was seen by some 200,000 visitors. Acknowledgement: This work is supported by the Serbian Ministry of Education and Science (project III44006-10). REFERENCES [1] M. M. Woolfson, Introduction to Computer Simulation, OUP Oxford. [2] Hartmann Alexander, Practical Guide to Computer Simulations, World Scientific Publishing; Pap/Cdr edition (21 Jun 2009). [3] M.W. Marcellin , M.J. Gormish, A. Bilgin, M.P. Boliek, “An overview of JPEG-2000”, In Proceedings of the IEEE Data Compression Conference, DCC 2000, 28 Mar 2000-30 Mar 2000. [4] D.S. Taubman, M.W. Marcellin, “JPEG2000: standard for interactive imaging”, In Proceedings of the IEEE, vol. 90 , Issue 8, pp. 1336 - 1357. [5] V. Vuĉković, N. Stojanović “The Complete 3-D Modeling and Real-Time Simulation of the Tesla’s Boat”, International Journal of Emerging Sciences IJES, vol. 1, no. 4, pp. 535-544, December 2011. [6] V. Vuĉković, “3-D Modeling and Simulation of the Tesla’s Wireless Controlled Boat” (Invited Lecture and Paper), In Proceedings of the 7th International Symposium Nikola Tesla, Belgrade, pp. 37-44, November 23, 2011. [7] V. Vuĉković, N. Stojanović, "Mathematical 3D Modeling and Real-Time Simulation of the Tesla’s Wireless Controlled Vehicle - Boat", The Scientific Journal Facta Universitatis, Series Electronics and Energetics, vol. 24, no. 2, Niš, pp. 257-270. August 2011. [8] V. Vuĉković, "Virtual Models of Tesla Patents", In Proceedings of the XVII Telecommunications Forum TELFOR 2009, CD ROM Proceedings, Belgrade, 24-26. November 2009, pp. 1335-1338. [9] Z. Zhang, "Determining the Epipolar Geometry and its Uncertainty", A Review. International Journal of Computer Vision. vol. 27, no. 2, 1998, pp. 161 - 198. [10] Bo Peng, Lei Zhang, Automatic Image Segmentation by Dynamic Region Merging, Department of Computing, The Hong Kong Polytechnic University, Hong Kong. https://www.cs.auckland.ac.nz/courses/compsci773s1c/resources/IJCV-Review.pdf