BIOMATH https://biomath.math.bas.bg/biomath/index.php/biomath B f Biomath Forum ORIGINAL ARTICLE Spike timing neural network model of conscious visual perception Petia Koprinkova-Hristova, Simona Nedelcheva Institute of Information and Communication Technologies Bulgarian Academy of Sciences Sofia, Bulgaria petia.koprinkova@iict.bas.bg simona.nedelcheva@iict.bas.bg Received: 19 November 2021, accepted: 25 February 2022, published: 8 April 2022 Abstract—The aim of the paper is to investigate the influence of thalamo-cortical connectivity on the conscious perception of visual stimuli. We conducted simulation experiments changing the key parame- ters of our spike timing neural network model of visual perception and decision making that are sup- posed to be related to conscious perception, namely bottom-up and top-down connections between tha- lamic relay, including Thalamic reticulate nucleus (TRN) and Lateral geniculate nucleus (LGN), and primary visual cortex (V1). The model output, that is perceptual based decision in the lateral intrapare- ital (LIP) area of the brain for left or right saccade generation, was observed. Conclusions about the influence of altered key parameters on the ability of our model to take proper decision were commented in respect to the observed activity in the brain areas responsible for conscious visual perception and decision making. Keywords-spike timing neural network; conscious- ness; thalamo-cortical connections; visual percep- tion; I. INTRODUCTION Since the earliest days of psycho-physiology, there has been a debate about the link between sen- sation, perception, attention, and consciousness. The main question is: what happens to a sensory signal in the brain when it reaches a conscious stage of processing as opposed to being processed outside of awareness? In search of an answer to this question the concept of “Neural correlates of consciousness” is introduced that represents the set of neuronal events and mechanisms generating a specific conscious perception. Based on it in [3], [4] consciousness is viewed as a state-dependent property of some complex, adaptive, and highly interconnected biological structures in the brain. A model of consciousness is a theoretical description that relates brain phenomena such as fast irregular electrical activity and widespread brain activation to expressions of consciousness such as qualia [19]. Copyright: © 2022 Koprinkova et al. This article is distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Citation: Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious visual perception, Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 1 of 10 https://biomath.math.bas.bg/biomath/index.php/biomath mailto:petia.koprinkova@iict.bas.bg mailto:simona.nedelcheva@iict.bas.bg https://creativecommons.org/licenses/by/4.0/ https://creativecommons.org/licenses/by/4.0/ https://doi.org/10.55630/j.biomath.2022.02.258 Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious ... Recent studies on neural correlates of con- sciousness are a continuation of the research ini- tiated at the end of the 19th century [12]. The contemporary trend in this intensively developing nowadays area of research was initiated in the 1990s by the development of an empirical ap- proach focusing on visual awareness because the visual system was already intensively investigated [2], [3], [4], [10]. Since then, consciousness re- search became more diverse but its link to visual perception continued [17], [18]. Irrespective of the intense interest and research efforts in studying the consciousness, there is still no consensus about the neural correlates of consciousness, i.e. what are the minimal neural mechanisms that are jointly suffi- cient for any one conscious perception, thought or memory, under constant background conditions [3]. It is still unclear which brain regions are essential for conscious experience. Recently, a dominant trend is to view conscious- ness as emerging from interactions between dis- tributed networks of neurons and especially from the global activity patterns of cortico-cortical and thalamo-cortical loops [5], [6], [7], [14], [21], [22], [23]. In [15] a hypothesis was proposed that the thalamus is the primary candidate for the location of consciousness since it has been referred to as the gateway of nearly all sensory inputs to the corre- sponding cortical areas. This theory was supported by numerous works. Lately, a view of thalamo- cortical processing is proposed in [20] where two types of thalamic relays are defined: first-order re- lays receiving subcortical driver input, e.g. retinal input to the lateral geniculate nucleus, and higher- order relays, receiving driver input from layer 5 of the cortex, that participate in cortico-thalamo- cortical circuits. Recent findings [8] support the important role of the lateral geniculate nucleus in the emergence of consciousness and provide a more complex view of its connections to the other parts of the thalamus and the visual cortex. In [1] it was suggested that the hallmark of conscious processing is the flexible integration of bottom- up and top-down thalamo-cortical streams and a novel neurobiology theory of consciousness called Dendritic Information Theory, was proposed. We have already developed a hierarchical spike timing model of visual information perception and decision making including the detailed structure of the thalamic relay composed by laretal geniculate nucleus, thalamic reticulate nucleus and interneu- rons [11]. The model was implemented in NEST Simulator [9] on the supercomputer Avitohol. Here we investigate the influence of the thalamo-cortical connectivity on the conscious per- ception of visual stimuli by changing the key parameters of our model that are supposed to be related to conscious perception, namely the bottom-up and top-down connections between the thalamic relay and primary visual cortex (V1). The model output - perceptual based decision for left or right saccade generation - was observed. Conclusions about the influence of the altered key parameters on the ability of our model to take proper decision were commented in respect to the observed activity in the areas V1, Middle temporal (MT), Medial superior temporal (MST) and LIP areas. The structure of the rest of the paper is as follows: next section presents briefly our model architecture and its parameters under investigation; the simulation results are presented and discussed in sections III and IV; the paper finishes with the concluding remarks pointing out the directions of our further work. II. MODEL STRUCTURE The details of the structure of our spike timing hierarchical model of visual information process- ing and perceptual decision making with reinforce- ment learning were reported in [11]. For the aim of the current simulation investigations we use only its perceptual-based decision part shown on Fig. 1. Each colored rectangle on Fig. 1 represents a group of neurons positioned on a regular two- dimensional grid. Each group, called further layer, corresponds to a brain structure involved in visual information perception as follows: retinal ganglion cells (RGC); Lateral geniculate nucleus (LGN); Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 2 of 10 https://doi.org/10.55630/j.biomath.2022.02.258 Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious ... Fig. 1. Model structure as in [11]. Thalamic reticulate nucleus (TRN) and Interneu- rons (IN); Primary visual cortex (V1); Middle tem- poral (MT) area; Medial superior temporal (MST) area; Lateral intraparietal cortex (LIP). Each layer has a special role in the visual information pro- cessing as follows: RGC in eyes transform the light to electrical signal fed into the brain via the optic nerve; LGN, TRN and IN has a role of relay structure forwarding the information to the visual cortex; V1 detects orientation of the visual stimulus; further MT detects the direction of movements while MST has more complicated role to detect the patterns of movement (here expansion/contraction from/to a given focal point); finally LIP collects the processed visual informa- tion and takes a perceptual based decision (in this case left or right expansion center of the stimulus). The arrows denote the connections between layers, called synapses. The details of connectivity, described in our previous works [11], are based on the literature information. Briefly, each neuron has its own receptive field - area of neurons from a given layer that it is connected to - depending on the function of the layer it belongs to as well as on its position within its layer. The dimensions of all model building blocks are shown in Table I TABLE I VISUAL PERCEPTION AND DECISION MAKING MODEL DIMENSIONALITY. Layer Size Neurons number RGC 2 × 20 × 20 800 LGN 2 × 20 × 20 800 IN 2 × 20 × 20 800 TRN 2 × 20 × 20 800 V1 E 2 × 20 × 20 800 V1 I 2 × 10 × 10 200 MT E 2 × 20 × 20 800 MT I 2 × 10 × 10 200 MST 2 × 20 × 20 800 LIP 2 × 20 × 20 800 The connections of interest in current investi- gation are those between the thalamus (including LGN, TRN and IN layers) and the visual cor- tex (V1 layer). The V1 neurons have orientation sensitivity due to their elongated receptive fields defined by Gabor functions. Their orientation and phase parameters were determined so as to achieve the typical for the mammalian brain “pinwheel- structure” (for more information see [16]). They are separated into four groups - two excitatory (E1 and E2) and two inhibitory (I1 and I2) - connected via lateral connections based on their distance and roles as in [13]. Fig. 2 shows the designed in this way strengths of forward connections (from the thalamus to V1). The feedback connections from V1 to the tha- lamus are proportional to the feedforward once. In the current work we scale the feedforward and feedback connections to investigate their role in the conscious visual information perception. III. SIMULATION EXPERIMENTS The model input stimulus consisted of moving dots expanding from a left focal point as shown on Fig. 3 (for more details see our previous works [11], [16]). The visual stimulation lasted Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 3 of 10 https://doi.org/10.55630/j.biomath.2022.02.258 Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious ... Fig. 2. Connectivity between thalamus and V1 as in [16]. Fig. 3. Visual stimulus screen shot. Blue arrows denote the imaginary expansion center of the moving white dots on the gray screen. for 1670 milliseconds. First 150 milliseconds were “washed out” since they were needed for RGC spatio-temporal filters to accumulate the visual information at the beginning of the stimulation. We conducted simulation experiments by vary- ing the scaling factors of both feedforward and feedback connections between the thalamus and V1. Table II summarizes the experiments. Sign + denotes the cases with successful propagation of the visual information from RGC up to LIP area while those marked with − are the cases without transmission of the visual information to V1. The smallest values of the feedforward connec- tions (scaled by 0.1 and below) do not allow prop- agation of the visual information to the primary visual cortex thus preventing the perceptual based decision while the minimal feedback connectivity does not have such deterioration effect. In order to investigate in details the effect of varying feedforward/feedback connectivity we TABLE II SIMULATION EXPERIMENTS Feedback Feedforward scaling scaling 0.1 and below 0.5 1.0 2.0 5.0 0.0 − + + + + 0.01 − + + + + 0.1 − + + + + 0.5 − + + + + 1.0 − + + + + 2.0 − + + + + 5.0 − + + + + monitored the spiking activity in all layers - from primary visual cortex V1 up to the decision LIP areas. The next section summarizes and discusses the results of the simulation experiments carried out. IV. RESULTS AND DISCUSSION Figures 4-7 show the simulation results for the experiments with successful propagation of the visual information (marked by + in Table II) for all considered model layers (V1, MT, MSTe and LIPleft respectively). The top parts of the figures 4-7 show firing rates during stimulation in the observed layers of the model. In order to distinguish clearly differences in spiking activity, bottom parts of the figures 4- 7 show the mean and the variance of the above spiking frequencies. As it was expected, the biggest differences in spiking activity caused by scaling of the feed- back/feedforward connectivity were observed in V1 since it is the first structure influenced directly by the thalamus. Fig. 4 shows that with the in- crease of both feedforward and feedback connec- tions the initiation of spiking activity, e.g. first reaction to the visual stimulus, begins earlier. In case of smallest feedforward connectivity without feedback connections (feedback scaling 0.0 and feedforward scaling 0.5) it begins at about 107th millisecond of stimulation, while in the last case (feedback/feedforward scaling 5.0) it begins at about 61st millisecond. The increased feedforward connectivity also led to the increase in spiking activity by the end of stimulation. Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 4 of 10 https://doi.org/10.55630/j.biomath.2022.02.258 Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious ... Fig. 4. Spiking activity in V1 (top) and its mean (bottom left) and variance (bottom right). Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 5 of 10 https://doi.org/10.55630/j.biomath.2022.02.258 Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious ... Fig. 5. Spiking activity in MT and its mean (bottom left) and variance (bottom right). Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 6 of 10 https://doi.org/10.55630/j.biomath.2022.02.258 Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious ... Fig. 6. Spiking activity in MSTe and its mean (bottom left) and variance (bottom right). Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 7 of 10 https://doi.org/10.55630/j.biomath.2022.02.258 Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious ... Fig. 7. Spiking activity in LIPleft and its mean (bottom left) and variance (bottom right). Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 8 of 10 https://doi.org/10.55630/j.biomath.2022.02.258 Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious ... The spiking activity in the next (MT) layer was influenced in similar manner like that in V1 while the MSTe and LIP areas show almost similar activity for all experimental cases. While the mean of the V1 firing rate increases with both scaling parameters (Fig. 4), its variance shows lowest values in cases of middle (2.0) and lowest (0.5) feedforward scaling coefficients. The mean spiking frequency in the MT area shows a bit different tendency: the biggest mean frequency was observed in the case of the biggest feedforward scaling coefficient in combination with lowest feedback scaling coefficient. The vari- ance of spiking activity in MT has also irregu- lar dependence on the thalamus-V1 connectivity showing bigger values not only in the case with the biggest scaling parameters but also in the cases with middle feedforward and low feedback scaling as well as with low feedforward and high feedback scaling coefficients (Fig. 5). The observed in the MT area effect spreads also to the MSTe and LIP areas (figures 6 and 7). While the biggest mean frequencies in the MSTe area were observed for the biggest feed- forward and lowest feedback scaling coefficients, its biggest variance was in the case of middle feedforward scaling (2.0) in combination with low feedback scaling (0.01) coefficients. The strongest mean firing activity in the decision area (left LIP area in our example) moves further to the low- est feedforward scaling (0.5) in combination with middle (2.0) feedback scaling while the biggest variance in LIP spiking activity was observed for relatively low feedback/feedforward connectivity with scaling coefficients 0.5 and 1.0 respectively. In summary, the biggest mean firing rates move through the model areas from the case of biggest feedforward/feedback scaling coefficients in V1 to the case of lowest feedforward and middle feedback scaling coefficients in the LIP area. This result shows the significance of feedback con- nectivity from V1 to the thalamus. In all cases even small feedback connectivity increases the mean spiking activity through all the model areas in comparison with cases of missing feedback thalamo-cortical connectivity. However, after some threshold the higher feedback scaling coefficients have suppressing effect on the spiking activity in the MT, MST and LIP areas. V. CONCLUSION The presented in this paper simulation ex- periments with varying feedforward/feedback thalamo-cortical connectivity demonstrated the significance of the feedback from the visual cortex to the thalamus. Our results revealed that the re- lation between the two way connectivity strengths and the spiking activity in the visual information processing areas MT, MST and LIP following the visual cortex V1 is not straightforward and should be investigated in deeper details. We have to account that our model connectivity was designed based on the literature information and needs further experimental clarification. The dimension of the test model was minimal so its scaling up to more realistic number of neurons will make the simulation experiments more realistic too. Our planes for future work include exploitation of the experimental data (MRI and fMRI) reveal- ing the real brain connectivity in order to make our model more realistic. We expect to obtain such a data by cooperation with the partners from COST action “Neural architectures of consciousness” in the near future. The implementation of the model on the supercomputer Avitohol will allow a signif- icant increase of its dimension that is our second aim for the future work. Acknowledgments: The reported work is sup- ported by the Bulgarian Science Fund grant No KP-06-COST/9 from 2021 “Supercomputer sim- ulation investigation of a neural model of brain structures involved in conscious visual perception” by a co-funding scheme of COST action CA18106 “Neural architectures of consciousness”. SN was supported by the Bulgarian National Scientific Program Young Scientists and Postdocs, Module Young Scientists. Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 9 of 10 https://doi.org/10.55630/j.biomath.2022.02.258 Petia Koprinkova-Hristova, Simona Nedelcheva, Spike timing neural network model of conscious ... REFERENCES [1] Aru, J., Suzuki, M., Larkum, M. E. (2020) Cellular mechanisms of conscious processing. Trends in Cog- nitive Sciences 24(10), 814-825. [2] Block, N. (1995). How many concepts of conscious- ness? Behavioral and Brain Sciences 18(2), 272-287. DOI: 10.1017/S0140525X00038486. [3] Crick, F. and Koch, C. (1990) Towards a neurobiological theory of consciousness. Semin. Neurosci. 2, 263-275. [4] Crick, F. and Koch, C. (1992) The problem of con- sciousness. Sci. Am. 267, 152-159. [5] Dehaene, S. et al. (2003). A neuronal network model linking subjective reports and objective physiological data during conscious perception. PNAS 100 (14), 8520- 8525, DOI: 10.1073/pnas.1332574100. [6] Dehaene, S. et al. (2006) Conscious, preconscious, and subliminal processing: a testable taxonomy. Trends Cogn. Sci. 10, 204-211. [7] Dehaene, S. and Changeux, J.-P. (2011) Experimen- tal and theoretical approaches to conscious processing. Neuron 70, 200-227. [8] Ghodratia, M. et al. (2017). Towards building a more complex view of the lateral geniculate nucleus: Recent advances in understanding its role. Progress in Neuro- biology, vol. 156 pp. 214-255, 2017. [9] Jordan, J. et al. (2019) NEST 2.18.0, Zenodo, DOI: 10.5281/zenodo.2605422. [10] Koch, C. (2004) The Quest for Consciousness: A Neu- robiological Approach, Roberts Publishers. [11] Koprinkova-Hristova P., Bocheva N. (2020) Brain- Inspired Spike Timing Model of Dynamic Visual Infor- mation Perception and Decision Making with STDP and Reinforcement Learning. Lecture Notes in Computer Science, vol 12566, 421-435. [16] Nedelcheva, S., and Koprinkova-Hristova, P.: Orien- tation Selectivity Tuning of a Spike Timing Neural Network Model of the First Layer of the Human Visual Cortex. in Advanced Computing in Industrial Mathe- matics. BGSIAM 2017. Studies in Computational Intel- ligence 793, eds. K. Georgiev, M. Todorov, I. Georgiev, (Springer, Cham), 291-303 (2019). [12] LeDoux, J. E., Michel, M., Lau, H. (2020) A little history goes a long way toward understanding why we study consciousness the way we do today, PNAS 117(13), pp. 6976-6984. [13] Kremkow, J., Perrinet, L. U., Monier, C., Alonso, J.- M., Aertsen, A. Frégnac, Y., Masson, G. S.: Push-Pull Receptive Field Organization and Synaptic Depression: Mechanisms for Reliably Encoding Naturalistic Stim- uli in V1. Frontiers in Neural Circuits (2016) DOI: 10.3389/fncir.2016.00037. [14] Mashour, G.A. et al. (2020) Conscious processing and the global neuronal workspace hypothesis. Neuron 105, 776-798. [15] Min, B. K. (2010). A thalamic reticular networking model of consciousness. Theoretical Biology and Medi- cal Modelling, 7(1), 10. DOI: 10.1186/1742-4682-7-10. [17] Palva, J.M., Monto, S., Kulashekhar, S. and Palva, S. (2010) Neuronal synchrony reveals working memory networks and predicts individual memory capacity. Proc. Natl. Acad. Sci. USA, 107, 7580-7585. [18] Panagiotaropoulos, T.I., Kapoor, V., Logothetis, N.K. (2014). Subjective visual perception: From local pro- cessing to emergent phenomena of brain activity, Philo- sophical Transactions of the Royal Society B: Biological Sciences, 369 (1641); DOI: 10.1098/rstb.2013.0534. [19] Seth, A.K. 2009 Explanatory correlates of conscious- ness: Theoretical and computational challenges. Cogni- tive Computation 1(1):50-63. [20] Sherman, S. M. (2016) Thalamus plays a central role in ongoing cortical functioning. Nature Neuroscience 19(4), pp. 533-541. [21] Tononi, G. and Edelman, G.M. (1998) Consciousness and complexity. Science 282, 1846-1851. [22] Tononi, G. (2008) Consciousness as integrated informa- tion: a provisional manifesto. Biol. Bull. 215, 216-242. [23] Tononi, G. et al. (2016) Integrated information theory: from consciousness to its physical substrate. Nat. Rev. Neurosci. 17, 450-461. Biomath 11 (2022), 2202258, https://doi.org/10.55630/j.biomath.2022.02.258 Page 10 of 10 https://doi.org/10.55630/j.biomath.2022.02.258 Introduction Model Structure Simulation experiments Results and discussion Conclusion References